Most of the data published on this portal is updated via our automation system, which allows us to provide daily updates to hundreds of files. This would not be possible if we manually updated the data. The other side of this coin is that because we are not manually updating data, we are not manually checking the quality. Things happen to the data that we don’t always catch until someone in the community spots the problem and lets us know.

We’ve been working on a project to build quality checks into our automation process so that this happens less frequently. As a result of this, we’ve had to make some changes across all of our data files, including standardizing field names.

We know that changing field names could cause scripts that pull our data to break, so starting June 10, 2019, we are outputting the updated data files with file names that have a version number added to the end. The old files will still be available under the existing file names, but the data will not be updated at that location.

If you normally download our data by clicking the little blue arrow button on the dataset pages, then you don’t need to worry about this! However, if you ingest our data by making a request via URL, you will need to update your scripts. The links to all of our data files will have “_v1” appended to indicate they are version 1. If we ever need to make changes to the schemas in the future, we will increase the version number for that data file.

Some of the quality issues we will be addressing by the end of the project include unexpected missing values, dates that are outside of the temporal coverage for that file, geocodes that are outside of the bounding box for that file, numeric values that are suddenly much higher or much lower than the average or median, and many others. The automation system will alert us when it finds these problems, and we’ll look into it and fix it if needed.

When I wrote about how we launched a homemade Open Data portal, I made the claim that a data portal is just a website with a catalog of datasets.

A few companies sell expensive portal products to governments. These companies take care of hosting the datasets and running that website I’m referring to, where people visit to download the data. Some of these portals also give users limited ability to search, filter, and chart data without downloading it.

We wanted to have more control over our portal. We had ideas for how it should look and how it should work. Some of these ideas are included in a piece of guidance the Civic Analytics Network wrote for Open Data programs.

We could also save the City of San Diego money, so we figured we would make our own website and find a place to host our datasets. We used a bunch of tools to make accomplishing these simple objectives even easier. What follows is an overview of those tools.

Our work to automate data updates won us a spot with 16 other finalists in Amazon’s City on a Cloud Innovation Challenge.

Finalists, announced earlier this week, include other governments and schools that are competing in the Best Practices category. This category honors a completed project that focuses on a citizen service, uses Amazon web services, and employs an innovative solution that other public entities might be able to use.

Be honest. When you read the words data automation you get a sudden rush of melatonin to your brain, your eyelids get heavy, and you get an uncontrollable urge to fall asleep. Don’t be ashamed; you are reacting to these words in a similar manner to 99.5% of people on the planet. Bear with me though for just a few paragraphs while I try to explain why it matters, and how we do it here at the City of San Diego.

Hey San Diego! Your open data portal just got a LOT faster!

One of the reasons we wanted to run our own data portal is the flexibility we have to change it and add functionality.

Today, we’re putting the pedal to the metal on those desires. We initially launched the portal based on JKAN, but with modified schemas, layouts, and branding. Because of how fast we moved, we put off thinking about speed and performance.

Since the dust settled a bit, we had a chance to do that.

StreetsSD was an interesting project us from an organizational and technical perspective. Let’s peek behind the scenes to see how this all came together.

The City of San Diego is not only a smart city, it’s a solar friendly city. Our 2015 Climate Action Plan sets an ambitious goal to achieve 100% clean and renewable electricity in the city by 2035. If 2016 is an indicator, we’re well on our way to reaching our goals.

Even before City Council passed Mayor Faulconer’s Climate Action Plan, our Development Services Department was ahead of the game and began expediting solar permits several years ago. This is also thanks to supportive financing programs such as PACE and other incentives.

The City’s street sweeping schedule might not be the first thing on your mind when you’re looking for a parking spot - unless you’ve gotten one of those parking tickets.

The City aims to sweep commercial streets about once per week, and residential streets at least once per month. Sweeping removes debris that might clog the the storm water system and cause flooding and prevents harmful particles from entering the ocean. If the block you’re parked on has signage restricting parking during a certain day and time for street sweeping, you could get a ticket when the sweeper goes by.

The best thing to do is check for signs before you park, but this guide will give you an overview of street sweeping in general, and where you might get a ticket.

Use this map of the sampling sites where the City’s Public Utilities Department tests for indicator bacteria.

Public Utilities currently has 160 sample sites for the City of San Diego drinking water system and is required to test 85 sites per week. Temperature, chlorine and pH are measured on site, and then a sample is brought to a lab to test for the presence of the indicator bacteria coliform and E.coli. Coliform and the strains of E.coli tested serve as indicators of the presence of potentially harmful bacteria.

The Office of the City Treasurer recently completed upgrading all on-street parking meters to SMART parking meters!

Our IPS meters provide us with real-time data in the back office, helping to streamline operations like coin collection and repairs. Our back office system also provides up-to-date transactional data, giving us additional insight into how effectively the meters are being used on-street.

As a result, we have been working with Maksim since he started with the City supporting the open data initiative to share this data with you!