Category: Tools

Oh, how Chicago land use is controlled by spot zoning

If you only had a zoning map to try and understand how the different blocks in the City of Chicago relate to their neighborhoods and the city at large, you might have the idea that the city has no neighborhoods, but is actually a collection of tiny, randomly dispersed zones of differing land uses.

And then when you walked those areas you’d find that the zones, which attempt to prescribe a land use, at least nominally, don’t have anything to do with the restaurant, housing, and commercial building mix of uses actually present.

No plan would have been devised to create a map like this.

Over the last five years, and surely over the last 14, the City of Chicago has been divided (really, split) into an increasing number of distinct zoning districts.

The city’s zoning map is updated after each monthly city council meeting, to reflect the numerous changes that the 50 alders have approved individually. (Their collective approval occurs unanimously in an omnibus bill.)

Every few months I ask the Chicago Department of Innovation and Technology (DoIT) for the latest zoning map, in the form of a shapefile (a kind of file that holds geographic information that can be analyzed by many computer programs). While Chicago has one of the country’s best open data offerings, some datasets, like zoning, don’t get updated in the catalog.

There are two ways I can analyze and present the data about the quantity of zoning districts. Both, however, show that the number of distinct zoning districts has increased. This means that the city is divided even more finely than it was just six months ago.

Analysis 1: Period snapshots

I have the zoning shapefile for five periods, snapshots of the city’s zoning map at that time. From August 2012 to now, May 2016, the number of discrete zoning districts (the sum of all B3-5, RS-1, DX-7, etc. zoning classes) has increased 7.8 percent.

Period Zoning districts change

August 2012

11,278

September 2014

11,677

3.42%

June 2015

11,918

2.02%

November 2015

12,015

0.81%

May 2016

12,162

1.21%

I collect the period snapshots to show the history of zoning at a specific address or building in Chicago, which is listed on Chicago Cityscape. For example, the zoning for the site of the new mixed-use development in Bucktown that includes a reconstructed Aldi has changed four times in four years.

aldi zoning history

Analysis 2: Creation date

The zoning shapefiles also have the date at which a zoning district was split or combined to create a new district, either with a different zoning class (RT-4, C1-1, etc.) or a different shape.

With the most recent zoning shapefile I can tell how many new zoning districts were split or combined and a record representing it was added to the list. The records start in 2002, and by the end of the year 7,717 records were created.

The following year, only 14 records were added, and in 2004, only 6. The Chicago City Council adopted a rewritten zoning code in 2004, and I guess that the zoning map was modified prior to adoption. After 2004, the number of new zoning districts picks up:

year zoning districts added by splitting/combining cumulative change

2002

7717

7717

2003

14

7731

0.18%

2004

6

7737

0.08%

2005

267

8004

3.45%

2006

497

8501

6.21%

2007

561

9062

6.60%

2008

592

9654

6.53%

2009

304

9958

3.15%

2010

245

10203

2.46%

2011

271

10474

2.66%

2012

277

10751

2.64%

2013

299

11050

2.78%

2014

397

11447

3.59%

2015

367

11814

3.21%

2016

173

11987

1.46%

none listed

175

12,162

It seems there’s a light relationship between the recession that started in 2008 and the number of zoning changes made. There are more made annually before the recession than after it. It actually seems to track with building permits (sorry, no chart handy).

The U.S. DOT should collaborate with existing “National Transit Maps” makers

The U.S. DOT demonstrated one idea for how a National Transit Map might look and work at a conference in February.

The Washington Post reported this month that the United States Department of Transportation is going to develop a “National Transit Map” because, frankly, one doesn’t exist. The U.S. DOT said such a map could reveal “transit deserts” (the screen capture above shows one example from Salt Lake City, discussed below).

Secretary Anthony Foxx wrote in an open letter to say that the department and the nation’s transit agencies “have yet to recognize the full potential” of a data standard called the General Transit Feed Specification that Google promoted in order to integrate transit routing on its maps. Foxx described two problems that arose out of not using “GTFS”.

  1. Transit vehicles have significantly greater capacity than passenger cars, but are often considered just vehicles because we are unable to show where and when the transit vehicles are scheduled to operate. The realistic treatment of transit for planning, performance measures, and resiliency requires real data on transit system operations.
  2. One of the most important social values of transit is that it makes transportation available to people who do not have access to private automobiles, and provides transportation options for those who do. Yet, we cannot describe this value at a national level and in many regions because we do not have a national map of fixed transit routes.

“The solution is straightforward”, Foxx continued, “[is] a national repository of voluntarily provided, public domain GTFS feed data that is compiled into a common format with data from fixed route systems.”

The letter went on to explain exactly how the DOT would compile the GTFS files, and said the first “collection day” will be March 31, this week. As of this writing, the website to which transit agencies must submit their GTFS files is unavailable.

What Foxx is asking for has already been done to some degree. Two national transit maps and one data warehouse already exist and the DOT should engage those producers, and others who would use the map, to determine the best way to build a useful but inexpensive map and database. Each of the two existing maps and databases was created by volunteers and are already-funded projects so it would make sense to maximize the use of existing projects and data.

“Transitland” is a project to host transit maps and timetables for transit systems around the world. It was created by Mapzen, a company funded by Samsung to build open source mapping and geodata tools. Transitland is also built upon GTFS data from agencies all over the world. Its data APIs and public map can help answer the question: How many transit operators serve Bay Area residents, and what areas does each service?

For the United States, Transitland hosts and queries data from transit agencies in 31 states and the District of Columbia. In Washington, D.C., Transitland is aware of four transit agencies. It’s a great tool in that respect: Not all of the four transit agencies are headquartered in D.C. or primarily serve that city. The app is capable of understanding spatial overlaps between municipal and regional geographies and transit agencies.

Transitland has a “GUI” to show you how much transit data it has around the world.

“Transit Explorer” is an interactive map of all rail transit and bus rapid transit lines in the United States, Mexico, and Canada. Yonah Freemark, author of The Transport Politic, created the map using data culled from OpenStreetMap, the National Transit Atlas Database (administered by the DOT and which shows fixed-guideway transit), and his own research. I wrote the custom JavaScript code for the Leaflet-powered map.

No other agency or project has collected this much data about fixed-guideway transit lines in any of the three countries, since the map includes detailed information about line lengths, ridership, and other characteristics that are not included in GTFS data. Transit Explorer, though, does not include local bus service or service frequencies, which the DOT’s map may if it incorporates the full breadth of GTFS data.

Transit Explorer also goes a step further by providing data about under construction and proposed fixed-guideway transit lines, which is information that is very relevant to understanding future neighborhood accessibility to transit, but which is not available through GTFS sources.

Finally, “GTFS Data Exchange” is a website that has been storing snapshots of GTFS feeds from agencies around the world for almost a decade, or about as long as GTFS has been used in Google Maps. The snapshots allow for service comparisons of a single agency across time. For example, there are over 100 versions of the GTFS data for the Chicago Transit Authority, stretching back to November 2009; new versions are added – by “cta-archiver” – twice a month.

Josh Cohen, writing in Next City, highlighted the significance of Google’s invention of GTFS, saying, “Prior to the adoption of GTFS, creating such a map would’ve been unwieldy and likely produced an out-of-date product by the time it was completed.” The DOT’s own National Transit Atlas Database includes only fixed-guideway (a.k.a. trains) routes, and hasn’t been updated since 2004.

Not all GTFS feeds are created equal, though. Some transit agencies don’t include all of the data, some of which is optional for Google Map’s purpose, that would make the National Transit Map useful for the spatial analysis the DOT intends. Many agencies don’t include the “route shapes”, or the geographic lines between train stations and bus stops. Researchers are able to see where the vehicles stop, but not which streets or routes they take. Foxx’s letter doesn’t acknowledge this. It does, however, mention that transit agencies can use some federal funds to create the GTFS data.

David Levinson, professor at the University of Minnesota, believes the map will bias coverage (geographic reach of transit service) over frequency (how many buses are run each day that someone could ride).

The U.S. DOT’s chief data officer, Dan Morgan, whom I met at Transportation Camp 2015 in Washington, D.C., presented at the FedGIS Conference this year one idea to demonstrate coverage and frequency in Salt Lake City, using the GTFS data from the Utah Transit Authority.

Levinson also tweeted that it will be difficult for a national map to show service because of the struggles individual transit providers have symbolizing their own service patterns.

Foxx’s letter doesn’t describe how planners will be able to download the data in the collection, but whichever app they build or modify will cost money. Before going much further, and before spending any significant funds, Foxx should consult potential users and researchers to avoid duplicating existing projects that may ultimately be superior resources.

Foxx can also take advantage of “18F” a new agency within the General Services Administration to overcome government’s reputation for creating costly and difficult to use apps. The GSA procures all kinds of things the federal government needs, and 18F may be able to help the DOT create the National Transit Map (and database) in a modern, tech and user-friendly way – or write a good RFP for someone else to make it.

Look for the National Transit Map this summer.

Converting a transit agency’s GTFS to shapefile and GeoJSON with QGIS

Many years ago I wrote a tutorial on how to use an ArcGIS plugin to convert a transit agency’s GTFS package – a group of files that describe when and where their buses and trains stop – into files that could easily be manipulated by popular GIS desktop software.

That was so long ago, before I became an expert in using QGIS, a free and open source alternative to ArcGIS.

This tutorial will show you how to convert GTFS to a shapefile and to GeoJSON so you can edit and visualize the transit data in QGIS.

Prerequisites

First you’ll need to have QGIS installed on your computer (it works with Linux, Mac, and Windows). Second you’ll need a GTFS package for the transit agency of your choice (here’s the one for Pace Suburban Bus*, which operates all suburban transit buses in Chicagoland). You can find another transit agency around the world on the GTFS Data Exchange website.

Section 1: Let’s start

  1. Open QGIS.
  2. Load your GTFS data into the QGIS table of contents (also called the Layers Panel). Click Layer>Add Layer>Add Delimited Text Layer. You will be adding one or two files depending on which ones are provided.

    QGIS add delimited text layer

    Add delimited text layer.

  3. Now, here it can get tricky. Not all transit agencies provide a “shapes.txt” file. The shapes.txt file draws out the routes of buses and trains. If it’s not provided, that’s fine, but if you turn them into routes based on the stops.txt data, then you will have funny looking and impossible routes.

    QGIs browse for the stops.txt file

    Browse for the stops.txt file

  4. Click on “Browse…” and find the “stops.txt”. QGIS will read the file very quickly and determine which fields hold the latitude and longitude coordinates. If its determination is wrong, you can choose a different “X field” (longitude) and “Y field” (latitude).
  5. Click “OK”. A new dialog box will appear asking you to choose a coordinate reference system (EPSG). Choose or filter for “WGS 84, EPSG:4326”. Then click “OK”.
  6. The Pace bus stops in the Chicagoland region are now drawn in QGIS!

    Pace bus stops are shown

    Pace bus stops are shown

  7. If the GTFS package you downloaded includes a “shapes.txt” file (that represents the physical routes and paths that the buses or trains take), import that file also by repeating steps 4 and 5.

Section 2: Converting the stops

It’s really easy now to convert the bus or train stops into a shapefile or GeoJSON representing all of those points.

  1. Right-click the layer “stops” in the table of contents (Layers Panel) and click “Save As…”.
  2. In the “Save vector layer as…” dialog box, choose the format you want, either “ESRI Shapefile” or “GeoJSON”. **
  3. Then click “Browse” to tell QGIS where in your computer’s file browser you want to save the file. Leave the “CRS” as-is (EPSG:4326).

    Convert the bus stops to a shapefile or GeoJSON.

    Convert the Pace bus stops to a shapefile or GeoJSON.

  4. Then click “OK” and QGIS will quickly report that the file has been converted and saved where you specified in step 3.

Section 3: Converting the bus or train routes

The “shapes.txt” file is a collection of points that when grouped by their route number, show the physical routes and paths that buses and trains take. You’ll need a plugin to make the lines from this data.

  1. Install the plugin “Points to Paths”. Click on Plugins>Manage and Install Plugins… Then click “All” and search for “points”. Click the “Points to Paths” plugin and then click the “Install plugin” button. Then click “Close”.

    Install the Points to Paths plugin.

    Install the Points to Paths plugin.

  2. Pace bus doesn’t provide the “shapes.txt” file so we’ll need to find a new GTFS package. Download the GTFS package provided by the Chicago Transit Authority, which has bus and rail service in Chicago and the surrounding municipalities.
  3. Load the CTA’s “shapes.txt” file into the table of contents (Layers Panel) by following steps 4 and 5 in the first section of this tutorial.  Note that this data includes both the bus routes and the train routes.

    QGIS load CTA bus and train stops

    Import CTA bus and train stops into QGIS

  4. Now let’s start the conversion process. Click on Plugins>Points to Paths. In the next dialog box choose the “shapes” layer as your “Input point layer”.
  5. Select “shape_id” as the field with which you want to “Point group field”. This tells the plugin how to distinguish one bus route from the next.
  6. Select “shape_pt_sequence” as the field with which you want to “Point order field”. This tells the plugin in what order the points should be connected to form the route’s line.
  7. Click “Browse” to give the converted output shapefile a name and a location with your computer’s file browser.
  8. Make sure all  of the options look like the one in this screenshot and then click “OK”. QGIS and the plugin will start working to piece together the points into lines and create a new shapefile from this work.

    These are the options you need to set to convert the CTA points (stops) to paths (routes).

    These are the options you need to set to convert the CTA points to paths (routes).

  9. You’ll know it’s finished when the hourglass or “waiting” cursor returns to a pointer, and when you see a question asking if you would like the resulting shapefile added to your table of contents (Layers Panel). Go ahead and choose “Yes”.

    QGIS: CTA bus and train points are converted to paths (routes)

    The CTA bus and train points, provided in a GTFS package, have been converted to paths (routes/lines).

  10. Now follow steps 1-4 from Section 3 to convert the routes/lines data to a shapefile or GeoJSON file.**

Notes

* As of this writing, the schedules in Pace’s GTFS package are accurate as of January 18, 2016. It appears their download link always points to the latest version. Transit schedules typically change several times each year. Pace says, “Only one package is posted at any given time, typically representing Pace service from now until a couple of months in the future. Use the Calendar table to see on which days and dates service in the Trips table are effective.”

** Choose GeoJSON if you want to show this data on a web map (like in Leaflet or the Google Maps API), or if you want to share the data on GitHub.

Updated: 311 is back in the spotlight after 3 years

Updated October 21: Rahm has dropped the plan to privatize 311, but explained that the offer was less about saving $1 million annually than it was to upgrade the tech. I still don’t think it was explained well enough – or challenged enough by City Council – why a privatization was the “only” way to get an upgrade worth $25-50 million, or why the upgrade would cost that much. Then, there’s still the open question as to what happened to the RFP in 2014, looking for bidders to create a new 311 system. I’m also curious if any of the nascent civic-oriented CRM and Open311 startups would have a reasonable chance at responding to an RFP or privatization effort, or if the RFP process would largely exclude new or small players. 

Last week, for budget “season” in Chicago, Mayor Rahm Emanuel proposed privatizing the 311 system – where citizens and aldermen request services from any city department, and citizens get information about city services – as a way to pay for the technology upgrade the system desperately needs.

There are over 300 service request types (codes), of which many are available to the public (the rest being for internal or aldermanic use). Here’s the full list of codes and the department that administers or manages the requests.

The topic of analyzing 311 data came up tonight at ChiHackNight, and I talked to a DePaul University Student interested in using their predictive analytics study on this rich (maybe) dataset.

I brought them up to speed on the changes made to the 311 technology system and created this “status” document to demonstrate the wild and whacky year of changes in 2012, and the stagnation since then. I’m publishing this because I also want to collect ideas and feedback about things I don’t know regarding the progress in redeveloping the tech that powers Chicago’s 311.

311 technology chronology

2012

Code for America fellows come to Chicago and work with DoIT to develop 13 service request types (codes) for Open311, a layer on 311 that enables third-party apps to submit new service requests – with photos. See what other cities have adopted Open311.

I pled with the Chief Technology Officer at the time, John Tolva, holding a new position in the mayor’s office, to enable 5 of the bicycle and pedestrian-related codes in the Open311 system, so that I can build apps for Grid Chicago, and later, Streetsblog Chicago, readers to request city services that impact the safety and quality around walking and biking.

Ben Sheldon, one of the four Code for America fellows, made SuperMayor.io, and it’s one of the few Open311-based apps that’s still working (most of the apps linked to from this Smart Chicago Collaborative blog post are dead).

2013

The City of Chicago releases an RFP to remake the 311 system (September). One aspect of operating the system I noticed that better tech could probably have a big impact on is the number of service request taken by phone: 74%!

Channels through which 311 service requests are submitted

According to the RFP issued in 2013, 74% of service requests are taken by phone. Only about 6% are done through the web.

2014

Companies respond to the RFP (I assume). Are these responses hosted anywhere? Smart Chicago Collaborative made Chicago Works For You, a dashboard that’s also  the only Chicago + Open311 map.

The RFP never made it past “Step 4” in the Bid Tracker, or, “Short List Determination”. The Bid Tracker is really unhelpful at this point, though, because it doesn’t mark the date at which Step 4 was reached.

Bid tracker for the 311 RFP

The helpful (to a point) Bid Tracker for the 311 RFP.

2015

Mayor Emanuel announces a proposal to privatize the 311 system, including the technology and the staff, to be able to pay for the tech upgrade. Emanuel says it would save the city about $1 million annually.

Aldermen don’t like this. While I haven’t read up on their specific complaints, I believe they have to do with the quality of the jobs that would exist under a private operator.

Currently the 73** employees work for the city and have benefits like healthcare and probably some retirement assistance. The new jobs could be, well, anything, and if they’re like other call centers, probably pay less and could have worse benefits.

There’s also the question of whether anyone else can actually do the same (or better) job for less money (a question that 6th Ward Alderman Roderick Sawyer raised). That means someone other than people in the mayor’s office need to be able to get a detailed look at the expenses of the 311 call center and tech.

Andy Shaw questions why the privatization proposal is coming up now: “The timing is odd because the administration recently sent an enlightened privatization ordinance to the Council for approval, so it’s logical for the city to wait until it’s analyzed, and then adopted, before embarking on new privatization initiatives” and “the ordinance has 40 aldermanic co-sponsors but hasn’t been heard in committee yet, so it’s premature to consider privatizing 3-1-1”.

Shaw suggests using the do-nothing legally-separate-entity Chicago Infrastructure Trust to upgrade the service. (The new board members Emanuel installed this year have “set an ambitious goal”, Shaw said, to convert the city’s streetlights to something more energy efficient. It would likely work by splitting the cost savings with a private contractor.

** The city’s salary dataset on the open data portal lists 56 employees with “311” in the job title. They collectively earn $3,519,732.00 annually.

Thank you to Derek Eder, Eric Sherman, George Nakhleh, and Dan O’Neil for the discussion and links.

Updated Monday, Oct. 12 to add quotes from Better Government Association’s Andy Shaw (a former investigative reporter for a TV station).

How to make a map of places of worship in Cook County using OpenStreetMap data

The screenshot shows the configuration you need to find and download places of worship in Cook County, Illinois, using the Overpass Turbo website.

If you’re looking to make a map of churches, mosques, synagogues and other places of worship, you’ll need data.

  • The Yellow Pages won’t help because you can’t download that.
  • And Google Maps doesn’t let you have a slice of their database, either.
  • There’s no open data* for this because churches don’t pay taxes, don’t have business licenses, and aren’t required to register in any way

This where OpenStreetMap (OSM) comes in. It’s a virtual planet that anyone can edit and anyone can have for free.

First we need to figure out what tag people use to identify these places. Sometimes on OSM there are multiple tags that identify the same kind of place. You should prefer the one that’s either more accurate (and mentioned as such in the wiki) or widespread.

The OSM taginfo website says that editors have added over 1.2 million places of worship to the planet using amenity=place_of_worship.

Now that we know which tag to look for, we need an app that will help us get those places, but only within our desired boundary. Open up Overpass Turbo, which is a website that helps construct calls to the Overpass API, which is one way to find and download data from OSM.

Tutorial

In the default Overpass Turbo query, there’s probably a tag in brackets that says [amenity=drinking_fountain]. Change that to say [amenity=place_of_worship] (without the quotes). Now change the viewport of the map to show only the area in which you want Overpass Turbo to look for these places of worship. In the query this argument is listed as ({{bbox}}).

The map has a search bar to find boundaries (cities, counties, principalities, neighborhoods, etc.) so type in “Cook County” and press Enter. The Cook County in Illinois, United States of America, will probably appear first. Select that one and the map will zoom to show the whole county in the viewport.

Now that we’ve set the tag to [amenity=place_of_worship] and moved the map to show Cook County we can click “Run”. In a few seconds you’ll see a circle over each place of worship.

It’s now simple to download: Click on the “Export” button and click “KML” to be able to load the data into Google Earth, “GeoJSON” to load it into a GIS app like QGIS, or “save GeoJSON to gist” to create an instant map within GitHub.

*You could probably get creative and ask a municipality for a list of certificates of occupancy or building permits that had marked “religious assembly” as the zoning use for the property.