Page 37 of 171

Converting a transit agency’s GTFS to shapefile and GeoJSON with QGIS

Many years ago I wrote a tutorial on how to use an ArcGIS plugin to convert a transit agency’s GTFS package – a group of files that describe when and where their buses and trains stop – into files that could easily be manipulated by popular GIS desktop software.

That was so long ago, before I became an expert in using QGIS, a free and open source alternative to ArcGIS.

This tutorial will show you how to convert GTFS to a shapefile and to GeoJSON so you can edit and visualize the transit data in QGIS.

Prerequisites

First you’ll need to have QGIS installed on your computer (it works with Linux, Mac, and Windows). Second you’ll need a GTFS package for the transit agency of your choice (here’s the one for Pace Suburban Bus*, which operates all suburban transit buses in Chicagoland). You can find another transit agency around the world on the GTFS Data Exchange website.

Section 1: Let’s start

  1. Open QGIS.
  2. Load your GTFS data into the QGIS table of contents (also called the Layers Panel). Click Layer>Add Layer>Add Delimited Text Layer. You will be adding one or two files depending on which ones are provided.

    QGIS add delimited text layer

    Add delimited text layer.

  3. Now, here it can get tricky. Not all transit agencies provide a “shapes.txt” file. The shapes.txt file draws out the routes of buses and trains. If it’s not provided, that’s fine, but if you turn them into routes based on the stops.txt data, then you will have funny looking and impossible routes.

    QGIs browse for the stops.txt file

    Browse for the stops.txt file

  4. Click on “Browse…” and find the “stops.txt”. QGIS will read the file very quickly and determine which fields hold the latitude and longitude coordinates. If its determination is wrong, you can choose a different “X field” (longitude) and “Y field” (latitude).
  5. Click “OK”. A new dialog box will appear asking you to choose a coordinate reference system (EPSG). Choose or filter for “WGS 84, EPSG:4326”. Then click “OK”.
  6. The Pace bus stops in the Chicagoland region are now drawn in QGIS!

    Pace bus stops are shown

    Pace bus stops are shown

  7. If the GTFS package you downloaded includes a “shapes.txt” file (that represents the physical routes and paths that the buses or trains take), import that file also by repeating steps 4 and 5.

Section 2: Converting the stops

It’s really easy now to convert the bus or train stops into a shapefile or GeoJSON representing all of those points.

  1. Right-click the layer “stops” in the table of contents (Layers Panel) and click “Save As…”.
  2. In the “Save vector layer as…” dialog box, choose the format you want, either “ESRI Shapefile” or “GeoJSON”. **
  3. Then click “Browse” to tell QGIS where in your computer’s file browser you want to save the file. Leave the “CRS” as-is (EPSG:4326).

    Convert the bus stops to a shapefile or GeoJSON.

    Convert the Pace bus stops to a shapefile or GeoJSON.

  4. Then click “OK” and QGIS will quickly report that the file has been converted and saved where you specified in step 3.

Section 3: Converting the bus or train routes

The “shapes.txt” file is a collection of points that when grouped by their route number, show the physical routes and paths that buses and trains take. You’ll need a plugin to make the lines from this data.

  1. Install the plugin “Points to Paths”. Click on Plugins>Manage and Install Plugins… Then click “All” and search for “points”. Click the “Points to Paths” plugin and then click the “Install plugin” button. Then click “Close”.

    Install the Points to Paths plugin.

    Install the Points to Paths plugin.

  2. Pace bus doesn’t provide the “shapes.txt” file so we’ll need to find a new GTFS package. Download the GTFS package provided by the Chicago Transit Authority, which has bus and rail service in Chicago and the surrounding municipalities.
  3. Load the CTA’s “shapes.txt” file into the table of contents (Layers Panel) by following steps 4 and 5 in the first section of this tutorial.  Note that this data includes both the bus routes and the train routes.

    QGIS load CTA bus and train stops

    Import CTA bus and train stops into QGIS

  4. Now let’s start the conversion process. Click on Plugins>Points to Paths. In the next dialog box choose the “shapes” layer as your “Input point layer”.
  5. Select “shape_id” as the field with which you want to “Point group field”. This tells the plugin how to distinguish one bus route from the next.
  6. Select “shape_pt_sequence” as the field with which you want to “Point order field”. This tells the plugin in what order the points should be connected to form the route’s line.
  7. Click “Browse” to give the converted output shapefile a name and a location with your computer’s file browser.
  8. Make sure all  of the options look like the one in this screenshot and then click “OK”. QGIS and the plugin will start working to piece together the points into lines and create a new shapefile from this work.

    These are the options you need to set to convert the CTA points (stops) to paths (routes).

    These are the options you need to set to convert the CTA points to paths (routes).

  9. You’ll know it’s finished when the hourglass or “waiting” cursor returns to a pointer, and when you see a question asking if you would like the resulting shapefile added to your table of contents (Layers Panel). Go ahead and choose “Yes”.

    QGIS: CTA bus and train points are converted to paths (routes)

    The CTA bus and train points, provided in a GTFS package, have been converted to paths (routes/lines).

  10. Now follow steps 1-4 from Section 3 to convert the routes/lines data to a shapefile or GeoJSON file.**

Notes

* As of this writing, the schedules in Pace’s GTFS package are accurate as of January 18, 2016. It appears their download link always points to the latest version. Transit schedules typically change several times each year. Pace says, “Only one package is posted at any given time, typically representing Pace service from now until a couple of months in the future. Use the Calendar table to see on which days and dates service in the Trips table are effective.”

** Choose GeoJSON if you want to show this data on a web map (like in Leaflet or the Google Maps API), or if you want to share the data on GitHub.

Cataloging the city’s emails about what staff said regarding Laquan McDonald’s death

Screen grab from the released Laquan McDonald video

The City of Chicago released a trove emails spread over eight PDF files containing 3,000 pages, on New Year’s Eve, wherein city staff, including from the mayor’s office, police department, law department, discussed how they should handle countless requests from the media for information about the police shooting of Laquan McDonald.

In the middle of the night on October 20, 2014, Chicago police officer Jason Van Dyke shot 17-year-old Laquan McDonald 16 times, killing him.

Over a year later, a dashcam video of the shooting was released, sparking protests, theresignation of Chicago’s Police Chief, a federal investigation, and calls for the Mayor and State’s Attorney to resign.

I set to work that night reading as many as I could, and gathering strangers on Twitter to help read and catalog them.

This blog post is intended to point to the Google Sheet where we – over a dozen people who congregated here via Twitter – recorded the details of each message, including what it said, what we think the communicators meant, and what information was missing.

Read the Laquan McDonald emails catalog.

Swiss transit journey planners can guide you to the top of any mountain

Steven’s note: I originally wrote this in January 2017 for Transitland, my contract employer at the time. Links may be broken.

Looking west from Mount Rigi-Kulm and you can see the cloud layer that prevents you from seeing Lake Lucerne. The two cog railways are parked in the middle.

Looking west from Mount Rigi-Kulm and you can see the cloud layer that prevents you from seeing Lake Lucerne. The two cog railways are parked in the middle.

A month ago I hopped over to Germany to start a holiday trip over Christmas and the New Year. I flew into Frankfurt but I would be returning to Chicago from Zurich, Switzerland, almost three weeks later. I had spent two hours in Zurich in 2016 on a layover, and I was struck by the city’s beauty and their amazing public transport system. I made it a priority to revisit Zurich, to have a proper stay.

Before I left, I was already working to import the single GTFS transit feed for the whole country into Transitland, so I was aware of some of the transit systems. That work continues because the feed is massive; it has more than 400 operators and I need to add metadata about each of them.

I arrived the night before my mountain trip to a hotel – a 3 minute walk to the nearest entrance to Zurich’s hauptbahnhof (main station) – and I spent that whole evening planning an epic transit and mountain adventure the next day. (I stayed in because it’s also pretty expensive to go out in Zurich, so I was also saving my money for what turned out to be an _expensive _ epic trip.)

When in Switzerland, I figured, you should spend time outside on a mountain. And there’s no exception in the winter.

a view of Lake Lucerne from inside the cog railway train that's going up the mountain

It’s a cog railway up a Swiss mountain, of course it’s going to look steep like this.

I googled “nearest mountain to Zurich” and found Mount Rigi. I never validated if Mount Rigi is the nearest mountain, but after reviewing details on how to get to the base and how to get to the top (the mountain has its own website), I could tell it would be possible to go there and return in the same day.

Mount Rigi has multiple peaks, the tallest of which is Rigi Kulm at 1,798 meters, and you can plan a trip directly there with a single app.

You can use the Swiss Federal Railways (SBB) smartphone app or website to plan a trip from anywhere in Switzerland to the cog railway station below the restaurant atop Rigi Kulm. Seriously.

I wanted to use as many modes as possible, and I don’t like going on the same route more than once, so I adjusted SBB’s recommended route to travel from Zurich to Rigi Kulm via Lucerne and Vitznau. This was my outgoing itinerary:

  • Depart Zurich HB at 10:04 on InterRegio 2637 to Lucerne, arriving at 10:49
  • Depart Luzern Bahnhofquai (train station dock) on a boat across Lake Lucerne at 11:12 to Vitznau at the base of the mountain, arriving at 12:09
  • Depart Vitznau on Rigi-Bahnen 1127 at 12:15, arriving to the peak at 12:47

After spending about five hours on the mountain – I took a small cable car to a second peak – I heaaded down the mountain on a different cog railway to Arth-Goldau, a valley town with InterCity train service direct to Zurich.

screenshot of the SBB journey planner showing the trip from Zurich to the top of Mount Rigi-Kulm, changing from an intercity train to a boat to a cog railway.

The SBB website shows my actual itinerary. This isn’t the first recommended itinerary because there are more direct and faster ways to get to Rigi Kulm from Switzerland, but I wanted to ride in a boat so I added the “via” stop in Lucerne.

What was more fascinating than the legendary on-time performance and convenient and short connection times of the Swiss public transport network was that I bought trips for the boat, two cog railways, and the return train on a single ticket.

I could have bought a single ticket for the entire trip back in Zurich before I departed but I was in a hurry to catch that 10:04 train and it takes a bit longer to buy a multi-stop journey from the ticket vending machines. (You can also buy the ticket on the website and app, which quoted 98 Swiss Francs, or $96, without the return from Arth-Goldau.)

The second cog railway I took on this trip, to Arth-Goldau, opened in 1875, four years after the first cog railway of the day from Vitznau. That one opened in 1871, the first cog railway in Europe.

If I had missed the 10:04 train, there would have been another train leaving for Lucerne less than 30 minutes later, but I would arrive about 30 minutes early for the next boat and cog railway because they run less frequently.

On the day I traveled, Friday, January 6, the journey took 2 hours and 43 minutes. I checked SBB’s website for this blog post and they recommend a differently, slightly longer journey on weekends, at 3 hours and 1 minute. And they really mean that 1 minute.

The Swiss railway clock’s second hand waits at the 58.5 second mark and proceeds when it receives a “minute impulse” signal from the SBB’s master clock. Train operators then depart.

Get to know the Swiss timetable

The single feed includes the Swiss Federal Railways (SBB), city transit systems, intercity buses like PostAuto, funiculars, cable cars, cog railways, and even chair lifts.

You can take a sesselbahn (chair/ski lift) from Feldis/Veulden to avoid an uphill hike to Mutta; it’s operated by Sesselbahn und Skilifte Feldis AG. You can find its two stops and straight route up the mountain in Transitland’s Feed Registry.

We’re working to import all of them into the Transitland datastore, and we’ll get there eventually (it takes a lot of time to add metadata like an operator’s metropolitan coverage area and canton). For now, though, we’ve added the stops and routes for 11 operators, including all of the ones that covered my trip to Mount Rigi.

Steven’s note: there used to be an embedded map hosted at the following URL:

https://tangrams.github.io/tangram-frame/?noscroll&maxbounds=46.891,7.667,47.501,9.198&url=https://transit.land/images/switzerland-transit/scene.yaml#10.6461/47.1304/8.4492

Edit this map yourself in Tangram Play. These routes were extracted via Mobility Explorer and its direct connection to the Transitland API and I edited some of them because many of routes in the Swiss feed are very simplified.

#Space4Cycling: Chicago needs intuitive bike lanes and other street markings

Two bicyclists take different routes around this driver blocking the bike lane with their car

In this case at Milwaukee and Green, space was made and well-marked for cycling but no space was outlined for driving. The driver of the black car must pull up this far to see beyond the parked silver car. In the Netherlands they’ve come up with a solution that would work here: shift the green bike lane toward the crosswalk so that the motorist crosses the crosswalk and bike lane at the same time and has space to wait to turn left between the bike lane and the travel lane.

What does an intuitive bike lane or other street marking mean?

It means that the street user can reasonably (yeah) guess, and guess right, what they’re supposed to do.

For bicyclists in Chicago, the lack of bike lane markings that continue to the edge of an intersection (often demarcated at the stop bar) creates an unintuitive bike lane design.

At intersections, an intuitive bike lane design would mean that the bicyclist and the motorist know where and how to position their vehicles in respect to the other, even if there isn’t a car there yet, or there’s not a bike there yet. Many intersections in Chicago that have protected bike lanes do this; especially the ones with separate signal phases. And these intersections work really well for bicyclists: they stand safely away from motorists, and motorists don’t attempt to occupy these spaces.

Inverted sharrow

The “sharrow before and after the intersection because the city dropped the bike lane” is the most common “didn’t make space for cycling” problem. There was plenty of space to make for cycling here, and nearly every other “sharrow…” situation: it’s along the curb and it’s subsidized, curbside parking for drivers.

But currently at dozens, if not hundreds, of Chicago intersections where the bike lane drops before the intersection, you’ll see bicyclists behave and maneuver in several ways, none of which are accommodated by the street’s design.

Some people will bike between two lanes of cars to the front of the line, and when they get there, lacking a bike box or advanced stop line, they’ll stand with their bike in the area between the crosswalk and the stop bar. If the first car is over the stop bar, then people will usually stand with their bike on the crosswalk.

Riding north on Damen towards Fullerton-Elston

The sharrow painted on the pavement, and an accompanying sign saying, “shared lane – yield to bikes” are unintuitive because no one can occupy the same space at the same time, and the symbols don’t communicate who gets first right to a specific part of the road space. In the end, though, in a situation like this, I’ve never seen someone wait back this far on their bike, and many will consider riding on the sidewalk to get to the front. When they get there, though, they won’t find any #space4cycling.

Others will bike between a lane of cars and the curb to get to the front of the line.

New buffered bike lane on Halsted just ends

This is another version of the “sharrow before and after the intersection because the city dropped the bike lane”. Why’d they drop it in this instance? To make space for Halsted Street drivers turning right, and to push more drivers northward through its intersection with Clybourn Avenue.

Others will wait to the side of drivers, and other still will wait behind a line of cars, putting themselves at a major time disadvantage as the people who biked up to the front. Not to mention they’ll choke on more fumes.

Then, when the light turns green, motorists behave differently. Some will follow behind the first bicyclist, while others will try to pass but closely because they’re essentially sharing a lane side-by-side – this exerts a lot of mental stress on the bicyclist.

Where the city has built space that’s absolutely not to be shared (meaning it’s for the exclusive use by people bicycling), then the designs are substandard because they still allow or seem open to driving. Otherwise, though, space for cycling that’s “part time” is only usable space for those holding the most power and not for the people riding bikes who need it.

frankling at washington bike lane (composite image)

In this new design that built a “protected intersection” for bicyclists going north on Franklin and east on Washington Street, the bike space is still a drivable area. (Top photo by Kevin Zolkiewicz; bottom photo by Skip Montanaro)

These deficiencies in Chicago’s bike lane network are often the result of failing to make, or make well, space for cycling from space used for parking or turn lanes.

Bicycling on the Dearborn Street bike lane

Three years after the City of Chicago built the novel and well-used two-way cycle track on one-way Dearborn, this situation north of the track still exists. And somehow they expect drivers on a 4-lane road to travel at 20 MPH.

This is 2015 and we continue to “not make space for cycling” despite every policy that calls for making bicycling in Chicago safe and convenient so that more people will do it. It’s just that in the unwritten policies it says that you can implement that policy if it doesn’t impede driving*.

* The City of Chicago has built many road diets (a reduction in the number of travel lanes) in the last four years, and some before that. A few of these have worked well for bicyclists, like on 55th and Vincennes where they built protected and buffered bike lanes, respectively (and Dearborn through the Loop).

I put road diets in a note after “impede driving” because they’re only done where they also won’t make local traffic more congested on that street or an intersecting streets.

On the face of it, that’s exactly what many people believe they’ll do because a road diet removes or converts lanes and that’s seen as the same as reducing car capacity which will shift that car traffic to other streets. That pretty much doesn’t happen and the city only implements road diets on streets that have MORE capacity than is used.

How to extract highways and subway lines from OpenStreetMap as a shapefile

It’s possible to use Overpass Turbo to extract any object from the OpenStreetMap “planet” and convert it from a GeoJSON or KML file to a shapefile for manipulation and analysis in GIS.

Say you want the subway lines for Mexico City, and you can’t find a GTFS file that you could convert to shapefile, and you can’t find the right files on Sistema de Transporte Colectivo’s website (I didn’t look for it).

Here’s how to extract the subway lines that are shown in OpenStreetMap and save them as a GIS shapefile.

This is my second tutorial to describe using Overpass Turbo. The first extracted places of worship in Cook County. I’ve also used Overpass Turbo to extract a map of campgrounds

Extract free and open source data from OpenStreetMap

  1. Open the Overpass Turbo website and, on the map, search for the city from which you want to extract data. (The Overpass query will be generated in such a way that it’ll only search for data in the current map view.)
  2. Click the “Wizard” button in the top toolbar. (Alternatively you can copy the code below and paste it into the text area on the website and click the “Run” button.)
  3. In the Wizard dialog box, type in “railway=subway” in order to find metro, subway, or rapid transit lines. (If you want to download interstate highways, or what they call motorways in the UK, use “highway=motorway“.) Then click the “build and run query” button.
  4. In a few seconds you’ll see lines and dots (representing the metro or subway stations) on the map, and a new query in the text area. Notice that the query has looked for three kinds of objects: node (points/stations), way (the subway tracks), relation (the subway routes).
  5. If you don’t want a particular kind of object, then delete its line from the query and click the “Run” button. (You probably don’t want relation if you’re just needing GIS data for mapping purposes, and because routes are not always well-defined by OpenStreetMap contributors.)
  6. Download the data by clicking the “Export” button. Choose from one of the first three options (GeoJSON, GPX, KML). If you’re going to use a desktop GIS software, or place this data in a web map (like Leaflet), then choose GeoJSON. Now, depending on what browser you’re using, a couple things could happen after you click on GeoJSON. If you’re using Chrome then clicking it will download a file. If you’re using Safari then clicking it will open a new tab and put the GeoJSON text in there. Copy and paste this text into TextEdit and save the file as “mexico_city_subway.geojson”.

Overpass Turbo screenshot 1 of 2

Screenshot 1: After searching for the city for which you want to extract data (Mexico City in this case), click the “Wizard” button and type “railway=subway” and click run.

Overpass Turbo screenshot 2

Screenshot 2: After building and running the query from the Wizard you’ll see subway lines and stations.

Overpass Turbo screenshot 3

Screenshot 3: Click the Export button and click GeoJSON. In Chrome, a file will download. In Safari, a new tab with the GeoJSON text will open (copy and paste this into TextEdit and save it as “mexico_city_subway.geojson”).

Convert the free and open source data into a shapefile

  1. After you’ve downloaded (via Chrome) or re-saved (Safari) a GeoJSON file of subway data from OpenStreetMap, open QGIS, the free and open source GIS desktop application for Linux, Windows, and Mac.
  2. In QGIS, add the GeoJSON file to the table of contents by either dragging the file in from the Finder (Mac) or Explorer (Windows), or by clicking File>Open and browsing and selecting the file.
  3. Convert it to GeoJSON by right-clicking on the layer in the table of contents and clicking “Save As…”
  4. In the “Save As…” dialog box choose “ESRI Shapefile” from the dropdown menu. Then click “Browse” to find a place to save this file, check “Add saved file to map”, and click the “OK” button.
  5. A new layer will appear in your table of contents. In the map this new layer will be layered directly above your GeoJSON data.

Overpass Turbo screenshot 4

Screenshot 4: The GeoJSON file exported from Overpass Turbo has now been loaded into the QGIS table of contents.

Overpass Turbo screenshot 5

Screenshot 5: In QGIS, right-click the layer, select “Save As…” and set the dialog box to have these settings before clicking OK.

Query for finding subways in your current Overpass Turbo map view

/*
This has been generated by the overpass-turbo wizard.
The original search was:
“railway=subway”
*/
[out:json][timeout:25];
// gather results
(
// query part for: “railway=subway”
node["railway"="subway"]({{bbox}});
way["railway"="subway"]({{bbox}});
relation["railway"="subway"]({{bbox}});
/*relation is for "routes", which are not always
well-defined, so I would ignore it*/
);
// print results
out body;
>;
out skel qt;