CategoryOpen Access

How to download data from ArcGIS MapServers using your computer’s command line

A lot of geospatial data (GIS) is stored on ArcGIS MapServers, which is part of the Esri “stack” of products that municipalities use to manage and publish GIS data. And a lot of people want that data. If you have ArcGIS software on your Windows computer, then it can be pretty easy to plug in the map server URL and manipulate and extract the data.

For the rest of us who don’t have an extremely expensive license to that software, you can use a “command line” tool (written in Python) on any computer to download any layer of GIS data hosted on the ArcGIS MapServer and automatically convert it to GeoJSON.

You’ll need to install the Python package pyesridump, from the OpenAddresses GitHub repository, created by Ian Dees and other contributors.

Installing pyesridump is easy if you have pip installed, using the command pip install esridump.

The next thing you’ll need is the URL to a layer in a MapServer, and these are not easy to find.

Finding data to download

I can guarantee the county where you live has one. Before you continue, check to see if your county (or other jurisdiction) has the “open data portal” add-on to their ArcGIS stack.

Here are links to the open data portals enabled by Esri for Lake County, Illinois, and Broomfield County, Colorado). This is much easier to browse and find data to download (in shapefile and other formats) and you can skip this tutorial.

I don’t have a good recommendation to find the MapServer URL, though. A reader suggested looking for MapServers for jurisdictions around the world by looking through Esri’s portal of open data called ArcGIS Hub. Once you locate a dataset you want, you can find the MapServer URL under About>Data Source on the right side of the page.

I normally find them by looking at the HTML source code of a MapServer I already know about.

For this example I’ll use one of the GIS layers in the Cook County, Illinois, election service MapServer – here’s the layer for the Cook County commissioners districts.

Fetch the data

Once you have the URL the command is simple:

esri2geojson http://cookviewer1.cookcountyil.gov/ArcGIS/rest/services/cookElectnSrvc/MapServer/11 cookcounty_commissioners.geojson

  • The first term, esri2geojson tells your computer which program to load.
  • The second term is the URL of the MapServer URL.
  • The third term is the filename and location where you want to store the file. I prefer running the command “inside” the folder where I want the file to be stored. You can also specify a full path of the file. On a Mac this would look like ~/Users/username/Documents/GIS/projectname/cookcounty_commissioners.geojson

After you enter the command into your computer’s terminal, press enter. esri2geojson will report back once, after it finds and understands the MapServer URL you gave it. When it’s done, the command will “close” and your computer’s terminal will wait for the next command.

Do you have questions, or need some help? Leave a comment below.

Cataloging the city’s emails about what staff said regarding Laquan McDonald’s death

Screen grab from the released Laquan McDonald video

The City of Chicago released a trove emails spread over eight PDF files containing 3,000 pages, on New Year’s Eve, wherein city staff, including from the mayor’s office, police department, law department, discussed how they should handle countless requests from the media for information about the police shooting of Laquan McDonald.

In the middle of the night on October 20, 2014, Chicago police officer Jason Van Dyke shot 17-year-old Laquan McDonald 16 times, killing him.

Over a year later, a dashcam video of the shooting was released, sparking protests, theresignation of Chicago’s Police Chief, a federal investigation, and calls for the Mayor and State’s Attorney to resign.

I set to work that night reading as many as I could, and gathering strangers on Twitter to help read and catalog them.

This blog post is intended to point to the Google Sheet where we – over a dozen people who congregated here via Twitter – recorded the details of each message, including what it said, what we think the communicators meant, and what information was missing.

Read the Laquan McDonald emails catalog.

Links between Emanuel’s campaign donors and their building projects

The Tribune called out Emanuel’s appearance at a press conference as an endorsement of a locally-designed skyscraper (Studio Gang and bKL Architecture) to be built by Wanda, a Chinese development company – it has yet to receive any approval. Photo: Ted Cox, DNAinfo.

The Chicago Tribune reviewed the campaign contributions of Mayor Rahm Emanuel’s top donors and linked each donor to how it does business with Emanuel or the city. The article overall discussed how easy it is for Rahm to raise more money than what’s probably necessary to be elected a second time.

The Tribune graciously provided this data as a simple table which I’ve republished here in order to add links to building permit information from Chicago Cityscape. The website I’ve developed lists company and person names in an immediately searchable form. Currently there are over 90,000 companies, architects, and property owners that have received a building permit since 2010. Use the Illinois Sunshine database to find out who’s contributing to whom in the Chicago election.

Note: You’ll see “listed under [many] names” for several companies; this indicates that the Chicago building permit database uses different spellings, or the company has changed their name.

[table id=1 /]

Neither the article nor this table are meant to indicate any wrongdoing – campaign donations are public and it’s common to receive them from companies that do business in Chicago. It’s the extent that the donation appears to pay for favors or favoritism over other donors (which may be competing companies), or what’s right, that determines when immorality becomes an issue (a connection that’s hard to demonstrate).

Get out of Googleville: my presentation on web mapping

Alternate headlines: Google Maps versus OpenStreetMap; why OpenStreetMap is better than Google Maps

I presented to the Chicago GIS Network Meetup group on February 5,2013, about alternatives to Google when it comes to mapping on the web. I created the presentation and outline a couple hours before giving it and came up with this slideshow with three frames.

Googleville 1 of 3

Google Maps and its data is a one-way street (or many one-way streets). Google will take data but won’t give it back.

Googleville 2 of 3

Google Maps has all of these features, but they’re easier to manipulate when you use an alternative. Alternatives like: MapBox, TileMill, OpenLayers, OpenStreetMap (made easy with JOSM), GeoCommons – I’m sure there are plenty more.

Googleville 3 of 3

OpenStreetMap is the Wikipedia of online mapping and geographic data. Considering switching to OSM.

Stolen Bike Registry data: Which train stations have the most bike theft?

If you can help it, don’t park your bike on the sidewalk under the tracks at the Clybourn Metra Station. Too many opportunities for theft here. 

The Stolen Bike Registry is a website created by Chicagoans for people to notify the community that their bike has been stolen. I make no claims to the accuracy or completeness (or anything) about this list or the dataset from which it was created. Because of less than optimal data collection practices, and a diversity of website users, the location information is difficult to comb through and present. I’ve used Google Refine to clean up some of the location data so that I can pick out the theft locations that represent CTA or Metra stations.

This is a list of the most reported bike theft locations that are CTA or Metra stations, from about June 13, 2006, to April 2, 2011, representing 1,740 bike theft reports*. It’s not known how many bike thefts were reported to the police because they don’t know.

CTA (13 stations)

Logan Square Blue Line CTA 8
Rockwell Brown Line CTA 5
Addison Brown Line CTA 2
Fullerton Red/Brown Line CTA 2
Paulina Brown Line CTA 2
Western & Milwaukee (Blue Line) CTA 2
Western Brown Line CTA 2
Addison Blue Line CTA 1
Chicago Brown Line CTA 1
Damen Blue Line CTA 1
Ashland Orange Line CTA  1
Cumberland Blue Line CTA 1
Wellington Brown Line CTA 1

The new bike racks at Clybourn Metra station are in a more visible spot. Maybe there’s even a security camera pointed at them some of the time. 

Metra (24 stations)

Clybourn Metra 19
Ravenswood Metra 18
Edgebrook Metra 4
Evanston Main Street Metra 2
Forest Glen Metra 2
Healy Metra 2
Lake Cook Metra 2
Ogilvie Metra 2
57th Street Metra 1
College Avenue Metra Train Station 1
Corner of Maple & Church in downtown Evanston, near Metra 1
Glenview Metra Station 1
Harlem Metra Station Berwyn, IL 1
Irving Park Metra Stop 1
Jefferson Park Metra 1
LaSalle Street Metra 1
Mayfair Metra 1
Metra Station at Davis Street, Evanston 1
Morton Grove Metra Station 1
Prairie Crossing Metra Station 1
Rogers Park Metra 1
Union Station Metra 1
Western Metra Station 1
Wilmette Metra 1

* Reports come from around the world. 10 dates have been excluded because their dates were anomalous, empty, or not possible.

Updated September 30 to correct a Metra station and combine it with another.

Using Google Refine to get the stories out of your data

Let’s say you’re perusing the 309,425 crash reports for automobile crashes in Chicago from 2007 to 2009 and you want to know a few things quickly.

Like how many REAR END crashes there were in January 2007 that had more than 1 injury in the report. With Google Refine, you could do that in about 60 seconds. You just need to know which “facets” to setup.

By the way, there are 90 crash reports meeting those criteria. Look at the screenshot below for how to set that up.

Facets to choose to filter the data

  1. Get your January facet
  2. Add your 2007 facet
  3. Select the collision type of “REAR END” facet
  4. Choose to include all the reports where injury is greater than 1 (click “include” next to each number higher than 1)

After we do this, we can quickly create a map using another Google tool, Fusion Tables.

Make a map

  1. Click Export… and select “Comma-separated value.” The file will download. (Make sure your latitude and longitude columns are called latitude and longitude instead of XCOORD and YCOORD or sometimes Fusion Tables will choke on the location and try to geocode your records, which is redundant.)
  2. Go to Google Fusion Tables and click New Table>Import Table and select your file.
  3. Give the new table a descriptive title, like “January 2007 rear end crashes with more than 1 injury”
  4. In the table view, click Visualize>Map.
  5. BAM!

I completed all the tasks on this page in under 5 minutes and then spent 5 more minutes writing this blog. “The power of Google.”

Be specific. Be, be specific.

Update September 5, 2011: I gave a short speech to Moving Design participants about language and word choice, a kind of follow up to this article, as a “policy insight of the day.”

When speaking or presenting, be as specific as possible. The following are examples specific to the course of transportation discussions.

“Car traffic banned from this road.” Are you also banning trucks and SUVs?

“Vehicles will be rerouted.” Does this include those riding bicycles? Here’s an example of a current detour that only mentions cars, buses, and trucks. Which route should someone riding a bicycle take? Sometimes state and local laws will classify a bicycle as a vehicle, but then exclude it in specific passages – it’s weird. Better just call out specific vehicles, be they of the motorized or human-powered variety.

“Cars are aggressive to bikes.” Cars and bikes don’t operate themselves.

“We plan to narrow the road to calm traffic.” Are you going to narrow the road, or narrow certain lanes and reassign portions of the road to different uses, like a protected bike lane, or wider sidewalk? Then give the measurement of lanes, the sidewalk, and the curb face-to-curb face width. Consider that “street” is not a synonym for “road.” Road often represents what’s between the curbs, and the pavement, while street includes the road as well as the sidewalk. Street is a bit more abstract as well, sometimes meaning the activity that occurs on or around roads (like “street life”).

“Ignorant drivers…” Or do they lack specific education and relevant information?

This bikeway in Bremen, Germany, uses both color and pavement design to delineate space for people bicycling (like me) and people walking.

Better bike crash map now available for Chicago

I met Derek at a get together for “urban geeks” last Tuesday where he told me he was making a filterable/searchable version of my Chicago bike crash map using the Google Fusion Tables API. It essentially allows you to perform SQL-like queries to show different results on the map than one view. It’s possible to do this yourself if you open the bike crash map in the full Google Fusion Tables interface (do that now).

You can use it now!

New bike crash map, click through to view

Derek’s map has the benefit of great interface to drill down to the data you want. You can select a day, a surface condition, and the injury type. To download the data yourself, you’ll still have to access the full Fusion Tables interface.

Door lane photo and graphic by Gary Kavanagh in Santa Monica, California.

And since the data is the same as my original map, crash reports involving motor vehicle doors are not included. Here’s why doorings are excluded.

Reminder about open data and Obama’s Open Government Directive

Quickly after taking office, President Obama issued a memorandum about open government and opening government data. Then came the Open Government Directive* which said:

To the extent practicable and subject to valid restrictions, agencies should publish information online in an open format that can be retrieved, downloaded, indexed, and searched by commonly used web search applications. An open format is one that is platform independent, machine readable, and made available to the public without restrictions that would impede the re-use of that information.

Essentially, the executive government (er, Obama Administration) adopts the presumption of openness, that distributing public data is the default position and action to take.

Don’t squat on the data. Don’t fret over how people will view or manipulate the data – this is not your concern. Don’t delay its release. If you do this, you are a frigid dataist and I will remember this.

Photo of visual note taking at an open data seminar by Karen Quinn.

*The Directive has a little more backbone than the original memorandum: “This memorandum requires executive departments and agencies to take the following steps toward the goal of creating a more open government.”

Thank you to Tech President.

Trying out uDig, a free, multi-platform GIS application

ArcGIS is the standard in geographic information system applications. I don’t like that it’s expensive, unwieldy to install and update, and its user interface is stymying and slow*. I also use Mac OS X most of the time and ArcGIS is not available for Mac. It doesn’t have to be the standard.

I’ve tried my hand at Cartographica and QGIS. I really like QGIS because there’re many plugins, it’s open source, there’s a diverse community supporting it, and best of all, it’s free. I’ve written about Cartographica once – I’m not a fan right now.

My project

  • The data: Bicycle crashes in the City of Chicago as reported to IDOT for 2007-2009
  • Goal: Publish an interactive map of this data using Google Fusion Tables and its instant mapping feature.
  • Visualizing it: Added streets (prepared beforehand to exclude highways), water features, and city boundary (get that here)
  • Process: Combine bike crash data; reproject to WGS84 for Google; remove extraneous information; add latitude/longitude coordinates; export as CSV; upload to Google Fusion Tables; map it!
  • View the final product

Trying out uDig

In reaching my goal I had a task that I couldn’t figure out how to complete with QGIS: I needed to combine three shapefiles with identical table schemes into one shapefile – this one shapefile would eventually be published as one map. The join feature in fTools wasn’t working so I looked for a new solution, uDig, or “User-friendly Desktop Internet GIS.”

The solution was very easy. Highlight all the records in the attribute table of one shapefile, click Edit>Copy, then select the destination table and click Edit>Paste. The new records were added within a couple seconds. I could then bring this data back into QGIS to finish the process (outlined above under Project). I did use fTools later in the process to add lat/long coordinates to my single shapefile.

After adding more data to better visualize the crashes in Chicago, I noticed that uDig renders maps to look smoother and slightly prettier than QGIS or ArcGIS. See the screenshot below.

A screenshot of the three bicycle crash datasets (2007, 2008, 2009) with the visualization data added.

The end product: three years of police reported bicycle crashes in the City of Chicago on an interactive map powered by Google Fusion Tables, another product in Google’s arsenal of GIS for the poor man. View the final product.

*I haven’t used ArcGIS version 10 yet, which I see and read has an improved user interface; it’s unclear to me and other users if the program’s been updated to take advantage of multi-core processors. ESRI has a roundabout way of describing their support.

© 2017 Steven Can Plan

Theme by Anders NorénUp ↑