TagArcGIS

How to download data from ArcGIS MapServers using your computer’s command line

A lot of geospatial data (GIS) is stored on ArcGIS MapServers, which is part of the Esri “stack” of products that municipalities use to manage and publish GIS data. And a lot of people want that data. If you have ArcGIS software on your Windows computer, then it can be pretty easy to plug in the map server URL and manipulate and extract the data.

For the rest of us who don’t have an extremely expensive license to that software, you can use a “command line” tool (written in Python) on any computer to download any layer of GIS data hosted on the ArcGIS MapServer and automatically convert it to GeoJSON.

You’ll need to install the Python package pyesridump, from the OpenAddresses GitHub repository, created by Ian Dees and other contributors.

Installing pyesridump is easy if you have pip installed, using the command pip install esridump.

The next thing you’ll need is the URL to a layer in a MapServer, and these are not easy to find.

Finding data to download

I can guarantee the county where you live has one. Before you continue, check to see if your county (or other jurisdiction) has the “open data portal” add-on to their ArcGIS stack.

Here are links to the open data portals enabled by Esri for Lake County, Illinois, and Broomfield County, Colorado). This is much easier to browse and find data to download (in shapefile and other formats) and you can skip this tutorial.

I don’t have a good recommendation to find the MapServer URL, though. A reader suggested looking for MapServers for jurisdictions around the world by looking through Esri’s portal of open data called ArcGIS Hub. Once you locate a dataset you want, you can find the MapServer URL under About>Data Source on the right side of the page.

I normally find them by looking at the HTML source code of a MapServer I already know about.

For this example I’ll use one of the GIS layers in the Cook County, Illinois, election service MapServer – here’s the layer for the Cook County commissioners districts.

Fetch the data

Once you have the URL the command is simple:

esri2geojson http://cookviewer1.cookcountyil.gov/ArcGIS/rest/services/cookElectnSrvc/MapServer/11 cookcounty_commissioners.geojson

  • The first term, esri2geojson tells your computer which program to load.
  • The second term is the URL of the MapServer URL.
  • The third term is the filename and location where you want to store the file. I prefer running the command “inside” the folder where I want the file to be stored. You can also specify a full path of the file. On a Mac this would look like ~/Users/username/Documents/GIS/projectname/cookcounty_commissioners.geojson

After you enter the command into your computer’s terminal, press enter. esri2geojson will report back once, after it finds and understands the MapServer URL you gave it. When it’s done, the command will “close” and your computer’s terminal will wait for the next command.

Do you have questions, or need some help? Leave a comment below.

Trying out uDig, a free, multi-platform GIS application

ArcGIS is the standard in geographic information system applications. I don’t like that it’s expensive, unwieldy to install and update, and its user interface is stymying and slow*. I also use Mac OS X most of the time and ArcGIS is not available for Mac. It doesn’t have to be the standard.

I’ve tried my hand at Cartographica and QGIS. I really like QGIS because there’re many plugins, it’s open source, there’s a diverse community supporting it, and best of all, it’s free. I’ve written about Cartographica once – I’m not a fan right now.

My project

  • The data: Bicycle crashes in the City of Chicago as reported to IDOT for 2007-2009
  • Goal: Publish an interactive map of this data using Google Fusion Tables and its instant mapping feature.
  • Visualizing it: Added streets (prepared beforehand to exclude highways), water features, and city boundary (get that here)
  • Process: Combine bike crash data; reproject to WGS84 for Google; remove extraneous information; add latitude/longitude coordinates; export as CSV; upload to Google Fusion Tables; map it!
  • View the final product

Trying out uDig

In reaching my goal I had a task that I couldn’t figure out how to complete with QGIS: I needed to combine three shapefiles with identical table schemes into one shapefile – this one shapefile would eventually be published as one map. The join feature in fTools wasn’t working so I looked for a new solution, uDig, or “User-friendly Desktop Internet GIS.”

The solution was very easy. Highlight all the records in the attribute table of one shapefile, click Edit>Copy, then select the destination table and click Edit>Paste. The new records were added within a couple seconds. I could then bring this data back into QGIS to finish the process (outlined above under Project). I did use fTools later in the process to add lat/long coordinates to my single shapefile.

After adding more data to better visualize the crashes in Chicago, I noticed that uDig renders maps to look smoother and slightly prettier than QGIS or ArcGIS. See the screenshot below.

A screenshot of the three bicycle crash datasets (2007, 2008, 2009) with the visualization data added.

The end product: three years of police reported bicycle crashes in the City of Chicago on an interactive map powered by Google Fusion Tables, another product in Google’s arsenal of GIS for the poor man. View the final product.

*I haven’t used ArcGIS version 10 yet, which I see and read has an improved user interface; it’s unclear to me and other users if the program’s been updated to take advantage of multi-core processors. ESRI has a roundabout way of describing their support.

How to convert GTFS to GIS shapefiles and KML

This tutorial will teach how you to convert any transit agency’s General Transit Feed Specification (GTFS) data into ESRI ArcGIS-compatible shapefiles (.shp), KML, or XML. This is simple to do because GTFS data is essentially a collection of CSV (comma separated values) text files (really, really large text files).

Note: I don’t know how to do the reverse, converting shapefiles or other geodata into GTFS data. I’m not sure if this is possible and I’m still investigating it. If you have tips, let me know.

Converting GTFS to GIS shapefiles

Instructions require the use of ArcGIS (Windows only) and a free plugin called ET GeoWizards GIS for any version of ArcGIS. I do not have instructions for Mac users at this time.

I wrote these instructions while converting the Chicago Transit Authority’s GTFS files into shapefiles based on a reader’s request. “Field names” are quoted and layer names are italicized.

  1. Download the GTFS data you want. Find data from agencies around the world (although not many from Europe) on GTFS Data Exchange.
  2. Import into ArcGIS the shapes.txt file using Tools>Add XY Data. Specify Y=lat and X=lon
  3. Using ET GeoWizards GIS tools, in the Convert tab, convert the points shapefile to polyline.
  4. Select the shapes layer in the wizard, then create a destination file. Click Next.
  5. Select the “shape_id” field
  6. Click the checkbox next to Order and select the field “shape_pt_sequence” and click Finish.
  7. Depending on the number of records (the CTA has 466,000 shapes), it may take a while.
  8. The new shapefile will be added to your Table of Contents and appear in your map.
  9. Import the trips.txt and routes.txt files. Inspect them for any NULL values in the “route_id” field. You will be using this field to join the routes and trips table. It may be a case that ArcGIS imported them incorrectly; the text files will show the correct data. If NULL values appear, follow steps 10 and 11 and continue. If not, follow steps 10 and 12 and continue. This happens because ArcGIS inspected some of the data and determined they were integers and ignored text. However, this is not the case.
  10. Export the text files as DBF files so that ArcGIS operates on them better. Then remove the text files from the Table of Contents.
  11. (Only if NULL values appear) Go into editing mode and fix the NULL values you noticed in step 9. You may have to make a new column with a more forgiving data type (string) and then copy the “route_id” column into the new column. Then continue to step 12.
  12. Join routes and trips based on the field “route_id” – export as trips_routes.dbf
  13. Add a new column to shapes.shp called “shape_id2”, with data type double 18, 11. This is so we can perform step 14. Use the field calculator to copy the values from “shape_id” (also known as ET_ID) to “shape_id2”
  14. Join routes_trips with shapes into routes_poly based on the field “shape_id” (and “shape_id2”)
  15. Dissolve routes_poly on “route_id.” Make sure all selections are cleared. Use statistics/summary fields: “route_long,” “route_url.” Save as routes_diss.shp
  16. Inspect the new shapefile to ensure it was created correctly. You may notice that some bus routes don’t have names. Since these routes are well documented on the CTA website, I’m not going to fill in their names.

Click on the screenshot to see various steps in the tutorials.

Converting GTFS to KML

After you have it in shapefile form, converting to KML is easy – follow these instructions for using QGIS. Or if you want to skip the shapefile-creation process (quite involved!), you can use KMLWriter, a Python script. Also, I think the latest version of ArcGIS has built-in KML exporting.

Converting GTFS to XML

If you want to convert the GTFS data (which are essentially comma-separated value – CSV – files) to XML, that’s easier and you can avoid using GIS programs.

  • First try Mr. Data Converter (very user friendly).
  • If that doesn’t work, try this website form on Creativyst. I tested it by converting the CTA’s smallest GTFS table, frequencies.txt, and it worked properly. However, it has a data size limit. (User friendly.)
  • Next try csv2xml, a command line tool. (Not user friendly.)
  • You can also use Microsoft Excel, but read these tips and caveats first. (I haven’t found a Microsoft application I like or think is user friendly.)

Trying out new GIS software

I want to draw 50 and 120 feet buffers around the points of store entrances to show where bike parking should and shouldn’t be installed. I want to follow this example:

walgreens with bike parking buffers

Aerial photo of a Tucson, Arizona, Walgreens showing the location of existing bike parking and two buffers (50 and 120 feet) where proposed city rules would allow bike parking. I advocate for ratifying the 50 feet rule, which I’ve discussed on this blog and elsewhere many times.

I want to do this easily and accurately, so I will use GIS software to create a “buffer.” I use QGIS occasionally, but I want to try out other Mac-friendly applications. I’m getting my orthoimagery (geometrically corrected aerial photography) from the United States Geological Survey (USGS) using a web protocol called Web Map Server. I’m trying:

  • Cartographica, $495, with free trial license.
  • uDig, completely free software. UPDATE: I have had NO success getting any data to load from a WMS connection into uDig. I would like to understand why. Cartographica can obtain some of the WMS-stored data I want, although it messes up often.

I’m having success with neither – both are having issues downloading or maintaining a connection to the USGS orthoimagery. In one case, Cartographica trims the Bing Maps imagery to match the extent of my other objects (the buffer). In another case, it won’t even download the USGS imagery (and gives no indication that anything is happening). uDig hasn’t been able to download anything so far – I hope it’s asking for the current extent, instead of all data because it’s taking a looong time to do anything (so long that I just quit in the  middle of it).

This screenshot shows how to add new WMS connections to Cartographica.

UPDATE: I did it! I successfully used Cartographica (and the integrated Bing Maps) to create this drawing that shows the current (abysmal) bike parking at a Chicago Home Depot outside the 50 feet line.

How GIS helps earthquake relief efforts for Haiti

While Geographic Information systems software can definitely produce pretty maps, its power lies in analyzing data and plotting or comparing sensory or observed data to spatial data (like roads or terrain). The earthquake in Haiti rocked the capital city, Port-au-Prince with a shock of magnitude 7.0 on Tuesday, January 12, 2010.

A photo from a United States military flyover shows damage in the Port of Port-au-Prince, Haiti. Photo taken by Petty Officer 2nd Class Sondra-Kay Kneen and uploaded by Chuck Simmins.

There are several applications for GIS to help with earthquake response, and two blog posts that appeared this morning shed light on how.

The first article came from ESRI, the California-based makers of ArcGIS, the most used GIS application. The article linked to a user-built map on their ArcGIS Online service showing on Bing maps where the earthquake and its aftershocks struck (the map sits behind a registration wall). ESRI even has a disaster response team that helps organizations get their response projects off the ground quickly.

Infrastructurist posted the second article, showing some before and after satellite imagery of Haiti, provided by Google and GeoEye.

So what can GIS do? From ESRI’s list, “GIS for Disaster Response“:

  • Rapid identification of potential shelter/housing locations (schools, libraries, churches, public buildings) appropriate for supporting affected populations.
  • Determine how many tents will be needed based on the location of populations affected by the disaster.
  • Analyze areas where large numbers of refugees can establish camps out of harm’s way that are accessible for supply delivery and have access to water and other resources necessary to support large numbers of people.
  • Many more examples.

Want more information? Here’s where to get it:

A map of buildings designed by Frank Lloyd Wright in Oak Park

I would like to hear feedback on the design of this map I made for school. It shows the location of buildings in Oak Park, Illinois, by American architect Frank Lloyd Wright.

The map is accurate; the building listing is from Oak Park Tourist. Feel free to print out the map and go on your own walking tour!

I created this map based on data provided to me by my GIS for Planners class instructors. Also in the assignment they gave limited background on who is commissioning the map, who will use the map, and the information it should describe and display. As the class has progressed, the assignments have become more open ended.

However, in making maps, there are always certain elements you cannot do without. Map makers include a scale bar, north arrow, and source information so that their map appears trustworthy. A legend is most often required, but many times maps can be designed intuitively so that users do not require a legend. I have attempted to design this map like that: I excluded a legend. Map users should be able to discern that the black and gray lines represent streets and also that the larger map with the gray background is a zoomed in portion of the smaller map.

I have posted other maps to my Flickr that you can also browse, some I made because of assignments, and others for personal interest and practice.

Converting shapefiles and KML files

Google Earth Pro is a slightly more advanced version than the free edition of the popular satellite imagery application (okay, it does way more, but many people just use that feature). One major additional feature it includes is the ability to import GIS shapefiles and display their features on top of the imagery, including terrain. It’s useful to have your data as KML (Keyhole Markup Language) because KML (or KMZ) is easier to share and Google Earth standard edition is free. But then again, it’s useful to have your KML files as shapefiles because proper GIS software is more powerful at analyzing data. Also, someone might ask you for your data in shapefile format (but they could easily follow these instructions).

Good data management requires options. Options mean your data won’t be locked into a proprietary format. Data want to be free! Read on for ways to convert your KML and shapefile data:

Converting KML files to shapefiles

Like Google Earth Pro, Quantum GIS (QGIS) can convert KML or KMZ to shapefile, and best of all – it doesn’t cost $400 per year (it’s free!). QGIS is a cross-platform application meaning it will run on Windows, Mac OS X, and Linux.

Use QGIS to convert a KML or KMZ file to shapefile:

  1. Click on Layer > Add Vector Layer
  2. Find your KML or KMZ file.
  3. Right-click your new layer and click “Save as shapefile.”

Zonums provides online conversion tools. Or, use ArcGIS and this plugin to convert KML files to shapefiles.

Converting shapefiles to KML files

The freeware Shp2kml 2.o (Windows only) from Zonums will convert shapefiles to KML files. Want some free, interesting data to try it out? Check my ever expanding repository.

ESRI’s ArcGIS can convert KML files to shapefiles using this plugin and then import the shapefile as a layer onto your map.

Creating KML files online

As I described in this post, BatchGecode will generate a KML file for you by inputting a list of addresses and names. Additionally, Google Earth (part of the rising Google GIS platform) creates KML files. Google’s My Maps feature also allows you to generate KML files (for sharing or download) by clicking and drawing points and lines on a map and inviting you to describe the features you create. Use this to get a map of your church congregation, or a map of people who voted for your candidate.

GeoCommons Finder lets you upload geodata in many formats, save it to your profile, and then download it into multiple formats. You can upload a shapefile (.shp) and its accompany files (shx, dbf, and prj), verify that it read your data correctly,

More choices for converting

Additional software with conversion capabilities:

  • MapWindow (another free software choice; Windows only) – An alternative to QuantumGIS and ArcGIS.
  • ExpertGPS (Windows only, not free) – Ideal for GPS device owners, or for researchers using GPS devices in projects. But it can convert the GPS and shapefile data into KML, shapefiles, or a spreadsheet, amongst other functions.
  • Zonums, creator of the standalone Shp2kml software converter, now offers many online tools for KML users, including one that reverses the conversion and exports shapefiles from KML files. I found the link on FreeGeographyTools.com.
  • OpenGeo Suite – Commercial software with non-profit licenses.
  • uDig – Free GIS software, but I haven’t had good experiences with it on my computers.

GeoCommunity has a good article, with screenshots, on how some of these programs work.

Need to work with General Transit Feed Spec (GTFS) data?

Geocoding in Quantum GIS – QGIS

Geocoding is the process of turning street addresses into geographic coordinates. You can geocode easily in QGIS using several methods.

If you just want to geocode and you don’t need to see the addresses plotted on a map in QGIS, then follow these instructions. If you don’t need to see them on a map nor do you need the geographic coordinates, then use BatchGeocode.

If you only need to geocode a single address and get its coordinates immediately, use geocoder.us.

An example geocoded address on the map using the “single address” method.

How to geocode multiple addresses in QGIS

UPDATE April 11, 2013: Updated the directions because the “Add delimited text layer” function moved from the Plugins to Layer menu. 

UPDATE March 24, 2011: I updated the directions to use GPS Visualizer instead of BatchGeocode.com because BG stopped giving geographic coordinates in its output.

Get directions on geocoding a single address in QGIS with a plugin.

QGIS is an open-source Geographic Information Systems (GIS) application that has been gaining ground since 2004. It runs on all operating systems (it began as a Linux project) and you can download it for free.

I use it often because ESRI doesn’t make the popular ArcGIS software for Mac. That’s unfortunate, but like I said here, software, technology and mapping issues can be easily overcome – we can use QGIS to create maps. QGIS, though, is missing one major feature for basic map building: geocoding.

Here’s a step-by-step tutorial on how to bring in multiple street addresses and their XY coordinates into your QGIS map en masse: Continue reading

© 2019 Steven Can Plan

Theme by Anders NorénUp ↑