Search "kml"

Converting Google My Maps to KML and GPX

Convert your routes that you made in Google My Maps to GPX so that you can view them on Garmin GPS devices, or upload them to MapMyRide.

  1. Access your My Map. Your My Map must have lines or routes in it. It appears that a My Map with only points doesn’t convert correctly.
  2. Click on View in Google Earth. Your web browser will download a KML file. It may automatically open in Google Earth, but this is not necessary.
  3. Visit GPS Visualizer to convert your KML file to GPX
  4. Select GPX as your output.
  5. For the input, choose the KML file you just downloaded from Google My Maps.
  6. Click Convert. Your file will be uploaded and your GPX file will be presented for download on the next page.
  7. Download your GPX file from the link on the page.

You can now transfer the GPX file to your GPS device, or upload it to MapMyRide. I confirmed that MapMyRide successfully imports the Google My Map I converted following these instructions.

How to convert GTFS to GIS shapefiles and KML

This tutorial will teach how you to convert any transit agency’s General Transit Feed Specification (GTFS) data into ESRI ArcGIS-compatible shapefiles (.shp), KML, or XML. This is simple to do because GTFS data is essentially a collection of CSV (comma separated values) text files (really, really large text files).

Note: I don’t know how to do the reverse, converting shapefiles or other geodata into GTFS data. I’m not sure if this is possible and I’m still investigating it. If you have tips, let me know.

Converting GTFS to GIS shapefiles

Instructions require the use of ArcGIS (Windows only) and a free plugin called ET GeoWizards GIS for any version of ArcGIS. I do not have instructions for Mac users at this time.

I wrote these instructions while converting the Chicago Transit Authority’s GTFS files into shapefiles based on a reader’s request. “Field names” are quoted and layer names are italicized.

  1. Download the GTFS data you want. Find data from agencies around the world (although not many from Europe) on GTFS Data Exchange.
  2. Import into ArcGIS the shapes.txt file using Tools>Add XY Data. Specify Y=lat and X=lon
  3. Using ET GeoWizards GIS tools, in the Convert tab, convert the points shapefile to polyline.
  4. Select the shapes layer in the wizard, then create a destination file. Click Next.
  5. Select the “shape_id” field
  6. Click the checkbox next to Order and select the field “shape_pt_sequence” and click Finish.
  7. Depending on the number of records (the CTA has 466,000 shapes), it may take a while.
  8. The new shapefile will be added to your Table of Contents and appear in your map.
  9. Import the trips.txt and routes.txt files. Inspect them for any NULL values in the “route_id” field. You will be using this field to join the routes and trips table. It may be a case that ArcGIS imported them incorrectly; the text files will show the correct data. If NULL values appear, follow steps 10 and 11 and continue. If not, follow steps 10 and 12 and continue. This happens because ArcGIS inspected some of the data and determined they were integers and ignored text. However, this is not the case.
  10. Export the text files as DBF files so that ArcGIS operates on them better. Then remove the text files from the Table of Contents.
  11. (Only if NULL values appear) Go into editing mode and fix the NULL values you noticed in step 9. You may have to make a new column with a more forgiving data type (string) and then copy the “route_id” column into the new column. Then continue to step 12.
  12. Join routes and trips based on the field “route_id” – export as trips_routes.dbf
  13. Add a new column to shapes.shp called “shape_id2″, with data type double 18, 11. This is so we can perform step 14. Use the field calculator to copy the values from “shape_id” (also known as ET_ID) to “shape_id2″
  14. Join routes_trips with shapes into routes_poly based on the field “shape_id” (and “shape_id2″)
  15. Dissolve routes_poly on “route_id.” Make sure all selections are cleared. Use statistics/summary fields: “route_long,” “route_url.” Save as routes_diss.shp
  16. Inspect the new shapefile to ensure it was created correctly. You may notice that some bus routes don’t have names. Since these routes are well documented on the CTA website, I’m not going to fill in their names.

Click on the screenshot to see various steps in the tutorials.

Converting GTFS to KML

After you have it in shapefile form, converting to KML is easy – follow these instructions for using QGIS. Or if you want to skip the shapefile-creation process (quite involved!), you can use KMLWriter, a Python script. Also, I think the latest version of ArcGIS has built-in KML exporting.

Converting GTFS to XML

If you want to convert the GTFS data (which are essentially comma-separated value – CSV – files) to XML, that’s easier and you can avoid using GIS programs.

  • First try Mr. Data Converter (very user friendly).
  • If that doesn’t work, try this website form on Creativyst. I tested it by converting the CTA’s smallest GTFS table, frequencies.txt, and it worked properly. However, it has a data size limit. (User friendly.)
  • Next try csv2xml, a command line tool. (Not user friendly.)
  • You can also use Microsoft Excel, but read these tips and caveats first. (I haven’t found a Microsoft application I like or think is user friendly.)

The importance of sharing data in KML format

The KML file is an important format in which to share locational data. KML was developed by a company called Keyhole, which Google purchased in 2004, and subsequently released Keyhole’s flagship product: Earth.

A Keyhole Markup Language file is a way to display on a map (particularly a 3D globe of Earth) a collection of points with a defined style. Google has added more functionality and style to the KML format, expanding the styles that can be applied and the information that can be embedded.

KML, like XML (eXtensible Markup Language), is extremely web-friendly. For a web application at work I developed, I included this PHP class that creates an KML file on-demand based on a predefined database query. The file contains locations and attributes of recently installed bike racks in Chicago. EveryBlock imports the file and its information into their location-based service, aggregating many news types around your block.

But a KML file is more important than being the native file for use within Google Earth. It’s an open source text file that can be manipulated by a number of software programs on any computer system on earth (or read on a printed page). It’s not encoded, like shapefiles, so I can read the file with my own mind and understand the data it would present in a compatible map viewer. I see lines of organized syntax describing points and polygons, listing their attributes in plain language.

Have you ever tried to see the “inside” of a shapefile? Only GIS programs can read them for you. KML provides data producers and consumers the opportunities to keep data open, available, and easy to use. We need locational data for our work, and we need tools to help us use it, not hide it.

Converting shapefiles and KML files

Google Earth Pro is a slightly more advanced version than the free edition of the popular satellite imagery application (okay, it does way more, but many people just use that feature). One major additional feature it includes is the ability to import GIS shapefiles and display their features on top of the imagery, including terrain. It’s useful to have your data as KML (Keyhole Markup Language) because KML (or KMZ) is easier to share and Google Earth standard edition is free. But then again, it’s useful to have your KML files as shapefiles because proper GIS software is more powerful at analyzing data. Also, someone might ask you for your data in shapefile format (but they could easily follow these instructions).

Good data management requires options. Options mean your data won’t be locked into a proprietary format. Data want to be free! Read on for ways to convert your KML and shapefile data:

Converting KML files to shapefiles

Like Google Earth Pro, Quantum GIS (QGIS) can convert KML or KMZ to shapefile, and best of all – it doesn’t cost $400 per year (it’s free!). QGIS is a cross-platform application meaning it will run on Windows, Mac OS X, and Linux.

Use QGIS to convert a KML or KMZ file to shapefile:

  1. Click on Layer > Add Vector Layer
  2. Find your KML or KMZ file.
  3. Right-click your new layer and click “Save as shapefile.”

Zonums provides online conversion tools. Or, use ArcGIS and this plugin to convert KML files to shapefiles.

Converting shapefiles to KML files

The freeware Shp2kml 2.o (Windows only) from Zonums will convert shapefiles to KML files. Want some free, interesting data to try it out? Check my ever expanding repository.

ESRI’s ArcGIS can convert KML files to shapefiles using this plugin and then import the shapefile as a layer onto your map.

Creating KML files online

As I described in this post, BatchGecode will generate a KML file for you by inputting a list of addresses and names. Additionally, Google Earth (part of the rising Google GIS platform) creates KML files. Google’s My Maps feature also allows you to generate KML files (for sharing or download) by clicking and drawing points and lines on a map and inviting you to describe the features you create. Use this to get a map of your church congregation, or a map of people who voted for your candidate.

GeoCommons Finder lets you upload geodata in many formats, save it to your profile, and then download it into multiple formats. You can upload a shapefile (.shp) and its accompany files (shx, dbf, and prj), verify that it read your data correctly,

More choices for converting

Additional software with conversion capabilities:

  • MapWindow (another free software choice; Windows only) – An alternative to QuantumGIS and ArcGIS.
  • ExpertGPS (Windows only, not free) – Ideal for GPS device owners, or for researchers using GPS devices in projects. But it can convert the GPS and shapefile data into KML, shapefiles, or a spreadsheet, amongst other functions.
  • Zonums, creator of the standalone Shp2kml software converter, now offers many online tools for KML users, including one that reverses the conversion and exports shapefiles from KML files. I found the link on FreeGeographyTools.com.
  • OpenGeo Suite – Commercial software with non-profit licenses.
  • uDig – Free GIS software, but I haven’t had good experiences with it on my computers.

GeoCommunity has a good article, with screenshots, on how some of these programs work.

Need to work with General Transit Feed Spec (GTFS) data?

Converting shapefiles to GeoJSON, and other format conversions

To develop the Chicago Bike Map app, I had a problem I thought would be simple to solve: load train lines into a Leaflet-powered map. I had the train lines stored as a polyline shapefile but Leaflet can only read the GeoJSON format or a string of geographic coordinates representing lines.

I eventually found a solution (I can’t remember how) and I need to share it with you. The converter can do more than ESRI shapefiles to GeoJSON. It can reproject the data in the conversion. It can convert from several formats to several other formats.

The site is called MyGeodata Converter. You upload a ZIP file of geographic files – .shp and its companion files (.prj, .dbf, .shx), .kml, and .gpx. Let’s take the Chicago Transit Authority train lines shapefile straight from the City of Chicago’s open data portal. It downloads as a zipped collection of a shapefile and its buddies and we can take this file straight to the Converter and upload it. The Converter will unzip it and read the data; it will even identify the projection system (for Chicago-based geographic data, its common to use NAD83 Illinois StatePlane East FIPS 1201 Feet (SRID 102671, the same as SRID 3435).

The Converter will convert to one of the following formats, with same or new projection; accepts SQL statements to extract a subset of data:

  • ESRI shapefile
  • GML
  • KML, KMZ
  • GeoJSON
  • Microstation DGN
  • MapInfo File
  • GPX
  • CSV

BikeLock app based on dataset I opened up

Bike parking at Daley Plaza, downtown Chicago. 

It’s really cool to see work you did “go places”. A friend of mine who works at Groupon just linked me to an iOS app called BikeLock that finds bike racks near you on iPhones, iPads, and iPod touches. It’s based on bike rack location data in the City of Chicago’s Data Portal. (The data on there is old, while the data in the public API I built is real time.)

Download it from the iTunes Store for 99 cents. The developer is Mike Jahn, another Groupon staffer. You can get the same information for free, though, on my mostly mobile-friendly Can I bring my bike on Metra? web app, and a website I made for the Chicago Department of Transportation (CDOT).

A screenshot of the Can I bring my bike on Metra? bike rack finder website. 

That data comes straight from the Bike Parking Web Application I started developing in 2008 soon after I started working in the Chicago Bicycle Program. It was good that my supervisor had the same perspectives I did about open and transparent data and work. But it didn’t start like that; here’s the full story:

My first job at the Bicycle Parking Program was to deal with abandoned bikes, get them off the street. I was taught the existing method of keeping track of my work, but I used my programming skills (in PHP, MySQL, and with the Google Maps API) to develop a web application that tracked it faster and mapped out the abandoned bikes I had to visit and tag with a notice. I was using this for a few days or few weeks and then show my boss. His reaction was something like, “Great! Now make one for bike racks!”

Why? Well, let’s take this quote from Judy Baar Topinka, Illinois comptroller, speaking Tuesday about her office’s new website, The Ledger, which lists the state’s unpaid bills among other financial data.

“The object of the exercise is to make everything that we know of in the comptroller’s office public. If we know it, you’ll know it.” WBEZ

I made one for bike racks. I created two environments, one for private administration at the office (“Bike Parking Web Application”) and one for the public (“public interface”). A later feature I added to the public interface was the Advanced Search. This allows you to filter by Ward, Community Area, and Status. You can then choose your sorting method. A map will appear above the results. You can download the results as either an XLS file, and XLS file that’s designed to be imported in GIS programs (like QGIS), or a KML file.

I’m aware of just one other app that uses this data set: MassUp.us. I don’t know if MassUp uses the real-time API that my Metra bike rack finder uses.

Logan Square McDonald’s crash map

This is part of a series of articles on the issue of lifting the pedestrian street designation on a part of Milwaukee Avenue in Logan Square so that the McDonald’s franchise owner can demolish the building, build a new building, and build a double order point (“tandem”) drive through. Read the first post

At the hearing on December 13, 2011, Alderman Reilly asked if there was evidence of injuries or crashes due to the drive through. No one brought this data to the hearing. I cannot directly attribute the crashes to the existence of the drive through (unless I had the original crash reports), the drive through probably generates traffic that would not be there without the drive through, and it causes people to have to turn across a lane of traffic, either to enter the driveway on Milwaukee, or when exiting the driveway onto Sawyer, or when turning onto Milwaukee from Sawyer. I am looking for studies that research the impacts of drive throughs at fast food restaurants and pharmacies.

37 people were involved in 13 crashes within 100 feet of the center of the McDonald’s driveway from 2007-2010. Seven people were injured, one was a pedestrian. Double the search radius to 200 feet and we see 87 people involved in 35 crashes. Now, four pedestrians and cyclist were injured in addition to the 10 drivers and passengers injured.

Download the data in this map. View a larger map

This was my testimony at the zoning committee hearing (this may not be verbatim, but it’s really close):

Hello, my name is Steven Vance. I work as a consultant and writer on sustainable transportation advocacy and planning projects. The text amendment to modify the pedestrian street designation may negatively impact the continuity and safety in traffic of all modes along Milwaukee Avenue, which happens to be the city’s most popular bike route.

I ask that McDonald’s provide a traffic impact study before this matter is discussed further.

Lynn, a Logan Square neighbor, describes more of what happened at the hearing, as well as the next step at the Zoning Board of Appeals.

Here’s a map of all pedestrian streets in Chicago. View larger map.

Download a KML file of all the pedestrian streets. Download the shapefile of all the pedestrian streets. Thank you to Azad Amir-Ghassemi and Bill Vassilakis for their help in digitizing the table of pedestrian streets in the zoning code.

Update January 10, 2013

Driving danger

Crash data from the Illinois Department of Transportation show several crashes along Milwaukee Avenue from 2005 to 2011. If this location hadn’t been removed from the P-Street ordinance, McDonald’s would have been required to install both the drive-thru’s entrance and exit on Sawyer, where there is markedly less traffic than on Milwaukee (or not build them at all). This project has not only allowed a documented hazard to persist (despite the P-Street designation), but perhaps to be worsened.

From 2005-2011, there were 3 bike-automobile crashes and 5 pedestrian-automobile crashes within 200 feet of the drive-thru entrance, which includes the intersection of Sawyer and Milwaukee (where many people will drive back onto Milwaukee from the drive-thru exit). There were 82 car-car crashes in the same period. At a nearby intersection, Milwaukee/Dawson, an intersection with a similar retail makeup and traffic count, shows about half the number of crashes.

How to upload shapefiles to Google Fusion Tables

It is now possible to upload a shapefile (and its companion files SHX, PRJ, and DBF) to Google Fusion Tables (GFT).

Before we go any further, keep in mind that the application that does this will only process 100,000 rows. Additionally, GFT only gives each user 200 MB of storage (and they don’t tell you your current status, that I can see).

  1. Login to your Google account (at Gmail, or at GFT).
  2. Prepare your data. Ensure it has fewer than 100,000 rows.
  3. ZIP up your dataX.shp, dataX.shx, dataX.prj, and dataX.dbf. Use WinZip for Windows, or for Mac, right-click the selection of files and select “Compress 4 items”.
  4. Visit the Shape to Fusion website. You will have to authorize the web application to “grant access” to your GFT tables. It needs this access so that after the web application processes your data, it can insert it into GFT.
  5. If you want a Centroid Geometry column or a Simplified Geometry column added, click “Advanced Options” and check their checkboxes – see notes below for an explanation.
  6. Choose the file to upload and click Upload.
  7. Leave the window open until it says it has processed all of the rows. It will report “Processed Y rows and inserted Y rows”. You will be given a link to the GFT the web application created.

Sample Data

If you’re looking to give this a try and see results quickly, try some sample data from the City of Chicago data portal:

Notes

I had trouble many times while using Shape to Fusion in that after I chose the file to upload and clicked Upload, I had to grant access to the web application again and start over (choose the file and click Upload a second time).

Centroid Geometry – This creates a column with the geographic coordinates of the centroid in a polygon. It lists it in the original projection system. So if your projection is in feet, the value will be in feet. This is a function that can easily be performed in free and open source QGIS, where you can also reproject files to get latitude and longitude values (in WGS84 project, EPSG 4326). The centroid value is surrounded in the field by KML syntax “<Point><coordinates>X,Y</coordinates></Point>”.

Simplified Geometry – A geometry column is automatically created by the web application (or GFT, I’m not sure). This function will create a simpler version of that geometry, with fewer lines and vertices. It also creates columns to list the vertices count for the simple and regular geometry columns.

Using Google Fusion Tables to create individual Chicago Ward maps

I wanted to create a map of the 35th Ward boundaries using Google My Maps for a story on Grid Chicago. I planned to create this by taking the Chicago Wards boundary shapefile and exporting just the 35th Ward using QGIS into a KML file. I ran into many problems and ended up using Google Fusion Tables as the final solution.

The problems

First, QGIS creates invalid KML files. Google Earth will tell you this. I opened the KML file in a text editor and removed the offending parts (Google Earth mildly tells you what these are; you can use this validator to get more information).

Second, Google My Maps would not import the KML file. I tried a different browser and a different KML file; a friend ran into the same issue. I reported this problem to Google.

The solution

I uploaded to Google Fusion Tables a KML file containing all wards. I did this instead of uploading the single Ward because, like a database, I can filter values in the column, selecting only the row I want with “ward=35″.

After applying the filter, the map will show the boundary for just that ward. I grab the HTML code for an embeddable map and voila, the article now displays an interactive map of the 35th Ward.

Whenever I want to create a map for a different ward, I go back to this Fusion Table, make a new filter and copy the new HTML code.

A screenshot of the embedded map, showing just 1 of 50 wards, in the Grid Chicago article. 

Elsewhere

I had the same problems with QGIS exporting and uploading the KML files to My Maps the other day when I was creating maps for the abandoned railroads for Monday’s Grid Chicago article. Not thinking about Fusion Tables, I drew on the map with my mouse the lines.

Screenshot of the map of abandoned railroads. 

Free online GIS tools: An introduction to GeoCommons

Read my tutorial on how I created the pedestrian map with GeoCommons. Read on for an introduction to GeoCommons and online GIS tools.

GeoCommons, like Google My Maps and Earth, is part of the “poor man’s GIS package.” It’s another tool that provides (few) of the functions that desktop GIS software offers. But it excels at making simple and somewhat complex maps.

I first used GeoCommons over a year ago. I started using it because it would convert whatever data you uploaded into another format that was probably more useful. I mentioned it in this article about converting files. For example, if you have a KML file, you can upload it and export it as a shapefile for GIS programs, or a CSV file to load into a table editor or spreadsheet application.

After creating the Chicago bike crash maps using Google Fusion Tables, I wanted to try out another map-making web application, one that provided more customization and prettier maps.

I found that web application and created a version of the bike crash maps, with several other data layers, in GeoCommons. I overlaid bike counts and bikeways so you can observe some relationships between each visual dataset. My latest map (screenshot below), created Wednesday, shows pedestrian counts in downtown Chicago overlaid with CTA and downtown Metra stations, as well as the 48 intersections with the most pedestrian collisions (from this UNC study, PDF).

Screenshot of pedestrian count map described above.

How these online GIS tools can be useful to you

I bet there’s a way you can use Google Fusion Tables and GeoCommons for your job or project. They’re extremely simple to use: they can take in data from the spreadsheets you’re already working on and turn them into themed reference maps. With mapping, you can do simple, visual analysis that doesn’t require statistical software or knowledge.

Imagine plotting your client list on a map and grouping them by age to see if perhaps your younger clients tend to live in the same neighborhoods of town, or if they’re more diverse (should you do this, keep the map private, something that you can’t do in GeoCommons – yet).

You may also find it useful if you want to create a route for your salespeople or for visiting church members at their homes. Plot all the addresses on a map, then manually filter them into different groups based on the clusters you see. With Google Fusion Tables, you can easily add a new column with the GROUP information and apply a numbered or lettered group and then re-sort.

Other things you can do in GeoCommons

  • Merge tables with geography – I uploaded two datasets: a table containing census tract IDs and demographic information for Cook County I downloaded from the American FactFinder 2; and a shapefile containing Cook County census tracts boundary information. After merging them, I could download a NEW shapefile that contained both datasets.
  • Make multi-layer maps
  • Symbolize based on frequency/rate
  • Convert data – This is by far the most useful feature. It imports “shapefiles (SHP), comma separated values (CSV), Keyhole Markup Language (KML), and GeoRSS” and exports “Shapefile, CSV, KML, GeoRSS Atom, Spatialite, and JSON” (from the GeoCommons user manual).

Read my tutorial on how I created the pedestrian map with GeoCommons.

« Older posts

© 2014 Steven Can Plan

Theme by Anders NorenUp ↑