CategoryTools

Use Turf to perform GIS functions in a web browser

Turf's merge function joins invisible buffers around each Divvy station into a single, super buffer.

Turf’s merge function joins invisible buffers around each Divvy station into a single, super buffer –all client-side, in your web browser.

I’m leading the development of a website for Slow Roll Chicago that shows the distribution of bike lane infrastructure in Chicago relative to key and specific demographics to demonstrate if the investment has been equitable.

We’re using GitHub to store code, publish meeting notes, and host discussions with the issues tracker. Communication is done almost entirely in GitHub issues. I chose GitHub over Slack and Google Groups because:

  1. All of our research and code should be public and open source so it’s clear how we made our assumptions and came to our conclusions (“show your work”).
  2. Using git, GitHub, and version control is a desirable skill and more people should learn it; this project will help people apply that skill.
  3. There are no emails involved. I deplore using email for group communication.*

The website focuses on using empirical research, maps, geographic analysis to tell the story of bike lane distribution and requires processing this data using GIS functions. Normally the data would be transformed in a desktop GIS software like QGIS and then converted to a format that can be used in Leaflet, an open source web mapping library.

Relying on desktop software, though, slows down development of new ways to slice and dice geographic data, which, in our map, includes bike lanes, wards, Census tracts, Divvy stations, and grocery stores (so far). One would have to generate a new dataset if our goals or needs changed .

I’ve built maps for images and the web that way enough in the past and I wanted to move away from that method for this project and we’re using Turf.js to replicate many GIS functions – but in the browser.

Yep, Turf makes it possible to merge, buffer, contain, calculate distance, transform, dissolve, and perform dozens of other functions all within the browser, “on the fly”, without any software.

After dilly-dallying in Turf for several weeks, our group started making progress this month. We have now pushed to our in-progress website a map with three features made possible by Turf:

  1. Buffer and dissolving buffers to show the Divvy station walk shed, the distance a reasonable person would walk from their home or office to check out a Divvy station. A buffer of 0.25 miles (two Chicago blocks) is created around each of the 300 Divvy stations, hidden from display, and then merged (dissolved in traditional GIS parlance) into a single buffer. The single buffer –called a “super buffer” in our source code – is used for another feature. Currently the projection is messed up and you see ellipsoid shapes instead of circles.
  2. Counting grocery stores in the Divvy station walk shed. We use the “feature collection” function to convert the super buffer into an object that the “within” function can use to compare to a GeoJSON object of grocery stores. This process is similar to the “select by location” function in GIS software. Right now this number is printed only to the console as we look for the best way to display stats like this to the user. A future version of the map could allow the user to change the 0.25 miles distance to an arbitrary distance they prefer.
  3. Find the nearest Divvy station from any place on the map. Using Turf’s “nearest” function and the Context Menu plugin for Leaflet, the user can right-click anywhere on the map and choose “Find nearby Divvy stations”. The “nearest” function compares the place where the user clicked against the GeoJSON object of Divvy stations to select the nearest one. The problem of locating 2+ nearby Divvy stations remains. The original issue asked to find the number of Divvy stations near the point; we’ll likely accomplish this by drawing an invisible, temporary buffer around the point and then using “within” to count the number of stations inside that buffer and then destroy the buffer.
Right-click the map and select "Find nearby Divvy stations" and Turf will locate the nearest Divvy station.

Right-click the map and select “Find nearby Divvy stations” and Turf will locate the nearest Divvy station.

* I send one email to new people who join us at Open Gov Hack Night on Tuesdays at the Mart to send them a link to our GitHub repository, and to invite them to a Dropbox folder to share large files for those who don’t learn to use git for file management.

Meet Chicago’s newest street view fleet: bikes

Bicycle holds an iPhone to take street view-style images

This position gives the smartphone an unimpeded view of the street but prevents the user from manipulating it.

I first used the Mapillary app on iPhone last fall, in August, and I uploaded one photo, of my arm, which I can’t delete from the website. I uploaded a couple more photos from a street in Roscoe Village in November.

This week, though, I uploaded 500 photos from a three mile journey on California and Milwaukee Avenues in Chicago – streets that no one else had photographed for the Mapillary street view service.

Mapillary is an open source (sort of) street view service, originally developed in Sweden, which allows the public to contribute photos taken with their smartphone app.

What’s “sort of” about Mapillary being open source is that it appears that the company owns the photos once you upload them. People are free to use the photos to edit OpenStreetMap, or publish elsewhere – for personal use only – with attribution that adheres to Creative Commons 4.0. People who want to use the photos in a commercial application must subscribe to a pay service.

Mounting an iPhone to a bicycle

I took the jump from contributing nothing to uploading a whole lot because I bought an iPhone mount for my bicycle. After months of research – okay, chalk it up to my being lazy and it being really cold outside – I settled on the DgRock Universal Bicycle Mount from Amazon for $9. I was perplexed that there was a gap in choices between this decent $9 product and the next group, hovering around $30-40.

After three days of use, I’m satisfied, despite limitations that are present in all mounts I surveyed. The DgRock mount is solid, barely moves even as the bike bounces along Chicago’s pothole-ridden streets, and securely holds the iPhone with a strong, spring-loaded grip. It’s universal in two ways: it holds nearly any smartphone (it probably can’t hold one with a screen 5″ or larger) and it attaches to most bicycle handlebars.

The first day I used the DgRock mount Mapillary complained with a red icon that it couldn’t get a proper fix on its location and therefore it wouldn’t start photographing. Fine, I was in downtown Chicago where connecting to GPS satellites can be hard. I figured the wifi positioning system that all smartphones and tablets use would suffice.

There are problems with the mount, but I think these apply to all bicycle smartphone mounts: When the phone is in position to take photos, meaning its horizontal and level to the ground, you can’t see the screen. That’s because the screen, mounted on the handlebars, is much lower than your eyes and faces vertically, instead of angled towards your face. The only way around this, I believe, is to either get an upright bicycle (like my WorkCycles Fr8) or an adjustable lens (I can’t find any).

Smartphone mount holding an iPhone on a bicycle

This position allows the user to manipulate the smartphone but you cannot take street view-style images.

The possible position angles of the smartphone when held by the mount was my main concern as I was shopping on Amazon: The mount need to have the flexibility to position the smartphone so its rear camera could be level with the ground. Smartphone mounts, though, are made to put the device in a position to be used and viewed frequently by the bicycle rider – it was unclear if many of the other smartphone mounts could accommodate the street view angles requirement.

The DgRock has no issue moving the iPhone into the right position, as you can see in the photos from my journey (or scroll to the end). Its issue, though, is that you have to put the smartphone in “backwards” so that the claw covers up part of the screen. I call it an issue but it doesn’t disturb the mount’s primary purpose when using Mapillary – the phone still has a clear view of the street.

Even with an upright bike like mine, though, it’s difficult to see the screen. I believe that Mapillary can actually design its app to help overcome this physical limitation.

Using Mapillary

The Mapillary app has improved greatly since the first version. It allows you to delete bad or undesired photos before uploading, and it has a simpler interface to go from opening the app to making your own street view. There are a couple changes I think would improve the user experience and lead to more contributions.

I would like to be able to turn off the screen while using Mapillary to save battery life. I think that the screen could fade to black and a small white dot or halo appears frequently to remind you that it’s working. I’d also like it to chime when iOS throws the “low storage” warning. Otherwise I may be riding along, thinking Mapillary is capturing the street, when iOS had actually run out of storage 10 minutes ago.

This is why we need more people editing OpenStreetMap

Unmapped homes in the Irving Park community area

These homes were built after the City of Chicago’s building footprints dataset was created (2010?). Ian Dees imported the dataset in 2012. Many of the buildings that you can now see on Bing Maps have not been present on Bing’s satellite imagery since at least 2012.

1. OpenStreetMap is the world’s most complete free map, to which anyone can contribute their “ground truth” data (the location of wells and convenience stores, road names, and whether Lula Café at 2537 N Kedzie Boulevard in Logan Square has outdoor seating).

2. OpenStreetMap is used by thousands of non-profit and non-governmental organizations, corporations, apps, and people daily to locate themselves, locate others, get directions, and find places.

3. Nearly every map is out of date the moment it is published, including online, “current” maps like Google Maps, Bing Maps, their competitors, and OpenStreetMap.

4. Bing Maps provides its satellite imagery to OpenStreetMap editors – you and me – so that we can trace (copy) things on the planet to be things on the map. Google Maps doesn’t allow tracing (copying).

5. Bing updated its satellite imagery for Chicago (and probably a lot of other places) within the last six weeks…and there are hundreds of objects that aren’t yet mapped in OpenStreetMap. In Chicago most of these buildings are newly constructed houses.

Those hundreds of houses now need to be added to OpenStreetMap, with addresses, to complete the buildings collection in Chicago, and to expand the gazetteer (an address book) of places in Chicago.

I’m glad you want to help me do it! Here are two helpful things you can do:

  1. Start tracing the buildings yourself (here’s how new mappers can get started), or
  2. Leave notes at buildings which aren’t yet mapped so that map editors like myself know where to look to trace buildings.

Update: There’s a bonus third thing you can do, and that’s come to the next MaptimeCHI event on Thursday, February 26th, at the Chicago Community Trust (225 N Michigan, 22nd floor). RSVP for Anatomy of a Web Map. The Trust will also provide food and beverages. I’ll be there to teach new mappers and assist generally.

Adding notes is extremely helpful

You can contribute without editing by adding notes describing new things, or identifying problems with existing things. Click the “Add a note” button on OpenStreetMap.org.

Mapping a campground that doesn’t exist: a before and after view of OpenStreetMap

Pretty soon there will be a campground shown in OpenStreetMap, and added to its geocoding database, when I’m done adding it.

I temporarily become addicted to mapping places on OpenStreetMap. In my quest to find and map all campgrounds in Chicagoland – in order to publish them in the Chicago Bike Guide – I came across a campground that was constructed this year and opened in August 2013. This is the story of figuring out how to map the Big Rock Forest Preserve campground in Big Rock, Illinois.

I found on the Kane County Forest Preserve District website that the organization operated a campground at Big Rock Forest Preserve. I couldn’t locate the campground in Google Maps by the address the website gave. I couldn’t find it in OpenStreetMap, either, because no one had mapped it, but it’s there now.

When I searched for the park by name, Google Maps zoomed me to the main entrance of the park, but I still couldn’t see a campground. I downloaded the forest preserve district’s park map (always as a PDF) and followed the roads in Google Maps until I came across the campgrounds approximate position. There was a new road here so I followed that to find a campground under construction.

Google Maps shows the campground and artificial lake under construction.

Google’s imagery of the under-construction campground was taken on May 23, 2013 (get the date from Google Earth). This was great because now I could open JOSM, a powerful desktop OpenStreetMap editor, and locate the site, load Bing’s imagery and start tracing the campground to upload to OSM.

Bing’s imagery in JOSM, the OpenStreetMap-editing app, doesn’t show the campground.

The problem was that Bing’s imagery – and this is typical – was outdated. I could easily compare the imagery side-by-side and based on other landscape features (like the forest edge) guess where to trace the campground, but OSM needs better quality data. Enter MapWarper.

Read the rest of this post on Web Map Academy.

CDOT misses the lesson on open data transparency

Publishing the wrong measurement as a PDF isn’t transparency.

The Chicago Department of Transportation released the first progress report to its Chicago Forward Action Agenda in October, two and a half years after the plan – the first of its kind – was published. I’ve spent an inordinate amount of time reading it and putting off a review. Why? It’s been a difficult to compare the original and update documents. The update is extremely light on specifics and details for the many goals in the Action Agenda, which should have organizational (like record keeping and efficiency improvements) and public impacts (like figuring out which intersections have the most crashes). I’ll publish my in-depth review this week.

Aside from missing specifics and details, the update presents information differently and is missing status updates for the three to five “performance measures” in each chapter. It was difficult to understand CDOT’s reporter progress without holding the original and update side-by-side. I think listing the original action item, the progress symbol, and then a status update would have been an easier way to read the document.

The update measures some action items differently than originally called for, and the way pothole repair was presented, a problem for people bicycling and driving, caught my analytical eye.

CDOT states a pothole-filling performance measure of the percentage, which it desires to be increased, “patched or fixed within 72 hours of being reported” but the average, according to the website Chicago Potholes, which tracks the city’s open data, is 101 days*. The update doesn’t necessarily explain why, writing “the 72 hour goal for filling potholes is not always feasible due to asphalt plant schedules” and nothing related to the performance measure.

As originally written, the only way to note the performance would be to list the percentage of potholes filled within the goal time, at the beginning and in the update. This performance measure has a complementary action item – an online dashboard – which could have provided the answer, but didn’t.

CDOT published that dashboard this summer as a series of six PDF files that update daily and you can hardly call it useful.

Publishing PDF files in the day and age of open government data – popular with President Obama and Mayor Rahm Emanuel – is unacceptable. Even if they are accessible – meaning you can copy/paste the text – they are poor outlets for data given the nationally-renowned civic innovation changes that Emanuel has succeeded in establishing.

There’s another problem: the dashboard file for pothole tracking doesn’t track the time it takes to close a pothole request, nor the number of pothole requests that are patched within 72 hours. It simply tells the number completed yesterday, the year to date, and the number of unpatched requests. (I’ve posted the pothole-tracking file to Scribd because the dashboard [PDF] doesn’t work in Safari; I also notified city staff to this problem which they acknowledged over three weeks ago.)

The “Chicago Works For You” website reports a different metric, that of the number of requests made each day, distributed by ward.

I discussed the proposed dashboard with former commissioner Gabe Klein over two years ago. He said he wanted to create a dashboard of projects “we’re working on that’s updated once a week.” Given Klein’s high professional accessibility to myself, John Greenfield and other reporters, I’ll give him and CDOT a pass for not doing this. But Klein also said, “I’m really big on transparency and good communication. When I left [Washington,] D.C. our [Freedom of Information Act Requests] were dramatically lowered.”

I’ll consider the pothole performance measure and action item “in need of major progress.”

* For stats geeks, the median is 86 and standard deviation is ±84.

Android versus iOS: my Chicago Bike Laws experience

Chicago Bike Laws: dooring info

Screenshot of Chicago Bike Laws, highlighting the dooring law.

For all this talk that more people use Android, and Android has the biggest and still-growing chunk of the smartphone market (in the United States and the world), I’m not seeing that reflected in how many downloads there are of my two apps.

Chicago Bike Laws is a free app that has been available for Android (download) since November 4 and on iOS (download) since November 9, a 5-day difference. Yet the iOS version has had more installations (58) since then while people have downloaded the Android version 32 times (but 5 people removed it).

What gives? The app is exactly the same for both platforms.

Are Chicago bicyclists more likely to have iPhones? I don’t do any platform-specific promotion so you can count that out.

(The experience I’ve had with Chicago Bike Guide activity is different because the Android version came online over a year later and has always been a paid app – Android apps cannot switch between free and paid while iOS apps can. By the way, the Chicago Bike Guide is free for iOS right now and half-off for Android. The comparison is that the adoption rate is much slower for the Android version.)

On Active Transportation Alliance’s transportation summit

Active Transportation Alliance invited Eric Hanns and I to speak about “using data for advocacy” at their first annual transportation summit held after a member meeting two Saturdays ago. My and Eric’s talks were complementary and centered around the data tool I built and which Eric and the other volunteers in the 46th Ward participatory budgeting program used to prioritize and market infrastructure projects in Uptown.

The tool in question is the Chicago Crash Browser I made last year and improved this year to load data faster, with great help from the Smart Chicago Collaborative and several members of the OpenGov Hack Night group I cherish.

Click or tap a spot in Chicago to retrieve the number of bicyclist-car and pedestrian-car crashes within 150 feet. With this information, the PB volunteers could show the alderman how important it was for him to support bike and pedestrian infrastructure projects in the ward, and to persuade ward voters to fund these projects.

Find more information about the four other summit “breakout groups” on Active Trans’s website. Eric and I prepared a “Using Data for Advocacy: Making the Case with Compelling Facts” handout which you can download as a PDF or see on our Google Doc. I’ve conveniently listed the links from the handout below but if you want more pointed advice on where to look for specific data, or get an answer to questions you have but don’t grok the context of each of these tools, leave me a comment.

How to convert bike-share JSON data to CSV and then to shapefile

Update January 4, 2013: The easiest way to do this is to use Ian Dees’s Divvy API as it outputs straight to GeoJSON (which QGIS likes). See below.

For Michael Carney’s Divvy bike-share stations + Census tract + unbanked Chicagoans analysis and map he needed the Divvy station locations as a shapefile. I copied the JSON-formatted text of the Divvy real-time station API, converted it to CSV with OpenRefine, and then created a shapefile with QGIS.

Here’s how to create a shapefile of any bike-share system that uses hardware from Public Bike System Co based on Montréal and is operated by Alta Bicycle Share (this includes New York City, Chattanooga, Bay Area, Melbourne, and Chicago): Continue reading

Update on my work this summer: Chicago Bike Guide now on Android

Yesterday I released the first version of my app for the Android operating system. The Chicago Bike Guide (born the Chicago Offline Bike Map) had been available for iOS since April 2012 and the most frequent question I heard was, “When will you have an Android version?” At first I was probably joking when I said it was coming soon as I had no interest in it. But things change: more people kept asking, I was slowly learning how to publish an app for Android, I bought a tablet myself on which to test it, and Android eclipsing iOS as the most dominant mobile OS had some effect.

I use PhoneGap software to compile the Chicago Bike Guide for iOS. My app is actually an HTML5 compliant website using jQueryMobile and PhoneGap allows this website to interact with some of the hardware on the iPhone (like the GPS components) and software (native notification buttons). PhoneGap compiles for Android and other operating systems, but my experience using the Android emulator put a damper on my progress.

I would install Google’s software development kit (SDK), including Eclipse, and load the app (er, website) into the emulator and it would be terribly slow. It didn’t emulate the Android experience very well. I did this twice; I don’t remember making any additional progress the second time but I probably made different progress, slightly expanding my understanding of how to make Android apps.

PhoneGap 3.0.0 came around, and its use requires focusing on the command line to build apps. In prior versions, for iOS, you would add “helper software” to Xcode but now it creates the Xcode project for you. It was easier to use this time around, as the process in this version was simpler to understand, use, and the documentation had improved. I felt it was time to try again to make an Android version so I “made the plunge” and bought the Asus MeMO Pad HD 7″ tablet from my local Micro Center.

At first I used the tablet to test my app (website) as a website, loading it over the network from my Mac’s web server. After fixing a lot of the display bugs I moved into native app testing. One command, “cordova build android”, and an APK I can email to my tablet appears 30 seconds later. This is one of the few areas where Android has iOS beat in terms of testing.

The iOS development environment requires so much setting up with developer profiles, team profiles, and other gunk I forget the names of. With Android, you simply email the ChicagoBikeGuide-debug.apk file to your device and open it in the Gmail app. Voilá.

After getting this far I stopped progress on developing for Android as I wanted to issue a new version for iOS with features and bug fixes I’d been working on since the last release in July (version 0.8.2). With that out of the way last week, and the new version waiting for Apple’s review, I worked on the Android version on Sunday and finished it today.

Did you catch that? I uploaded version 0.8.3 for iOS on October 3 and it’s been in review since then while in less than 24 hours I set up my Google Play Developer Console ($25 per year), merchant account, store listing, and started selling the Android version.

N.B. Google Play doesn’t allow you to switch the app between free and not free like the iTunes App Store does, so I cannot release it for free during a short promotional period like I did for the iOS platform.

After a sufficient period (a few days, perhaps) of no reports of it crashing, I will promote the Android version heavily.

New iOS app offers most advanced Divvy route directions

Chicago Bike Route for iOS

Walking directions from my house to the Divvy station at the CTA California Blue Line station, and then from there to the Divvy station at LaSalle/Illinois Streets. Lastly, there’s walking directions to some arbitrary N LaSalle Street address.

Adam Gluck and Andrew Beinstein showed up at OpenGov Hack Night on July 16, 2013, to show off the technical concept of their forthcoming app for iOS devices. I looped them into the Divvy app-making progress I and others were undertaking (documented on a shared Google Doc).

They said they would make their app was going quite different from all of the eight apps for using Divvy that have since launched before theirs: it would offer directions for walking to the nearest Divvy station with available bikes, directions to the Divvy station nearest their destination with open docks, and then walking directions from that end station to their destination.

Chicago Bike Route launched Friday last week. Currently only three of the eight iOS apps released before Chicago Bike Route have routing. CBR takes directions to a new level by giving you directions from where you are to where you want to go, and not necessarily from a specific Divvy station (like my Chicago Bike Guide does). Instead, CBR gives you complete directions between origin and destination and smartly picks the nearest Divvy station with available bikes. Now, I believe most often this will just be the nearest Divvy station, period, as it’s relatively rare for a station to lack bicycles.

The app uses Google Directions and for every trip makes a maximum of three calls to their API; counts against the app’s free quota from Google. The first call gets walking directions from the origin to the nearest Divvy station with available bikes, and the second call gets bicycling directions to the Divvy station with available docks nearest the destination, and the third call (assuming the destination isn’t that Divvy station) gets walking directions from the end Divvy station to your destination. The next step, I believe, is to have the app use a prediction model to accurately choose the end Divvy station. A lot can happen at that Divvy station in the 30 minutes (or whatever) it takes to get there. It may not have open docks when you arrive.

Two other suggestions I have: an improvement to the autocomplete destination function because it didn’t recognize “Chicago city hall” or its address, “121 N LaSalle Street”; and adding a “locate me” button. Additionally I’d like them to add some basic resources to advise users on where they can get more information about Divvy or bicycling in Chicago.

Adam and Andrew are going to publish a “dock surfing” function in the app that will incorporate multiple segments on Divvy to make a trip longer than the free 30 minute period. This would probably mean a fourth call to the Google Directions API. I emailed Adam and Andrew to learn more about the app development.

Video of Beinstein and Gluck presenting their app to Hack Night. Created by Christopher Whitaker for Smart Chicago Collaborative.

Why did you make Chicago Bike Route?

We made the app because we wanted to make something civic related. We thought that Divvy was an exciting new civic program coming into existence, and we kept seeing it all over the place. It also solves a real problem in public transportation that we notice and hear about a lot living in Hyde Park called the “last mile problem.” We also had the data in our hands from having attended civic hack night at 1871 when Divvy came and we thought “let’s make a native Divvy app!” And that’s what we did. We also released a framework for interacting with the Divvy API natively for developers who don’t want to get their hands dirty playing around with the iOS frameworks.

What makes your app stand out from the pack?

I think the routing but also the simplicity of design of the app.  We wanted it to be something you could just open up and use and was like all the other mapping utilities that one has on their phone (Google Maps, Apple Maps). And that’s what we did. You open it, enter an address, and you get routed to that address. Something that people could use to get up and running with Divvy with basically no familiarity with the system.

What features are you planning for the future?

Bike surfing! Seriously though. We think that it would be a really useful feature for some people, and also help reduce the cost of using the bikes. It would be useful for the regular riders where the $2 additional charge could really add up but also if you are someone who is not part of the program and are just taking the bike out for a joy ride. It can actually get kind of expensive, since every half hour after the first hour in a half is an additional $8, rather than $4.50 for members. You would also be less familiar with the bike stations under that situation. We also need to integrate with Chicago public transportation. But, we also want to keep with the simplicity, and create a user experience with basically no learning curve, and we are a little cautious to throw something in that could complicate things.

© 2017 Steven Can Plan

Theme by Anders NorénUp ↑