Category: Data

Download GIS data from ArcGIS servers using a simple command

A lot of geospatial data (GIS) in the world is stored on ArcGIS MapServers, a part of the Esri “stack” of products that municipalities and other governments use to manage and publish GIS data.

A lot of people want that data, and not every government has enabled the ArcGIS data hub features that make it easier to get. For an example data hub, check out Cook County, Illinois.

If you have ArcGIS software on your Windows computer, then it can be pretty easy to plug in the map server URL and manipulate and extract the data.

For the rest of us who don’t have a very expensive license for that software, you can use a “command line” tool (written in Python) on any computer to download any layer of GIS data hosted on the ArcGIS MapServer and automatically convert it to GeoJSON.

Tutorial

You’ll need to install the Python package pyesridump, from the OpenAddresses GitHub repository, created by Ian Dees and other contributors.

Installing pyesridump is easy if you have pip installed, using the command pip install esridump.

The next thing you’ll need is the URL to a layer in a MapServer, and these are not easy to find.

Finding data to download

I can almost guarantee the county where you live has one. Before you continue, check to see if your county (or other jurisdiction) has the “open data portal” add-on to their ArcGIS stack.

A handful of suggestions:

  • Here are links to the open data portals enabled by Esri for Lake County, Illinois, and Broomfield County, Colorado.
  • A reader suggested looking for MapServers for jurisdictions around the world by looking through Esri’s portal of open data called ArcGIS Hub.
  • Use a search engine to search for the URL pattern that’s common to ArcGIS servers: paste “inurl:arcgis/rest/services” into a search engine. (You can also limit this search to a particular domain name, say, “chicago.gov”, by adding “site:chicago.gov” to the search query, making the full search query “inurl:arcgis/rest/services site:chicago.gov”.

Once you locate a dataset you want, you can find the MapServer URL under About>Data Source on the right side of the page.

I normally find them by looking at the HTML source code of a MapServer I already know about.

For this example I’ll use one of the GIS layers in the Cook County, Illinois, election service MapServer – here’s the layer for the Cook County commissioners districts.

Fetch the data

Once you have the URL the command is simple:

esri2geojson http://cookviewer1.cookcountyil.gov/ArcGIS/rest/services/cookElectnSrvc/MapServer/11 cookcounty_commissioners.geojson

  • The first term, esri2geojson tells your computer which program to load.
  • The second term is the URL of the MapServer URL.
  • The third term is the filename and location where you want to store the file. I prefer running the command “inside” the folder where I want the file to be stored. You can also specify a full path of the file. On a Mac this would look like ~/Users/username/Documents/GIS/projectname/cookcounty_commissioners.geojson

After you enter the command into your computer’s terminal, press enter. esri2geojson will report back once, after it finds and understands the MapServer URL you gave it. When it’s done, the command will “close” and your computer’s terminal will wait for the next command.

Do you have questions, or need some help? Leave a comment below.

Fun with stats: Building permits by street name and number edition

John Hancock Center

The John Hancock Center. Photo by Kevin Dickert.

 

On which street are the most building permits issued?

Michigan Avenue!

But where on Michigan Avenue are the most building permits issued?

Take a guess!

First, can you answer: Are most building permits issued to North Michigan Avenue (between Madison Street, 0 north/south, and Oak Street, 1000 north), or South Michigan Avenue (between Madison Street, 0 north/south, and um, somewhere south of 130th Street, 13000 south)?

Here’s the answer…

Even though South Michigan Avenue is at least 13x longer than North Michigan Avenue, South Michigan Avenue has 39 percent fewer building permits!

From 2006 to yesterday (Saturday), there were 7,828 building permits issued to projects on North Michigan Avenue and 4,714 building permits issued on South Michigan Avenue.

The most common address on North Michigan Avenue to receive building permits was 875 N Michigan Avenue. It’s also the most common address to receive building permits on all Chicago streets.

What’s there? The John Hancock Center (tower)!

The average building address number on North Michigan Avenue is 540.6. That means that building permits on North Michigan Avenue concentrate around Grand Avenue, which is near the city’s biggest Marriott hotel, and is where the Under Armor flagship store is.

The next most common street – after South Michigan Avenue – is North Clark Street, which extends from Madison Street (0 north/south) to the northern edge of the city at Howard Street, which is 7600 north, about 7.6 times longer than North Michigan Avenue.

S. Clark Street Signs

Businesses in the 400 block of South Clark Street, as of when the photo was taken in November 2008. I believe the hotel is still there. This is the busiest block of South Clark Street, for building permits. Photo by Bruce Laker.

South Clark Street doesn’t register in the top 10 or even the top 100. It comes it at number 162, with 772 building permits. This is surprising to me because South Clark Street runs from Madison Street (0 north/south) in downtown and goes to 2200 south, and has a lot of downtown office buildings.

South LaSalle Street (3,613 building permits), South Wabash Avenue (2,916), and South Dearborn (1,611) are all in the top 50. The data could be wrong somehow.

Inclusionary zoning calculator will tell you how many units a developer can afford to make “affordable”

An “inclusionary zoning” calculator can help you determine how much affordable housing your town should require that developers build in their new construction residential buildings.

I learned about Grounded Solutions Network’s Inclusionary Housing Calculator at the second-ever YIMBYtown conference in Oakland, California, two weeks ago.

YIMBY (yes in my back yard) is a movement to reduce barriers to building more housing in order to be able to house everyone at a level they can afford. It’s a movement for other things, and it means a lot of different things to a lot of different people but the end result is that more housing needs to be built.

An interested person inputs a lot of values relevant to their local housing market into the IHC and it will calculate the cost of construction per unit and the rental income from those units, and then will figure the profit margin for the developer. What makes this “inclusionary” is that one also needs to enter the desired portion of units that are set aside as “affordable” (to people making a certain income) and subsidized by the developer’s rental income.

I put the IHC through a real world exercise by inputting as much data as I knew about a rejected proposal in Pilsen.

The first proposal from Property Markets Group had 500 units, and 16 percent of them were set aside (news on this and their subsequent proposals) [I cannot find the source of the “16 percent on-site” factor]. Chicago’s Affordable Requirements Ordinance, or ARO, requires that 10 percent of the units are affordable, and that 25 percent of those 10 percent must be built on site. The other 75 percent can be built on site, or the developer can pay an in-lieu fee per unit.

Needless to say, 16 percent on-site is much, much higher than 25 percent of 10 percent. A neighborhood organization, the Pilsen Land Use Committee, however, requires 21 percent in the area, and the city council member, Danny Solis, 25th Ward, adheres to.

PMG said they couldn’t go that high, and that’s what I wanted to test.

According to this Inclusionary Housing Calculator, could the developer make enough profit (considered as 10 percent) if the building had 21 percent of units as affordable?

In this exercise, the answer was “no, PMG could not make a profit if they had to set aside 21 percent of the units as affordable.”

But the calculator showed that they could earn a 12 percent profit if 16 percent of the units were affordable. 

Some of the inputs are actual, like the sale price of the land (found in the Illinois Department of Revenue’s transactions database), but I had to make up some inputs, including the apartments’ bedroom mix, and the future rental prices of those apartments.

Further reading

  • It’s tough for people to move into one of these set-aside apartments in Chicago (DNAinfo Chicago, July 28, 2017)
  • Inclusionary zoning cannot create enough affordable units (City Observatory, February 11, 2016)
  • Other housing cost calculators like this one (City Observatory, July 26, 2016)

The U.S. DOT should collaborate with existing “National Transit Maps” makers

The U.S. DOT demonstrated one idea for how a National Transit Map might look and work at a conference in February.

The Washington Post reported this month that the United States Department of Transportation is going to develop a “National Transit Map” because, frankly, one doesn’t exist. The U.S. DOT said such a map could reveal “transit deserts” (the screen capture above shows one example from Salt Lake City, discussed below).

Secretary Anthony Foxx wrote in an open letter to say that the department and the nation’s transit agencies “have yet to recognize the full potential” of a data standard called the General Transit Feed Specification that Google promoted in order to integrate transit routing on its maps. Foxx described two problems that arose out of not using “GTFS”.

  1. Transit vehicles have significantly greater capacity than passenger cars, but are often considered just vehicles because we are unable to show where and when the transit vehicles are scheduled to operate. The realistic treatment of transit for planning, performance measures, and resiliency requires real data on transit system operations.
  2. One of the most important social values of transit is that it makes transportation available to people who do not have access to private automobiles, and provides transportation options for those who do. Yet, we cannot describe this value at a national level and in many regions because we do not have a national map of fixed transit routes.

“The solution is straightforward”, Foxx continued, “[is] a national repository of voluntarily provided, public domain GTFS feed data that is compiled into a common format with data from fixed route systems.”

The letter went on to explain exactly how the DOT would compile the GTFS files, and said the first “collection day” will be March 31, this week. As of this writing, the website to which transit agencies must submit their GTFS files is unavailable.

What Foxx is asking for has already been done to some degree. Two national transit maps and one data warehouse already exist and the DOT should engage those producers, and others who would use the map, to determine the best way to build a useful but inexpensive map and database. Each of the two existing maps and databases was created by volunteers and are already-funded projects so it would make sense to maximize the use of existing projects and data.

“Transitland” is a project to host transit maps and timetables for transit systems around the world. It was created by Mapzen, a company funded by Samsung to build open source mapping and geodata tools. Transitland is also built upon GTFS data from agencies all over the world. Its data APIs and public map can help answer the question: How many transit operators serve Bay Area residents, and what areas does each service?

For the United States, Transitland hosts and queries data from transit agencies in 31 states and the District of Columbia. In Washington, D.C., Transitland is aware of four transit agencies. It’s a great tool in that respect: Not all of the four transit agencies are headquartered in D.C. or primarily serve that city. The app is capable of understanding spatial overlaps between municipal and regional geographies and transit agencies.

Transitland has a “GUI” to show you how much transit data it has around the world.

“Transit Explorer” is an interactive map of all rail transit and bus rapid transit lines in the United States, Mexico, and Canada. Yonah Freemark, author of The Transport Politic, created the map using data culled from OpenStreetMap, the National Transit Atlas Database (administered by the DOT and which shows fixed-guideway transit), and his own research. I wrote the custom JavaScript code for the Leaflet-powered map.

No other agency or project has collected this much data about fixed-guideway transit lines in any of the three countries, since the map includes detailed information about line lengths, ridership, and other characteristics that are not included in GTFS data. Transit Explorer, though, does not include local bus service or service frequencies, which the DOT’s map may if it incorporates the full breadth of GTFS data.

Transit Explorer also goes a step further by providing data about under construction and proposed fixed-guideway transit lines, which is information that is very relevant to understanding future neighborhood accessibility to transit, but which is not available through GTFS sources.

Finally, “GTFS Data Exchange” is a website that has been storing snapshots of GTFS feeds from agencies around the world for almost a decade, or about as long as GTFS has been used in Google Maps. The snapshots allow for service comparisons of a single agency across time. For example, there are over 100 versions of the GTFS data for the Chicago Transit Authority, stretching back to November 2009; new versions are added – by “cta-archiver” – twice a month.

Josh Cohen, writing in Next City, highlighted the significance of Google’s invention of GTFS, saying, “Prior to the adoption of GTFS, creating such a map would’ve been unwieldy and likely produced an out-of-date product by the time it was completed.” The DOT’s own National Transit Atlas Database includes only fixed-guideway (a.k.a. trains) routes, and hasn’t been updated since 2004.

Not all GTFS feeds are created equal, though. Some transit agencies don’t include all of the data, some of which is optional for Google Map’s purpose, that would make the National Transit Map useful for the spatial analysis the DOT intends. Many agencies don’t include the “route shapes”, or the geographic lines between train stations and bus stops. Researchers are able to see where the vehicles stop, but not which streets or routes they take. Foxx’s letter doesn’t acknowledge this. It does, however, mention that transit agencies can use some federal funds to create the GTFS data.

David Levinson, professor at the University of Minnesota, believes the map will bias coverage (geographic reach of transit service) over frequency (how many buses are run each day that someone could ride).

The U.S. DOT’s chief data officer, Dan Morgan, whom I met at Transportation Camp 2015 in Washington, D.C., presented at the FedGIS Conference this year one idea to demonstrate coverage and frequency in Salt Lake City, using the GTFS data from the Utah Transit Authority.

Levinson also tweeted that it will be difficult for a national map to show service because of the struggles individual transit providers have symbolizing their own service patterns.

Foxx’s letter doesn’t describe how planners will be able to download the data in the collection, but whichever app they build or modify will cost money. Before going much further, and before spending any significant funds, Foxx should consult potential users and researchers to avoid duplicating existing projects that may ultimately be superior resources.

Foxx can also take advantage of “18F” a new agency within the General Services Administration to overcome government’s reputation for creating costly and difficult to use apps. The GSA procures all kinds of things the federal government needs, and 18F may be able to help the DOT create the National Transit Map (and database) in a modern, tech and user-friendly way – or write a good RFP for someone else to make it.

Look for the National Transit Map this summer.

Which places in Chicago get the most building permits?

View from the CTA green roof

The Merchandise Mart in the Near North Side community area ranks second place in locations receiving the most building permits.

Ed. note: I changed the title of this blog post because one interpretation of the original, “Where are the most building permits issued in Chicago?”, has the answer “City Hall”, the location of the issuer. My bad. 

Without regard to type or construction cost, the most building permits in the City of Chicago are issued at 11601 W Touhy Ave.

Where is that? It depends on which geocoder you use.

Two buildings at 11601 W Touhy Ave from Google Street View. The City of Chicago has issued thousands of building permits to this address, but the work is actually distributed across the O'Hare airport grounds. Google Maps and the Cook County parcel map places these buildings in Des Plaines.

Two buildings at 11601 W Touhy Ave from Google Street View. The City of Chicago has issued thousands of building permits to this address, but the work is actually distributed across the O’Hare airport grounds. Google Maps and the Cook County parcel map places these buildings in Des Plaines.

Google Maps puts it on this building that’s on a street called “Upper Express Drive” and in the city of Des Plaines, Illinois. But the City of Chicago wouldn’t issue building permits in another city.

Our own geocoder converts the geographic coordinates given in the city’s building permits database for these permits to the address “399 E Touhy Ave, Des Plaines, IL”. The Cook County parcel for the same location has the address “385 E Touhy Ave, Des Plaines, IL”.

Now where is this building?

It’s at O’Hare airport, and it’s one of a handful of addresses* the city’s buildings departments uses to denote permits issued to work at O’Hare. Since 2006 to Saturday, December 12, 2015, there’ve been 2,403 building permits issued here. The permits’ work descriptions indicate that a lot of the work occurs elsewhere on the airport grounds.

13 buildings have had more than 400 permits issued since 2006 to yesterday.

address community area count
11601 W Touhy Ave O’Hare

2403

222 W Merchandise Mart Plz Near North Side

802

141 W Jackson Blvd Loop

538

233 S Wacker Dr Loop

518

2301 S Lake Shore Dr Near South Side

516

30 S Wacker Dr Loop

510

5700 S Cicero Ave Garfield Ridge

495

500 W Madison St Near West Side

482

227 W Monroe St Loop

422

55 E Monroe St Loop

421

875 N Michigan Ave Near North Side

408

151 E Wacker Dr Loop

407

350 N Orleans St Near North Side

401

A pattern emerges: 10 of these 13 buildings are in the Central Business District and the other three are O’Hare airport, McCormick Place (2301 S Lake Shore Drive), and Midway airport (5700 S Cicero Ave).

The first location that’s outside the Central Business District and not one of the city’s airport or its convention center is at 1060 W Addison St – better known as Wrigley Field – in the Lake View community area with 321 building permits issued. It ranks #30. If you keep running down the list, the next most frequently issued location is 7601 S Cicero Ave – that’s the Ford City Mall and I think the city’s only sprawl-style indoor mall. It ranks #39 because it pulls monthly electric maintenance permits.

The Merchandise Mart’s position at #2 is notable because the majority of its permits are for small amounts of work: there is a lot of electrical rewiring done because of the frequent shows and exhibitions in the interior design materials “mall”.

The Mart sees other activity, though, including multi-million renovations for technology companies like Motorola Mobility and Braintree. The Mart also received a permit this year for a new $3 million staircase construction, part of its building-wide renovation project.

Rendering of new main (south) lobby staircase at the Merchandise Mart

This rendering shows a new grand staircase that will be built in the Merchandise Mart’s south lobby jutting from the side of the lobby that’s between the doors on the Chicago River side, and the reception desk and central elevator bank. A building permit issued this fall puts the construction cost at $3 million.

If you want to know more about building trends in Chicago, send me a message through the Chicago Cityscape website and I can put together a custom report for you.

* Other addresses I’ve noticed are:

  • 10000 N Bessie Coleman Dr
  • 10000 W Ohare St
  • 11600 W Touhy Ave
  • 11555 W Touhy Ave

Of these only the two Touhy Ave addresses are logical: O’Hare Street isn’t a real road, and 10000 N Bessie Coleman Dr is much further north than the northernmost point in Chicago.