City of pointless secrecy

Opinion

Hey there, time traveller!
This article was published 07/04/2014 (4190 days ago), so information in it may no longer be current.

Let me preface this rather long and nerdy post by saying I have rarely had a more head-explodingly frustrating experience with any government in my 15 years of journalism. To illustrate my feelings, please enjoy the short animated GIF at the end of this blog.

Also let me preface this with one important definition: Open data refers to the idea that government information ought to be, to the fullest extent possible, available to the public. Data – everything from mosquito trap counts to oil well licenses – should be downloadable for free in an easy-to-use format that allows citizens to sort and search and map. Data don’t lie. Open data is how citizens hold their government to account. Open data is how entrepreneurs create cool apps like this and this.

The city of Winnipeg (and the province, for that matter) is years behind other governments, even the uber-controlling Harperites, on open data. Very, very little information is available for download.

What we do get is only available after months and months of FOIs. I’ve been arguing for the last seven months with the city, for example, over a bit of harmless info nearly every other municipality in Canada released years ago.

To illustrate my point, let me tell you about my Kafkaesque experience trying to get what are called boundary shapefiles for data available on the Now Winnipeg site. This site is kinda nifty. It appears to be the start of some kind of public service portal based on Winnipeg’s 236 neighbourhoods. It includes an  export tool that supposedly allows the public to "download your own datasets based on your current slice-and-dice settings, made available in a variety of machine-readable formats."

Holy moly! When we read those words, my online boss, Wendy Sawatzky and I nearly raced down the street for a celebratory beer at the Lincoln. Those words are music to a data nerd’s ears.

Wendy and I then immediately tried to download some data, specifically the boundary files that would allow us to recreate maps of each of the city’s 236 neighbourhoods and then overlay other cool data on top.

We hoped to, for example, match the neighbourhood shapefiles to the city’s regular list of vacant buildings to create a map that shows which neighbourhoods are hotspots, or which neighbourhoods improved. Or, we could map census data, to get a really clear visual portrait of poverty.

With those shapefiles, we could map so much cool stuff, and uncover so many cool stories that illustrate otherwise invisible trends.

The trouble with the Now Winnipeg site, we soon found, was twofold.

First, the data was downloadable in obscure formats (irJSON? SPARQL?) that even our smartest geeks downstairs could not parse into a format useable by our industry-standard mapping program. That breaks one of the cardinal rules of open data: stuff has to be user-friendly.

Second, the ever-skeptical eye of my boss fell on this long and unwelcoming list of rules about how the data can be used.

Basically, the rules say the data can’t be used. You can download it. But you can’t modify or distribute it. You can’t republish it. You can’t even give it to a buddy. It is entirely useless to anyone, and, again, at odds with all modern open data principals, unless you get written permission.

So, we asked the city for two things. Would it waive the conditions of use, and could we have the data in usable formats such as ArcGIS or Excel?

We thought the answer to both questions would be a quick and cheerful yes. That’s because, over a year ago, the city passed a policy on open data that mandates the city do exactly what we asked:  allow the public free and unfettered access to the biggest number of databases possible in easy-to-use formats.

Here’s the text of our questions and the replies from the city’s top communications boss, Steve West:

Q: We want permission to export all the datasets on the Now Winnipeg site. And we want to be able to use, disseminate and display that data with no restrictions.

A. We are pleased to approve access to the data on the NOW site under its terms of use.

Q: As per the Open Data plan, we request the data in "prevailing open standards" for machine-readable formats. For databases, we request standard formats such as: tab- or comma-delimited text (.txt or .csv), or other standard spreadsheet formats (.xml, .xls, .accdb, etc) For geographic data we request standard formats such as: ArcGIS (.shp), MapInfo (.tab), Geography Markup Language (.gml) or Keyhole Markup Language (.kml).

A. As the Open Data initiative has not yet been launched, we are not in the position to provide this data. An announcement would be made when the site is available.

Okay, I said to my boss, we’ll have to have another crack at deciphering those obscure Borg formats, but at least the conditions of use have been waived.

Not so fast, said Wendy. West said "we’re pleased to approve access," but also "under its terms of use," and those are the very restrictions we’re trying to have waived.

Jeez, that’s true, I said. So, I went back to the city and asked for clarification, even though my repeated questions seemed pretty simple to me. In response I got this:

The NOW site is currently governed by the overall City of Winnipeg terms of use. If the requested data is for personal use there is no concern or restriction; if being used for other than personal use, we require an explanation of the intended use.

And that’s when I did this:

After wasting weeks on this, we still have data Steve Jobs himself wouldn’t be able to decipher, and we aren’t allowed to use it for anything, anyway.

This is a system entirely focused on thwarting all modern methods of government accountability and transparency.

You’ll find me and Wendy at the Linc, drowning our sorrows.