#melbdataguru

[melbdataguru]

How to publish a real-time VicRoads road closure feed?

Ever seen VicTraffic? It's an interactive map showing roadworks, public events, traffic alerts, natural disasters and other events causing disruption on roads across the state. In addition to this site, VicRoads provides the data to a number of data consumers on a commercial basis, with individual agreements and SLA's. For some consumers, like ESTA (who manage 000 calls), correct, up-to-date information is critical for safe routing of ambulances and other emergency services. For others, like Australia Post, it's a matter of efficient service delivery.

It's easy to see why this data would be useful to an even wider range of consumers: that is, why it should be open data. But let's look at the complexities.

This data comes from several sources, combined:

Planned roadworks

Planned events (such as marathons)

Traffic alerts (some of which are triggered by smart freeway sensors)

Unplanned road closures (eg, due to an accident, landslide or a bushfire)

Unplanned road closures provided by local councils. (For reasons I don't quite understand, councils have to notify VicRoads of unplanned road closures, but not planned ones.)

VicRoads doesn't develop or host many software applications, and VicTraffic is no exception: it's developed and hosted by WaveDigital. The VicTraffic application requests data, in the form of an XML SOAP feed, every couple of minutes, from VicRoads servers, over a secure, IP-controlled link.

So, how can VicRoads release their data in realtime to the world?

Option 1: Remove the IP whitelisting, and publish the URL of their data feed

This is the simplest option, but completely reckless. Their servers are not built to be accessible to the public, and could easily be crippled during an emergency. Lives could easily be at stake.

Option 2: WaveDigital should provide a machine-readable feed of the data

Two problems: first, obviously this would cost money for the change, which could be significant. Secondly, this now makes WaveDigital's servers the single point of failure, and the gateway to a potentially much larger ecosystem than they reckoned on. If the servers became swamped by non-critical needs during an emergency, then critical needs could also be compromised. Clever architecting would perhaps be required.

Option 3: Publish a parallel feed to another location

Perhaps a better solution is to also provide an IP-controlled feed to another server designed to provide just this static content to large numbers of clients. The biggest risk in this solution is that applications could come to depend on this new endpoint, when there is in fact no SLA between VicRoads and the application or its consumers.

Also, since the current XML feed doesn't obey any standards, it's not that easy for any consumer to use. It would be better to publish as a standard like GeoJSON, so it's as easy as possible to mash up into other applications.

[melbdataguru]

First steps on the Open Data Journey

Over the last few months, I've had meetings with a few state government bodies that are just starting their open data journeys: EPA, VicHealth, Sustainability Victoria, Transport Safety Victoria. These first steps seem to recur:

1. Make a list of datasets available in the organisation.

Often this already exists, called an Information Asset Register, or Data Inventory. The usual driver for these is security: making sure that important information is both protected from loss, and from inappropriate sharing.

2. Add new fields

Ask these questions:

a) Published: Has this information been published before? If data has already been included in a report, then it's easier to publish as data.

b) Confidential: Is there anything confidential in this data? Individuals' names, addresses, etc. Political sensitivities get recorded here, too. There's no need to go near any confidential or sensitive datasets to begin with.

c) Processing required: What would need to happen to turn this dataset into something that could be uploaded to DataVic? Maybe it already exists as a spreadsheet, maybe it needs to be exported from a database, maybe some complex aggregation and de-identification would be required.

d) Sign-off: Who would need to sign off on this dataset being released?

e) Useful for: What reasons would this data be valuable for? App development? Research? It's hard to predict every use, but a quick brainstorm helps discussion.

Other, organisation-specific fields get added. For EPA, we added fields like how often a dataset is collected, how many data attributes, and so on.

3. Prioritise

Once the spreadsheet is more or less complete, you can start assigning priorities. You balance:

Value: "Useful for" column and previous demand.

Ease of publishing: "Processing required"

Sensitivity: "Confidential" and "sign-off" columns.

The first couple of datasets should be low-sensitivity, easy to publish, and preferably at least moderately usfeul.

4. Publish, gather feedback and repeat!

Upload datasets wherever is easiest. For state and federal government bodies, the choice of portal is mandated. For councils, there are several options: pick whichever seems easiest.

[melbdataguru]

Open council data standards!

I've created a draft set of standards for some common council datasets: trees and garbage collection zones so far, with more to come. Update: Building accessibility ratings added!

The primary goal: to allow datasets from different councils to be easily joined up into something bigger, to support research, app development, knowledge exchange between councils, and use by the public.

Your feedback is needed!

At the Municipal Association of Victoria's technical forum in Healesville on the 5th of December, we proposed 4 datasets for councils to focus on publishing. Already, Ballarat has released their garbage collection data at data.gov.au:

And here's what the current state of trees datasets looks like:

Let's get the whole state filled in. I'm tracking the release of these datasets here: LGA published datasets.

[melbdataguru]

Slides from my open data talks

Open Data Hackfests and Health (for Department of Health, March 21, 2014)

Rethinking the Smart CIty with better open data (for Melbourne Knowledge Week, 30 October, 2014)

The Open Road (for ~20 staff at VicRoads, 25 November 2014)

Open Data for Corangamite (for directors and CEO of Corangamite Shire Council, 2 December 2014)

Open data together (for ~30 council IT staff at MAV technical forum, 5 December 2014)

Open Data (~70 VicRoads staff, 9 December 2014)

[melbdataguru]

"Unstructuring" a pretty government spreadsheet

Government bodies are big fans of the pretty spreadsheet:

It's no good for any kind of data work like that, however. Here's what you do.

1. Strip the formatting, remove header fluff and blank columns.

2. Create cell formulas to construct one row from all the cells around it.

3. Add an IF expression so that cells only get populated once per block of text. Here I check whether the "Tel:" cell contains exactly that text. If it doesn't, remain blank.

4. Fill down.

5. Copy all the new cells, and paste-special so that they contain values, not formulas.

6. Remove the old columns.

7. Remove all the interleaved blank rows with the magic touch: sort!

Voilà. Now we have something we can work with.

[melbdataguru]

Vegetating the state with DEPI

The states' government data portals contain all kinds of data goodies that are pretty useful, if you know it's there and how to get it. Here's one: the Department of Enivornment Primary Industries' "Vicmap Vegetation 1:25,000" dataset.

What is it? A detailed geospatial data file showing areas that have trees, across the whole state.

Why is it useful for me? I run http://cycletour.org, a map-based planning tool for cycle tours. It's nice to know which areas are forested, as they're often nicer than farmland, and better for stealth camping.

Searching for "trees" on http://data.vic.gov.au turns up lots of stuff, like this:

There's a lot of technical knowledge assumed here. See the text in blue?

WMS: This actually isn't a data file you can download. It actually means the first two "datasets" are just locations of a Web Map Service - an older way of serving map tiles over the web. So it's a URL that you could give to a GIS program to change your basemap, but you're stuck with however DEPI chose to render this vegetation layer. And you can't do any actual analysis with the data - it's just a set of images.

SHP: This is a downloadable Shapefile, a very widely supported proprietary geospatial format.

And the 1:25,000? Government departments still seem to be dominated by the idea of printing maps, where scales are expressed as ratios. A dataset described as 1:25,000 means that the data is precise enough to be printed on a map at that scale. In this case, that's good enough for a pretty detailed hiking map, for instance. (The datasets also include more detailed statements like "For topographic base derived data this represents an error of 8.3m on the ground for 1:25,000 data")

Ok, let's download it.

Not so fast.

Data.vic.gov.au doesn't actually contain downloadable geospatial files from DEPI. Instead, after entering your email address, you get redirected to the Spatial Datamart, a somewhat clunky geospatial data portal. Here's what you have to do to get all the data for Victoria:

1.Choose area type "Whole of State"

2.Click "VIC", then ">". (So now it knows which whole of state...)

3. Click Apply and Submit.

You could choose a different projection, but since it doesn't offer anything useful, like, oh, the world's most popular web mapping projection, Spherical Mercator (EPSG 3857), there's not much point.

 4. Sit back and wait. You'll get an email in an hour or so, letting you download the file. This one's about 1.5GB.

Then, you can do what you want with it. In my case, styling the data with TileMill, and adding a new vegetation layer to cycletour.org:

[melbdataguru]

Visualise the City - researchers and government learn Tableau

How do you get researchers and government talking to each other? Maybe you stick them in a workshop together. Last week, I ran Visualise the City, a one day workshop to learn Tableau, a pretty awesome visualisation and data analysis tool. I opened the event to researchers at the University of Melbourne, staff at the City of Melbourne, and a small number of "public" - mostly GIS people interested in open data.

The first few hours were almost a standard Tableau workshop, run by Tableau sales consultant Cynthia Halim. Almost - instead of the dry and off-topic sales data normally used for corporate audiences, we substituted in the City of Melbourne's excellent Urban Forest dataset. This rich dataset of 70,000 trees with species, diameters, year of planting and estimated life expectancy quickly piqued participants' interest.

Lunch at nearby Thai restaurant Middlefish was excellent and got everyone talking. Vigorous debates over statewide infrastructure policy at one end of the table were balanced by technical questions on Tableau's capabilities at the other.

Then, the group challenge in teams of three: a researcher, a City of Melbourne staff member, and an "other". In one hour, each team had to:

find an open dataset, on data.melbourne.gov.au or elsewhere;

ask a question of the data; and

answer the question with a Tableau dashboard

They did great. We had questions like "when will we get relief from the London Plane?", a novel use of the Urban Forest dataset from an allergy-sufferer. Or how about "is manufacturing shrinking in Melbourne", looking at the change in floor space usage reported in the Census of Land Use and Employment.

But best of all, researchers got a good chance to talk to government directly about data that could be relevant to them, ask for more data, and even just share the experience of research.

[melbdataguru]

Towards an Open Data City Census for Australia

What does a well organised open data movement look like at city council level? Perhaps like this: http://us-city.census.okfn.org/

<p>aoeu</p>

This impressive chart is the combined work of Code for America (citizens working on technology problems with government), the Open Knowledge Foundation (promoting openness), and the Sunlight Foundation (promoting transparency). They identified 18 kinds of data that every US city is likely to collect.

They then assess whether each city publishes this data as open data, using 9 separate criteria: data exists; is digital; is publicly available; is free of charge; is online; is machine readable; is available in bulk; is openly licensed; and is up to date.

So an annual budget published once every year as a PDF would probably score 7 or 8 points, lacking machine readability and perhaps open licensing.

Global Census

Similarly, there is a Global Census, measuring open data availability for different countries. The Global Open Data Index Survey, ranks us 10th, somewhat improbably ahead of Germany:

(The choice of categories here is a bit odd. "Public transport timetables" don't always make a lot of sense at national level, being predominantly state or municipal in many cases. The requirement for a 1:250,000 scale "national map" is just plain weird in the era of OpenStreetMap)

Open Data City Census for Australia

As local governments around Australia begin their open data journey, the time has probably come for an “Australian city open data census”. It would both chart and guide their progress. The US list is a good starting point but some categories don’t apply well here:

Crime stats are probably collected at state level, not city.

City elections are less important here, so are campaign donations so important?

Public transport may be managed at a state rather than city level.

I had already started working on something similar with MAV, the Municipal Association of Victoria: a list of “High Priority Local Government Datasets" that most councils in Victoria could provide. The ultimate goal of that list is somewhat different however: joining up data from the geographically small councils into bigger, more useful datasets. And it focuses on “usefulness” rather than transparency.

Now, local versions of the Open Data Census have already been created for more than 25 different countries: the UK  France  even Burkina Faso and Bangladesh.

But there's still no Australian City Census. So, let's get on it.

Alex Sadleir, Open Knowledge Foundation Australia volunteer, has signed up to coordinate the effort. Get in touch if you can help out!