Liberating Civic Data

Liberating Civic Data

The Knight-Mozilla OpenNews project brought 40 hackers to the MIT Media Lab this weekend to liberate civic data.

The Knight-Mozilla OpenNews project brought 40 hackers to the MIT Media Lab this weekend to apply their skills to civic data problems. This is one person's experience with his first serious Hackathon.

3:00 PM. Opening Circle

Dan Sinker, who heads the OpenNews project, convenes the group, talking about the schedule of the next 24 hours, logistics of keeping people fed, and turns to introductions. People have come from around the country as well as overseas to attend and report they're feeling anywhere from "jazzed" to "exhausted" from long travel. He explains the purpose of the Hackathon, to do projects to "liberate" civic data, that is, make data that is theoretically open actually accessible in useful meaningful ways. The group turns to "pitches", ideas that attendees have that they'd like others in the group to work with them on. The most imaginative of pitches is one that suggests using data being collected on Mars to build a massive role playing game, allowing kids to design their own Mars-exploring robots and excite them about the prospect of being astronauts. The amazing thing about this idea is, even though offered in jest, its plausibility. It could work, it could inspire future astronauts, and it could be done by the people in the room, only not in the 24 hours we have to work. Other pitches are more grounded. Develop design principles for usable government data sites. Visualize the New York State prescription drug pricing database. Wolfram Alpha for government data. Work the Texas campaign contribution data.

After a dozen or so ideas are pitched, then comes the team formation. Those pitching ideas stand up and people interested in working on them join them and talk. People ebb and flow as they get a sense of what it means to be part of the team, the problems being solved, the skills required. Having not written a line of code for years, I start first with the design principles group which quickly decides to merge with another group, those who want to write a guide for opening government data. While these are really two very different problems, in the give and take of a hackathon they're declared similar enough in the sense that they are primarily writing projects, not coding projects. There's no formal "hey, let's start working on this" declaration, but the conversation quickly shifts from the hypothetical to the practical and we find ourselves in front of a whiteboard wall, and start talking about what our deliverable is.
Data Guide whiteboard work In a space of a couple of hours, we've gone from being (mostly) strangers to collaborators. Nobody asks about your qualifications to do the work, all that matters is your willingness to do something relevant.

5:00 PM

Our discussions take on an interesting rhythm. We talk, settle on what we're about to do, pause, and then start talking again about what we're going to do. This isn't going around in circles, it's more the group mind, having come to a rough consensus, taking a step back, finding itself unsatisfied, trying again. We end up doing research on existing guides to opening government data, searching for them, communicating with each other in verbal shorthand. One of our team members sets us up on GITHUB, an open source repository site, and we start a collaborative document collecting links. Another team member gets us focused on milestones and deadlines, and we aim to have a skeleton document sketched out before dinner. Dinner comes - Indian food - but we keep working away. The notion emerges that, while there are a lot of open data guides for government, they're abstract, and don't help with what you need to do to get data online once you've decided to do it. We talk about document structure and settle on a template into which we can plug various domain-specific guides.
I pick environmentalism and climate change, thinking that this is an important topic and one that there might be some good examples whose larger adoption would be a helpful step forward. But web searching finds nothing worth emulating. There are very limited examples of cities reporting energy usage of their own buildings but nothing that feels interesting or ambitious.

10:30 PM Closing circle

Sinker reminds us that he likes to run "humane" hackathons, one where people leave to sleep, rather than work round the clock. This is a check-in on what teams are working on, if they have any needs, either logistics or technical, which might be helped by others. I ask if anyone knows of a good example of open environmental data.

9:15 AM Sunday

I walk back into the Hackathon a few minutes after starting time. There were already 20 or so people heads-down at their computers, working quietly. Erika Owens, the Knight-Mozilla OpenNews community manager, has laid out a first "Saturday comics" breakfast of cereal and poptarts. Later, there'd be a real breakfast, and even later, pizza. Napoleon Bonaparte said that armies move on their bellies. Hackathons code on them.

Sunday's Saturday Morning Cartoon breakfast

One of the team members who'd authored a document about putting restaurant health inspections online requests someone review her work. Having hit a dead end on my task, I go to it. A few weeks before the Hackathon, I'd moderated a panel about Cambridge as a news desert and included, as an example of oases in the desert, the MIT Tech's report on Cambridge restaurant inspections, so I'm modestly familiar with the topic. I add detail, expand a few items, include pointers to other work.

Somehow, working on this piece breaks my block and I realize that, framed correctly, I have gathered enough information to do something arguably useful. I author a document about opening energy utilization data that starts with "Congratulations! You're a pioneer" and points to a white paper about utility data ownership and stewardship, an emerging standard for utilization data, and an amazing UCLA-generated interactive map of Los Angeles electrical utilization.

12:30 PM Sunday

I return from some personal business and check in with the team. More documents have been authored and team members are working furiously on the framework that will turn our works into a web site. This is work to which I can offer nothing except encouragement as the tools being used are not anything I've had experience with.

2:45 PM Show and tell

The room gets set up "science fair" style, so folks can wander through and get demos of the various pieces of work. Beyond fellow Hackathon attendees, we get a stream of people who'll be attending the MIT-Knight Civic Media Conference that starts shortly after our hackathon ends. People look respectfully at our work, probing why we really think this is useful. One person notes that he doesn't think this will help him in his job, then pauses to note that he's the Chief Technology Officer of the Federal Communications Commission. We observe that he's not the audience, it's aimed more a modest sized cities without the resources of the FCC. The idea seems to have resonance.

I take the time to see the work of other teams. One group has developed a web site they call "Judgmental" to take Virginia court decisions locked up in PDF files and turn them into a searchable database. Another has built something called "OpenOpenNewsNews", designed to scrape news web sites and learn what you can about them. But the most fascinating has to be the app built on top of the New York State prescription drug price database. You can look at price discrepancies between different pharmacies for the same drug and locate the pharmacy with the cheapest prices. Not only does the app answer interesting questions, it's well designed and just looks good.
Demo'ing the NY State Prescription Drug Price database app

4:00 PM Closing Circle

Sinker wraps up the day saying this was one of the most productive Hackathons he's had the pleasure to run. The tendency of some hackathons is for teams to talk rather than do but this one, he notes, had moved to doing and building quickly.

Hackathons serve two purposes. First, they build community. People have met each other, old ties renwed, new ones forged. Second, they build things that, under the best of circumstances, live on past the Hackathon. There's already been interest outside the Hackathon for using Judgmental, folks on the Data Guide team have committed to continuing the work on the guide. Both goals have been achieved.

OpenNews-MIT Data Hack (Day 1) from Laurian Gridinoc on Vimeo.

OpenNews-MIT Data Hack (Day 2) from Laurian Gridinoc on Vimeo.

Creative Commons License
Liberating Civic Data by Saul Tannenbaum is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.


This is really interesting!! What do we need to do to make CCTV a hacker/maker space? Or maybe it has been for the past 25 years!