The crisis for government data in a 21st century shutdown

It was just three months ago that President Obama celebrated the startups built on open government data. With the help of freely available public information, these companies can build sustainable businesses while creating jobs and solving problems. That’s the idea, anyway. Today, walking through the strangely empty streets of downtown D.C., it’s hard to imagine an entrepreneur naive enough to take that offer.

Across the government’s data offerings, the now ten-day shutdown has meant disruptions, downtime and confusion. A sampling of what’s available and what isn’t: is down; so is the Census. is up, but it’s not releasing any new data. Federal Register staffers say they’re keeping the system up, and that some new data might come through, but that they can’t correct errors. The Library of Congress said its THOMAS system would be going offline, then decided it would stay up. is still online, thanks to a technicality about how it’s funded. And it appears that you can still pull historical records about train accidents, for some reason. But countless other data services are down or display vague messages about the possibility of bad data.

The fate of each service was determined by its parent agency based on a confusing set of administration instructions, political considerations and, seemingly, a dose of random chance. The ensuing disruption has been considerable, unnecessary, and chaotic.

The damage done by the shutdown extends far beyond the open data community, of course, and in many cases its repercussions are far more severe. I’m not going to tell you that the loss of could ever compare to a cancer patient delaying treatment.

Still, our society depends on open government data to a startling extent — so much so that even milliseconds can matter. From the maps on our phones to the recall data that keeps our food safe, government-produced information and the systems that distribute it grow more essential every day.

It’s tempting to dismiss the dot-gov disruption as a small part of a larger crisis, but this isn’t the first time that our government has treated its role as an information provider recklessly. The last few years have seen the end of both the Consolidated Federal Funds Report and the Statistical Abstract, both of which were canceled over the howls of researchers (the latter after more than 130 continuous years of publication). In 2011, Sunlight was forced to lead the charge to save the E-Government fund, a small pool of money that sustained, and many of the Obama administration’s other signature open data achievements. And Congress continues to sporadically attempt to defund the American Community Survey, seemingly because of a few members’ paranoia.

Confusingly, the House Republican majority that has engineered this shutdown has, until now, been among the most reliable and forward-thinking sources of open data leadership in our government. One can only hope that it will soon resolve the internal tensions that have led to this crisis.

What we need to do better

Although the Obama administration is not directly responsible for the shutdown, it can take action to restore faith in government data as a civic resource and platform for business.

First, the administration should level with data users about the sustainability and probable degradation path of individual systems. That the United States government has to make such hedges is a national embarrassment, but here we are. The administration should commit, in advance, to telling users which services could fail and how.

Second, the administration should consider data consumers when making decisions about which services to shutter. The Office of Management and Budget already weighs the preservation of life and property when making shutdown decisions; they should expand these criteria to include private industry. Yes, it’s reasonable for the administration to use the political levers available to it when responding to political attacks. But unnecessarily disrupting data services that people depend on does long-term damage to the idea of open data — an idea that the administration has spent considerable energy promoting. It can’t have it both ways.

Third, bulk data distribution has to be made a priority. The White House’s Digital Government strategy mandates a greater use of APIs within government. APIs are great — Sunlight offers a number of them — but they can encourage dependence among their users, and should only rarely be government’s first choice. Bulk data is easy to host, easy to mirror and allows third parties to lend a hand when government stumbles.

None of these steps will resolve the ludicrous spectacle of a wealthy nation failing to fund its government. But they can help — at least until we find legislators who are prepared to be responsible stewards to their country and its data.

Tom Lee is Director of Sunlight Labs at the Sunlight Foundation, a nonpartisan nonprofit that uses technology and ideas to make government transparent and accountable. Follow him on Twitter @tjl.