Making Government Transparent and Accountable
The Sunlight Foundation uses cutting-edge technology and ideas to make government transparent and accountable. Underlying all of our efforts is a fundamental belief that increased transparency will improve the public's confidence in government
The Sunlight Foundation Blog
-
Introducing the Cycle of Transparency
Government transparency is that rarest of political phenomena — a great idea with support across the political spectrum and popularity among the public. Yet, here we are in the 21st century with every tool we would need to make government more transparent and accountable, and still we are operating with a government that often behaves as it did in the 19th century.
So, transparent government is a good thing, but we do not yet have one. Now what?
It’s clear that there is a breakdown between conceptual support for the idea of government transparency and enacting the changes necessary to make it so. There is fear and resistance to change inside government that requires cultural, political, and attitude adjustments. And there’s a large gap between the good intentions of citizens and watchdog groups and think tanks and reporters, and translating those good intentions into effective results. Many people want to act, but they rarely know how or where to begin.
For many, the concept of transparency still simply feels too vague to get behind in a meaningful way. People strongly support transparency in theory, but don’t know what they would need to do, or how they would need to think, to create the “open, transparent government” we talk about. (Continue reading…)
-
Improvements Needed For High Value Datasets On Data.gov
This morning a number of organizations — POGO, OMB Watch, CREW, National Security Archive, the Center for Democracy and Technology and the Open The Government coalition– and Sunlight sent a letter to Vivek Kundra, Federal CIO, about improvements needed to the release of High Value Datasets on Data.gov. Here are the core recommendations included. Please tell us what you think in the comments below.
As advocates for government openness, we support the Administration’s efforts to provide the public with access to information through Data.gov. We are eager to work with you to ensure the success of Data.gov and, in that spirit, write to raise our concerns with the datasets submitted by agencies to fulfill their requirement under the Open Government Directive to post three high value datasets by January 22, and to offer constructive suggestions for improving their usefulness.
As an overall recommendation, we urge you to add public representatives to the Open Government Initiative interagency working committee and ask the committee to address the problems and recommendations identified below.
Release Format and Usability by the Public
We understand one of the primary purposes of Data.gov is to enable the technology community and transparency advocates to most effectively use the data to make a direct impact on the daily lives of the American people. The format of the data plays a key role in its usability; many within the community of advocates who re-use and repackage government data would prefer data in CSV format, rather than the XML format in which many of the posted databases are provided. Accordingly, we recommend that you strike an appropriate balance between formats (such as XML) that serve the coding community and web-based presentations by agencies that can be used and understood by the general public.
In addition, some of the currently posted files are quite large, ranging upward to several hundred megabytes. Their large size undermines their usefulness for most people or organizations. The large number of currently posted datasets also makes it difficult to find a particular database of interest. We therefore recommend that if a Data.gov dataset is available from an agency through a web-based interface, Data.gov link to that interface on the dataset’s Data.gov landing page. For a consumer looking for information on a car seat, for example, it would be far easier to search the Department of Transportation’s online database rather than scrolling through screen after screen of raw data in XML format. Additionally, as agencies continue to post datasets to Data.gov, efforts should be made to identify those of greatest public interest that lack such interfaces and develop web interfaces that allow the data to be explored online.
Further, while we agree there is value in aggregating government data in a single site, it is questionable how much the collocation of the currently posted information on Data.gov actually benefits the public. The site is not searchable by topic and does not provide any way to bring together data from different sources on similar topics.
As an enhancement to the organization of the site, we recommend that you use tagging or metadata to enable the public to bring together information on a topic. The thesaurus that USA.gov uses provides a useful example of the needed vocabulary.
Value of Data
The release of the datasets also has prompted discussions about the value and the quality of the released data, and the additional value provided by access to existing data in a new format. We believe repackaging old information is of marginal value, yet that is what many agencies have done with their recent postings on Data.gov. According to the Sunlight Foundation, of 58 datasets posted by major agencies, only 16 were previously unavailable in some format online. This leaves the impression that agencies posted easily available data, the proverbial low-hanging fruit, rather than seriously considering which of their datasets truly are of high value. While these initial postings can be considered a test run, more attention needs to be directed toward ensuring the overall quality and usefulness of the data.
In addition, sustained attention should be paid to the possibility of making some of the datasets available as feeds that are constantly up to date, rather than as static datasets that are pulled down and then reposted on an occasional basis. We recommend that agencies be required to explain why the data is high value by having them designate which of the “high value criteria” the data meets: information that can be used to increase agency accountability and responsiveness; improve public knowledge of the agency and its operations; further the core mission of the agency; create economic opportunity; or respond to need and demand as identified through public consultation. Similarly, we recommend requiring agencies to indicate whether a high value dataset was previously unavailable, available only with a FOIA request, available only for purchase, or available, but in a less user-friendly format. Going forward, this will make it much easier to track how agencies are complying with the other requirements of the Open Government Directive. While we appreciate the value of data that furthers the mission of an agency, we believe it is equally important to make available to the public data that holds an agency accountable for its policy and spending decisions. We hope to see more datasets of this type available in the near future.
Quality
As is to be expected in efforts of this type, there were a number of glitches–datasets that could not be downloaded or, once downloaded, could not be opened (the Central Contractor Registration FOIA extract from the General Services Administration seems to have caused several users problems). Additionally, some datasets were incomplete (the Hazard Grant Mitigation Program data released by FEMA is missing 23 years of data between 1966 and 1989). Even more troubling, some did not have header rows, and for those that did, their Data.gov pages did not always link to code sheets explaining what those header rows meant. Without this information, the data cannot be used.
We therefore urge the implementation of a responsive feedback mechanism that allows the public to alert an agency that a specific dataset is not working, lacks information, or is missing explanatory material and provides a response to the concerns within a specified time. One way to address this may be to include an agency contact with the ability to resolve any database problems or provide information about the database. The interagency working group could sample the quality of these agency-specific dialogues to ensure that they are having an impact and to develop recommendations on best practices to improve the responsiveness. Additionally, we strongly recommend that all datasets on Data.gov be directly associated with their code sheets.
Finally, we are concerned with the current lack of public notice when data is removed from the site. We respectfully urge you to note all raw tools and data that are removed from Data.gov, and to provide an explanation for their removal.
Many of the concerns outlined above apply across all or many of the agencies’ datasets. Accordingly, we think that standards for handling these types of problems can easily be addressed through the interagency working group and then disseminated amongst the agencies.
-
White House Asks for Help with Data.Gov and OGD Dashboard
The White House is soliciting feedback on Data.Gov and its Open Government Directive Dashboard. Here is the nub of their request for your participation:
1. Open Government Dashboard: The Open Government Directive calls for the creation of an Open Government Dashboard to measure progress and impact. Deputy Chief Technology Officer, Beth Noveck is looking for your input, including as to the metrics by which we measure success. Click here to participate.
2. Future of Data.gov: The Open Government Directive instructs all federal agencies to make available high-value data that promote national priorities and improve the lives of everyday Americans through Data.gov. Yet the current version of Data.gov is just the beginning. Chief Information Officer Vivek Kundra asks for your help in shaping the future of this key open government platform. As part of the Data.gov Dialogue, you can download the draft plans, submit a new idea, or comment on someone else’s. We look forward to Evolving Data.gov with you.
-
Real-Time Data Program Wins Innovation Award
I know this is a couple days old, but it hasn’t been mentioned here yet. The District of Columbia’s real-time online data disclosure project was one of six winners of the Innovations in American Government awards given out by the Harvard Kennedy School’s Ash Institute for Democratic Governance and Innovation. The project was spearheaded by then-D.C. Chief Technology Officer (CTO) and current federal Chief Information Officer (CIO) Vivek Kundra. You can see the two sites singled out for praise below:
According to the Ash Institute, “this is the first initiative in the country that makes virtually all current district government operational data available to the public in its raw form rather than in static, edited reports.” Real-time data disclosure is becoming far more common in cities across the nation with San Francisco introducing DataSF.org and the New York City legislature examining open data legislation. (Vancouver, Canada has also endorsed the release of city data in raw form.)
Real-time, raw data disclosure is the cutting edge in transparency and government innovation. While the federal government has released Data.gov, a raw data site similar to D.C.’s, there are countless sets of public data compiled by the federal government that are in one or more of the following three categories: 1) Not online; 2) Not in a structured format; 3) Not compiled and disclosed in real-time. As many public data sets as possible should meet these three criteria. For some data it is unreasonable to ask for real-time disclosure. These sets should then, at least, meet the first two.
Side note: It’s great to see my city defy our Rodney Dangerfield-like existence and finally get some respect.
-
This Week in Transparency – August 14, 2009
Here are some of the more interesting media mentions of Sunlight and our friends and allies over the past week:
Jonathan D. Salant and Lizzie O’Leary with Bloomberg.com have an article showing how there are six lobbyists attempting to influence the health care reform debate for each of the 535 members of the House and Senate. That figure is three times the number of lobbyists registered to lobby on defense. They used data from the Center for Responsive Politics to illustrate how every one of the 10 biggest lobbying firms by revenue is attempting to influence the debate on behalf of some interest or another, spending $263.4 million on lobbying during the first six months of 2009 alone. They quote Bill Allison, Sunlight’s senior fellow, “Whenever you have a big piece of legislation like this, it’s like ringing the dinner bell for K Street.” Multiple other outlets picked up the article and Bill’s quote, including Kate Barrett at ABC News. And David Schechter, CNN’s senior national editor, wrote a column about the lobbying feeding frenzy surrounding the health care reform debate. He lists Sunlight and OpenSecrets.org as good sources for information on the “lobbying largesse.”
In light of the increasingly heated debate over how to reform health care policy, Lisa Stone at BlogHer wrote about the new partnership between BlogHer and OpenCongress, the joint project between the Participatory Politics Foundation and Sunlight, to provide a forum to move the discourse in a more civil and positive direction. They have asked Nancy Watzman, Sunlight’s director of the Party Time project, to share her investigations on their site multiple times a week. Be sure to check their coverage out, which starts today.
Writing at Forbes, Tim O’Reilly, founder and CEO of O’Reilly Media, wrote about what he calls the promise of innovation provided by Government 2.0. And he asked, “How does government itself become an open platform that allows people inside and outside government to innovate?” O’Reilly points to the Apps for America contests as an example of the “virtuous circle of citizen innovation” using the information made available through the White House’s Data.gov. PC World published a piece by Grant Gross with IDB News Service on how the contest is asking developer to use the raw data released on Data.gov and elsewhere to demonstrate the power of data-publishing and number-crunching services. Gross discussed with Clay Johnson, Sunlight Labs’ director, about how the Labs works to assist traditional and citizen journalists with investigative reporting. “As the Obama administration begins to release more data, there aren’t enough fingers on keyboards here in Sunlight Labs to handle all this,” Clay said. “Has the Obama administration succeeded in making more government data available? You’re talking to the guy with the most unquenchable thirst for that, who will never say that they’re successful.” (Continue reading…)
-
Ten Great Government Web Sites
Joab Jackson at Government Computer News has, for the second year in a row, pulled together a compendium of 10 government Web sites that he says are embracing both social networking tools and transparency. (He produced his first list last August). This year’s list includes sites that embrace the Web’s full potential, Jackson writes, and they can offer ideas for other agencies seeking to improve their own sites.
Governmental agencies have realized that a Web presence is essential, since most citizens are now beginning to interact with government online. “By and large, agencies have responded to that demand by creating richer, more interactive sites,” Jackson wrote. He quotes Sheila Campbell, co-chair of the Federal Web Managers Council, saying agencies are starting to see that they need to social media revolution and its larger information ecosystem. “Managing the Web isn’t just managing the Web site. It means putting the content out where people are on the Web.”
Making Jackson’s list are:
Data.gov for fundamentally shifting how government interacts with the Web.
Forge.mil which provides an online meeting place for military agencies to build software in a collaborative fashion.
San Francisco’s Trasit.511.org combines the schedules of dozens of subway, light-rail, trolley and bus systems to provide a one-stop shop that can help users plan a route from doorstep to doorstep.
The State Department’s State.gov for using Facebook, Twitter, YouTube, Flickr and other social media tools to get the word out about the agency’s activities.
Government Printing Office’s Federal Digital System offers public access to documents from all three branches of government through a single portal. “It is a Web site of sweeping scope,” Jackson writes.
The State of Utah’s Web site for pulling off what “is perhaps the most amazing trick of all: not looking like a state-run Web site.”
Science.gov, out of the Office of Scientific and Technical Information within the Energy Department, presents information and makes it accessible by subject matter rather than by the office or agency that generated the information.
The U.S. Postal Service’s site for making it possible to do online about 80 percent of everything you can do by taking a trip to the post office.
The Department of Health and Human Services’ two sites Women’s Health/Girl’s Health that uses plain, easy-to-understand language, to address more than 800 topics related to women’s health, such as fitness, nutrition, breastfeeding, pregnancy and reproductive health.
Federal Web Managers Council’s WebContent.gov provides most of the information Web managers need to bring “uniformity and quiet sophistication” for federal Web sites.
It’s highly encouraging to see government agencies responding to the public’s demand for more helpful and easy-to-use online tools.
-
This Week in Transparency – July 24, 2009
Here are some of the more interesting media mentions of Sunlight and our friends and allies over the past week:
CQ Weekly’s Maura Reynolds wrote about the Obama administration’s successes and failures in achieving its transparency goals six months into the term. Reynolds quoted Ellen Miller, Sunlight’s director, about how many of their transparency initiatives are still in development and how the kinks are being worked out. “A default position that government data will be accessible to the public in machine-readable format is a huge step forward,” Ellen said. “Is it moving as fast as I’d like? Of course not. But I can be patient while this unfolds.” Ellen also commented on some of the administration’s initiatives, such as “town hall” meetings, that have been tightly controlled. “There is real transparency, and then there is transparency theater,” she said. “I can distinguish between the two.” Reynolds wrote that the more people expect the Internet to deliver the information they want, the more kinds of information they will expect to access that way. “It’s kind of a genie out of the bottle,” Ellen said. “The Internet has raised expectations. I fundamentally believe that the way technology pushes information out to the edges will have a powerful effect on the power structure.” Reynolds reports that open government advocates praise two federal Web sites, USAspending.gov, a site that tracks all federal spending and was set up as a result of a bill co-sponsored by then-Sen. Obama, and Data.gov, the site the new administration designed as a “one-stop shop for number crunchers that consolidates statistics across federal agencies in standard, machine-readable formats.” The article quotes Gary Bass, director of OMB Watch, saying the sites could be vehicles for connecting government performance to spending. “From the point of view of the average user, there has been nothing like this before. That is truly a credit to this administration.” Reynolds notes that it was OMB Watch’s FedSpending.org that served as the technical platform for USAspending.gov.
Despite the existence of rules requiring congressional lawmakers to disclose earmarks they request, rules do not exist requiring them to disclose items classified as “program support.” The Washington Post’s Carol Leonnig illustrates this problem with a report on how $160 million intended to help Mexico’s police buy U.S.-made first-responder radios was tucked into the voluminous congressional plan for U.S. military spending next year. Leonnig quotes Bill Allison, Sunlight’s senior fellow, “It kind of makes a mockery of the disclosure requirements we have. They will disclose the little things, the $1 million projects, but when you have the big-ticket items, you don’t have members willing to take responsibility for those.”
Stephanie Condon, writing at CBS News‘ “Political Hotsheet” column, cited a report from Taxpayers for Common Sense that found that lawmakers serving on the the House Appropriations Subcommittee on Defense included 1,080 earmarks worth $2.7 billion dollars in the fiscal-year 2010 defense appropriations bill they approved last week. The lawmakers specifically requested more than $1.6 billion in earmarks for their campaign contributors, entities who had donated nearly $1 million to the committee members.
-
This Week in Transparency – July 17, 2009
Here are a few of the more interesting media mentions of Sunlight and our friends and allies from the week:
Jeff Jacoby, columnist for The Boston Globe, mentioned ReadTheBill.org in a piece he wrote calling on congressional lawmakers read legislation before they vote on it. Glenn Reynolds, at his Instapundit blog, linked to Jacoby’s column. Andrew Sullivan’s blog, The Daily Dish, followed by linking to Reynolds.
In Washington Monthly’s July/August edition, Charles Homans wrote about the Obama administration’s “experiments with data-driven democracy.” The article centers on the work of Vivek Kundra, the White House’s chief information officer, and mentions both the District of Columbia’s Apps for Democracy contest and Sunlight’s Apps for America contest. Homans quotes Clay Johnson, Sunlight Labs’ director, saying Kundra has his work cut out for him. “I have nothing but respect for what he’s trying to do. But it’s a hard job, and it’s going to take some time for this to actually happen right. I mean years.” While discussing Kundra’s launch of Data.gov, Homans again quotes Clay, “The top data source is on the world’s copper smelters, which isn’t going to tell us very much about what’s going on inside of our government.”
As Ellen Miller, Sunlight’s director, wrote earlier this week, “When it comes to following the money that’s flowing to power on Capitol Hill, no one does it better than the Center for Responsive Politics.” For instance, MAPLight.org used CRP data to show how money watered down the energy bill, the American Clean Energy and Security Act of 2009 (HR 2454). With Congress debating health care reform, Forbes used CRP data to show how America’s Health Insurance Plans, the political advocacy and trade group for the health insurance industry, has spent nearly $10 million on lobbying Congress in the past two years. Robert J. S. Ross, writing at The Huffington Post, quotes CRP about how the insurance industry has contributed $568 million to political campaigns since 1998. CNN’s Jonathan Mann used CRP data in noting how doctors have spent roughly two-thirds of a billion dollars lobbying lawmakers in the last 10 years.
-
This Week in Transparency – July 2, 2009
Here are a few of the more interesting media mentions of Sunlight and our friends and allies from the week:
Last Friday evening’s June 26th program, CNN’s Lou Dobbs broadcasted a piece by correspondent Louise Schiavone about the Cap and Trade Energy Bill that the House of Representatives was to vote on and pass later that evening. Schiavone interviewed Jake Brewer, Sunlight’s engagement director, who said, “This is the kind of bill that’s going to affect our economy on a massive scale, our climate, our national security, and is not the kind of thing to be taken lightly. The opacity of this process is — to be perfectly honest, it’s infuriating.” Schiavone then stated erroneously that Sunlight opposed the bill. For the record, Sunlight has no position on the content of the bill itself, but advocates for the Congress to put all non-emergency legislation online for 72 hours before voting on it. The transcript can be read here, and the video is below.
-
Guardian gives Brits the data goods
We’ve had our share of political scandals in this country, but there’s something so–British–about the expense scandal swirling around members of parliament lately. A floating duck island? A little life coaching for a girlfriend? Poppy wreaths? And lots and lots of second home allowances.
While the story was broken by the Telegraph (and the word is that the paper probably paid big bucks for the information), the UK Guardian newspaper’s “Data Store” is the spot to go to get google spreadsheets detailing all the details on which members of parliament spent how much on what. And much more, it turns out, from tax databases to swine flu to school performance records. The paper’s Data Blog (slogan: facts are sacred), points out interesting trends and uses of the data.
That would include Tony Hirsch, of the Open University, who took the expense data and loaded it into the “Many Eyes” visualization tool. He came up with a number of intriguing charts and graphs, including this “scatter plot” that allows you to compare members’ expenses on two axes. (See screen shot below.)
This is great stuff, and it looks as if the British government may be poised to take the concept further. Today TechPresident reported that the British government is considering its own version of Data.gov. While a good amount of data is already made available by the British government, it comes in a variety of awkward formats (sound familiar)? Making these data available in easily downloadable form in one place would be a huge step forward. And it all will have that disctinctly British tone. My favorite data description listed? A health care database that gives you “conditions by body-part.”
