Note: Jeremy Weinstein & Joshua Goldstein’s response to this Article can be viewed here: The Benefits of a Big Tent: Opening Up Government in Developing Countries
The internet’s power to make government information more available and useful has, in the last several years, become a topic of keen interest for citizens, scholars, and policymakers alike. In the United States, volunteers and activists have harnessed information that the government puts online in key domains, ranging from the federal legislative branch to local city services, and have created dynamic new tools and interfaces that make the information dramatically more useful to citizens. These new tools have sparked significant academic and popular interest and have begun to prompt a fundamental shift in thinking: Policymakers have begun to consider not only the citizens who may ultimately benefit from government information, but also the third parties who can play a valuable mediating role in getting the information to citizens.
The primary concrete result of this trend is that governments have made a growing range of public sector data available in machine-processable electronic formats that are easier for others to reuse. Information that enhances civic accountability, including pending congressional legislation and federal regulations, is indeed more readily available. But more mundane and practical government information, from bus schedules to restaurant health inspection data, is also being provided in friendlier formats. Such data can be used to improve quality of life and enhance public service delivery, but may have little impact on political accountability.
Recent policy initiatives that promote or reinforce this trend have been described as “open government” projects. These initiatives usually include the provision of reusable data as one among a range of steps designed to increase overall governmental transparency. For example, President Obama’s Open Government Directive, which was designed to implement the new administration’s overall “principles of transparency, participation and collaboration,”1 instructed executive branch agencies, inter alia, to “publish information online in an open format . . . . An open format is one that is platform independent, machine readable, and made available to the public without restrictions that would impede the re-use of that information.”2 Similarly, the multilateral Open Government Declaration,3 signed by the United States and seven other countries in September 2011,4 situates these new technologies of data sharing in the context of political accountability.5 It begins with an acknowledgement that “people all around the world are demanding more openness in government.”6 Among their promises, the signatories commit to “provide high-value information, including raw data, in a timely manner, in formats that the public can easily locate, understand and use, and in formats that facilitate reuse.”7
These new “open government” policies have blurred the distinction between the technologies of open data and the politics of open government. Open government and open data can each exist without the other: A government can be an open government, in the sense of being transparent, even if it does not embrace new technology (the key question is whether stakeholders know what they need to know to keep the system honest).8 And a government can provide open data on politically neutral topics even as it remains deeply opaque and unaccountable. The Hungarian cities of Budapest and Szeged, for example, both provide online, machine-readable transit schedules,9 allowing Google Maps to route users on local trips. Such data is both open and governmental, but has no bearing on the Hungarian government’s troubling lack of accountability. The data may be opening up, but the country itself is “sliding into authoritarianism.”10
The popular term “open government data” is, therefore, deeply ambiguous—it might mean either of two very different things. If “open government” is a phrase that modifies the noun “data,” we are talking about politically important disclosures, whether or not they are delivered by computer. On the other hand, if the words “open” and “government” are separate adjectives modifying “data,” we are talking about data that is both easily accessed and government related, but that might or might not be politically important. (Or the term might have a third meaning, as a shorthand reference to the intersection of data meeting both definitions: governmental data that is both politically sensitive and computer provided.)
In this Essay, we acknowledge that this ambiguity may sometimes be beneficial, but ultimately argue that the term “open government” has become too vague to be a useful label in most policy conversations. Open data can be a powerful force for public accountability—it can make existing information easier to analyze, process, and combine than ever before, allowing a new level of public scrutiny. At the same time, open data technologies can also enhance service delivery in any regime, even an opaque one. When policymakers and the public use the same term for both of these important benefits, governments may be able to take credit for increased public accountability simply by delivering open data technology.
In place of this confusion, we offer a stylized framework to consider each of these two questions independently. One dimension describes technology: How is the disclosed data structured, organized, and published? We describe the data itself as being on a spectrum between adaptable and inert, depending on how easy or hard it is for new actors to make innovative uses of the data. The other dimension describes the actual or anticipated benefits of the data disclosure; the goals of disclosure run on a spectrum between service delivery and public accountability. This is admittedly a simplification of reality: In practice, many disclosures serve both objectives. However, it is common for one of the two motives to predominate over the other, and we believe this provides a useful starting point for thinking about the competing goals of disclosure.
On the diagram, the vertical axis describes the data itself, and the horizontal axis describes the extent to which service delivery or public accountability predominates as a goal or anticipated result of the disclosure. Along the vertical dimension, there is broad political consensus in favor of adaptable data; but, horizontally, there are differences of opinion about the relative political importance of service delivery and public accountability as end goals for public disclosure. (Our discussion in Part III, below, illustrates these dimensions by populating the graph with examples of concrete public policies.)
We have organized our discussion as follows: Part I.A explains the conceptual origins of the relatively modern idea of open government as a public policy, starting with the first recognized use of the term in the mid–twentieth century. The phrase is of fairly recent vintage, but it reflects a particular perspective on the issues it describes—and it was well established before the internet came into being. Part I.B, correspondingly, explores the conceptual roots of open data, an idea that has always included, but has always applied far beyond, the kinds of information associated with civic transparency. Part II follows the story forward in time, as these concepts begin to merge and give rise to the ambiguous idea of open government data, and details some of the confusions that have ensued in the wake of this ambiguity. Part III describes our alternative proposal, which differentiates the widely shared goal of adaptable data from the more controversial choice between enhanced service delivery and enhanced public accountability as the end goals of disclosure.
I. Conceptual Origins
Open government and open data each have rich conceptual histories with independent origins. These histories are indispensable tools for understanding the current debate.
A. Conceptual Origins of Open Government
The idea of open government, as a synonym for public accountability, is part of the peacetime dividend that America reaped after the Second World War. After the war ended, the federal government was left in a state of relative opacity. Having grown accustomed to wartime information restrictions, the federal workforce was “fearful of Cold War spies, intimidated by zealous loyalty investigators within and outside of government, and anxious about” workforce reductions following the war.11 As a result, “the federal bureaucracy generally was not eager to have its activities and operations disclosed to the public, the press, or other governmental entities.”12
The opacity surrounding World War II was not, as wartime opacity might be today, a deviation from a clearly established statutory requirement of federal government transparency. Instead, prior to World War II, the key federal law controlling disclosure of government information was the archaic Housekeeping Statute of 1789,13 which gave “[g]overnment officials general authority to operate their agencies” and withhold records from the public.14 The Administrative Procedure Act of 1946,15 while it did contain a general requirement of access to public records, empowered agencies to restrict access “in the public interest,” with or without “good cause found”—a faint precursor of the robust justificatory requirements and procedural assurances of modern administrative law.16
The period from 1945 to 1955 was a “crucial decade” of early pressure toward greater openness, driven in part by the American Society of Newspaper Editors (ASNE).17 In 1953, ASNE commissioned a report, prepared by a prominent newspaper attorney named Harold Cross, titled The People’s Right to Know: Legal Access to Public Records and Proceedings.18 The report’s foreword noted that Cross had “written with full understanding of the public stake in open government”19—one of the earliest known uses of the term. The report became “the Bible of the press and ultimately a roadmap for Congress regarding freedom of information,”20 and it served as “a call to battle . . . aimed primarily at the needs of news editors and reporters.”21
In 1955, the U.S. Congress created the Special Subcommittee on Government Information, also known as the Moss Committee,22 which incubated the legislation that became the Freedom of Information Act a decade later.23 Wallace Parks, who served as counsel to the subcommittee,24 gets credit as the first to expound on the term “open government” in print, thanks to his posthumous 1957 article, The Open Government Principle: Applying the Right to Know Under the Constitution.25 Parks does not explicitly define the term “open government” in the article (in fact, he uses the phrase just four times in twenty-two pages), but his usage makes clear that he sees open government as a matter of accountability:
From the standpoint of the principles of good government under accepted American political ideas, there can be little question but that open government and information availability should be the general rule from which exceptions should be made only where there are substantial rights, interests, and considerations requiring secrecy or confidentiality and these are held by competent authority to overbalance the general public interest in openness and availability.26
Parks’s thinking, and perhaps his choice of words,27 was part of a long campaign of legislative pressure that would culminate with the passage of the Freedom of Information Act (FOIA)28 in 1966. Although President Lyndon B. Johnson “hated the very idea of journalists rummaging in government closets, hated them challenging the authorized view of reality, [and] hated them knowing what he didn’t want them to know,”29 he nonetheless signed the FOIA bill, professing “a deep sense of pride that the United States is an open society in which the people’s right to know is cherished and guarded.”30
Over the next several decades, policy stakeholders used the term “open government” primarily as a synonym for public access to previously undisclosed government information. When Congress amended FOIA in 1974,31 it noted that “[o]pen government has been recognized as the best insurance that government is being conducted in the public interest.”32 Similarly, the Privacy Act of 1974 aimed to achieve the ideals of “accountability, responsibility, legislative oversight, and open government” together, while respecting citizen privacy in government-held information.33 Congress also considered open-meeting laws—like the Government in the Sunshine Act,34 which threw open the doors of federal agency meetings—to be under the umbrella of open government.35 As the case law of FOIA and related statutes developed through the 1970s and 1980s, federal court decisions began to use the term “open government” as well, likewise referring to governmental transparency.36
B. Conceptual Origins of Open Data
The internet holds obvious promise as a tool for sharing more data, more widely, than has ever been possible before. Across a wide range of technical fields, the adjective “open” has become a powerful, compact prefix that captures information technologies’ transformative potential to enhance the availability and usefulness of information.
Parallel explorations of the possibilities have been unfolding in a number of areas, accelerating in tandem with the growing uptake of the internet. For example, the Open Access movement aims to make peer-reviewed scientific literature freely available online.37 The Open Educational Resources campaign seeks to create digital repositories of free learning materials to support global access to knowledge.38 Open technological standards create pools of patent rights, relieving individual innovators of the need to negotiate patent licenses.39 The Creative Commons system of copyleft licenses, which makes it easier for creative artists to share and reuse each other’s work, aims toward “an Internet full of open content, where users are participants in innovative culture, education, and science.”40
Similarly, a programmer’s decision to release her software under an “open source” license means that the program’s source code will be freely available to its users.41 The phrase “open source” has also, more broadly, become shorthand for the collaborative innovation strategy that underlies many open source software projects—an ethos in which anyone can contribute, abundant scrutiny can help to find and resolve bugs,42 and a community of creators can take pride in a useful, freely available end product.43
Across each area, there is a common thread: When many individuals or groups are able to access information themselves and interact with it on their own terms (rather than in ways prescribed by others), significant benefits can accrue. Each of these movements is focused on certain classes of information, and each one leverages new technology to make that information more freely and readily available and useful.
The label “open,” as applied to various kinds of information, thus inherits both a technological and a philosophical meaning. At a technological level, the term suggests using computers to handle information efficiently in place of manual human processing, greatly extending the range of logistically feasible ways in which information can be used. The extent to which this is possible often turns on technical details, as computers can more readily transform information that is provided in standard, structured formats.
Philosophically, the term suggests participation and engagement—all the people who might benefit from information can share and reuse it in a democratized, accessible way. This implies an absence of legal barriers to innovative new projects, and a larger cultural enthusiasm for innovative and sometimes unexpected developments.44
The label “open data” combines both senses of the word “open”—both the term’s technological meaning and its philosophical meaning—with a focus on raw, unprocessed information that allows individuals to reach their own conclusions. Before its civic uses, scientists used the term to refer to raw, unprocessed scientific data.
The earliest appearance of the term “open data” in a policy context appears to come from science policy in the 1970s: When international partners helped NASA operate the ground control stations for American satellites, the operative international agreements required those partners to adopt an “open-data policy comparable to that of NASA and other U.S. agencies participating in the program, particularly with respect to the public availability of data.”45 The agreements also required that data be made available to NASA “in the NASA-preferred format.”46
Later, a 1995 National Academy of Sciences report titled On the Full and Open Exchange of Scientific Data elaborated the idea of sharing data from environmental monitoring satellites, perhaps reflecting its shared lineage with those earlier NASA agreements: “International programs for global change research and environmental monitoring crucially depend on the principle of full and open exchange . . . . Experience has shown that increased access to scientific data, information, and related products has often led to significant scientific discoveries and the opportunity for educational enhancement.”47
The term “open data” has also appeared in the life sciences context, principally in relation to genetic data. A feature on Jim Kent, the graduate student whose programming work allowed the publicly funded Human Genome Project to finish its work before competing private efforts did, said in part: “Kent’s work illustrates the need to think about more than just open source code; in the scientific community there is a growing awareness of the importance of open data.”48
II. “Open Government” Meets “Open Data”
A. Early Roots of the Convergence
Government data started going online almost as soon as the internet opened to individual users in the early 1990s. The earliest pioneer was Jim Warren, a sixties radical from Silicon Valley. Warren was well known as the founder of the West Coast Computer Faire, one of the first venues to showcase the personal computer.49 He was also known as an open government activist, but his particular flavor of transparency had a high-tech twist.50 In 1993, he “show[ed] California Assembly Member Debra Bowen how public access to state legislative records could be accomplished via the Internet at low cost and high benefit to the public.”51 Bowen introduced A.B. 162452 in March 1993, and Warren “single-handedly launched a crusade to ensure the bill’s passage,” which succeeded later that year.53 California became “the first state in the nation to put its legislative information, voting records, and state laws online.”54 Following California’s lead, open government advocates in at least a dozen other states began to push similar grassroots proposals.55
At the federal level, when the Republicans gained control of Congress in 1994, they enjoyed a fresh opportunity to overhaul that body’s infrastructure—the first such opportunity since widespread public use of the internet began. The website THOMAS, launched in 1995, provided public access to proposed legislation, directory information about members and committees, and daily hearing schedules, among other useful documents.56 Although today discussions of open government in Congress often begin with THOMAS, the website was not clothed in the language of “open government” at its launch.57 Before the convergence, “open government” referred narrowly to the initial release of previously undisclosed government information or the effort to get such information released. At its inception, THOMAS simply increased the accessibility of congressional work that was already publicly available.58 While this increase in accessibility was dramatic, it arguably did not fall within the then-current meaning of the term “open government,” because it did not disclose any previously unavailable material.
THOMAS was not what would now be called an open data project either, because the information it provided was accessible only via a government-supplied interface. The website was designed to serve the needs of citizens—not to open the door for third parties to innovate. By contrast, although they may not have used the term “open data,” several other key government offices have long pursued open data policies, providing key data online in machine-readable formats that (unlike THOMAS) facilitate third-party analysis and reuse. The greatest example may be the U.S. Census, which was providing public information through Census.gov as early as 1996.59The first major project to take advantage of open data for an open government purpose—that is, to make data machine readable and accessible in order to promote government transparency and accountability—was OpenSecrets.org, a website that allows users to search and analyze campaign finance disclosures.60 It launched in 1998 under the auspices of the Center for Responsive Politics, combining government data with third-party innovation. From the beginning, the website aimed to let users adapt the data to their own purposes. On the site’s early home page, its creators explained that they planned on “expanding the interactivity of the site, making it possible for you to ask your own questions—how much did the tobacco industry give in the last election, for example, or where does your congressman rank in dollars from labor unions, defense contractors, or phone companies.”61 True to that promise, the site quickly emerged as a powerful and popular tool for members of the public, researchers, and journalists—a role it still enjoys today.
GovTrack.us, a website launched in 2004 as a side project of then–graduate student Joshua Tauberer,62 was a landmark in the convergence of open government and open data.63 It focused on the same core information as THOMAS: legislative data about Congress. The website included bills, votes, biographical information on members, and reusable digital maps of congressional districts, and it offered new functionality beyond THOMAS’s own for people to search, sort, and monitor legislation of interest to them.
The data in THOMAS was not freely available in bulk at the time of GovTrack’s launch—instead, Tauberer had to painstakingly write computer code to systematically scrape and reassemble the data in THOMAS. But once he had reassembled the data for his own use, Tauberer did not keep it to himself. Instead, he made it freely available, both in bulk and through an application programming interface (API) so that other websites could dynamically access his database and provide up-to-the-minute legislative information themselves, in whatever format or context they judged best. A partial inventory on GovTrack lists at least thirty current and former online projects that rely on GovTrack’s data, including prominent sites like OpenCongress and MAPLight.org.64
Well into the 2000s, however, the concept of “open government” among public officials still centered on fresh disclosures, rather than improved access to already-public data. The Honest Leadership and Open Government Act of 200765 dealt with requirements related to lobbying waiting periods and disclosures, earmark requests, and gifts to Congress. That same year, another law with a similar title, the OPEN Government Act of 2007,66 modified FOIA’s fee structure and established an ombudsman to oversee FOIA’s processes. Neither of these bills approached “open government” in the technologically innovative mode of sites like GovTrack.
B. “Open Government” Becomes a Label for Both Technological Innovation and Political Accountability
In recent years, participants in the policy debate—first in the United States, and then internationally—began to use the term “open government” in a more ambiguous way.
President Obama and his team, both during the campaign and in government, have shown a major commitment to both open government and open data—and they have also been the leading force behind the conceptual merger of the two ideas.
On the campaign trail, then-Senator Obama promised to “restore the American people’s trust in their government by making government more open and transparent,”67 responding in part to his predecessor’s perceived lack of transparency. At the same time, the technology and internet industries based in Silicon Valley served as a key source of financial and logistical support for the campaign, both through their own financial contributions and by helping to build a record-setting, web-based fundraising machine.68 Obama was no stranger to the power of the internet: As a Senator, he sponsored the legislation that established USASpending.gov, an online portal that gave internet users an unprecedented degree of insight into the federal budget.69 His background as a grassroots organizer also helped him appreciate the power of online networking to connect his supporters with the campaign and with each other.
Alongside their specific policy impulse toward transparency, therefore, the candidate and campaign harbored a powerful, if general, sense that internet technologies could open doors for innovation, efficiency, and flexibility in government. In effect, this was a commitment to open data. “From a policy standpoint, there [were] many reasons for tech-minded types to support Obama, including his pledge to establish a chief technology officer for the federal government and to radically increase its transparency by making most government data available online.”70 The campaign itself embraced a data-driven approach to its fundraising appeals, rigorously tested alternative fundraising and outreach messages, and devolved to its supporters a significant degree of autonomy in interacting with their friends to build support.71
The Obama transition team created a high-level working group on technology and innovation, alongside similar working groups on economics, national security, health care, and other major issues.72 The group had an ungainly name but an endearing acronym: the Technology, Innovation & Government Reform Policy Working Group, or TIGR (pronounced like Tigger, the friendly tiger from Winnie the Pooh). The group’s charter was to help prepare the incoming administration to implement its Innovation Agenda, which included a range of proposals to
create a 21st century government that is more open and effective; [that] leverages technology to grow the economy, create jobs, and solve our country’s most pressing problems; [that] respects the integrity of and renews our commitment to science; and [that] catalyzes active citizenship and partnerships in shared governance with civil society institutions.73
This charter was squarely focused on technological innovation rather than on civic accountability.74
Meanwhile, the communities of technological and political openness had continued to merge outside of government. A key meeting took place in the San Francisco Bay Area a year before the transition team’s work.75 The recommendations drawn up by attendees at the meeting speak in merged terms of “open government data”:
This weekend, 30 open government advocates gathered to develop a set of principles of open government data. The meeting . . . was designed to develop a more robust understanding of why open government data is essential to democracy.
. . . .
. . . The group is offering a set of fundamental principles for open government data. By embracing [these] eight principles, governments of the world can become more effective, transparent, and relevant to our lives.
. . . .
Government data shall be considered open if it is made public in a way that complies with the principles below.76
The language here is telling: Participants understood themselves as “open government advocates,” but the principles they produced specify circumstances under which “[g]overnment data shall be considered open” (emphasis added), rather than government itself. The eight principles, which include completeness, timeliness, and freedom from license restrictions, are requirements that attach to disclosures, not to regimes.77 It may be true in some sense that a regime becomes more open whenever it provides additional open data, even for mundane and apolitical topics,78 but it is easy to imagine that a closed regime might disclose large amounts of data conforming to these eight requirements without in any way advancing its actual accountability as a government.79
There was also an emerging scholarly literature on the benefits that government might enjoy from fuller use of the internet, encompassing but reaching well beyond technology-driven enhancements of public accountability. Beth Noveck, who played a leading role in the Obama administration’s open government initiatives, wrote a book in this vein arguing not only for transparency, but also for new modes of “collaborative participation” that leverage citizens’ expertise.80 We ourselves made similar arguments in our paper, Government Data and the Invisible Hand.81 There, we advocated for the release of machine-readable, structured government data to help close “the wide gap between the exciting uses of Internet technology by private parties, on the one hand, and the government’s lagging technical infrastructure, on the other.”82
On President Obama’s first day in office, he issued two memoranda that dealt with “open government,” using the term to refer both to increased transparency and to technological innovation. The first, a memorandum on the Freedom of Information Act,83 was designed to encourage agencies to be more responsive to FOIA requests. It stated that FOIA
encourages accountability through transparency [and] is the most prominent expression of a profound national commitment to ensuring an open Government. . . .
. . . .
All agencies should adopt a presumption in favor of disclosure, in order to renew their commitment to the principles embodied in FOIA, and to usher in a new era of open Government.84
The creators of FOIA, as described above, had political objectives, not technological ones, and this memorandum focuses squarely on those political goals—transparency and accountability.85 The word “innovation” does not appear, and technology earns a mention not as an end itself, but rather as one of the key means of achieving the political objective: “All agencies should use modern technology to inform citizens . . . . [Future Office of Management and Budget (OMB) guidance should] increase and improve information dissemination to the public, including through the use of new technologies.”86
The second memorandum, on Transparency and Open Government,87 took a much broader view. Whereas the FOIA memorandum suggested that a “new era of open Government” could be achieved through the transparency that FOIA compliance entails,88 the Open Government memorandum treated transparency as just one among a trio of goals, setting out in separate paragraphs that an open government is transparent, participatory, and collaborative.89 Transparency was just one of the features of open government, and public trust was just one of the benefits: “We will work together to ensure the public trust and establish a system of transparency, public participation, and collaboration. Openness will strengthen our democracy and promote efficiency and effectiveness in Government.”90
The new administration thus began to move toward a broader conception of open government than had existed before—one that drew on the technological and philosophical commitments to innovation that the word already carried in technical circles. The president’s memoranda set the stage for the Open Government Directive and the Initiative that were to follow. Being accountable was just one part of what made a government “open”—participatory or collaborative measures that enhanced efficiency or effectiveness might equally claim to be making the government more “open.”
The central practical mandate of the Open Government Directive,91 issued eleven months later, was an open data requirement, not a political transparency requirement: The directive required agencies to “publish online in an open format at least three high-value datasets” via the new federal data portal at Data.gov.92 High value, in turn, did not necessarily mean politically sensitive: Aside from making the agency more transparent or accountable, data might also be high value if it would “improve public knowledge of the agency and its operations”93 or “create economic opportunity.”94
Predictably, agencies responding to this mandate have tended to release data that helps them serve their existing goals without throwing open the doors for uncomfortable increases in public scrutiny. In many cases, agencies published datasets on Data.gov that were already available in other online locations.95 While agencies packaged some of these datasets into more usable machine-readable formats, critics questioned how these disclosures added to the public’s “insight into agency management, deliberations, or results.”96 Critics saw the repackaging of old information as providing only “marginal value” and urged the government to make available “public data that holds an agency accountable for its policy and spending decisions.”97 A broader study of Data.gov in 2011 noted a significant downward trend in agency dataset publication over the site’s first year.98 It concluded that most federal agencies “appear[ed] to cooperate with the program while in fact effectively ignoring it,” and that Data.gov had become “the playground for a tiny group of agencies.”99
Even as the administration’s political momentum for its Open Government Initiative waned, local and state governments began to adapt these ideas for their own purposes. From New York to San Francisco, city and state leaders launched new websites devoted to sharing public data, often describing them as “open data” projects.100 But the rhetoric among localities was more focused on service delivery than on accountability. City leaders in particular put an emphasis on improving communities through better services. San Francisco mayor Gavin Newsom expressed his hope that DataSF.org would “stimulate local industry, create jobs and highlight San Francisco’s creative culture and attractiveness as a place to live and work,” and only briefly acknowledged the possibility for greater accountability.101
Meanwhile, similar ideas have gained momentum internationally, reflecting other nations’ growing recognition of the new technological realities. The European Union’s 2003 Directive on the Re-use of Public Sector Information instructed that “[w]here possible, documents shall be made available through electronic means,”102 and the EU now operates a website and program to encourage member states to develop their own national data portals.103 Independent efforts were underway in the United Kingdom by 2007,104 leading to the creation in 2008 of a “Power of Information Task Force” to explore the benefits of adaptable government data.105 Data.gov.uk, launched in October 2009, appears to have been the first site of its kind outside the United States.106
A new multilateral initiative, instigated by the United States, has dramatically accelerated the spread of these ideas over the past year. In October 2010, President Obama addressed the United Nations General Assembly and urged member states: