Data in Whitehall: which UK departments are the least and most open
After all, we need information - whether it be on projects, salaries, decisions or spending - to hold Whitehall to account and track how public money is spent on our behalf. And a growing number of startups that hope to build businesses on the back of open data rely on it for their livelihood.
"Open data increases the transparency and accountability of the Government. It also benefits individuals and businesses by allowing them to analyse huge tranches of information to uncover useful trends," says Mike Weston, CEO of data science consultancy Profusion.
So it's heartening to hear how much of it is in the public domain now, at least compared to five years ago when the UK government first officially started promoting open data: the idea public data should be freely available to all.
All Whitehall departments produced open data strategies in June 2012. The amounts and types of data the different organisations committed to release varied widely, but a few have led the way.
"The government has been strong in opening up data in areas such as publicly funded research and environmental measurements," according to data analytics company Tessella's analytics lead, Nick Clarke.
Campaigners and entrepreneurs say the defence, environment, energy, transport and health departments are pioneers within Whitehall.
"I have nothing but pleasant things to say about Department of Energy and Climate Change (DECC) and the Department for Environment, Food and Rural Affairs (DEFRA)," says Ian Makgill, who set up open data startup Spend Network.
"In central government it can be very varied. For example the kind of people you'd expect to be least helpful like the Ministry of Defence and in fact incredibly efficient and get it. Considering the size of the department they provide information remarkably quickly," he adds.
"The transport and health [departments] have very good open data programmes," says open data campaigner Owen Boswarva.
"It's pertinent that they started early and developed their own policies rather than being led by Cabinet Office. Other than ONS, those departments have the most embedded open data cultures in the public sector," he adds.
Just last week Defra promised to publish 8,000 datasets in the next year, including its high-resolution maps covering 72 percent of England.
Soon-to-depart government digital director Mike Bracken welcomed the move as a sign Defra understands 'its future depends on being a more data-driven organisation'
"This is more than simply opening up some spreadsheets. It's about deep-rooted culture change so that it's open by default," he said in a blog post.
"They're treating data as a public asset, which is exactly the right thing to do. If other departments are wondering what to do with their own piles of data, I have four simple words of advice: get it out there," Bracken added.
Makgill suggests the situation would be vastly improved if departments 'took their own medicine'.
"The problem with open data on spending is it's not actually used by government, it's just put out there. If these departments were relying on that data week in week out, they would publish it every month without fail, and it would be usable," he says.
Despite this handful of best practitioners, the open data picture across Whitehall is not entirely rosy.
"We regularly have to submit Freedom of Information requests to get stuff that should be published, like spending or contract data, which is supposed to be open," says Makgill.
"Most departments are middling: there's some good and some bad practice. The general rule is departments see the virtue in open data when they can leverage it to support another policy or initiative, but lack a systematic approach," Boswarva says.
Another issue is the format of the data that gets released: it is not always the usable, clean, machine-readable data experts say you need to link datasets and analyse them.
"The raw data could just about be understood by a layman but to combine different datasets you require a high level of expertise," says Weston.
"For the data to be truly accessible and for the Government to be fully accountable this information needs to be presented in such a way that it is easier to handle and understand," he says.
"[Limitations on open data] include some data not being published, some not being recorded, some being suppressed or redacted, and some being published without useful categorisations," a report by Whitehall thinktank Institute for Government warned last year.
"This makes it difficult to analyse definitively the extent and effectiveness of government contracting," it concluded.
Several individuals told ComputerworldUK good performance on open data is - perhaps unsurprisingly - often linked to the enthusiasm of one or two senior people or teams within the department.
"If you could clone a few key teams around the civil service you could make a real difference with relatively little investment," Open Data Institute cofounder Sir Nigel Shadbolt says.
However "I rarely see much evidence of a coherent cross-government open data policy," Boswarva adds.
He says the Department for Communities and Local Government and the Ministry of Justice both have "pretty awful" records when it comes to data release.
DCLG has cut the amount of data it collects from local government as is refusing to release a number of important datasets, such as Energy Performance Certificates, he says.
The MOJ has led Whitehall efforts to water down open data measures within European Union directives and has "resisted attempts to open up case law", Boswarva adds.
Makgill says he has had trouble dragging data out of the MOJ - it took an Information Commissioner's Office judgement to compel them to release spending data, a process that took 13 months.
The Department for Business, Innovation and Skills' position where it has to defend the commercial interests -and thus data - of trading funds like the Ordnance Survey, Land Registry and Met Office make it "the most harmful countervailing force to the development of an open national information infrastructure", Boswarva claims.
For example, the department came under criticism from MPs in March last year for including the Postcode Address File - a database of all UK addresses - in its sale of the Royal Mail, meaning it would no longer be available to all.
It is the Cabinet Office, ironically the department responsible for open data policy, that is cited by all respondents as having the worst record in central government when it comes to publishing its own data.
Despite being in charge of cajoling the public sector into publishing more data and becoming more transparent, the Cabinet Office did not publish its spending data for a whole year from August 2014. This data is supposed to be released every month.
"The Cabinet Office itself doesn't hold much data. But as the lead department for open data policy, its failure to publish recent spending data and its notorious hostility to FOI have undermined its credibility," Boswarva says.
"The Cabinet Office's guiding principle seems to be 'Do as I say, not as I do'," one civil servant who wished to remain anonymous told ComputerworldUK.
"In the Cabinet Office, there is a whole team dedicated to transparency, yet they can't seem to even persuade their finance teams to regularly publish spending data," Makgill says.
"It's the only department I've seen where there is such a clear demarcation between what the policy people are trying to achieve and the service is actually delivering," he adds.
Anecdotal evidence from civil servants and startups suggests the Cabinet Office's laggard approach to releasing its own data and resistance to FOI is being used as an excuse by other departments who take a more lax approach to the issue.
"We've heard from people who are having problems getting their department to publish their data as their leaders are saying 'well, the Cabinet Office aren't doing it'. It corrodes from the head down," Makgill says.
Its easy to see open data as a niche issue. But without good quality data, we are all in the dark about the work government does: its impact, its value and its importance.
"The use of purchasing data in government has historically improved services. If you smother data, you smother innovation," he adds.
That means the public can't scrutinise the government properly, civil servants make decisions based on partial data and politicians cannot fully understand the consequences of their policies. With significant cuts expected to be made across public services this autumn, that is quite a scary thought.