
UK judge says ‘freedom of information’ means choice of digital file format - Libertatea
http://www.washingtonpost.com/blogs/the-switch/wp/2014/08/05/uk-judge-says-freedom-of-information-means-choice-of-digital-file-format/
======
dublinben
I can't think of anything more obtuse than releasing a database as a hundred
plus page PDF. It seems like the bureaucrats responding to these requests are
specifically trained to _not_ release information.

~~~
anigbrowl
They want to minimize the amount of time they spend on the FOIA requests,
because it comes out of their operating budget. So they select the releasable
records and then print them to pdf, where they can do any necessary redaction
before release.

Exporting them into another _database_ where they can be manipulated freely is
beyond the skills of most database users, and requires an expensive database
administrator, who will have to have meetings with the administrator of the
original database, which might result in travel requests, expenses...at which
point most organizations will invoke the clause in FOIA rules (at least in
America) of charging the requestor the reasonable costs of the data retrieval,
which in a case like this will be substantial - at least a few thousand. Bear
in mind that that FOIA requests are rarely so broad in scope as to justify the
release of an entire database; most requests are for a subset of information
about particular individuals, timeframes, political administrations or
suchlike.

I often dig through stuff like this in PDF format (because I enjoy reading
court cases), and while I can't say I like it, if I really need to get the
data out I set up some sort of scraping macro. Most people want something in
human readable form and I suspect that if they were given an optical disc with
an SQL dtabase on it, or even an XML file, they'd complain about the
impenetrability of it.

Perhaps you could suggest some good candidates for release formats that met
the criteria of being easily accessible/convertible as well as being
manipulable and easy to produce.

~~~
jon-wood
> Perhaps you could suggest some good candidates for release formats that met
> the criteria of being easily accessible/convertible as well as being
> manipulable and easy to produce.

The entire case here was around the requestor specifying he'd like the data in
Excel format (although presumably CSV would also be acceptable). I can't think
of many databases which can't export query results to CSV, or indeed many
civil servants who are so completely inept with a computer that they couldn't
redact information as (or more) easily in Excel than they could with a PDF.

~~~
incision
_> 'I can't think of many databases which can't export query results to
CSV...'_

This is pretty much irrelevant as these requests are not going to end up in
the hands of a DBA, or anyone with anything approaching direct access to the
databases in question.

It's going to end up with someone who has an adversarial relationship with
their own IT department as a 'maker of work'. Bureaucracy isn't something
these agencies foist on the public, it just as bad if not worse inside.

The request will go up and down through a series of managers, directors and
deputies before being put on the schedule for a change control board meeting.

In that meeting, a representative of the database administrators will explain
that this request is not only complex, but potentially quite dangerous. He
will point out that required data is actually completely accessible via a
clunky mainframe interface wrapped in Java.

Some plebe in the receiving office of the request will end up clicking through
that Java interface and printing/exporting the requested data _one page at a
time_.

This process will repeat itself until such requests are a weekly if not daily
occurrence at which point the plebe might be given a dedicated interface for
exporting such requests.

 _> '...or indeed many civil servants who are so completely inept with a
computer that they couldn't redact information as (or more) easily in Excel
than they could with a PDF.'_

I take it you've not met many civil servants then?

In any case, the mechanics aren't really the issue. It's the game of hot
potato behind the scenes and the eschewing of responsibility that takes place
at all levels.

I can virtually guarantee that even if all the stars aligned such that someone
did obtain and redact a fully electronic copy of such data it would be
rejected for release due to the 'unfamiliar' format. Someone in the chain
would fear that they couldn't be sure of what they were releasing because it's
not in the specific format they're used to.

~~~
eli
I know many civil servants. The ones with access to run queries on a database
know how to export to Excel. If anything, there's an over reliance on Excel.

------
binarymax
I'm sure there are political reasons, but it is very hard for me to understand
_why_ agencies purposefully keep the data obscure.

Opening data stimulates innovation around that data. It benefits society by
allowing analysis to provide value, and improves the economy through the sale
of related products.

Every time I see things like this, I cannot help but hope that if not my
children's generation, at least my grandchildren's will be one that takes for
granted accessible free information from public agencies. So many good things
would come from old world beaurocrats relinquishing their stranglehold on data
that is payed for by the people who need it the most.

~~~
incision
_> 'I'm sure there are political reasons, but it is very hard for me to
understand why agencies purposefully keep the data obscure.'_

I can understand / attest to both of the following stances by experience:

Thinking idealistically, it's a basic deterrent against lazy, disingenuous,
'data out of context' arguments. By keeping that data obscure you're building
in a selection process. Requiring potential detractors to put forth even the
small effort of cutting, pasting and sorting will deter many before they
begin.

Thinking cynically, it's a basic deterrent against uncovering widespread lies
and corruption. Keeping the data obscure will encourage the people who might
uncover your schemes to look for easier, juicier targets.

 _> 'Opening data stimulates innovation around that data.'_

It's an enabler sure, but I'm not so sure about a stimulator.

As best I can tell, most FOIA requests primarily serve the lawyers hired by
various agencies to sue each other or the Government in the process of
contract grievances.

 _> 'It benefits society by allowing analysis to provide value, and improves
the economy through the sale of related products.'_

It certainly could, but it's easy to forget that many people working in
Government are just as self-interested as anyone else. They're no more
interested in the big picture than the next man. More importantly, no one
below the level of perhaps a local Mayor and his/her lieutenants have their
jobs tied to economic performance or general well-being of their constituents.

------
incision
My general impression, having seen the 'inside' of such processes, is that
everyone would be better served by mandates around open data than continued
interpretations and amendments around FOIA.

1.) The most common sort of requests (salary schedules, the outcomes of past
competitive bid processes) are for information that could/should simply be
public in the first place.

All this stuff can simply be published as part of a normal workflow rather
than manually compiled on a per request basis.

2.) Compliance could be checked trivially via API rather than having to wait
for failures to bubble up over time and potentially raise scandal.

3.) Proper funding could be allocated and structure put in place to build and
maintain these services. As it is, there's a good chance any given request
lands on the desk of some plebe who lacks the knowledge, tools and authority
to handle the request correctly or efficiently.

------
allegory
Well that will just result in someone manually cutting and pasting stuff into
excel and exporting a CSV. We are doomed to live the following by specifying
it:

[http://wyorock.com/excelasadatabase.htm](http://wyorock.com/excelasadatabase.htm)

This is due to government assignment policy which is delegate it until there
is no one left to delegate it further resulting in the lowest paid and skilled
workers performing the above.

~~~
anigbrowl
I don't think that's peculiar to government - sounds like any other large
organization with a hierarchical structure.

~~~
allegory
I've worked in various capacities in government and non-government
organisations. It is much worse in government than not from experience because
they are cesspools for the incompetent at best. In a private sector company
people are considerably more aware of their standing and not heavily unionised
into artificially secure positions.

------
_kst_
I can see this backfiring if someone requests data in some ancient and obscure
format just to be annoying.

Data needs to be provided in a reasonable and usable format; that imposes some
responsibility on both the provider and the requester.

~~~
BrokenEnso
Would there be anything to prevent someone from creating a one off file format
that requires a license fee and just start making FOIA requests in that
format. If not, this seems like something that could be exploited for profit.

~~~
knodi123
Don't FOIA requests usually require you to pay reasonable fees? I know they're
not free....

------
7952
Recently I was trying to get hold of local plan housing allocation which
detail where new building is likely to be. The local authority would only give
me PDFs of maps with hundreds of insets which obfuscate everything. With a
proper digital format national mapping would be possible. The public could
easily find out where new developments are going, something that is very hard
to do now.

