
Congress votes to make open government data the default in the United States - danso
https://e-pluribusunum.org/2018/12/21/congress-made-open-government-data-the-default-in-the-united-states/
======
compute_me
Making public data open by default can arguable be an imporant step towards
fostering societal equity. However, it needs to be not only "open", which
typically means stashed away in some corner as a spreadsheet or database file,
but accessible and useful to people. The UK has been pushing open data for
years now and more and more institutions are now realizing this. Shameless
plug for a research project that is aiming to make open data more accessible
and to democratize data-science: [https://data-in.place/](https://data-
in.place/) ...

~~~
newleaf
In case you haven't seen it, I think
[https://www.data.gov/](https://www.data.gov/) is an attempt to answer your
point about making it "accessible and useful to people." There's room for
improvement, but it's a start.

~~~
org3432
I thought there would be a flood of projects analysing the data when it came
out, but it seems like the idea everyone applauded but not much came out of
it. Steve Ballmer's [http://usafacts.org](http://usafacts.org) seems like the
first real attempt though.

~~~
pfd1986
Is there a HN thread discussing these reports? I'd love to hear folks'
opinions about them... Interesting facts to me in the latest report [1]:

\- Page 44: "Our economy has grown at a steady rate despite changes in
economic policy". I expected to see a lot more fluctuation on this data.

\- Page 30: "There have been more suicide gun deaths than homicide gun deaths
every year since 1981". This is crazy to me given how much we hear about gun
homicide being a problem in this country.

\- Page 29: Crime rate has declined but "The number of incarcerated persons
has increased by 330% since 1980".

[1]
[https://static.usafacts.org/public/resources/USAFactsReport2...](https://static.usafacts.org/public/resources/USAFactsReport2018.pdf)

~~~
hueving
>Page 44: "Our economy has grown at a steady rate despite changes in economic
policy". I expected to see a lot more fluctuation on this data.

Barring some calamity, this is pretty much as expected. Tariffs impact a very
small percentage of the economy with large size, and the rest of it with a
small overhead, much like a fed interest rate hike. Even with this hawkish
fed, there hasn't been anything overly harmful to the economy from a policy
perspective.

> This is crazy to me given how much we hear about gun homicide being a
> problem in this country.

Anything that's politicized gets this special treatment. Kid kills brother
with car, blurb in local newspaper. Kid kills brother with gun, national news
and Tweets from Presidential candidates.

Texting and driving is as bad as drinking and driving when it comes to number
of deaths (dwarfing gun deaths as well), yet people do it like it's no big
deal. Most cities still only levy small fines (in comparisons to DUIs) for
doing it. Not all accidental death is created equal.

>Page 29: Crime rate has declined but "The number of incarcerated persons has
increased by 330% since 1980"

Welcome to the US where the war on drugs gives us authoritarian level
incarceration rates.

~~~
SllX
Every election cycle I look for the candidate with the balls to say the War on
Drugs is over, let’s wind this crap down, change our laws to reflect this
fact, change some sentences ex post facto to reflect this and get on with our
lives.

Every election cycle, I continue to be disppointed. Even if they were crazy in
every other regard, I would probably still vote for them. It’s like, Step 1
towards doing anything meaningful in regards to poverty, education, criminal
justice reform, et cetera.

------
dandare
All I want for this Christmass is an authoritative list of all US federal
agencies. I kid you not, there is no such list and the number of federal
agencies is uncertain. From Wikipedia:

>Legislative definitions of a federal agency are varied, and even
contradictory, and the official United States Government Manual offers no
definition. While the Administrative Procedure Act definition of "agency"
applies to most executive branch agencies, Congress may define an agency
however it chooses in enabling legislation, and subsequent litigation, often
involving the Freedom of Information Act and the Government in the Sunshine
Act. These further cloud attempts to enumerate a list of agencies.

~~~
dandare
Shameless plug: does anybody know where I could get detailed spending and
receipts data for US federal budget? Including all federal agencies? There is
very limited data at [https://www.whitehouse.gov/omb/supplemental-
materials/](https://www.whitehouse.gov/omb/supplemental-materials/) but I am
looking for more detailed data.

~~~
philipashlock
The main place to explore spending data is
[https://www.usaspending.gov/#/download_center/](https://www.usaspending.gov/#/download_center/)
and [https://datalab.usaspending.gov](https://datalab.usaspending.gov)

Let me know what you can't find or request via [https://www.data.gov/data-
request/](https://www.data.gov/data-request/)

------
dgellow
As a next step I would love to see an exploration of a legal system where a
change of a law or regulation is backed by both data (data and methodology
directly referenced by the law) and a description of the expected impact.
Something such as:

> By changing law A related to B, we expect the increase of C to be at least D
> in the next E months.

If not achieved, the change is reversed/reduced. Hopefully that would allow
experimentation without taking the risk of creating a system too bad in case
the implementation or policy isn’t good enough.

Just dreaming here :)

~~~
justin_oaks
I've dreamed the same thing.

Since every law is a trade-off, having both positive and negative effects, I
wish each law would enumerate the expected/possible positive and negative
effects. In other words, I want the trade-off to be explicit and for the
lawmakers to express why they believe the positive effects outweigh the
negatives ones.

------
havermeyer
I'm biased (I work on the BigQuery team) but I'm always excited to see more
public datasets made available in BigQuery:
[https://cloud.google.com/bigquery/public-
data/](https://cloud.google.com/bigquery/public-data/). It would be great to
have government data available through a variety of cloud services with free
exports.

Some personal favorites among BigQuery public datasets include NOAA GHCN[0],
the Census Bureau's Zip Code Tabulation Area [1], and FEC Campaign Finance
[2].

[0] [https://console.cloud.google.com/marketplace/details/noaa-
pu...](https://console.cloud.google.com/marketplace/details/noaa-
public/ghcn-d?filter=solution-
type:dataset&filter=category:climate&id=9d500d1d-fda4-4413-a789-d8786fd6592e)
[1]
[https://console.cloud.google.com/marketplace/details/bigquer...](https://console.cloud.google.com/marketplace/details/bigquery-
public-data/zipcode_area?filter=solution-
type:dataset&filter=category:maps&id=b0de5205-0a3f-4f1f-8cd4-47cfa001de63) [2]
[https://console.cloud.google.com/marketplace/details/bigquer...](https://console.cloud.google.com/marketplace/details/bigquery-
public-data/fec-campaign-finance?filter=solution-
type:dataset&filter=category:science-
research&id=af2dc7e0-f9bf-4a10-9047-fbc5e43eecb7)

------
todipa
Open data is the first step.

I wish I could do more business with my gov't (State and Local) through the
internet.

"Copies of documents cannot be ordered through this website, by email or over
the telephone." Only fax and snail mail...
[https://www.dos.ny.gov/corps/faq_copies.page.asp](https://www.dos.ny.gov/corps/faq_copies.page.asp)

~~~
weiming
Is this because they are required to make this info accessible, but are not
required to make it easy?

~~~
jaggederest
More importantly, there's no specific funding set aside to support the effort
to make it available. Government at all levels lives and dies by funding.

------
pratheekrebala
FOIA officers will still find a way to send me scanned PDFs of spreadsheets.

~~~
wilde
I’ll defend this practice. It’s the only way of knowing for sure that you’re
transmitting exactly the information you intend to send. Even copy/paste often
picks up other stuff you don’t intend.

~~~
prepend
It the only, but maybe the easiest.

Having a data review process with automated integrity, confidentiality, and
quality checks is not terribly difficult.

But having a prototocol to export the pdf to csv is also dead easy for
confirming only the data relevant is included. ASCII is just as “easy” as
scan, but it requires training clerks to be data-oriented rather than
document.

~~~
paulryanrogers
_ugh_ if only CSVs were standardized sooner and more completely. There are
many encoding, delimiter, escaping and truncation conventions to deal with in
real world data.

~~~
prepend
Definitely. They are better than PDFs, but still have lots of room for
improvement.

------
peterwwillis
Too bad thing related to "security" won't be included.

All I want for Christmas: accountability for the DoD budget, which for 2019
will be $717 Billion, the majority of the USG's discretionary spending.

In 2015, an audit of the budget revealed $125 Billion in wasteful spending,
and this was covered up. In 2016, the Office of Inspector General for DoD said
that the Army made 6.5 Trillion in wrongful adjustments to its 2015
accounting.
[https://en.wikipedia.org/wiki/United_States_Department_of_De...](https://en.wikipedia.org/wiki/United_States_Department_of_Defense#Criticism)

We still do not know how many contractors the DOD employs, or how much money
they are paid, because the numbers are just not recorded. We do know, though,
that the numbers and budgets we do have are often inaccurately reported,
according to the DOD OIG.

People whine a lot about paying taxes, but the politicians that always
complain about taxes are extracting record amounts of tax money for a military
that is mismanaged, doesn't do its accounting properly, can't build modern
fighting vehicles, and doesn't record basic information like _how many people
they employ_.

------
systematical
I've been working with the GPOs api. The engineer is quite responsive on
github and the api is pretty snappy. Constantly asking for feedback and
releasing new features. I think we're headed in the right direction. A shame
my project isn't further a long for a shameless plug.

If you're interested in informing the public on legislation, have experience
on the hill or UI/UX experience hit me up. I'm just some dude with an idea who
lives in a terminal. Money is secondary.

------
jcriddle4
Interesting that the Federal Reserve was exempted from this legislation.

~~~
lykr0n
The Federal Reserve [board] bases its data on data from the Reserve Banks
(think Federal Reserve of New York)- which are private and get data from banks
chartered in the region (so the New York Fed is partially owned by JP Morgan).
Requiring the Federal Reserve [board] to release underlying data could get
sticky and be a legal minefield.

~~~
zrail
Also the St Louis Fed provides provides FRED, which is pretty open for most
purposes.

[https://fred.stlouisfed.org/](https://fred.stlouisfed.org/)

------
crankylinuxuser
The first problem here is the data formats state and federal governments use.
You'll see a hodgepodge, but primarily MS Office $version .

The biggest problem with this terrible binary format is that metadata can leak
a great deal that should have not been released. So this leads to PDF output
of word/excel.

The next area is that especially local government offices have no way of
setting up a data portal. I'm working on this right now, where the only way to
get data out of Bloomington,IN is to do FOIA requests every week/month over
the data you want. This absolutely should be available via a portal, and not
locked behind "in person, mail, fax at cost of .10$ a page".

------
torgian
Well that’s cool, but the wording worries me:

public information _should_ be open by default to the public in a machine-
readable format, where such publication doesn’t harm privacy or security
federal agencies _should_ use evidence when they make public policy

The word “should” is used in both one and two. If my time in the government
taught me anything, it’s that “should” is only slightly stronger than “may”.
If an instruction says “shall”, then it is required to be done.

------
xivzgrev
Finally a bipartisan moment, looks like most reps were onboard.

~~~
friedman23
They have to be for the law to pass?

~~~
srtjstjsj
Not just most reps were onboard; nearly all reps voted in favor

------
erentz
"The Open, Public, Electronic, and Necessary Government Data Act (AKA the OPEN
Government Data Act)"

Please, could we end this obsession with backronyms in congress. Perhaps some
congressperson could create a suitably titled backronym act to rid us of
backronym acts. This should just be the one singular "Government Data Act" and
it should be amended as needed to cover any law changes around "Government
Data".

~~~
amyboyd
BACKRONYM Act: Banning Acronym Cleverness Keeping Representatives Occupied
Nutting Your Mom Act

(struggled with the NYM so gave up and went with something childish...)

~~~
missblit
Naming Your Measures / Mandates.

------
rpedela
Does anyone know if this applies to PACER and its fees?

~~~
digiphile
It does not. Only CFO Act agencies in the executive branch. But there is a
bill to open up PACER.

------
cantthinkofone
It's good to see the US govt making an effort to step up its technical level.
A vast if hidden problem in the political sphere is that most politicians do
not have a technical education. This creates a serious misconfiguration of the
govt alongside other centers of soft power like large cap tech companies.

There is no way around it that these companies have to work with the
government to secure public interests from 21st century threats. Just today I
read an article about how black hat hackers are targeting outdated industrial
control systems more vigorously than ever before. The government on its own
without technical upgrades cannot face down this problem in its current
condition. Which is why opening up data is a beneficial thing.

Openness of data is a double edged sword. It will make malicious agents' job
easier to have as much data as possible in a consistently machine readable
format, but it will also help those on the other side.

If tech is one of the things that can bolster and improve government, tech
needs to work in the optimal environment. Which is one with open data.

------
tehlike
With all the problems its government has, united states is still at it
fostering innovation.

Open data will bring innovation and accountability.

Great move

~~~
perfmode
> federal agencies should use evidence when they make public policy

the cynic in me just figures that this moves the goal post such that special
interest groups will adapt to produce the right evidence for the desired
outcomes

~~~
CptFribble
You're probably right, but at least in that case there will still be a paper-
trail of "data" that motivated entities can point to and make a case against.

I would rather have bogus evidence in the official record, that I can analyze
and challenge, than no evidence at all.

------
cptskippy
> there was a carve out “for data that does not concern monetary policy,”
> which relates to the Federal Reserve, among others.

Does that mean they won't open data that affects monetary policy or that they
will only open data that affects monetary policy?

Either way that seems huge.

------
mizzao
Can someone who is more familiar with the actual policy shed some light on
this?

~~~
danso
FWIW, the author, Alex Howard (who I'm friends-via-Twitter with), is as
familiar with this as anyone. He was previously a senior analyst at the
Sunlight Foundation, which is a prominent open-government organization:
[https://sunlightfoundation.com/author/ahoward/](https://sunlightfoundation.com/author/ahoward/)

The submitted post includes a link [0] to an article he wrote a couple days
back, which provides more context on "How did open government data get into
the US Code?", including the nitty-gritty of how the original bill was
proposed in the last session but ultimately left out of legislation. Howard
writes that the legislation was "one of the primary legislative priorities for
me during my years as a senior analyst and then deputy director at the
Sunlight Foundation"

[0] [https://e-pluribusunum.org/2018/12/20/senate-passes-
evidence...](https://e-pluribusunum.org/2018/12/20/senate-passes-evidence-
based-policymaking-bill-setting-up-historic-win-for-open-government-data/)

~~~
cptskippy
> (who I'm friends-via-Twitter with)

Full disclosure? Odd brag?

~~~
danso
It's someone who I mostly learned/knew through social media and am friendly
with, even though we've never worked together and maybe have met in person a
couple times at conferences.

~~~
cptskippy
I wasn't sure if you were disclosing the relationship because of the nature of
your comment or if you were just bragging about knowing them. It is/was odd.

~~~
danso
My opinion is that he’s an expert on the legislation, but I wanted to be clear
that I could be biased :)

~~~
cptskippy
Haha ok, as an outsider reading that it came off as a brag and I was confused.
That's why I asked.

------
tabtab
One has to be careful here. Lots of internal details may give trolls and
conspiracy theorists fodder to generate controversy and fake news, often by
taking things out of context. It may slow work-flow because workers will be
hesitant to write anything without expensive pre-vetting.

------
pteredactyl
Tangent, but related shameless plug. Especially to San Franciscans:
[https://theconstituent.net](https://theconstituent.net).

It makes legislation easier to access. And eventually easier to engage with.

------
ghaff
For those who don't know, as the article mentions in passing, there's already
quite a bit of data online available from data.gov (which was started by the
Obama administration).

------
pavon
Has anyone read any good analysis about Title I of HR4174 requiring evidence
when making public policy? Skimming the bill, I don't have a clue what the
implications are other than requiring agencies to make some reports. It sounds
like a good thing on the surface, but it has been a talking point of the Trump
administration, and put into practice in a rather Orwellian fashion.

Basically, the implied subtext is that regulators should not put any
restrictions on industry unless the evidence is completely unambiguously in
favor of the regulation. However, science is never 100% certain even at its
best. And you can always drag up some study that contradicts the strong
consensus of the field, whether by fluke or intentional design of the study.
In other words "evidence based policymaking" has been euphemism for "must give
alternative facts equal weight".

------
75dvtwin
Does anybody know a public source of

net worth + annual earnings+ board positions

of US Senator/congressman spouses+children ?

------
qwerty456127
Cool! And they should also adopt UK-like government web design principles.

~~~
philipashlock
See [https://designsystem.digital.gov](https://designsystem.digital.gov)

------
yostrovs
I'm glad they were able to do it before the new Congress came on board.

~~~
srtjstjsj
What has that to do with anything?

The bill had large majority bipartisan support (which makes me suspect it's
toothless...)

~~~
yostrovs
There's a reason it was passed during the lame duck session of the Senate,
just before change of control. That's when urgent matters that are guaranteed
to be ignored by the next Congress are handled.

I'm surprised this needs explaining.

------
burtonator
We should go the path of Sweden and make all IRS records public too...

------
HillaryBriss
when I first saw this story, I thought it was a joke ala The Onion. so I
looked for it on NYTimes, WaPo, CNN but didn't find it. i still can't find it
there.

~~~
danso
Open data is a relatively niche topic/policy in politics. It wouldn’t be front
page news on a normal day, and so it’s not a surprise if it doesn’t make the
weekend headlines, on a week in which the federal govt went into shutdown over
the border wall, nevermind the resignation of the defense secretary.

------
rubyfan
What type of policy implications does this actually have?

------
miguelrochefort
Hopefully they release it in RDF.

------
yohann305
I wonder if this open data can be (will be?) used by a foreign entity and be
turned against the american people? It should be open but not free-for-all

~~~
astazangasta
I'm not convinced there are really any important national security secrets
other than passcodes and so on. The rest seems to be excuses for hiding
corruption of various kinds. Secrecy in government is incompatible with
democracy by definition.

~~~
rjf72
You have to think about how people respond to things. Social media in its
current state has basically turned into a system that takes an event, removes
all context, spins it in the most negative way imaginable, and then dogpiles
on it to no end -- virtue signaling for imaginary points. There are often lots
of things in government, and in life in general, where you must choose between
two very negative choices. And the decision there does not necessarily imply
that you're in any way satisfied with it, but you see it as the decision least
likely to produce an awful outcome. Social media and this sort of logic are
wholly and completely incompatible.

Take Khashoggi as a contemporary example. Undoubtedly there have been
countless heavily classified conversations weighting the pros and cons of any
action against Saudi Arabia or MBS as a result of this. How we are _extremely_
dependent on Saudi Arabia for reasons outside the scope of this post, and they
know this. And this is all happening at a time when Saudi:US relations souring
would greatly stand to strengthen the geopolitical position and power of
nations such as China and Russia. In my opinion it's extremely likely we will
do nothing, but that's because it's better than doing something. If this was
not classified, social media would throw a nonstop hissy fit. Not because they
actually care or think we're making the wrong decision, but because it's an
incredibly easy way score those imaginary points and followers which are the
hottest commodity since sliced bread. Another issue here is that knowing the
mechanisms of our decision making here, and how close we came to 'breaking',
would be incredibly valuable information to nations such as Saudi Arabia which
they could then use to exploit the US.

So while I do agree with you that classification is very often abused, I also
think there are indeed vast amounts of information that must be classified.
People are not mature enough to impartially process our decision making
processes, and the details of such processes would provide invaluable
information to other nations which could then be used to exploit the US.

~~~
astazangasta
Khashoggi is obviously a canard; the Saudi Arabian government is currently
running torture chambers in Yemen where they roast people alive[1]. The US has
had no problem with this for years; we ourselves appointed a known torturer to
head the CIA, the organization which is supposedly agog about Khashoggi's
assassination. The CIA itself has been running an extrajudicial assassination
program for over a decade now. Talk about "virtue signaling".

These people, the ones who are part of the security apparatus, are not in any
position to make moral choices for the rest of us. They are highly immoral
actors. I want to see them removed from government. The idea that they are
capable of making good decisions in secret, without public input or oversight,
is belied by the two decades of war, torture, assassination, and general chaos
that they have actively fostered.

The problem, really, is that all of the secret policy conversations about
Saudi Arabia have been about how to get them to continue buying weapons from
American arms merchants, a policy greatly to the detriment of the American
public, which would, in general, favor a policy more like: disengage from the
Middle East, stop supporting torture, and transition away from fossil fuels.
If we had spent $6 trillion on that instead of funding pointless wars and
building up the Saudi torture state, we'd be much better off.

Let's stop having unaccountable, immoral people decide in secret what is good
for America, and get back to letting the American public decide. We're better
at it.

[1] [https://www.news.com.au/world/middle-east/inside-yemens-
secr...](https://www.news.com.au/world/middle-east/inside-yemens-secret-
prisons-you-can-hear-the-screams/news-story/6f810a5df25919c42bcec8905294f6aa)

~~~
burfog
About this: _" buying weapons from American arms merchants, a policy greatly
to the detriment of the American public"_

American arms merchants employ the American public. People get jobs. Taxes are
paid. Suppliers and subcontractors benefit too.

A detriment would be if Saudi Arabia bought from China or Russia, enriching
people in those nations instead of the American public. We'd also have far
less influence over Saudi Arabia... really, they could be a lot worse.

~~~
astazangasta
In the narrow sense I think you are correct; in the long term deeply
incorrect. There are many ways for the American public to be employed. A
society built on arms manufacturing seems the least desirable to me. This is
the detriment I speak of. Much of our effort and economy is wasted making guns
and bombs rather than more useful and less destructive things. This industry
sucks up a trillion dollars in government support annually; if it were reduced
and that effort (spending) directed elsewhere we would all benefit. We might,
e.g., direct it to developing fossil fuel independence and obviate the need
for anyone to sell arms to Saudis.

