
NSA OSS Technologies - andrewke
https://nationalsecurityagency.github.io
======
rdtsc
This caught my eye:

> [https://github.com/apache/incubator-
> pirk](https://github.com/apache/incubator-pirk)

> Employing homomorphic encryption techniques, PIR enables datasets to remain
> resident in their native locations while giving the ability to query the
> datasets with sensitive terms.

I can imagine a few scenarios there. One perhaps is when db admin should not
find out what someone, possibly working on a classified project is querying.

Or say one compartment / project collected the data and now they want to share
it with another project. Those read into the second project don't want to
reveal to the first one what they are querying because it would reveal
classified information.

Another scenario is a database which has results of possibly illegally
intercepted communications. If the NSA can argue that the Constitutionally
defined "search" doesn't occur until someone actually performs a search (as in
runs an SQL query over the data). Then having PIR capability means being able
to break the law but only let as few people as possible do it.

Also [https://github.com/redhawksdr](https://github.com/redhawksdr) is pretty
damn impressive. It looks like a complete parallel implementation of GNU
Radio. Completed with an IDE and such. Wonder how it compares?

~~~
Spearchucker
This is pretty common in the commercial world too, and something I've done
more than once myself. The obvious use case is storing medical records.

In the UK personal medical records are often stored by systems integrators in
datacentres with nebulous locations, and need to be accessed by third parties
for things like underwriting life insurance policies.

To protect the data (compliance with the EU data protection act) it's
encrypted in transit AND at rest. Access to data by third parties is managed
through AMRAs (access medical record authorisation), which are completed by
the third party, authorised by the data owner (private individual) and given
to the data owner's general/dental practitioner or pharmacist, who is able to
access and decrypt and appropriately share the sensitive data.

~~~
raverbashing
Are AMRAs kinda like "stored procedures" then?

~~~
Spearchucker
AMRAs are like physical documents with an instruction to the information
guardian, with a signature from the information owner, authorising access.

A lot of it is electronic these days, and is automated to the point that an
individual authorises access by clicking a link in an email that calls an
endpoint that in turn releases a token and URL to the requestor to view the
appropriate records.

~~~
Cakez0r
So like OAuth?

~~~
Spearchucker
Partly. OAuth is authentication (who I am), which is part of it, but the real
point is authorisation (what I can do).

------
hueving
Violating the sanctity of the captive portal license agreement!
[https://github.com/iadgov/goSecure/blob/master/scripts/wifi_...](https://github.com/iadgov/goSecure/blob/master/scripts/wifi_captive_portal.py)

:)

~~~
libeclipse
The captive portal license agreement?

~~~
MertsA
That code just bypasses a captive portal. It's basically automatically
clicking "I agree".

------
reiz
That's nice to see the NSA is contributing to the OSS community. I just
randomly picked one of the NSA GitHub repositories, analysed it with
VersionEye ([https://www.versioneye.com](https://www.versioneye.com)) and
found already 25 security vulnerabilities. Who is the best person to contact
in this case? Here is the security report:
[https://www.versioneye.com/user/projects/59479cd06725bd00123...](https://www.versioneye.com/user/projects/59479cd06725bd001230f152?child=summary#tab-
security).

~~~
0xADADA
They say it clearly themselves:

> The government benefits from the open source community’s enhancements to the
> technology.

They're hoping that by putting this code out there, unwitting dupes will then
collaborate with them to contribute to the surveillance state.

------
wfunction
Can someone explain how some of projects can be MIT-licensed (or anything-
else-licensed) as they claim? Aren't they necessarily in the public domain
given that they're works of the U.S. Government?

~~~
dagw
Certain government agencies and subsidiaries are exempt from having their work
considered "government work" and can thus claim copyright if they want. I'm
guessing the NSA is such and agency. Also if the work was actually done by a
contractor then there are other exemptions.

~~~
wfunction
Thanks, but I'm not sure that's it. For example, when I look at at [1], I see
an apparent contradiction with [2]. It almost seems like they don't know what
they're doing, but surely that's because I'm misunderstanding what's going on?

[1]
[https://github.com/NationalSecurityAgency/DCP/blob/21c8d3efe...](https://github.com/NationalSecurityAgency/DCP/blob/21c8d3efecd0ceb5fc38dfe8e2c3aa8011ed58cd/COPYING)

[2]
[https://github.com/NationalSecurityAgency/DCP/blob/496402fa9...](https://github.com/NationalSecurityAgency/DCP/blob/496402fa92aa61acfbb871f6faa0988b16d1e2c1/LICENSE)

~~~
ginreaper
Well if they are forking or contributing to other software or any type of
derivative work, they probably have to retain the original license by law

~~~
wfunction
That's not the case at least in the example I just gave though, right?

------
lsh
I've trialed Apache Nifi and it's very powerful. It's also a little unsettling
as you use it thinking about how the NSA used it ...

~~~
Toast_
NiFi looks pretty interesting, what's the learning curve like? Looks like a
supercharged yahoo pipes.

------
voltagex_
[https://github.com/ozoneplatform/owf-
framework](https://github.com/ozoneplatform/owf-framework) looks very
interesting - NSA wrote their own BI tool?

~~~
killjoywashere
OWF came up in a meeting recently -- may not be as awesome as NSA wants you to
believe.

~~~
voltagex_
Can you share any other info?

~~~
digitalzombie
reddit have a comment thread about this.

Once comment described it as a "shit show".

Apparently its terrible and they tried to get the company to rebuild it and
they made it the same.

Another comment comment on how "buggy" it was.

Or that the only dev that willing to work with it are contractors that want
money.

~~~
annnnd
> Or that the only dev that willing to work with it are contractors that want
> money.

Greedy b*stards! </sarcasm>

------
Cryptoholic
I wonder if all the people who are really suspicious of it in here realize
that this (releasing their projects as OSS) has been a thing for a while.

SELinux

Accumulo (a popular NoSQL distributed key-value store)

Apache NiFi (data processing system)

etc.

------
microwavecamera
It's nice to see a push for open security and solutions rather than secretive
offensive counter-security. Thanks. :)

~~~
alltakendamned
One does not exclude the other though

------
api
One of my favorites is the Speck cipher, which has been released before:

[https://en.wikipedia.org/wiki/Speck_(cipher)](https://en.wikipedia.org/wiki/Speck_\(cipher\))

I'd be very interested in more public cryptanalysis of this. It's a damn
simple cipher to implement, and if it were at least as secure as say
Salsa20/12 it'd be very nice for all kinds of applications.

------
nautilus12
Does the fact that many of these havent been updated in months or years mean
that these are really old projects that effectively hold no value to the NSA
and arent close to any of their core operations?

~~~
angry_octet
For every one of these projects someone spent a considerable effort do the
paperwork to get it pushed out, have it signed off as sanitized, non-
embarassing, etc. It is a lot of work to get that done in bureaucratic and
risk-averse organisations.

If there had been a community contributing back I expect that there would have
been more activity, but if it seems like noone will, would you spend your time
pushing out regular updates?

------
SomeStupidPoint
There's a lot of neat things there. (This one looks interesting:
[https://iadgov.github.io/goSecure/](https://iadgov.github.io/goSecure/))

Also interesting is splitting the repos: that the NSA and IAD have different
repos, and that one seems focused on defensive tech while the other is
publishing analysis tools.

I know there's a lot of people who aren't fans of the NSA (or what they do),
but I think most of us can see a need for a military-grade organization to
research defensive technologies for helping secure our infrastructure. I don't
think many of us would be unhappy with the NSA if that's all they did. (Or
phrased another way: most of us are unhappy because of how they conduct intel
work or compromise defensive capability for offensive ones, eg, that whole
business with ECC.)

So I think it's important to respond positively to things like the IAD github
page, even if we're not fans in general.

~~~
jamesfe
I think you're right. It's sad to see many people are looking at these tools
and performing a sort of "Allegory of the Cave" by extrapolating, then, the
evils that can be done with these tools.

Something, mostly common sense, tells me that we will not find some smoking
gun to a crime here in these OSS repos...if anyone wanted that, they can refer
to any number of leaks.

Ultimately, I'm happy to see this stuff shared, happy to see others use it and
happy to see the OSS community build on it.

~~~
LunaSea
There's multiple reasons why you wouldn't want to use these newly released
open-source projects. First one is, like you said, the danger of a backdoor.
The second one is that due to the very long list of non-ethical and illegal
practices of the organisation you don't want to contribute or depend on them.

~~~
ganoushoreilly
Bleach has been used in bombs around the world, should I not use it because
there's a chance it's been used for (perceived) evil?

~~~
yarrel
Cluster bombs have such blatant propaganda spread against them by well-meaning
but naive individuals. Don't be fooled - look at all their positive uses!

~~~
ganoushoreilly
That's not a like for like comparison, one is a bomb designed for destruction,
one is software designed to solve a problem. The software isn't designed to
kill, maim, or destroy, if it can be used as part of the process that doesn't
make the software itself evil. Should we throw out linux systems since linux
powers many control systems in war machines?

------
blazespin
I suspect there are a lot of very incredible computer programmers at the NSA
and they're probably using just regular open source non security related tools
every day. It's good to see that they're contributing back to the OS community
what they can.

~~~
sneak
Let's not forget: these people may be skilled, but they are working against
every principle of our community.

[https://www-androidauthority-com.cdn.ampproject.org/i/www.an...](https://www-
androidauthority-com.cdn.ampproject.org/i/www.androidauthority.com/wp-
content/uploads/2014/06/SSL-Added-and-Removed-Here.jpg)

~~~
angry_octet
And you pasted an amp link? Was that deep sarcasm?

The world is full of shades of grey, and black and white 'they are all evil!'
is just pointless and dumb.

~~~
sneak
It wasn't intentionally ironic.

Where did I call anyone evil? I didn't even vilify people - I specifically
focused on the work that these people are performing. It works to make the
world a worse place.

How should I criticize them?

~~~
angry_octet
For breaking the law and working to undermine the Constitution? Is that not a
big enough claim?

Maybe blame specific people (the DIR NSA Hayden, Bush, Obama). There was a
strong culture at NSA of NOT spying on Americans until those clowns came
along.

~~~
sneak
But spying on other people is okay, simply because where they had the audacity
to be born?

Nations are fictions and depriving rights to a group that you permit to others
based on nationality is entirely unjust.

I blame the specific people who built these systems to collect and process
data for the military.

~~~
angry_octet
Would you rather that nations didn't spy on each other? I wouldn't. Effective
spying helps prevent war, and lack of intel leads to wars.

If the British had had better intel they would never have invaded Afghanistan
to repulse an imaginary Russian annexation. If intel had been listened to,
instead of deliberately ignored/fabricated by Bush we wouldn't have invaded
Iraq because of imaginary WMDs. If Kennedy hadn't actively understood the
intel on Cuba, and just left it to the fears of the generals, the US would
have invaded Cuba. The biggest fear of the Warsaw Pact is that the West would
invade -- which was the West's fear also -- to the point that a lack of
intelligence on Western force movements almost resulted in a counter-premptive
invasion.

So maybe nations are fictions, but they are pretty powerful ones that most
people are happy to roll with, and that makes the consequences real. These
same people can agree on a set of collective norms that control who/what/where
can be surveilled.

The propaganda that there are many ISIS terrorists embedded in Western nations
is the key lever which will be used to transition from military/national
intelligence to the surveillance state. This can't be won by railing against
the NSA, but by countering (islamic and fascist) extremism, false reporting
and propaganda.

And yet in the mean time we are being outflanked by traditional nation state
adversaries using strategic propaganda campaigns. And they won't hesitate to
spy on us. I want NSA(/GCHQ/DGSE/8200) working hard to prevent that, instead
of navel gazing illegal programs about spying on
environmentalists/unionists/politicans.

~~~
feborges
What about using the state infrastructure/technology to spy in the Brazilian
state oil company and steal engineering secrets to hand to american
competitors?

------
bigon
Also don't forget SELinux
[https://github.com/SELinuxProject/](https://github.com/SELinuxProject/) which
is a project that comes from the NSA as well

~~~
Cakez0r
SELinux is on the list :)

------
deepnotderp
I see the PR department is getting smarter...

~~~
mtgx
Federal government is required to open source at least 20 percent of its code
now:

[https://sourcecode.cio.gov/](https://sourcecode.cio.gov/)

~~~
qeternity
The cynic in me wonders _which_ 20% we'll get. The repo linked here is pretty
impressive, so I may have to eat crow. I would have expected to get every html
template and vba macro ever written, instead of the value add stuff.

~~~
skynode
My thoughts exactly. That 20% will probably provide you with a map of Russia
while you're​ flying over the Swiss Alps. Some things are better not learned
(or known) at all than learned (or known) the wrong way.

------
headmelted
Femto ([https://github.com/femto-dev/femto](https://github.com/femto-
dev/femto)) looks pretty interesting to me just based on some work I've needed
to do in the past that it would've come in handy for.

It looks like the last commit was over a year ago, though. Is there
information I'm not seeing of whether these projects are actively maintained
(or still in use at NSA?).

~~~
nautilus12
Im also curious, without digging into it deeply is it something comparable to
Solr or Elasticsearch? If so i wonder how it stacks up

------
corpMaverick
This is nice. It should all be about protecting electronic systems. To help
individuals and companies build resilient systems. It protects the USA and the
world economy as a whole. It should never be about spying people or even
catching criminals IMHO.

------
thrillgore
If they're looking for pull requests, they won't find any salvation in the OSS
world.

------
aramas
It's generous to have all in royalty free license.

~~~
pvorb
Why should it be generous? It's been paid through taxes.

~~~
notfoss
Well, it's generous for non-americans ;)

~~~
Tepix
A small compensation for the invasion of their privacy...

------
r3bl
Fun sidenote: I was the very first civilian to contribute to their GitHub
project back in July 2015, when SIMP was the only project they had up on
GitHub.

It was literally a one letter change in the README file, but I still have the
privilege to call myself the very first civilian to contribute to the NSA's
open source project:
[https://github.com/NationalSecurityAgency/SIMP/pull/1](https://github.com/NationalSecurityAgency/SIMP/pull/1)

~~~
jameskegel
I'd shake your hand

------
forgottenacc57
Why are people so welcoming to the filthy spies invading citizen privacy?

~~~
Tepix
Some of the are defending against other filthy spies.

~~~
mmjaa
Down With All The Filthy Spies!

------
grandalf
Some very cool stuff.... useful from both a practical and anthropological
standpoint.

