
Gaffer: Large-scale graph database by GCHQ - kyrre
https://github.com/GovernmentCommunicationsHeadquarters/Gaffer
======
jwr
This can't be real. I needed something _exactly_ like this and was about to
start looking tomorrow. And, lo and behold, GCHQ delivers.

As for my future software needs, I expect the code to be written at MI-6 and
delivered just-in-time by James Bond.

~~~
jkxyz
Just don't mention anything about a new JavaScript framework in any of your
private communications, or they might spot it and make a new one of those too.

------
dignati
The name is ridiculous in german. A "gaffer" is a stalker or someone who looks
when he's not supposed to. Quite fitting.

~~~
sehr
Very weird! I've always heard it used to refer to English soccer club coaches

~~~
alessioalex
You mean English football clubs. Nobody says soccer in Europe mate :p

------
cwp
Really? The spies release a graph database where edges can have statistics
attached, like oh, say, counts?

We're being trolled.

Edit: Looking for forward to the release of RingTone, a system for processing
very high volumes of call detail records in real time.

------
nshstjgsbt
This is just a PR move to get the tech community to hate them less, even if
only by a little bit. They want to muddy the waters and insert the idea into
people's minds that "we're not all bad."

~~~
unfunco
I don't think that's the case, the software is taxpayer funded so it makes
sense to release it.

I think it's more likely that it will help with recruitment, pull requests of
high-quality might provide a potential interview or job offer.

~~~
EvanPlaice
If it's 'taxpayer funded' as you say, it should be licensed under the public
domain.

No 'if', 'ands', or 'buts'. If it's paid for by the people it's the property
of the people.

~~~
spyspy
And if you pay enough in taxes you should get your own tomahawk cruise
missile.

~~~
EvanPlaice
Not sure what you'd do with a tomahawk (besides hump it for fortitude).

Anyway...

In the US the policy is:

A United States government work is prepared by an officer or employee of the
United States government as part of that person's official duties.

It is not subject to copyright in the United States and there are no copyright
restrictions on reproduction, derivative works, distribution, performance, or
display of the work. Anyone may, without restriction under U.S. copyright
laws:

\- reproduce the work in print or digital form

\- create derivative works

\- perform the work publicly

\- display the work

\- distribute copies or digitally transfer the work to the public by sale or
other transfer of ownership, or by rental, lease, or lending.

Source: [https://www.usa.gov/government-works](https://www.usa.gov/government-
works)

US government based software developer agencies like 18f and USDS (United
States Digital Service) license all of their code to the public domain.

The UK is slightly different:
[https://github.com/GovernmentCommunicationsHeadquarters/Gaff...](https://github.com/GovernmentCommunicationsHeadquarters/Gaffer/issues/3)

But there _is_ the OGL (Open Government License):
[http://www.nationalarchives.gov.uk/information-
management/re...](http://www.nationalarchives.gov.uk/information-
management/re-using-public-sector-information/licensing-for-re-use/guidance-
for-information-providers/make-information-available-ogl/)

------
mark_l_watson
Interesting to get open source from the British sort-of equivalent to the NSA.

I just looked at the code: Java code that sits on top of the Hadoop file
system. Supports date-binned data storage so it looks applicable to systems
where you want to toss out old data occasionally.

~~~
acveilleux
Yep, and the Accummulo store it's built upon was open-sourced by the NSA.

------
vskarine
I like how it is on top of Accumulo which was created by NSA:
[https://en.wikipedia.org/wiki/Apache_Accumulo](https://en.wikipedia.org/wiki/Apache_Accumulo)

Great to see spy agencies cooperating in open source :P

------
typon
Do ends justify the means when it comes to knowledge being added to the world
open-source repository of software? Should we, as a community, reject these
people's hard work or just use it while also understanding that they're evil?
I'm conflicted.

~~~
throwaway94874
If you think what they do is evil then explain why. Don't pretend that
everyone thinks that way so much so that it doesn't even need an explanation.

~~~
typon
It has been discussed to death here why what they do is evil.

If you want a security expert's opinion, read Bruce Schneier's blog, and if
you are inclined to learn more about the ethics, this page is great:
[http://cs.stanford.edu/people/eroberts/cs201/projects/ethics...](http://cs.stanford.edu/people/eroberts/cs201/projects/ethics-
of-surveillance/index3.html)

------
aselzer
Love the way all the tests use "customer" and "product" nodes.

~~~
swalsh
Haha i wonder what the product is when you're a spy agency

~~~
samstave
Oppression and control and leverage

------
tangled
I'd be interested in seeing the buy vs. build analysis for this project. Are
there any pre-existing projects that have similar features? And assuming that
this project is used to process classified data, what impact does this have on
the selection process? e.g. is it possible to use closed-source solutions?

------
Ianvdl
From [1]: > Gaffer stores data in Accumulo, but inserting data and retrieving
it again requires the user to have no knowledge of Accumulo. As Gaffer stores
data in Accumulo, it is horizontally scalable so that very large data sets can
be dealt with. It has an API that allows users to retrieve the data they care
about, filtered according to their requirements and aggregated over the time
window of interest. It supports bulk update and continuous update.

Seems like a very useful tool, especially if you already have accumulo
infrastructure running. Docs need a bit more work I feel, but it's not
terrible for a single page.

[1]:
[https://github.com/GovernmentCommunicationsHeadquarters/Gaff...](https://github.com/GovernmentCommunicationsHeadquarters/Gaffer/blob/master/UserGuide.md)

------
sammorrowdrums
This is not long after MI6 were looking for NodeJS Devs
[https://news.ycombinator.com/item?id=10532855](https://news.ycombinator.com/item?id=10532855)

~~~
sammorrowdrums
Not that they're necessarily talking to each other about PR in the tech
community, but looks like maybe the British Gov is trying to attract some
talent and get devs engaged. Maybe they just want it improved for free :-p

~~~
DanBC
GCHQ / CESG have been putting details of some of the tech they work with on
their public webpage for a few years now.

[https://www.gchq-careers.co.uk/about-gchq.html](https://www.gchq-
careers.co.uk/about-gchq.html)

[https://www.gchq-careers.co.uk/departments/technology-and-
en...](https://www.gchq-careers.co.uk/departments/technology-and-
engineering.html)

The "applied research" is interesting: [https://www.gchq-
careers.co.uk/departments/applied-research....](https://www.gchq-
careers.co.uk/departments/applied-research.html)

I think the internet archive has some older pages with relevant information.

If this is GCHQ it's gently worrying - a github profile is pretty low on my
list of what I want in the way of transparency and accountability.

~~~
Cakez0r
It seems like they are making a recruitment push
([http://www.theinquirer.net/inquirer/news/2435685/gchq-is-
usi...](http://www.theinquirer.net/inquirer/news/2435685/gchq-is-using-
graffiti-to-find-hipster-coders)) and are trying to repair their reputation a
bit.

------
osense
[https://github.com/gchq](https://github.com/gchq)

Cybersquatters...

~~~
int_handler
It is curious to me why GCHQ didn't just contact GitHub to acquire
github.com/gchq but instead decided to go with the long and cumbersome
github.com/GovernmentCommunicationsHeadquarters. Perhaps it is a British thing
[1].

[1] [https://www.addedbytes.com/blog/if-php-were-
british/](https://www.addedbytes.com/blog/if-php-were-british/)

~~~
flurdy
They probably have the skills to acquire it without contacting GitHub as well.
But that may be bad PR...

~~~
sangnoir
> They probably have the skills to acquire it without contacting GitHub as
> well.

I'd say they _definitely_ have the skills to acquire it, if QUANTUM
{INSERT|DNS}[1] are still operational. As you said, it's probably not worth
it.

1\. [http://blog.fox-it.com/2015/04/20/deep-dive-into-quantum-
ins...](http://blog.fox-it.com/2015/04/20/deep-dive-into-quantum-insert/)

------
akerro
Any ideas how they could use it?

~~~
tomschlick
Graph databases are perfect for identifying connections between who talks to
who, relatives, contacts, etc.

~~~
Hermel
Ironically, "Gaffer" is the German word for rubberneck.

~~~
genericpseudo
British English word for "boss", traditionally in blue-collar jobs (building
sites, factories, that kind of thing) but used informally everywhere.

~~~
SixSigma
Gaffer is the name of the head electrician in film production, the assistant
is called the Best Boy.

[https://en.wikipedia.org/wiki/Gaffer_%28filmmaking%29](https://en.wikipedia.org/wiki/Gaffer_%28filmmaking%29)

Hence "Gaffer tape" which is black, rather than duct tape, which is the same
construction but grey.

~~~
samstave
PSA: Gaffer's tape makes the best ad-hoc mouse pad for surfaces that are not
mousable... (at a trade show and your new shiny glass counters are acting
weird with mice... make a small square with gaffers tape, which you already
use to secure cords under carpet...

Its my favorite tape for general use.

~~~
EvanPlaice
FYI, unrelated to the topic but they make mice that work on any surface how.

I currently use this one on my glass desk: [http://www.amazon.com/Logitech-
Wireless-Anywhere-Mouse-Mac/d...](http://www.amazon.com/Logitech-Wireless-
Anywhere-Mouse-Mac/dp/B0082D5660)

~~~
samstave
I replied to myself - but meant to reply to you:

> __ _Haha yeah I have at least ten of those... They do not work on certain
> glass surfaces, namely any shiny black granite desk._ __

~~~
EvanPlaice
Do you own one of the 'Darkfield Technology' ones?

That's specifically why I bought this one and it works flawlessly on every
glass surface I've tried.

~~~
samstave
Apparently not, I'll have to try one of those.

------
ExxKA
I wonder if this is for real

~~~
DanBC
It's not mentioned on the GCHQ / CESG website, as far as I can tell.

------
emehrkay
Get a
[TinekerPop3]([http://tinkerpop.incubator.apache.org](http://tinkerpop.incubator.apache.org))
interface and I'll update my python libs to support it (one day...in the
future...when I have time...and interest).

------
infinity0
Why is the British government doing research into something that terrorists
could use to further their extremist agendas?

