
Microsoft open-sources Graph Engine - dajoh
https://www.graphengine.io
======
wyldfire
I find that very abstract software packages like this are difficult to
visualize without an example. This page does not offer one, but TFM does --
see [1]. Also note that TFM describes the fact that this is Windows-only. That
makes it substantially less interesting, IMO.

[1] [https://www.graphengine.io/docs/manual/index.html#what-is-
ge](https://www.graphengine.io/docs/manual/index.html#what-is-ge)

~~~
v-yadli
Dev here. Cross-platform for Linux/BSD is on the way, and it's already in the
repo now.

~~~
halflings
That's great! You should definitely add that to the docs. I just went to the
same page and was extremely disappointed when I saw that it required Windows.
A benchmark comparing it with similar solutions (esp. Neo4j) would be much
welcome.

~~~
v-yadli
Sure we'll definitely update the docs! Or better to work out one way to put
the docs on GitHub and keep it synchronized with the site (they're markdown
anyway)

~~~
sah2ed
While you are at it, can you guys also do a feature comparison between Graph
Engine vs Neo4J vs JanusGraph (vs Apache Giraph)?

Thanks

------
bluejekyll
At this rate, are we going to see Windows open-sourced?

MS is on a roll. My bias since 1996 is being eroded with each OSS release they
have, and multi-platform targeted support.

I started using VSCode regularly as my main Rust IDE, and I feel dirty for
liking it. It's seemless across macOS and Linux.

~~~
WayneBro
Not yet. Their next strategy is to try and use ARM (again) to shove UWP down
your throat.

I love Windows but I seriously will not touch UWP until they loosen up the
sandbox restrictions so that I can do regular IPC with a win32 desktop app.
You can't even send an HTTP request to a little node.js Web server running on
your desktop right now. Fuck that. I'm not buying into it.

~~~
Someone1234
> I love Windows but I seriously will not touch UWP until they loosen up the
> sandbox restrictions so that I can do regular IPC with a win32 desktop app.

The whole reason for the UWP sandboxing is to make it so that users can
confidently install UWP applications with some assurances that it isn't
spyware/malware/blackmailware that will take over their PC or steal their
personal information.

UWP is Microsoft's answer to years of criticism they received for the way
process permissions work on Windows. They aren't going to just dump it all
because you and a few others find the priviledge model inconvenient.

> You can't even send an HTTP request to a little node.js Web server running
> on your desktop right now.

If your node.js web server was network accessible you could. But, no, it won't
allow unchecked IPC as that directly defeats the whole point of sandboxing to
begin with.

Personally I haven't used UWP much either, but that has nothing to do with
sandboxing and everything to do with poor performance and UI issues (e.g.
window controls, built in UI elements, etc). I still find the Windows 10 UWP
Calculator horrible to use, it was a MASSIVE downgrade over the old one.

~~~
jdmichal
Honestly, this particular complaint reminds me a lot of UAC. After _decades_
of people complaining that Windows account security was too lax, they
implement the graphical popup equivalent of `sudo`. Then proceed to get
skewered in public opinion for having done it.

Yes, proper security means you will be inconvenienced. In the same way that
you would be inconvenienced if your bank called you to confirm that $10000
wire you didn't initiate. It's inconvenient that you have to take a call and
resolve the issue, but it's a hell of a lot better than finding out at some
indeterminate point in the future that $10000 is missing from your account.

~~~
beagle3
If only. sudo, as configured out-of-the-box on Ubuntu, gets out of your way
and lets you work (Except if you haven't properly authenticated in the last 15
minutes); polkit is similarly out-of-your-way.

Sudo is also (relatively) easy to configure: "this command line can be run
under sudo by anybody", "this command must not be run under sudo even if they
know their password", etc.

UAC is always-in-your-face about everything, often unexpectedly - it hijacks
your desktops at inopportune times, and there's no way to tell it "Yes, this
software can do this again for the next 15 minutes without asking me again".

UAC is very, very far from being the graphical popup equivalent of sudo. If it
had been, people would not have complained as much. (Someone always complains.
But in the case of UAC, most of the complaints are justified)

~~~
Someone1234
While what you say was true about Vista, in Windows 7, 8, 8.1, and 10 UAC
prompts are suppressed when they're the direct result of user interactions
within the OS itself. Each OS reduced the number of UAC prompts a user
experiences.

~~~
poizan42
They also broke the security on the default configuration because now you can
just interact with the system components to bypass UAC, see e.g. [0]. This is
officially not a security vulnerability because UAC isn't actually a security
barrier unless you set it to "Always prompt", but just a feature to make
applications play nice. But note that it's not on "Always prompt" by
default...

[0] [https://github.com/hfiref0x/UACME](https://github.com/hfiref0x/UACME)

------
crudbug
Interesting - Linux Foundation with IBM, Google and others announced last week
JanusGraph [0]. Janus provides optional persistent storage option also.

What are the good use cases for these ?

[0] [http://janusgraph.org/](http://janusgraph.org/)

~~~
nitrogen
Old-school AI maybe, fraud detection, social network analysis, recommendation
engines, deanonymization, surveillance and tracking...

~~~
crudbug
I am using a time-series db where each metric measurement requires some static
context information to be included, e.g. Event 1 happened in Device A at Node
B, Location C. These static entities (A, B, C) are perfect for graph vertices.
But, when I am persisting the graph on disk - I store all the data statically,
No links just columns - the static entities become dimensional data.

Is there a better way to this ?

Ideally, we should persist the Object Graph directly without any data
conversion. I see a difference in - How we compute and How we store data.

~~~
rodionos
Not specific to GraphEngine, but you could store all meta-data about the
device separate from the time-series itself. The composite metric name (key?)
would then include the measurement name and the entity name (device name in
this case).

~~~
crudbug
That is one possible solution. But, for queries we will need some kind of JOIN
at runtime.

I do store the entity data separately in a transactional row store.

For analytical queries, we use a column store. For this, I have to include the
entity data (context) for each fired event.

The whole Compute & Storage semantics are different.

If only, we had a persistent data-aware compute & storage graph engine that
supported transactional & analytical workloads.

~~~
rodionos
Storing time-series in a columnar format (blobs o time-values) and entity data
in row format sounds good to me.

If you look at data historians (PI et al), that's how they lay out the data:
metrics storage is separate from the asset store, or Asset Framework as it's
called.

------
sidcool
For a noob like me, what are the applications of this framework? What are the
use cases? What current tools do this?

------
rch
This era of open source from MS is great, but my reaction is always "OK,
here's the Microsoft version of something I've been using for a couple of
years already".

I'd like to see the Spanner+Cyc GIS-capable global distributed real-time graph
engine with FPGA accelerated OLAP support MS Research is probably sitting on,
because we can almost / pretty-much hack that together with OSS now.

------
brilliantcode
Microsoft of 2017 makes me forget about Microsoft of 1997. It's insane how a
shakeup of CEO and new cultural shift can seemingly add another major boost to
it's brand.

I welcome open source, eventually, I believe that it will eat commercial
software, if the right economic incentives are in place. Microsoft may be
signaling this to the market.

~~~
webmaven
_> Microsoft of 2017 makes me forget about Microsoft of 1997._

Never Forget.

~~~
scott_karana
If I may paraphrase: "Never forget that large monopolies can wield disgusting
influence, but don't let hate for past monopolies blind you to modern
monopolies either" :-)

~~~
webmaven
_> don't let hate for past monopolies blind you to modern monopolies either_

Just because Microsoft is no longer the monopoly-du-jour, doesn't mean they
aren't still a monopoly...

Just because they have been making (successful) forays into open-source, does
not mean they won't still fiercely protect (with whatever means at their
disposal) their monopoly cash cows (Windows and Office).

The "standartization" of Office Open XML wasn't all that long ago, after all
(ECMA in 2006, ISO in 2008).

~~~
greggyb
Windows is a legitimate cash cow, but because of server installs. Office makes
a lot of money; SQL Server and other datacenter products make much more.

~~~
webmaven
_> Windows is a legitimate cash cow, but because of server installs. Office
makes a lot of money; SQL Server and other datacenter products make much
more._

And yet on the server Microsoft does not have a monopoly.

Consider that the main reason Microsoft _has_ a profitable business selling
server software is _because_ of their desktop monopolies through many diverse
paths.

------
mlmlmasd
Seems to be just a marketing page with no link to source code or even mention
of open source?

Here is the github page:

[https://github.com/Microsoft/GraphEngine](https://github.com/Microsoft/GraphEngine)

~~~
ygra
The repository is linked on the Download page.

------
cobookman
Am I the only one getting SSL cert authority invalid error

~~~
debatem1
Nope, I am as well.

~~~
EGreg
But I'm pretty sure you two are the only ones.

~~~
etelej
Nope. Same here. However, it's just Chrome on Android that doesn't recognize
the CA, even the native android browser is okay. Bizarre, may be coz they have
an incomplete cert chain
[https://www.ssllabs.com/ssltest/analyze.html?d=www.graphengi...](https://www.ssllabs.com/ssltest/analyze.html?d=www.graphengine.io)
and chrome on android has had issues with that

~~~
EGreg
I was being sarcastic :)

------
adamnemecek
I've been running into the idea of computational graphs a lot recently. It's
at the core of Tensorflow (and NN in general) but it also comes up for example
in Apple's AVFoundation where all audio processing happens in a graph of audio
units. Does anyone know what's the theoretical foundation of computational
graphs?

EDIT: I've created a wiki page for computational graphs.
[https://en.wikipedia.org/wiki/Computational_Graph](https://en.wikipedia.org/wiki/Computational_Graph).
Add your input.

~~~
mtrn
There is the biological analogue, which has inspired neural networks:

> Recall that in our general definition a feed-forward neural network is a
> _computational graph_ whose _nodes are computing units_ and whose directed
> edges _transmit numerical information from node to node._

> Each computing unit is capable of evaluating a single primitive function of
> its input. In fact the network represents a chain of function compositions
> which transfor m an input to an output vector (called a pattern).

From: [https://page.mi.fu-
berlin.de/rojas/neural/chapter/K7.pdf](https://page.mi.fu-
berlin.de/rojas/neural/chapter/K7.pdf) (1996)

Another ancestor would be the Data-Flow paradigm:

> .. programming paradigm that internally represents applications as a
> _directed graph_ , similarly to a dataflow diagram. Applications are
> represented as a set of nodes (also called blocks) with input and/or output
> ports in them. These nodes can either be sources, sinks or processing blocks
> to the information flowing in the system. _Nodes are connected by directed
> edges that define the flow of information between them_.

From:
[https://paginas.fe.up.pt/~prodei/dsie12/papers/paper_17.pdf](https://paginas.fe.up.pt/~prodei/dsie12/papers/paper_17.pdf)
(2012)

~~~
adamnemecek
> There is the biological analogue, which has inspired neural network

I'm somewhat aware but it seems like the idea of a computational graph is the
most generic computational idea I can think of and I'm surprised it's not more
explored.

> Another ancestor would be the Data-Flow paradigm:

Oh yeah, data flow is definitely another one.

~~~
Jasper_
Functional programming can be represented as a graph, where calls are directed
edges. Flow control is often analyzed in graph form in real compilers:

[https://en.wikipedia.org/wiki/Control_flow_graph](https://en.wikipedia.org/wiki/Control_flow_graph)

~~~
adamnemecek
It's not just functional programming (but I do agree that it's better in
functional programming). All the reverse engineering tools have some feature
to represent assembly as a CFG which is super helpful.

------
ronack
Does Microsoft use this for anything in production?

~~~
reubenbond
From the Readme, it seems they do:

> LIKQ is powering Academic Graph Search API, which is part of Microsoft
> Cognitive Services.

------
VeejayRampay
Funny how Microsoft has been rocking hard with their OS releases lately (and
good on them for doing so), but there's still a pervasive feeling in the dev
community about their true eventual motives and strategy.

"Fool me once..." I guess, or as we say in France "Cold water scares the
scalded cat".

------
vegabook
Is this in the same space as Storm, Flink, or Spark?

Does it do streaming data such as Flink or Storm? Or is it batch-optimized?

What languages does the compute engine support?

------
kensai
I still like the neat Pajek. Nifty little piece of software, unknown by most,
but really powerful! Especially if you are into social network analysis. Who
else uses it?

[http://mrvar.fdv.uni-lj.si/pajek/](http://mrvar.fdv.uni-lj.si/pajek/)

~~~
webmaven
Never heard of it before, and wow, that page is _seriously_ old-school.

------
zyztem
[http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...](http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html)

------
infocollector
Does this have a python interface?

~~~
onion2k
It provides a RESTful API for getting the data out.

~~~
simonw
Do you have a link to the docs on that? I didn't spot it.

~~~
onion2k
[https://www.graphengine.io/docs/manual/index.html#query-
serv...](https://www.graphengine.io/docs/manual/index.html#query-serving)

EDIT: Slightly more useful link -
[https://www.graphengine.io/docs/manual/TSL/MPProtocol.html#h...](https://www.graphengine.io/docs/manual/TSL/MPProtocol.html#http-
protocols)

------
dredmorbius
MIT License, for those curious:

[https://github.com/Microsoft/GraphEngine/blob/master/LICENSE...](https://github.com/Microsoft/GraphEngine/blob/master/LICENSE.md)

------
tempVariable
I see that the source code "os.h" file has directives to handle Linux and
Apple. Has anyone managed to build it and use it in the even most trivial way?

~~~
v-yadli
At the current state you could link Trinity.C with a custom Mono host and then
execute your GE assemblies/Trinity.Core. Check out CMakeFile.txt for details.
:) We're working to interface Trinity with CoreCLR. .NET standard 2.0 would
make this much easier.

------
purple-dragon
I'm a little confused. Is this competitive to something like Spark?

------
linux_devil
I am confused, how is it different from spark graphx?

------
zump
How is this different to D-Graph?

------
floopidydoopidy
Although astroturfing doesn't seem to be a problem on HN, I'd really
appreciate it if you didn't do it.

~~~
dang
Since you've done this repeatedly and ignored our request to stop, we've
banned your account.

Actual astroturfing, when it occurs, is an abuse of HN that we crack down hard
on. Defending this community against gaming and abuse is a huge priority for
us. Any user who thinks they might be seeing it happen on HN should email us
right away (hn@ycombinator.com) so we can investigate.

Imaginary astroturfing—the bug that causes some users to be certain that those
who disagree with them can only be nefarious shills because otherwise the pure
reason of their own point of view would be fully accepted—is also an abuse of
HN. This one is orders of magnitude more common, and it is poison. It eats
away the heart of civil, substantive discourse, the assumption of good faith
on the part of others.

Therefore we ban astroturfers, and we also ban users who accuse others of
astroturfing or shilling without evidence. An opposing view does not count as
evidence, and playing this card as a rhetorical device in an argument breaks
the HN guidelines.

We detached this subthread from
[https://news.ycombinator.com/item?id=13607640](https://news.ycombinator.com/item?id=13607640)
and marked it off-topic.

------
anon987
How the hell does this story have so many upvotes?

HN really needs to do something about Microsoft's vote manipulation - it's
becoming quite blatant at this point.

~~~
sctb
Please don't post baseless accusations of shillage. See more here:
[https://news.ycombinator.com/item?id=13610097](https://news.ycombinator.com/item?id=13610097).

------
bitwize
This is not a cuddly new Microsoft. First comes the embrace (look at all this
stuff on our github!), then comes the extend (run your Linux stack on Windows
and never have to give up Visual Studio!), I'm sure you know what comes next.
Hint: PC manufacturers no longer have to give you the option to disable Secure
Boot.

~~~
whodknee
>PC manufacturers no longer have to give you the option to disable Secure
Boot.

Then don't buy from those people. Simple

