
Beam: Distributed knowledge graph store written in Go by eBay - ngaut
https://github.com/eBay/beam
======
eternalban
Architectural decisions explained here:
[https://github.com/eBay/beam/blob/master/docs/central_log_ar...](https://github.com/eBay/beam/blob/master/docs/central_log_arch.md)

The architecture is: materialized views from a central append log.

I don't see a reference to Apache Bookkeeper[1][2] in the above. My
recommendation to anyone who wants to build their own variant of this
architecture is to build it on top of Bookkeeper instead of rolling your own.
It is designed for this precise workload [2] and addresses the write
throughput limitations of the OP 'Beam'.

[edit: pull quote from [2]]:

"what makes BookKeeper unique is its ability to offer a short-tailed, low-
latency, distributed scale-out storage solution. Although this is a CP system,
its greater availability makes it almost a C(A)P system. It is an apt storage
for immutable data."

[1]: [https://bookkeeper.apache.org/](https://bookkeeper.apache.org/)

[2]:[https://www.linux.com/news/event/apachecon/2017/4/high-
perfo...](https://www.linux.com/news/event/apachecon/2017/4/high-performance-
logging-apache-bookkeeper)

------
refset
"some functionality is lacking before Beam could be used as a critical
production data store, including deletion of facts"

It sounds like this needs a second central log to enable eviction of sensitive
data before it can be seriously considered usable. One log for transactions
and another other for the data itself. Or at least that is the conclusion we
reached when designing and building our own knowledge graph store (with a
similar use of Kafka):
[https://juxt.pro/crux/docs/index.html#_unbundled](https://juxt.pro/crux/docs/index.html#_unbundled)

------
fulafel
"Beam is designed to store large graphs that cannot fit on a single server. "

What would be a good alternative for less huge datasets?

~~~
mindcrime
Jena TDB -->
[https://jena.apache.org/documentation/tdb/index.html](https://jena.apache.org/documentation/tdb/index.html)

~~~
mtrn
We used Jena for a project with 500M triples, the uncompressed file was about
80G. It mostly works fine.

------
alexchamberlain
This is very interesting. The triple store space is crying out for a good
solution; a good _distributed_ solution - unheard of.

~~~
flarg
Is it common to combine triple store with event logging as this seems to do?

~~~
alexchamberlain
I may have misunderstood, but I think they're using the log as a write ahead
log, which is a similar, but subtly different idea to event logging.

A Write Ahead Log is an implementation detail of a distributed system, whereby
the system has a log of actions it applies against a state machine. Protocols,
such as Raft, exist to keep the log in sync, and the state machine is kept in
sync as a consequence. The log entries would be quite abstract and low-level,
like "tuple inserted".

An Event Log, on the other hand, is an application level architecture
technique. The log entries would be application-specific actions, like "user
logged in".

------
b_tterc_p
what is the best general use case open source knowledge graph, if any? The
actual graph, not the tech.

------
frou_dh
OBSERVATION: 7 out of 9 comments are bikeshedding the project's name.

~~~
bruxis
And if we include yours (and mine), we're at 9 out of 11!

------
isuckatcoding
Ah yes but not to be confused with apache beam.

~~~
Vosporos
And Erlang/OTP's BEAM. And Haskell's Beam. And probably a couple of other
things called Beam…

------
zenlot
I wonder if name is coincidence or it's on purpose to get some traffic from
searching on Erlang BEAM.

[https://en.m.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine...](https://en.m.wikipedia.org/wiki/BEAM_\(Erlang_virtual_machine\))

~~~
OrgNet
... or maybe they chose an English word to make it harder to find.

~~~
OrgNet
using two English words as the name would already be an improvement for the
common web-searcher... As an example, here is the name of two apps that do
similar things: "Blue iris" and "motion".... let me know which one that you
are able find more quickly.

~~~
zenlot
Don't like opinion or stressing too much? Relax

~~~
OrgNet
I was 100% serious... I don't think that the word BEAM helps... And I probably
need more stress, not less.

