
How the Dat Protocol Works - nanomonkey
https://datprotocol.github.io/how-dat-works/
======
tln
The diagrams on this page are FANTASTIC.

The dat homepage[1] is also nicely done. It would be interesting to see how
much and how it's being used. Since there is a central tracker does anyone
know if there are stats available?

[1]: [https://dat.foundation/](https://dat.foundation/)

~~~
social_quotient
Any specific tooling generating these or just a good designer?

~~~
nanomonkey
Duncan Keall is just talented. He also did the Scuttlebutt Protocol guide
([https://ssbc.github.io/scuttlebutt-protocol-
guide/](https://ssbc.github.io/scuttlebutt-protocol-guide/)).

Here is a blog post describing how it was made:
[https://blog.datproject.org/2019/01/21/how-dat-
works/](https://blog.datproject.org/2019/01/21/how-dat-works/)

------
mmcclure
Dat (and Beaker Browser) is one of those things that I think about playing
with every time I see it pop up somewhere. The technology truly feels like
something _special_.

Then I sit down to actually start playing with it I struggle to think of
anything to build that would actually take advantage of the tech aside from
purely static sites. I would really love to see more complex examples folks
have seen in the wild.

~~~
daffl
A while ago I made a demo of a P2P chat application that runs on the beaker
browser and uses the experimental P2P APIs. It's running at dat://feathers-
chat.hashbase.io (slides from the talk at [https://vanjs-
decentralized.hashbase.io](https://vanjs-decentralized.hashbase.io)). If the
peers can see each other you should be able to register with any email address
and send chat messages between browser instances.

What I realized is that this technology ticks a lot of the boxes of what we
right now think only the big cloud providers can do. By using a more
decentralized protocol it is by design

\- Actually Serverless

\- Offline-first

\- Real-time

\- Auto-deploying

\- Live-updating

\- 100% uptime

I really think there is something there from both, a developer and user
experience perspective. The problem is that a lot of it is still very
experimental and far from the usability and maturity of e.g. a Firebase or
Heroku.

~~~
mmcclure
This is a really cool idea, but how should I be using it? Trying to test it
out just now I opened up two tabs in Beaker but both rooms just have the one
user in them.

Love the design, by the way.

~~~
daffl
I believe this is happening because it is using the same peer connection. If
you start to separate instances of the browser (e.g. in a VM or another
machine) you should be able to see both users.

The design is taken from the Chat guide for
[https://feathersjs.com](https://feathersjs.com). Feathers is a JS library
that allows to architect APIs in a way that they are protocol independent.
Which worked great in this case because I just had to swap out the existing
REST/websocket Feathers adapter for a DAT/Beaker API Feathers adapter.

------
nanomonkey
Also see the Scuttlebutt Protocol guide: [https://ssbc.github.io/scuttlebutt-
protocol-guide/](https://ssbc.github.io/scuttlebutt-protocol-guide/)

Discuss:
[https://news.ycombinator.com/item?id=20364154](https://news.ycombinator.com/item?id=20364154)

------
kamfc
As a tech illiterate, what makes DAT superior to the standard p2p and
blockchain technology? If my understanding is wrong, can someone explain in
laymen term why DAT matters for the future?

~~~
ppm34
Say you have a 1 GB file that you want to share with everybody in this thread.
How do you do this? What if you have thousands of such files? In theory, DAT
and similar DHT-based file sharing protocols, can enable us to implement
something like YouTube, but without the enormous costs for running the
datacenters. In practice, ISPs won't let this happen and DAT-like protocols
will be limited to the world of VPSes that have static IP addresses.

~~~
macawfish
My hope is that asynchronous swarm based applications ("dweb") will work so
well without ISPs (e.g. on dynamic local networks and mesh networks) that they
will pose an existential threat to ISPs.

"Think WAN, act LAN"

or is it

"Think Wan, net LAN"?

I don't know.

------
elamje
Genuine question to hopefully help others as well: What are the major
differences with dat and IPFS?

~~~
guessmyname
> _What are the major differences with dat and IPFS?_

• [https://docs.datproject.org/docs/faq#how-is-dat-different-
th...](https://docs.datproject.org/docs/faq#how-is-dat-different-than-ipfs)

•
[https://github.com/ipfs/faq/issues/119](https://github.com/ipfs/faq/issues/119)

------
noodlesUK
DAT seems very well thought through, and having never seen beaker before that
looks amazing! However, how private is a P2P service like this? One major
problem for BitTorrent is how trivial it is to snoop other peoples history
(esp if you’re the RIAA or MPAA, or even iknowwhatyoudownload). Is overcoming
this a design goal for dat? Is it even possible to overcome it?

~~~
vjeux
There's some information about this in the protocol description:

"Discovery keys are used for finding other peers who are interested in the
same Dat as you.

If you know a Dat’s public key then you can calculate the discovery key
easily, however if you only know a discovery key you cannot work backwards to
find the corresponding public key. This prevents eavesdroppers learning of Dat
URLs (and therefore being able to read their contents) by observing network
traffic.

However eavesdroppers can confirm that peers are talking about a specific Dat
and read all communications between those peers if they know its public key
already. Eavesdroppers who do not know the public key can still get an idea of
how many Dats are popular on the network, their approximate sizes, which IP
addresses are interested in them and potentially the IP address of the creator
by observing handshakes, traffic timing and volumes. Dat makes no attempt to
hide IP addresses."

~~~
brokenmachine
So as I understand it, if there was a popular movie shared via dat and people
were sharing it (the key would have to be publicly known for people to
discover it at all), it would be trivial for the MPAA to snoop and see what
IPs were downloading it, just like torrents now.

Is that correct? Would those snoopers be able to prove who is sharing and who
is downloading?

------
guiomie
The first thing that popped to mind for me was that this seemed like a good
protocol to build some sort of decentralized/private social network, then in
their docs I managed to end up looking at this:
[https://github.com/Rotonde/rotonde-
client](https://github.com/Rotonde/rotonde-client)

------
eternalban
"Hypercore assumes a linear history. It can not support branches in the
history (multiple entries with the same sequence number) and will fail to
replicate when a branch occurs. This means that applications must be extremely
careful about ensuring correctness. Users can not copy private keys between
devices or processes without strict coordination between them, otherwise they
will generate branches and 'corrupt' the hypercore."

[https://www.datprotocol.com/deps/0002-hypercore/#the-
linear-...](https://www.datprotocol.com/deps/0002-hypercore/#the-linear-
history-requirement)

------
miguelmota
That's a beautiful guide. Anyone know how the diagrams were created?

------
miki123211
The accessibility of that guide with a screen reader is horrible. The text is
readable, but that's about it. Why, oh why couldn't people just write good,
textual articles like they used to, instead of all this picture based crap?
Instagram, Tinder and snapchat are bad anough as they are, but tech was mostly
free of this, but then we see stuff like this appearing. I won't really be
surprised if I will not be able to read 50, 60% of the content in 10 years if
current trends continue.

~~~
macawfish
What would help you screen read a visually oriented guide? Would detailed
textual descriptions of the figures help?

Personally I think that the way they're using diagrams to illustrate some of
these technical concepts is really helpful. I wonder if the format would
translate to spoken text? I.e. in parallel with the figures, having little
descriptive "imagination breaks" in the text which would provide opportunity
for focusing on some key concept or relationship that might otherwise be
illustrated using a visual figure. Maybe even using spatial language?

IMO there should be way more aurally oriented technical materials.

Recently I was listening to math lectures on YouTube while driving and
realized that they were much easier to follow than I expected. Still, I'd love
it if there was an aurally focused higher math practice. Sometimes I find it
easier to focus when I'm listening rather than using my eyes.

~~~
miki123211
In this case, not really descriptions, but equivalents. I don't care that a
figure shows two red boxes connected in a particular way, I care that packets
flow from node x to y, but not in the other direction (just an example).

------
cmroanirgo
I noticed the other day _GNUnet_ [0] here on HN, and it seems to me that there
could be a bit of overlap here, especially for the filesharing system... Is
there any collaboration happening between these two systems? From my couch
perspective, they both look great and seem as though they could compliment
each other really well.

[0]
[https://gnunet.org/en/#gnunet-0.11.5-release](https://gnunet.org/en/#gnunet-0.11.5-release)

------
IshKebab
They really need to work on a better explanation of what exactly this is!
Something like "It's similar to BitTorrent except..."

~~~
brokenmachine
[https://docs.datproject.org/docs/faq#how-is-dat-different-
th...](https://docs.datproject.org/docs/faq#how-is-dat-different-than-
academic-torrents-or-bittorrent)

>Academic Torrents [13] uses BitTorrent to share scientific datasets, and
BitTorrent has many drawbacks that hinder direct use by scientists. BitTorrent
is for sharing static files, that is, files that do not change over time. Dat,
on the other hand, has the ability to update and sync files over the peer-to-
peer network. BitTorrent is also inefficient at providing random access to
data in larger datasets, which is crucial for those who want to get only a
piece of a large dataset. BitTorrent comes close to the solution, but we have
been able to build something that is more efficient and better designed for
the data sharing use case.

~~~
IshKebab
Much better. They should stick that on the home page, not in the FAQ.

------
mongol
Is this similar to Syncthing?

~~~
macawfish
It's kinda like syncthing crossed with git. Lower level than syncthing,
focused on stuff like p2p shareable, versioned data structures and
cryptographic verification.

Although you can build applications on top of syncthing's API/protocol, it's
primarily a high level application, whereas Dat is a protocol meant for
building applications.

It would make sense to build something like Syncthing on top of Dat.

------
ryanmarsh
I didn't see IPFS mentioned on the page so I was wondering if someone here
could help me understand the difference between this and IPFS which seems to
share at least some of the same goals.

~~~
rtkwe
IPFS as I understand it doesn't have a built-in ability to alter files once
uploaded and has to use IPNS to handle pointing to newer versions over time.
Dat Protocol does this for you as I understand it haven't dug in too much.

------
torified
Dat seems like it could be a good way for those infosec people and others that
don't like youtube to share their videos.

Is there any way to stream video using OBS via dat?

------
IceDane
I'm sorry, but accessibility and UX is a primary focus, and then you explain
that urls consist of a gigantic hex string?

I can't share a file without having to copy paste the url, and I can't look at
two similar urls and immediately determine which is which. No thanks.

This seems like yet another product that seems good on paper, and seems great
if you're a nerd, but this will never catch on with the average consumer.

~~~
macawfish
The keys are akin to IP addresses, which are also not so human friendly.

People are still trying to wrap their heads around decentralized naming
systems. There are lots of ideas, and nothing has stuck yet in a
groundbreaking way.

But yeah, the naming problem is independent of what this protocol gives us.

~~~
rtkwe
I think the best idea for naming is probably going to be something like giphy
or what3words (though their implementation could use work since they're not
directly translatable between multiple languages) where a hex address gets
encoded in N words which are generally easier for people to grok than N*m (m
being 2/4/8 depending on how many hex characters are encoded in which words).
Choosing the words is hard though especially as you add more languages.

------
torified
Can you share on dat via tor?

------
deevolution
is this different from ipfs?

~~~
brokenmachine
Yes.

