
Scuttlebot: Peer-to-peer database, identity provider, and messaging system - pcr910303
http://scuttlebot.io/
======
Vendan
Scuttlebutt is a neat concept, burdened by a bad protocol. Signing a message
involves serializing a json object, signing it, adding the signature as a
field on that json object, and then serializing it again. To verify, you
deserialize the message into an object, remove the signature field, and then
reserialize it, and verify the signature against that new serialization. This
means that all the clients have to have a json serialization that 100% matches
the node.js serialization that the majority (totality?) of current clients
use. Any other implementation (I wrote one in Go) becomes a nightmare of
chasing rarer and rarer differences that prevent verification, and if you miss
one before you serialize your own message with it, suddenly you've poisoned
your own journal...

All in all, it's something to study and learn from, but I strongly recommend
not becoming involved unless you are 100% happy with being tied into node.js.

~~~
sneak
When I was faced with this (signing a structure), I serialized the json into
base64, then put that base64 string as a value (along with the MAC) into a new
json document. It of course increases deserialization overhead (json, verify,
unbase64, inner json) but sidesteps this issue.

I thought about sorting keys and other things like that, and the dozen edge
cases and potential malleability issues dissuaded me for the compatibility
issues mentioned above.

How have others solved it?

~~~
rakoo
Use Bencoding, like bittorrent does:
[https://en.wikipedia.org/wiki/Bencode](https://en.wikipedia.org/wiki/Bencode)

As I put in another comment, a torrent id is a hash of a map, where one of the
keys contains binary data. bencoding solved that decades ago already.

~~~
aabbcc1241
It looks similar to stackish and BON (binary object notation)
[https://github.com/bon-org/bon-
doc/blob/master/README.asciid...](https://github.com/bon-org/bon-
doc/blob/master/README.asciidoc)

BON is compatible with json+ and erlang data type, in specific, it allows any
data type for the map key. Json only allows string as map key.

------
asdkhadsj
Warning: A bit of a ramble about my experience trying to use SSB for my
"application".

I really enjoy SSB, conceptually. In reality I've had a fair amount of trouble
getting started writing an application in it. Notably, I have difficulty
knowing what patterns are good or bad in SSB.

As far as I can tell SSB utilizes effectively a commit ledger for user data.
This is fine, but some of the specifics of my application mean that I feel the
need to create many sub-identities so that revisions to a piece of content
don't bloat the parent identity when syncing to mobile/etc.

Strangely, I've got a fair bit of experience in these types of designs - as
I've written my own distributed storage for the application in question in
many different forms (various ledgers, CRDTs, etc). Yet my issue is that I'm
unsure what is considered "bad form" for the SSB crowd. I don't want to write
an application for SSB but written in some form of SSB anti-pattern.

Combine this with SSB not having well established[1] implementations in my
language of choice (Rust), and all around as much as I want to use SSB as the
foundation for my app the protocol was just a bit of a heavy burden currently.
I care what the SSB community thinks because I don't want to just "use" the
SSB protocol, I want my application to join the SSB hubs/networks and share
data.

I've looked a lot into this, but I still struggle to conceive exactly how my
application can fit into the SSB ecosystem, if at all. This isn't a complaint
post, just sharing some personal thoughts and my confusing history with SSB.

Love the project, and I especially love the design of gossip, person to person
based applications in this day and age.

[1]: Rust does have a handful of SSB libs, but I think I'd be looking for a
full, batteries included library. Everything I've seen is quite low level SSB.

~~~
moojd
I've been tinkering with SSB recently as well and I'm also finding it
difficult to figure out how to interface with the wider SSB network in the
'correct' way. The communication protocol is well documented and easy to
implement if you have libsodium available. Building applications on it should
be straight forward considering it's just an eventually consistent event
store.

I am having a few issues though. The 'api' that patchwork and other parts of
the network use on top of the communication protocol seems to not be
documented at all outside of the 'post' message. I'm not sure if it's not
documented or if I just haven't discovered it yet. For example, I'm having to
dive in to the code to see how a 'vote' message is formatted. I'm not sure how
all of the various clients are staying compatible. None of the message formats
seem to versioned in anyway either.

I'm also not sure what are the 'correct' ways to introduce my own message
types. Should I use a vendor prefix in the type or something? Is it bad form
to pollute the network with your own message types that other clients can't
understand?

I've just started diving in to all this so maybe I just haven't gotten deep
enough yet. I wrote a little toy SSB implementation last weekend to learn the
protocol. I've really enjoyed playing around with it so far but I'm still not
sure if it's the right fit for the applications I want to build yet.

------
TimJRobinson
Huge fan of Scuttlebutt and think it could (should) be the future of the
social internet. I recently quit my job to write about and work on
decentralized tech full time with scuttlebutt being my primary focus. I've
written about why I think it's so important here:
[https://adecentralizedworld.com/2020/03/what-is-
scuttlebutt/](https://adecentralizedworld.com/2020/03/what-is-scuttlebutt/)

~~~
alephnan
> Decentralized social networks have been tried before, the two most well
> known are Diaspora and Mastadon. With these services there [is still
> moderation]... Scuttlebutt is how I believe the social web should function
> in the future.

Even the Hackernews community which lean towards decentralization more so than
the general public would still argue there is value in moderation.

~~~
TimJRobinson
For sure, any network without any moderation is going to quickly fall into
chaos. I'm working on a post on decentralized personalized moderation at the
moment which should be out in the next day or two :)

~~~
moojd
I think this right here is a killer app for decentralized social networks and
is something I've been thinking about a lot. I'm looking forward to your post!

------
soapdog
Hey, this page is outdated. It is basically a list of APIs from ssb-server and
some other related plugins, but there are entries there that are old and
haven't been kept in sync with the actual modules.

[https://scuttlebutt.nz](https://scuttlebutt.nz) (nee
[https://ssb.nz](https://ssb.nz)) is a better link for all things SSB. The
protocol guide at [https://ssbc.github.io/scuttlebutt-protocol-
guide/](https://ssbc.github.io/scuttlebutt-protocol-guide/) is your friend in
understanding our little garden.

Be aware that SSB grew much like a garden. It is not a protocol and ecosystem
designed by committee with a cold and effective process. It grew from simple
stuff into more complex stuff, and yes we all understand some of the
challenges pilled upon all of us due to bad decisions in the past.

There is a lot to love in SSB. Instead of going "npm install, meh", you should
try it out. You don't need npm or nodejs to try SSB out, you can just pick any
of the clients listed in the first page I linked.

I develop one of those clients, patchfox, but it is not a full client so
you'll need to bring your own ssb-server.

------
miguelmota
Here's a getting started with scuttlebutt tutorial I wrote last year for
beginners [https://miguelmota.com/blog/getting-started-with-secure-
scut...](https://miguelmota.com/blog/getting-started-with-secure-scuttlebutt/)

~~~
myself248
There's an unstated assumption in all these things, that you already know some
people in the network. If not, you can see all sorts of interesting things,
but nobody knows you're replying to them, and it's a very lonely experience.

If there's a way around that, I haven't figured it out, and all these
tutorials aimed at people who've never heard of the network seem misplaced --
if you don't already have a personal invite, the network is useless to you,
but if you already have a personal invite, the article is useless to you.

------
dang
A thread from 2017:
[https://news.ycombinator.com/item?id=14409187](https://news.ycombinator.com/item?id=14409187)

Points to [https://scuttlebutt.nz/](https://scuttlebutt.nz/) which is related

~~~
masukomi
This HN post is linking to the protocol docs. Scuttlebutt.nz is about the
social network built on top of them. More importantly it's the most
significant thing that uses the protocol. The "apps" that use the protocol
are, in practice, just plugins that work in one particular app (somewhat geeky
app) and that particular app is primarily a client for the social network.

------
matlin
This is really cool. I dig the API and mechanisms to exchange data. I've been
working on a DBaaS (like Firebase) that works via message passing between
users and now I'm thinking Scuttlebot could be great tool to integrate with
it.

I wish there was more info on how the replication worked...

~~~
seanb
SSB Protocol Guide: [https://ssbc.github.io/scuttlebutt-protocol-
guide/](https://ssbc.github.io/scuttlebutt-protocol-guide/)

------
milansuk
So the homepage of this project is basically documentation. Where others
introduce use-cases, you actually teach us how it's done. I love it.

~~~
jariel
Except for those of us who have no idea really what it is, how it would be
used, or what the features really do.

~~~
SyrupThinker
Then you probably want [https://scuttlebutt.nz/](https://scuttlebutt.nz/)
instead, which is more of a user introduction.

------
arj
Can run directly ([https://github.com/arj03/ssb-browser-
demo](https://github.com/arj03/ssb-browser-demo)) in the browser as well.

------
jariel
Looks very cool - but to the author, you'll have to write some introductory
'what is this documentation' because the single sentence on the landing page
isn't quite enough.

"It's a magofinisticks peer-to-peer storage functionomatic" is how it might
read to some people. So what it is, what it does, the value of it, some
general overview of the features and how they work would go a long way.

Looks cool though.

------
audiodude
I've used patchwork before, but can someone comment on the other applications
of the scuttlebut network and where this scuttlebot fits in?

~~~
TimJRobinson
Scuttlebot is the core server and plugins for Scuttlebutt. The Github repo is
now called ssb-server. Patchwork uses it for replication and adds a gui and
new plugins on top of it.

------
dpc_pw
SSB is great conceptually, but is plagued with bad design decisions, and with
JS. I gave up on it, after trying it couple of times.

If someone took it, and did it right, it might be a huge hit.

------
thulecitizen
Awesome similar DWeb projects for those interested/inspired:

Ceptr/Holochain:
[http://developer.holochain.org/](http://developer.holochain.org/)

DAT protocol: [https://datprotocol.github.io/how-dat-
works/](https://datprotocol.github.io/how-dat-works/)

------
snicky
Why does the users' data have to be stored in a form of a linked list (or
"blockchain")? Couldn't every peer in the network just hold his own database
full of messages and blobs and sign them with their private key when requested
by an invited follower?

Edit: typo

~~~
capableweb
Basically Scuttlebot is a collection of linked lists (or blockchains if you
will) because you only have to discover one message to be able to fetch the
rest of the related content, since it's all linked as a DAG.

~~~
snicky
Yeah, so this is the design decision that I don't really understand. Each
message on the blockchain contains the author's public key representing his
identity. If I already know the identity of the author, I could just go ahead
and befriend him and later ask him or other peers who are his friends to share
his "feed" with me. If the feed lives in a relational DB, I could then use SQL
to ask for particular messages like:

"select * from posts order by created_at desc" or: "select * from posts order
by created_at asc where created_at > <timestamp>" or: "select * from blobs
where type = 'photo'" etc.

In other words, I could query the feeds the way I want to. Also, this allows
the author to delete the content, although this would also require to send
some DELETE requests to peers mirroring the author's feed and those requests
would have to be respected, but that's another story.

~~~
capableweb
The idea behind this is that it's streams all the way through, so there could
be something more efficient to use for these "rolling" database that basically
are streams of data. So instead of SQL, it's map-filter-reduce, a query engine
that works over streams. [https://github.com/ssbc/ssb-
query/blob/master/README.md](https://github.com/ssbc/ssb-
query/blob/master/README.md) \- [https://github.com/dominictarr/map-filter-
reduce](https://github.com/dominictarr/map-filter-reduce)

It's probably a result of going with pull-streams and basically architecture
all scuttlebot tools/libs around pull-streams, so you want something that
works over streams.

And yeah, Scuttlebot is not built with SQL as it's query language, but you
could probably easily write something that understands both SQL queries and
the file format scuttlebot stores the messages in. Thing is, you don't want to
send these queries to other peers (they are all local), because the system is
setup to be offline-first and only reach out to the network when it has to,
not just for listing latest messages.

In general, you want all the action of fetching/syncing of data to happen
exactly when you want it to happen, and then when you're done it doesn't
continue until next "sync". So everything you do is local.

But then, if "query the feeds the way I want to" is the reason you don't like
it, then so be it. But if it's just that you don't understand it, then reading
through and digging into the different repositories will give you a better
viewpoint. Start at the ssb-query and look what's using it and what it's
using.

~~~
snicky
I didn't mean to be provocative, I just wondered what's the advantage of that
particular setup. I guess your comment and the peer's explain it pretty well.
Also, thanks for the links to the repos. I'll start from there.

Is there any dev forum on ssb or other place I could catch you guys at if I
want to learn more or contribute?

------
nanomonkey
The protocol guide is quite beautiful: [https://ssbc.github.io/scuttlebutt-
protocol-guide/](https://ssbc.github.io/scuttlebutt-protocol-guide/)

------
z0mbie42
I recently discovered SSB and was really intrigued.

I would love to hear an experience report from actual users like what is
working great, and what is bad.

~~~
jekrb
It's been a while, but I used to be an active SSB user.

I hosted SSB pubs and used to post on patchwork semi-regularly.

I thought it worked pretty well as a social network. I discovered new and
interesting ideas from folks that I don't see much on mainstream social media.

I haven't followed the space much recently, and I'm curious about how it has
evolved over the last year or so.

My favorite memories on SSB:

Someone promoted a book that they had written, and we arranged for a sales
transaction by talking purely over the network. I sent them some amount of
Bitcoin, and they sent me the PDF of their book. It felt very personal to work
with the author directly, and side-step payment processors.

I loved taking my laptop out on the train or to a coffee shop, and replying to
threads and publishing a post to SSB while offline. Something about reading
other peoples ideas while disconnected, and then writing my thoughts, and
having them automatically sync to the network when I got back on my WiFi at
home, gave me a different perspective on ways to use technology.

~~~
zozbot234
> I loved taking my laptop out on the train or to a coffee shop, and replying
> to threads and publishing a post to SSB while offline. Something about
> reading other peoples ideas while disconnected, and then writing my
> thoughts, and having them automatically sync to the network when I got back
> on my WiFi at home, gave me a different perspective on ways to use
> technology.

You can do this with Usenet and most BBS's. Most native IM apps will also do
this for you, plus of course there's email.

~~~
capableweb
One cool feature of Scuttlebot is that if you and your friend are already
following each other, you only need a connection to each other P2P to be able
to send messages to each other. So if you're on a train with ad-hoc WiFi
connected to each other, you can still proceed as usual and sync stuff.

I don't think this feature exists in Usenet and BBS's where there is a central
server who masterminds the sync that everyone is doing. Same with email,
requires a server (local or remote) to send/receive stuff while in SSB both
local and remote are usually the same machine.

~~~
tlavoie
For BBSs, you're correct. Usenet (and email) used UUCP, which is actually much
closer in concept I think here.

UUCP is a store-and-forward mechanism, not dependent on a real-time connection
to a particular server. I used to run a node, connected to a guy I'd met who
worked for an ISP. He had, _gasp_ , a full-time network connection via ISDN;
pretty magical in these days of dial up.

So, Usenet feeds were configured on my own little system, essentially
subscribing to the newsgroups I wanted. Periodically, it would dial out to the
other gent, upload any new posts from me, and download anything new on those
newsgroups. My email came and went the same way. Naturally, what I got was a
subset of what he had accessible.

While I never used this functionality, I could have had others call up to me,
and I would just be an intermediate link in the chain. RFC 976
([https://tools.ietf.org/html/rfc976](https://tools.ietf.org/html/rfc976))
describes how this works for email, including SMTP over UUCP.

~~~
capableweb
Interesting, I didn't know that (BBS and Usenet was before my time), so thank
you for sharing.

That does sound a lot like how Scuttlebot treats feeds as well.

------
aabbcc1241
In addition, Manyverse is a social network built on SSB. Each user has a
chain, and they sync and relay their friends chain.

------
cachestash
npm install

sigh...

~~~
camdenlock
Yeah, this is where I lost interest too.

~~~
crispyporkbites
I don’t understand why npm install is so bad here? It’s just a package
manager. It’s not hard to write your implementation without npm if you prefer.

~~~
fabianhjr
it is, I tried with haskell and gave up when I noticed I couldn't easily get
json serialization to be in the same order as the javascript implementation
and therefor couldn't verify nor sign messages with interoperability with the
current clients.

~~~
codygman
> I tried with haskell and gave up when I noticed I couldn't easily get json
> serialization to be in the same order as the javascript implementation and
> therefor couldn't verify nor sign messages with interoperability with the
> current clients.

This gets you most of the way there:

> With the Aeson library, the order of keys in objects is undefined due to
> objects being implemented as HashMaps. To allow user-specified key orders in
> the pretty-printed JSON, encodePretty' can be configured with a comparison
> function.

\- [https://hackage.haskell.org/package/aeson-
pretty-0.8.7/docs/...](https://hackage.haskell.org/package/aeson-
pretty-0.8.7/docs/Data-Aeson-Encode-Pretty.html#g:3)

I'm not sure that messes with spacing or not though or you require specific
space preserving requirements to interoperate with current clients.

Even if not you can probably look at the source of encodePretty' and work out
how to not mess with spacing or keep it all on one line.

I've wanted a version of Aeson for a while that preserves order and all
formatting for things like linters that are very unobtrusive. It seems your
use case would have benefitted from that as well.

~~~
fabianhjr
Well, the problem is replicating the javascript order which isn't
lexicographical. :/

~~~
codygman
This gives you the ability to arbitrarily order via user defined list (the
first argument) and then to order by some comparison function if for instance
a new key were added (length in this example).

Given the example:

    
    
        {
          "baz": ...,
          "bar": ...,
          "foo": ...,
          "quux": ...,
        }
    

Then listing all of the keys in your desired order:

    
    
        comp :: Text -> Text -> Ordering
        comp = keyOrder ["foo","bar","baz","qux"] `mappend` comparing length
    

Note if you list all keys in the first argument of keyOrder it never goes to
comparing length. There is probably a cleaner way to denote that this is the
case but I can't be bothered to figure it out atm.

You can use this function:

    
    
        encodePretty' defConfig { confCompare = comp } YourType
    

And it will produce:

    
    
        {
          "foo": ...,
          "bar": ...,
          "baz": ...,
          "quux": ...,
        }
    

Can you give a sample of the javascript in the order necessary? This is from
NPM? They have to use some comparison function to figure out how to sort their
keys I'd imagine, so you could just copy that with Haskell.

------
magwa101
Are these new systems created just to get a fun name out there?

~~~
kseistrup
Scuttlebot is a pun on scuttlebutt, which is the cask used to serve water on a
ship, and the protocol was invented by a guy who lives on a boat.

Think of it as “water cooler”: a place where people can meet and smalltalk.

------
seemslegit
Uh, for something touting e2e encryption and security it would be better if
the site did not serve over plain http by default

~~~
progre
https adds nothing to a page when the only trafic is server -> client

~~~
seemslegit
Of course it does, HTTP is never only server -> client - from preventing a
passive eavesdropper from seeing what pages are being browsed on the server,
cookies, UA fingerprinting etc. to active content modification in transit.

~~~
progre
Pages are still visable in the tls handshake, no coockies on this page (that
would be client-> server traffic). But yeah, good point about the
fingerprinting and content modification

~~~
seemslegit
No, they aren't - just the hostname (domain name).

