Excited that you found PeerPad so fast. We've released its alfa just a week ago at Mozfest (https://github.com/MozillaFoundation/mozfest-program-2017/is...).
I see that there are some questions about PeerPad's architecture. If you would like to know how it works, I invite you to glance through the Architecture.md at https://github.com/ipfs-shipyard/peerpad/blob/master/docs/AR... If there are things that are not clear to you please open an issue so that we can make it better.
Also, you can expect to see soon a video tutorial of how to build a p2p flipchart using the same underlying kv store that PeerPad uses, y.js + IPFS. If you can't wait you can always take a look at https://github.com/ipfs-shipyard/p2p-flipchart and see how it works :)
We will continue working on PeerPad and add the remaining features necessary to make it usable for meeting notes, publishing and more. You can track the development and also help us built it through the Github Repo https://github.com/ipfs-shipyard/peerpad
Thank you \o/
> open https://ipfs.io$(ipfs resolve /ipns/peerpad.net)
(Or perhaps it uses one of the gateways?)
How do they prevent the data from disappearing if everyone disconnects?
> How do they prevent the data from disappearing if everyone disconnects?
Right now, we're not preventing that. You'll have to be online to share the content (or have someone else pin the content).
We do have a open issue for being able to select a pinning service, which would solve this issue: https://github.com/ipfs-shipyard/peerpad/issues/90
It uses WebRTC for peer-to-peer communication, which is yet unsupported / untested in some browsers.
To answer your question:
The collaborative data is saved locally by every participating peer.
There are plans to add remote tracking and pinning, increasing the persistence and availability guarantees:
If I understand correctly, each keystroke in the pad generates a new snapshot, which has a distinct IPFS ID.
Should the remote pinning service publish a list of all IPFS IDs on the p2p network, and store each individual document (each one keystroke away from the previous one) in the history graph?
Should it keep the complete list of CRDT operations, and map IPFS IDs to CRDT operations IDs, so that it can reconstruct any document when it is requested?
Is there a way to use the properties of CRDT that I am not seeing?
Each keystroke produces a change in the CRDT, which is then eventually propagated to all participating nodes.
Each CRDT message is signed and encrypted before being sent over the IPFS pubsub network.
This protocol is not IPFS-specific, but there are plans to change this:
The snapshotting occurs over IPFS, producing a static and encrypted self-contained snapshot, published over IPFS.
It is possible to access the CRDT properties inside the core library, where the CRDT is formed: https://github.com/ipfs-shipyard/peerpad-core/blob/master/sr... . This could be exposed if you would require it..
Nevermind. I was testing quickly in the airport between flights. Now I see I just had a flaky connection.
Usually I'm pretty skeptical of most systems, but I'm rather particularly impressed with yours. I also do not think you are getting the recognition you deserve, because the truly novel work here is Y-JS not IPFS (even though is IPFS is great), yet Y-JS is almost just a footnote on the page which is sad.
The first couple documents (on architecture and stuff) didn't seem to explore the CRDT that is used. I'll have to follow-through on other sub-documents/pages to read more, but wanted to get your thoughts first (and BTW, thanks for all the documentation you have already provided, it is impressive, and I know how hard it is to try and get everything shipped, so the last thing I want to be is an annoying HN person whining about why you didn't already have X or Y or Z finished, and wanted to give you a chance to reply directly).
I was able to rather quickly ( https://www.dropbox.com/s/8uxzq6d47szu4w1/Screenshot%202017-... ) get the document to go out of sync, but I understand this could just be a trivial implementation detail (and often is) NOT a problem with the CRDT, but wanted to check. Given your bachelor thesis, I know you are all too aware of these things, P2P collaborative rich-text is one of the hardest problems out there in the space. Your Y-JS system seems fairly "generalizable", no? Which is unique amongst CRDTs (ours at gun is as well, so I get the nuance). If that is true, usually P2P rich-text then requires a specialized CRDT on top of the generalizable one, is this what you all did?
For instance, Martin Kleppmann did an excellent explanation of explaining theirs here: https://youtu.be/yCcWpzY8dIA?t=16m26s . I was wondering if you did something similar?
We had to implement our own, which I did one of my infamous "cartoon explainers" on here: http://gun.js.org/explainers/school/class.html . However Martin actually brings up an edge case I hadn't thought of after 4 years of analyzing the system! But thankfully he provides a patch on how to fix it. ;)
Anyways, wanted to again repeat I'm very impressed with Y-JS (and say that it is not getting the attention it deserves for this being on top of HN, given that it is what is novel, not the other stuff). But I also wanted to inquire more about the generalizable-ness of Y-JS and then hear about any specific specialized-CRDT you may be using for P2P rich text, because it looks like it could be a breakthrough! Would love to hear more! Excellent work.
Congratz on developing Gun! I've looked at it before and I was super impressed! The decision to use Y.js instead was done due to its modularity. We added IPFS support by simply publishing a connector https://github.com/ipfs-shipyard/y-ipfs-connector which Kevin Jahns later added to the list of connectors - https://github.com/y-js/yjs/issues/77
We have linked Y.js from the page header, if you click the word CRDT it takes you to Y.js directly. Nevertheless, I agree that we need to make it more obvious How we appreciate all the work that Kevin has put into Y.js :) Do not worry, PeerPad nor our CRDT usage is finished, there will be a lot more! (In fact, PeerPad was built with little less than 4 weeks of actual dev time, there is so much more we can do).
I'm following up by email, I'm interested in testing Gun as well. Meanwhile, join the CRDT discussion on the Research CRDT (https://github.com/ipfs/research-crdt) Repo, would love to have your input there.
Thanks for your interest!
Off the CRDT libraries we analysed, Y.js was the one we picked up as it very modular, so that we could create our own connector and database layer.
These are all open-source: https://github.com/ipfs-shipyard/y-ipfs-connector and the encryption layer wrapping the database adaptor: https://github.com/pgte/y-indexeddb-encrypted/tree/encrypted
About the richtextdetails, the CRDT is based off of Quill.js deltas (https://github.com/quilljs/delta), composed by an array of such operations.
Anyway, this is all interim work, and we plan on making this independent of any specific library by implementing a generic CRDT on top of IPFS DAG and Pubsub APIs. If you're interested you can follow / chime in here: https://github.com/ipfs/research-CRDT/issues/11
I'm going to guess this is to establish the connections between peers. So this isn't really p2p because while there aren't any intermediaries for data transfer, there certainly are to establish the data connections.
If you want a p2p system because you care mostly about removing servers as a bottleneck, then this may not be an issue, but if you care about removing servers because you don't trust intermediaries, then this only gets you half way there.
This is, actually, truly peer-to-peer. Those are connections to the bootstrap nodes, you can set your own bootstrap nodes if you want but you do need to be seeded with a few initial peers to find new peers. If you want to establish a browser-to-browser connection, you will need some server to help setup the WebRTC connection; this is the unfortunate reality of browsers. However, you can (theoretically^) connect to arbitrary non-browser IPFS nodes over a websocket connection (that's what's happening here) without some "blessed" node acting as the rendezvous.
^Unfortunately, there's a bit of a wrinkle:
1. Most non-browser IPFS nodes don't listen on websockets by default. The transport is still experimental and and has some bugs.
2. If you load IPFS from an https origin, browsers won't generally allow you to establish connections to non-https websocket endpoints. Unfortunately, even if you enable listening on the websocket transport on your IPFS node, you'll end up listening on an http (not https) websocket. This is because IPFS manages the encryption of the connection itself (IPFS addresses, or "peer IDs", are cryptographic hashes of public keys) and doesn't use (or play well with) the CA system. To work around this, the bootstrap nodes have nginx proxies out in front that to handle HTTPs connections. Most (all other?) nodes don't.
But, it doesn't work in all scenarios. In the go-ipfs version of ipfs (or running js-ipfs) we can use mDNS for local discovery without any bootstraps, but unfortunately doesn't work in the browser (yet! FlyWeb would help https://github.com/ipfs/in-web-browsers/issues/45). Some sort of gossiping could work as well but we're always glad to hear other ideas of how we can make discovery work without having to hardcoded bootstrap nodes or signaling servers.