Edit: Another amusing anecdote. I've been following this on log2viz (the heroku realtime performance monitoring tool), and was aghast to see high response times and memory usage. Then I realized I was actually looking at the dashboard for one of my rails apps! Phoenix is sitting pretty at about 35MB memory usage, and a median response time of 5ms, so far.
And some corporate networks are too restrictive to be pierced by STUN, which means you need a TURN server, which is a relay server.
But yes, 98% of the time, WebRTC can go directly browser to browser. As long as neither browser is Internet Explorer. Sigh.
This library they also have makes WebRTC cross browser compatible.
If all popular torrent sites moved to in-browser downloads over WebRTC, virtually everybody would move away from IE :)
The HEAD of a repository is like a pointer to the hash of a commit. You may think of HEAD like a repointable alias. `cat .git/refs/heads/master` in any of your git repositories to see what I mean.
Not true! With git time-travel, you can refer to a future commits in the commit message. Eg:
$ git log --oneline HEAD~2..HEAD
9428c8c I am the child
cdd3ab5 I am the parent of 9428c8
For anyone curious, this uses the same idea as the "guess-and-check" approach spullara mentioned: https://news.ycombinator.com/item?id=953569
It changes the content of the commit (by iterating the hash in the commit message itself) until the hash of the child commit matches. A big part of the trick is that it only looks at a prefix of the hash, so the search space is much smaller than the full SHA1 hash.
Will this change propagate? Has anyone yet seen this modified page in their own browser?
From the original page: "The server double checks the hash of the content it gets and then passes it along to the new person." I guess the answer is that the change won't propagate?
>and you shouldn't be trusting me any more than any other random internet person.
So you could set it up to automatically recreate the page when your machine comes back online.
is that what you meant?
Send the link with the hash using your gmail account, stay tuned at the corresponding page on http://ephemeralp2p.durazo.us, watch “Currently viewing” pop to "2" immediately after you sent the mail. Obviously, the Google borg is slurping up each and every address on the Web it is fed.
If what you said makes sense to you, ask yourself how google can crawl any URL at all, considering communicating anything to any server could trigger a destructive action.
Then whoever created the link that does that is doing web wrong and needs to stop.
GET is, by definition, a safe method. (See RFC 7231, Sec. 4.2.1; RFC 2616, Sec. 9.1.1.) Doing destructive actions via a safe method is plain wrong.
There are links that do that and if google were to follow every linkt people would notice very fast.
(Or they get fed up and patch the problem without understanding it, by using robots.txt to block all crawlers from their site.)
That's true, but the relevant feature here is safety, not idempotence. (Though a safe method is also idempotent, not all idempotent methods are safe.)
No, of the "base" HTTP/1.1 methods, all safe methods (GET, HEAD, OPTIONS, TRACE) and some unsafe methods (PUT and DELETE) are idempotent.
POST is neither safe nor idempotent (and safety is the key feature here, rather than idempotence.)
GET, HEAD, and POST are cacheable methods, which you may be confusing with idempotent methods. These are very different categories, however.
9.1.1 Safe Methods
Implementors should be aware that the software represents the user in their interactions over the Internet, and should be careful to allow the user to be aware of any actions they might take which may have an unexpected significance to themselves or others.
In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.
Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them.
I think a better title would be "This page only exists AS LONG AS someone is looking at it"
Either way, cool demo!
Another potential method to pure-JS P2P is by using the microphone and speaker and building a mesh network at inaudible frequencies but this requires microphone permissions and won't work beyond the same room, and would be only several hundred bytes per second at best.
You talk about P2P uses, but would it be feasible to 'seed' a true P2P net? A site that delivers the application, and everything from there is P2P vs. 'ask the server so that the server asks potential peers (clients?)'?
I got this at the top:
Connecting to websocket.
Listening for content for hash 2bbbf21959178ef2f935e90fc60e5b6e368d27514fe305ca7dcecc32c0134838
Received content for hash 2bbbf21959178ef2f935e90fc60e5b6e368d27514fe305ca7dcecc32c0134838
Received content for hash 2bbbf21959178ef2f935e90fc60e5b6e368d27514fe305ca7dcecc32c0134838
Standing by... ready to share this content!
Two answers for the hash? Intentional? Fine? If that happens quite a lot, you might waste more bandwidth/resources than necessary in your experiment?
Regarding (2), you should have seen my first approach: Every new client broadcasted a request for the content, and everyone with it responded! Now that was a waste of bandwidth.
But what I've done here is I send the request for content out to everyone with it, with a probability of 1/N, where N is roughly the number of people with it. So in your case, it looks like it got sent to two folks. Sometimes it gets sent to none, in which case the client will retry in 2 seconds.
It was a little tricky to figure out since phoenix runs every socket connection in its own erlang process (great for robustness and concurrency, but a real mind bender if you're not used to it). So this probabilistic approach was the best I could come up with, instead of having some master process select the right person to send the request to.
True, and a very clear way of putting it. Interestingly, since Erlang is so well-suited for distributed computing, and Phoenix (the web framework I'm using) has been built to take advantage of that, it wouldn't be too hard to let someone else spin up this same service and take part in distributing the content.
Off the top of my head, the only thing that wouldn't work is the "Currently Viewing" counter, which relies on this all running on a single heroku dyno. Otherwise, the socket messages are routed over a distributed PubSub layer, which should be pretty easy to tap into.
That would make it even more etheral.
server 1abcdef is managing 300 sockets
server 2ghijkl is managing 200 sockets
then you just add those on the frontend
In that sense there needn't be a single service. Just a bunch of rendevous points for browsers-with-content to connect and announce what they have.
The first thing that came to my mind is that this could be a great tool to fight censorship
This is a test page with embedded media via base64 encoding. This shouldn't touch the server, so hopefully no bottlenecks for this page either :)
Example: a couple of Mark Twain quotes at
With source code: https://github.com/jbt/markdown-editor
Would be super cool if the URL encoding schemes of these two sites were compatible!
OTH, I'm guessing the markdown-editor "site" cannot be searched.
Here is the result for the Google search of channelurl.com for "bird" / http://www.google.ca/cse?q=bird&cx=partner-pub-1886386989888...
Kind of fun that way too!
If the content you get sent doesn't match that fingerprint, it will reject it.
Currently Viewing: 0
Connecting to websocket.
The actual way to do this an acctualy call it p2p would be with a small WebRTC framework that can bootstrap the actual site.
Curious: If you want to have a lot of concurrent connections to your Elixir/Phoenix server, do you need to up the OS file descriptor limit?
And can you do that on Heroku?
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="author" content="">
<link rel="stylesheet" href="/css/app.css?vsn=30EBB49">
<p>Currently Viewing: <span id="visitor-count">0</span></p>
<div id="js-console" class="web-console"><ul></ul></div>
ga('create', 'UA-61291061-1', 'auto');
If you ditched the hash that checks the content and made the whole page 'content-editable' (maybe with something that stripped certain tags like images and video for obvious reasons), that might be fun. :)
The SHA-256 checksum is actually part of what makes it very interesting to me, though. Since the content of the page is guaranteed by the location of it, it's kind of a shared web that anyone can help host.
You're talking about filesharing in P2P, again, totally something you can do with WebRTC. You should take a look at PeerJS if you want to experiment in no time.
You also talked about BitTorrent in the browser: you should definitely take a look at WebTorrent
I need to re-watch that movie.
Couldn't this be done without ever sending any content to the server, via in-browser hashing and client-to-client connections? (The server could still be used to help clients discover each other.)
It would be an interesting project to try something similar with WebRTC, to allow (as I understand it) actual P2P communication.
Good job for the innovation!
This page exists only if someone is looking at it
Hi! Welcome to Ephemeral P2P. Thank you for loading this content. Your browser retrieved it from the browser of someone currently viewing this page. You're now a part of the network and someone who loads this page in the future may get it from you!
The server does not store this content anywhere, so as soon as the last person closes their browser, it's gone. You can see a count of how "healthy" the page is (how many people are viewing the content) at the top.
How does it work?
At a high level, this is what happens:
From the homepage you enter the content you want to share.
When you submit it, you register the SHA-256 hash of the content on the server.
Your browser stands by with an open websocket to the server.
When someone else visits a link "/[sha256hash]", the server tries to retrieve the content from anyone registered with that hash. The server double checks the hash of the content it gets and then passes it along to the new person.
That new person now registers with the server as someone who knows the content for that hash.
Just a simple experiment to play with websockets and concurrency.
The app is built in Elixir (compiles to erlang) with the Phoenix framework, since it supports websockets out of the box. It's very "railsy" and in addition to rails-style "controllers", it has "channels" which are like controllers for websockets. Made building this thing a snap.
The app is hosted on a heroku 1X dyno and I'm hoping this hits the front page of HN to see how many concurrent connections I can squeeze out of it. Erlang is known for its concurrency, so I'd love to know how Elixir/Phoenix can serve as an alternative to my usual rails when highly concurrent solutions are needed. I plan to tweet my findings, so you can follow me (@losvedir) if you're interested in them.
Where do we go from here?
There are two aspects to this project that I've found quite interesting, that I hope people explore:
Peer-to-peer over browser websockets
Does something like this exist? I opted for P2P of HTML injected into a container div, since I didn't want to deal with the legalities of clients sharing binary files back and forth. But someone wanting to deal with DMCA and all that might have an interesting service here.
I could see this being a great alternative to something like sendfile (I think that's a thing?), or DropBox, or what have you, when you just want to send a file to a friend and it's too big for email. Big files could even be broken up into individual SHA-256'ed pieces, and the list of SHA-256 hashes could be the thing sent. The other side would then fetch each piece in turn and re-assemble.
But that's starting to sound kind of like BitTorrent... I wonder if someone could even make a web-based bittorrent client along these lines.
Content addressed web
The cool thing about the page content being represented by its SHA-256 hash is that it doesn't matter where the content comes from. If anyone sends you the content, you can verify that it's what you were looking for. This makes it well suited for peer-to-peer or otherwise distributed file serving.
Imagine essentially an archival service where all kinds of content (web pages, mp3s, videos, etc) are indexed according to their SHA-256. Hopefully this content would be distributed around the world and backed up and all that good stuff. Then if someone tweets a "hey, checkout this video I made [a2b89..]", it can be retrieved from this "global store of all content" using that hash. It's already very common to mention the SHA-256 alongside a download. Just think if you could take that and download from this service.
Wikipedia is an amazing collection of knowledge in terms of articles. It seems like it would be valuable to have a similar nonprofit service that was a repository of "notable" files.
A quick warning
I don't do any sanitization of the shared HTML content, so be wary of other links that folks may post. But I don't think it's too great of a security risk, since there's nothing private here (no https), and you shouldn't be trusting me any more than any other random internet person.
Thanks for checking this out! Feel free to fork the repo on github and play around with it yourself!
And a big thanks to the friendly folks on the #elixir-lang IRC channel who have been very helpful in building this.
A (more) permament version of this content can be found here.
First and foremost, the community is absolutely the friendliest, most helpful group of folks.
I'm happy with its performance. Phoenix feels similar to rails to me (I'm primarily a rails dev), but with easily 10X performance. And the concurrency. Oh, the concurrency. This is Erlang's bread and butter, and Elixir and Phoenix are built to take complete advantage of this. I love that every request gets its own Erlang process, and you don't have to worry about it blocking anything else. The Erlang scheduler will keep it fair, cutting off any long running computation if it has to, to keep the latency down on new requests.
I really like how interesting and mind bending Elixir's purely functional, process-oriented approach is. Nothing (well, very little) is mutable! You can sort of simulate mutation by spinning up a little infinitely looping tail-recursive process and sending messages to it. I encourage you to go through the excellent Elixir Getting Started guide , and in particular the "Processes" section for more on this.
But I think the thing I like most about it so far, is the introduction to battle-tested Erlang approaches to robustness (OTP). This is a set of abstractions and libraries that have been iterated on over the years and are a fantastic way of building an app that's resilient to failure. Elixir, as it does, takes these abstractions and libraries, and puts enough sugar and consistency over them to make them a joy to use. I find this approach  supremely elegant (it's erlang, but applicable to elixir).
Previously I stayed away from things like Go and Node/Express, because I felt like it wasn't as easy for the type of work I do as Django, and was missing key things like the admin CRUD interface. But when I looked at what's in the Phoenix framework and read the source, it started to feel like I'm witnessing the dawn of the next Ruby on Rails revolution.
See http://c2.com/cgi/wiki?HomoiconicLanguages "Languages in which program code is represented as the language's fundamental data type are called 'homoiconic'."
The Phoenix framework is really well thought out that making CRUD applications is easy and has all the bells and whistles you'd expect from a framework like Rails. But obviously that's just the surface. You have access to OTP with erlang's processes, supervisors, ETS, etc. You have access to every erlang module that's been written. You also have access to Elixir modules that load in to a Mix project like ruby gems. Also, it's wicked fast and scales.
But maybe best is the documentation is great. From elixir's docs to Phoenix to code docs, everything is clear. The community is quick to respond on IRC or GitHub.
For downsides, just newness: the lack of good blog posts make me feel like I'm re-inventing the wheel sometimes. Additionally, there's not many questions up on StackOverflow so my errors often come up with 0 google hits.
That was rewarding :)
So, when I load the page how is it decided that which client should serve me the content?