
Websockets 101 - the_mitsuhiko
http://lucumr.pocoo.org/2012/9/24/websockets-101/
======
peterwwillis
Encryption is not a panacea. As more websites use HTTPS, more corporate
networks are installing content filters that passively filter HTTPS content.

Yes. I just said corporate networks are spying on HTTPS connections.

It's really simple to implement. Get some kind of web proxy that has support
(Websense is pretty popular, amazingly) and generate a root cert on the box.
Then use a group policy on your AD server to distribute the root cert to all
the client systems. Now Websense can decode the HTTPS traffic by issuing its
own fake certs to clients and handle the "real" HTTPS handshake on the
frontend proxy side.

Result? Your websockets are still going to get fucked with by proxies. Can we
please stop building fake protocols on top of real protocols now?

~~~
kibwen
Interesting, I actually hadn't considered the possibility of an employer
issuing fake certificates to peek at its employees' encrypted sessions.

For anyone unnerved by this, you can still get around it by routing your
traffic via SOCKS proxy to an external host, but of course this assumes that
1) your employer isn't blocking SSH, 2) you have the authority to install
PuTTY and Firefox (not sure if Chrome has SOCKS proxy support) and 3) you have
an external server to proxy through (if you have a Linux machine somewhere,
you're probably already configured to allow this). And make sure you route
your DNS through SOCKS as well.

~~~
peterwwillis
If they're proxying HTTP traffic usually they'll block SSH and other
protocols. To work around this use an HTTPS/SSH multiplexer[1][2][3] or a
simple reverse proxy to accept HTTP[S] connections and connect them to an SSH
server on the backend. For the ssh client use a ProxyCommand tunneling
app[4][5][6] to turn an HTTP[S] proxy request into a two-way socket, and you
have an SSH connection over HTTP proxies.

[1] <http://www.rutschle.net/tech/sslh.shtml> [2] <http://www.pond-
weed.com/multiplex/> [3] <https://github.com/stealth/sshttp> [4]
[http://zwitterion.org/software/ssh-https-tunnel/ssh-https-
tu...](http://zwitterion.org/software/ssh-https-tunnel/ssh-https-tunnel) [5]
<http://proxytunnel.sourceforge.net/> [6] <http://www.agroman.net/corkscrew/>

------
tinco
Very nice readable article. Having dabbled with implementation a bit I agree
with most of his points.

The author complains about websockets being frame-based instead of stream-
based, because he argues if he would want a frame-based system he could easily
model it on top of it.

I think frames are awesome. It is easier to build a stream on top of frames
than it is to build frames on top of a stream in my opinion (you don't need an
extra protocol for starters). Besides that almost anything you would like to
do can be expressed in frames, saving most of us a lot of headaches.

I would like to add that websockets are not the end-all answer for web-game
programmers. It makes some stuff easier, and reduces some overhead, but it's
still on TCP, meaning realtime multiplayer games are still as good as
impossible to implement.

Sadly all the issues with proxies go for UDP and then some, so I don't think
we'll see any ubiquitous browser UDP protocol anytime soon :(

~~~
the_mitsuhiko
> I think frames are awesome. It is easier to build a stream on top of frames
> than it is to build frames on top of a stream in my opinion (you don't need
> an extra protocol for starters). Besides that almost anything you would like
> to do can be expressed in frames, saving most of us a lot of headaches.

Maybe I'm just old fashioned but I really don't like the idea of frames forced
onto me. I much rather get my data in small pieces and can already start
processing data as it comes in than having something buffer up everything and
then giving it me as a large chunk. Streaming json for instance over
websockets is a lot harder than TCP directly.

~~~
ishbits
IMO the built in framing is one of the things I really like about websockets.
I'm sure it will become my transport of choice for non performance critical
messaging even when a browser is not one of the peers.

------
btmorex
If anyone finds themselves needing to write a Websockets implementation,
there's an awesome protocol test suite at <http://autobahn.ws/testsuite>

I wrote a C++ implementation for a side project and with the aforementioned
test suite I actually found it pretty easy to get to 100% compliance. There's
some ugliness in the protocol because of proxies, but it's definitely not the
worst protocol in the world. The only big missing feature is compression and
there's a proposal for that (you could certainly do application level
compression, but I'd rather avoid writing compression code in JS).

------
jorangreef
Browser vendors must start to move away from top-down innovation, where they
hoard APIs such as TCP and UDP and release sub-standard specs in their stead.
Instead they must expose only the bare minimum OS low-level APIs (TCP, UDP,
POSIX), keeping the surface area as small and powerful and direct as possible,
and then let open source grow around this. Innovation needs to be
decentralized, bottom-up. Not designed by committee.

For that to happen, we need to stop conflating "web apps" (trusted, installed,
annointed with machine power) with "web pages" (accessed by single link
click). At present, Web Apps are suffering and crippled by being lumped under
the same security policy as web pages. But Web Apps need to have access to raw
machine resources in the same way that Native Apps have access.

Those that don't seem to care for any of this insistence, tend also to be
naive as to the massive differences between TCP and WebSockets, and IP and TCP
and the whole stack in general. The WebSocket spec is a good example of people
doing things in the most indirect way possible, with a maximum of red tape, as
opposed to people doing things in the most direct way possible, with a minimum
of red tape.

The Web as we have it in these respects is very much Animal Farm and 1984.
There appears to be little thought leadership from the major stakeholders in
this regard. People like Tim Berners-Lee are asking for change
([http://lists.w3.org/Archives/Public/public-
webapps/2012JanMa...](http://lists.w3.org/Archives/Public/public-
webapps/2012JanMar/0464.html)), but the new incumbents don't seem to want to
see.

~~~
milkshakes
I don't understand the point you're trying to make. Are you against
standardization? What specific actions would you take to "minimize red tape"?
You do know that TCP, UDP, and POSIX were all developed by committees right?
How exactly is the Web in any way like Animal Farm or 1984?

~~~
jorangreef
Standardisation for the web targets too much surface area. Browser vendors
need to agree on a smaller set of more powerful, more dangerous, more direct
APIs so as to put the responsibility for innovation back in the hands of
developers. Developers are being wedged further and further away from bare
metal, with no right to return, even if they should so wish - this process
needs to be reversed.

~~~
aapl
Raw TCP was considered for WebSockets, however briefly. Due to security
reasons, WebSocket was designed so that it's impossible for a WebSocket client
to spoof any existing protocol.

I would argue that things are actually swinging to the opposite direction
right now. WebGL is very close to OpenGL ES 2.0, Web Audio API includes low
level features, WebRTC 1.0 is slated to include datagram channels... Recent
standards efforts are striking pretty good balance of flexibility,
interoperability and security, IMHO.

~~~
jorangreef
I am referring to networking and storage, not graphics or audio.

At first glance it looks like the top-down massive surface area standards
approach is making progress. But the APIs are really sub-par when compared to
TCP, UDP, POSIX and what open source could do if given the chance to grow
around them.

The security reasons keeping raw TCP out of the browser apply to web pages,
not web apps (trusted, installed, annointed with raw machine power to act on
the user's behalf). This means that it is not possible to build a
SMTP/POP/IMAP etc. client in the browser without the use of a third-party
proxy (which introduces additional security concerns). This is a terrible blow
to web apps. WebSockets are a diversion.

WebRTC is a herculean effort. UDP should be beneath WebRTC not bundled
alongside it. Again, web page security issues have been projected onto web
apps, hence no directly exposed UDP.

There is no decent offline storage mechanism in the browser. No POSIX. No
fsync. No way to build any kind of performant database. IndexedDB is the only
thing available. It is poorly designed to begin with and implementations at
present are slow, buggy or lacking. It looks good in the to-do list demo's but
beyond that is a pain to work with. I have seen Chrome's IndexedDB reboot
Windows over and over again on occasion. All the great open source database
implementations are locked out. Everyone is forced to grow over IndexedDB.
Much better if LevelDB were directly exposed. But POSIX is what is needed.

The major issue going forward with regard to web apps, is that they need to be
seen as distinct from web pages, and when installed by the user, given access
to the last 40 years of computing progress.

------
jschrf
Excellent article, thank you for sharing.

As someone who has implemented a WS server and client and would like to be
able to host them in Amazon's cloud, the notes about ELB are worrysome.

~~~
gabrtv
Websockets through ELB works fine if you use the TCP listener. You lose out on
some HTTP-specific features like Cookie-based session stickiness, but it works
great otherwise. We've been using Websockets/ELB in production for almost a
year now.

------
romaniv
Why do standards like WebSockets don't include anything that can be used in
pure HTML? This would be a great tool for building efficient dynamic apps, if
you could "submit" forms without going to a new page, and then get incremental
page updates as a response.

~~~
xionon
I'm not sure I see the use-case there as being more useful than JS based
WebSockets. Writing a general JS library to do what you suggest would be
pretty straightforward, and the server-side work wouldn't be any different
(serving partial HTML based on request type).

If your users are worried about security / performance enough to turn off
JavaScript, they probably wouldn't want "HTMLSockets" enabled, either.

If your users are browsing with a browser so old it doesn't have a reasonable
JavaScript engine, then it wouldn't support "HTMLSockets," either.

Basically, there's nothing you could do with "HTMLSockets" that you can't do
with WebSockets, and with only a small layer to make it so. However, there's a
whole host of utility that you get from WebSockets that you could never get
from HTMLSockets, or would require a huge amount of hacking in order to get it
to work.

And all this is beside the "pureness" argument, where one might say that HTML
is for data, CSS is for design, JavaScript is for behavior; WebSockets are
behavior, not data.

~~~
romaniv
This is like asking why we should bother having any (new?) HTML or CSS
standards at all, when you could write a custom rendering engine in JavaScript
for every website.

There is tremendous value in having a solid, usable foundation for web
development that doesn't rely on an imperative language to work. To start
with, declarative features are _semantic_ and allow for semantic upgrades on
wide-scale basis. Do you like your Firefox/Chrome spell-checking in text
areas? Do you like them being resizable? Well, this wouldn't be possible if
all textareas were some JavaScript hack. (Hey, with only small layer you
_could_ actually fake a textarea. Does that seem like a good idea in
retrospect, though?)

Also, it would be great to be able to develop simple dynamic web applications
(or prototypes) without doing something complex. Progressive enhancement is
buried far too early.

 _If your users are worried about security / performance enough to turn off
JavaScript, they probably wouldn't want "HTMLSockets" enabled, either._

I frequently browse with JavaScript blocked by default, and I can tell you
right here that this assumption is invalid. There is a world of difference
between JavaScript library and a standard, declarative technology. The latter
is more likely to be faster, have less bugs, no side-effects and be secure.

------
zafriedman
An even quicker introduction to the final specification of WebSockets:
<http://socket.io>.

P.S. I'm aware that a) this doesn't help anyone not using Node.js on their
server (it's not even part of my production stack at work (yet!), even though
I'm bringing it to light here) and b) it's more than just WebSockets, for
instance it will gracefully degrade on legacy browsers.

I just can't miss an opportunity to sing its praises because it has so many
benefits over the simple implementation.

~~~
chrisduesing
It is not actually limited to Node.
<https://github.com/learnboost/socket.io/wiki>

Though some of these libs do tend to lag in version support.

------
TazeTSchnitzel
(Shameless plug...)

I have a list of websocket libraries and a small echo server demo here:
<http://ajf.me/websocket/#libs>

~~~
kibwen
That first link there to your Github repo is broken:
<https://github.com/TazeTSchnitzel/WebSocket.us>

~~~
TazeTSchnitzel
Oops. I moved it to my main website's repo:
[https://github.com/TazeTSchnitzel/ajf.me/tree/master/htdocs/...](https://github.com/TazeTSchnitzel/ajf.me/tree/master/htdocs/websocket)

------
rizwan
For folks thinking about using Socket.IO for websockets, it works very well,
but remember that most open-source implementations _ONLY_ implement WebSockets
and _NONE_ of the other fall-backs that are in the Socket.IO spec. So without
some work, these libraries are often useless for mobile development, for
instance.

~~~
digisth
I'm using SockJS (client and server: <https://github.com/sockjs/sockjs-
client>) for project I'm working on, which does implement a variety of
fallback methods (iframe tricks, JSON polling, etc.) and it's been working
pretty well (some work better than others.)

For those who to opt for these libs, I recommend the following for
protocols/fallbacks:

protocols_whitelist: ['websocket', 'xhr-streaming', 'iframe-eventsource',
'iframe-htmlfile', 'xhr-polling', 'iframe-xhr-polling']

This gets you a working system in pretty much every modern browser supporting
WS, and /usually/ works pretty well in IE8/9. Just try to avoid page refreshes
in those two browsers (very spotty results if you don't - I'd recommend
putting up a message for your users) and implement the workarounds for finicky
browsers (things like catching ESC in Firefox.) I've been pretty happy with it
so far.

------
chadrs
Man, I would have loved to have this about a year ago. Although then I would
have missed out on some of the hilarious drama on the websocket mailing list
(e.g. [http://www.ietf.org/mail-
archive/web/hybi/current/msg08668.h...](http://www.ietf.org/mail-
archive/web/hybi/current/msg08668.html))

------
kaiserama
Pop-up Video Excerpt:

The author (Armin Ronacher) also authored Flask (Python web micro framework)
and its template engine Jinja2. I've used Flask for a number of my most recent
projects and really have enjoyed working with it.

Back to your scheduled broadcast!

------
bowmessage
Loving the article! Just thought I'd point out: "needles complexity"

------
Evbn
> Messages sent from the client to the server are obfuscated with a basic
> transmission mask of 4 bytes which is sent within each websocket package.

What is this? Some pretend security feature?

~~~
pjscott
The purpose of the masking is to prevent malicious applications from being
able to send arbitrary bytes over the wire, for fear that they may be able to
screw with buggy proxies.

