I was just looking into what SDCH is (an accept-encoding option from Chrome) and it sounds very, very similar: It generates a dictionary and then uses VCDIFF between requests. Is this related somehow?
Vaguely. Both Railgun and SDHC work by compressing web pages against an external dictionary. In SDHC the dictionary must be generated (somehow), and it is intended for use between a web server and browser. Railgun is back-end for our network and automatically generates dictionaries.
Is anyone aware of a performance analysis between SDCH and one of the dynamic compressions like deflate?
I google, but all I find is people complaining their proxy/filter/appliance/diagnostic is breaking because it doesn't understand SDCH.
It seems like SDCH has been around for 4 years, I presume the lack of data means it hasn't worked out.
(I imagine that you could drastically reduce the CPU load of compression by making simple hard coded state machines for each dictionary. For content like XML or json you could easily make your field names and surrounding punctuation minimal. For many very short messages sharing a dictionary that would beat deflate on compression ratio, and for long messages of non-repeating field values it wouldn't be much worse. CPU use of expansion is probably comparable, though you might get better memory access behavior out of SDCH.)
What you describe is exactly what I've been looking for. There are remarkably few resources on this.
We have users in Singapore who access various XML-heavy web services in our NY office. A dictionary-style over-multiple-requests compression technique would be brilliant for their case.
Take a look at the various WAN accelerator appliances (Cisco, Silverpeak, Riverbed). They do almost exactly what it sounds like you want (if I'm remembering back to my evaluations, Cisco at least uses a multi-request dictionary for their compression)
Riverbed looks ideal but aren't they incredibly expensive? We looked at it years ago and I believe the necessary endpoints in our data center and in Singapore pushed past $140,000.
Riverbed (and all of the players in this space, really) are quite expensive, but this is where you get into the whole "Total ROI" argument for justifying the purchase.
Most companies depreciate hardware over 3 years. How much WAN/Internet bandwidth will you NOT use over the next 3 years, and how does that translate into upgrades you won't need to make?
There are also arguments for these boxes along the lines of "right now we use really expensive WAN links, but these boxes do end-to-end encryption too, so we can put the traffic on the Internet instead" but that opens up a few obvious cans of worms (and can of course be done without an accelerator with VPNs and whatnot).
Then you get into the more nebulous arguments that big bosses tend to like, such as "The average user makes Y XML requests per day to process X widgets. Each request takes Q seconds now. If we lower that to Q*0.5 with WAN acceleration, each user can now process N more widgets per day". Fluffy argument, but can have a big impact on business decision makers, especially if you can tie it to a dollar amount.
Note that WAN Accelerator salespeople are really, really good at coming up with arguments like this for/with you during the sales process.
Railgun is used between CDN and http server, while this one seems to be between browser and http server.
Railgun only requires website to deploy a client, and cooperate with cloudflare. User's client doesn't have to be Chrome or whatever; web server doesn't need to be aware of Railgun. It's transparent to both HTTP clients and HTTP servers. SDCH, however, requires a modification to HTTP/1.1 protocol, which implies changes in both HTTP clients and HTTP servers.
Both are quite promising, though, Railgun is easier to adopt.
