> newlines make sense as indicators for the end of a message
Until someone decides to inject a newline into your JSON, then you've got a problem . Just use a length-prefixed encoding, there are tons: bencode, tnetstrings, etc. Also, JSON objects delimit theselves, although you could use a proper stream parser to handle fragmented messages better. A secondary transport like websockets (as others have mentioned) would also give you framing for free.
> Load balancing
Just use DNS. You're already making a DNS request for S3, so here you have a redundant HTTP request, and leak network-level operational details in application logic.
 Ignoring HTTP-style newlines here for moment; common implementations are battle-hardened, but it doesn't mean newline-terminated protocols are really a good idea...
Would length-prefixed also cause issue if malicious client send numbers that are too big, just like malicious clients now can send json with newlines?
I been using websockets in my editor for a while, but for a mobile app they seamed like they just had extra stuff we did not need. But I am warming up to them now.
Honestly we thought about it, did not use DNS because its scary. The errors and propagation slowness can cost you a ton of uptime. I feel safer with a static DNS.
Load balancing is ultimately your call, and HTTP might be the simplest solution if very quick turnaround is critical. But don't be afraid of the Internet's plumbing, understanding it will make you a better engineer. For example, you can still have fail-over to any of the DNS A records (completely transparent with any decent HTTP/websocket client), or if you're seriously concerned about availability, anycast/multihoming. But I'd put money on you having more downtime due to errors on your end or with your provider, than with DNS.
Basically, these problems have been studied for a long time, so it's valuable to be comfortable with the various approaches. FWIW, adding extra A records is just a couple clicks on Linode, and they don't make you pay for each request to boot, unlike S3.
Also, some combination of websockets and/or keep-alive would yield most of the benefits of using just TCP.
Get rid of HTTP, and pretty much all of the complexity of the HTTP stack evaporates.
When you say "an HTTP-based API for some hardware (Hue lights) that could save 98% of its bandwidth using a TCP interface", I really raised an eyebrow. You're already using a TCP interface. I think what you meant to say was that you it could save bandwidth by using a stateful protocol instead of HTTP, or even a custom binary protocol.
Has anybody done HTTP over IPX?
IPX is virtually dead, but it would be entirely possible to do HTTP over IPX if you had a web server and client that could bind to an IPX interface.
On HN, I would hope that anyone confused by the terms knows they can look them up on Wikipedia:
A couple of years ago I had a temporary run in telecom, working with a company on a nation wide VoIP roll out. This was the first project where I was working on technologies that were completely outside the realm of your typical web application. I was surprised to find so many similarities. SIP, for example, looks a lot like HTTP. The distinction between messaging and media gave me new insights in to web programming.
As I learned more and more about telephony, it became obvious that the common language between web and telecom technologies was the separation of layers in the various protocols that make up each stack.
Knowing, and respecting, the language that is used to communicate these concepts helped me transition quickly between technologies. Because of that, I try to make an effort to share this knowledge with others.
My intent is to inspire, not to chastise :) I didn't do a very good job of that in my original comment.
Google Cloud Messaging and (to a lesser extent) websockets would have been better solutions.