Hacker News new | past | comments | ask | show | jobs | submit login

Everything at web scale is embarrassingly parallel, at least on some fronts. There's still often the bottleneck of maintaining state (in a database, probably), but a heavily loaded server will have hundreds or thousands of processes or threads or some kind of async (probably epoll or something reasonably efficient on modern systems) to serve many clients. Which is why Erlang (and Elixir) has some very enthusiastic proponents.

Serving the web is, I think, a very big part of the "general computing" space, today.




Serving the web seems to split between a) IO-bound problems and b) databases. It is certainly not embarassingly parallel in the "would benefit from GPUs" sense.

Those thousands of threads just sit there waiting for the database to respond a lot of the time. The database is busy trying to keep things synchronized across nodes. It is just that the hard stuff is pushed down to the database layer.


Both can be true. Database development, both the way they're built and the way people use them, has been focused on scaling across machines and storage for most of the history of the web; it's possible it'll never quite catch up with CPU, but it will continue to move that way. I don't know what the next evolution of databases looks like (I'm definitely not a database expert), but if Google and facebook and Amazon and Microsoft and many others need it to scale beyond what is currently possible to keep up with CPU parallelization, they'll figure out how to make it scale.


aren't databases in turn being held back by the hardware limitations like IO bandwidth?


Yeah, the Erlang ecosystem is so much better (for web) than everything else available today, it's not even funny.

Though it seems like there is some very major obstacle in people understanding it. It's so frustrating, I've already stopped advocating for Erlang/Elixir. Currently I'm concentrating in just reaping the benefits and doing my stuff.


Yep, I've built two very nice servers in erlang, but main problem - only I can extend and modify them in my company, my coworkers can't learn this. One made some inroads into this, but there's just too many cheap .net and java developers around for erlang to be successfull.

Upside for me - I'm irreplaceable. Downside - there's no one to help when something is wrong and company progress is throttled by my performance.


Elixir is partly fixing this problem. Though it's still not enough because the hard part is not the syntax (as most newcomers tend to blame their failure on), but the things you write with the language, which are the same for Elixir and Erlang.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: