What specifically was bad about the Erlang code? Isn't this just saying that if nobody in your company really understands a language, don't use it?
This is more about the technical competency of a specific company than general technical issues. Or, to put it bluntly, it's more "Mixpanel sucks at Erlang" than "Erlang sucks". Don't get me wrong, I'd be really, really interested in a good analysis why in this case Erlang was the wrong choice, but this article didn't even get close to anything technically interesting (Even with the ubiquitous requests/s graph).
I can't help but think when I spot such code smells that perhaps the issue runs deeper. So when I then read about Erlang vs Python I also can't but think that I bet it's not the language choice that is the problem.
I don't think I fully understood that I judge code at first sight like that, but I clearly do.
Additional error indicators are fine (much better than overloading the meaning of HTTP codes), but they should be sane, meaningful, consistent with developer expectations, etc. It's that part where the implementation is weird.
I looked at their site and saw they're hiring. Being a problem solver I tend to think "I can fix your problems, let me help", so I looked to see if they need an architect or chief scientist or something... but when the architect position ( http://mixpanel.theresumator.com/apply/Eoh3qJ/Solutions-Arch... ) has the minimum experience of "Student (College)"... I just face palm and am not surprised by any of this - I bet their definition of solutions architect is really just sales support.
I think they just need to raise the calibre of their backend, the signals here (this article, the error codes, the job advert) are not things that inspire confidence in the product. As the main thing that they do is crunching data I'm taken aback that even their backend engineer position advert ( http://mixpanel.theresumator.com/apply/CiOzuu/Software-Engin... ) doesn't even have the word algorithm in it, but maybe they have a wizard who does that for them.
Am I being harsh? Probably... I don't know the guys there and I may just have a very narrow reading of too little info to make such assumptions, so to be a bit balanced (I'm British, it's what we do), they clearly "get" reporting needs.
Their interfaces are show that they understand the kind of thing people want to see. I was impressed by those, I've done reporting a lot, and so many times you see reports that forget that the reader is trying to understand something. So it's refreshing to see reports that appear to remember that.
EDIT: Actually, I see that the spec says that authentication should do nothing to fix a Forbidden error, so you might be right. Unauthorized is for authentication.
Perhaps this is what buro9 means about overloading error codes, but the rest map so cleanly on the errors that it's hard to ignore them. You can't very well return 200 and say "There was error X" in the response.
Maybe we should use 418?
What's the HTTP error code for "ambiguous input, please select from one of the following unambiguous suggestions"?
Just returning an error isn't helpful, you want to help them resolve it. So you want to return something, and this isn't success (in an application sense) as you didn't do what they asked of you, you're doing something else. But it IS success in HTTP terms, the request made it to the server, got processed fine, and came back fine.
Application codes != HTTP codes.
In web applications, HTTP codes and application codes are quite integrated. Of course, you do need some extra error info, as HTTP error codes can't possibly cover every error in your app, but 422 with extra info as a catch-all sounds like a reasonable compromise.
Erlang is widely regarded as a great platform for servers with lots of concurrent connections because threads are so lightweight (Facebook chat is built on Erlang). And yet Mixpanel decided to rewrite in a language that would be more maintainable for them in the future.
How - by writing a WSGI app that uses gevent and simplejson
Why - because the Erlang code was old, no one knew Erlang well enough to maintain, and Python is the language used everywhere else at Mixpanel
The title wasn't "OMG PYTHON RULEZ ERLANG SUCKZ!!". The only real ambiguity in the title was that Mixpanel (as a whole codebase) didn't switch from Erlang to Python, just a small module switched.
What did you expect with this title? More language criticism?
Yeah, the erlang server code was pretty gross. It was one of the very first things written when we started Mixpanel over 2 years ago, and it's only been updated a few times since then.
I feel like the big thing you guys are missing is how little time we have. It's not like we don't know when we have bad code, or we don't realize that we made mistakes in the initial server design - we just have a million things on our collective plate. Fixing a very simple server - (accept request, validate json, put on queue) that is doing its job okay hasn't been a high priority thing.
When this code was written, Mixpanel had zero customers and we weren't sure what we were building yet. In that regard, Erlang has been a rock. We've barely had to touch it during the rampup from 0 to thousands of requests per second.
Now that we have the manpower, and we've learned what we really need, we can rewrite it to make things easier on ourselves. If we can get acceptable performance in python, there is no reason to use erlang.
I think there's some merit to the other complaints (error codes, etc), but that's another symptom of this thing being written so long ago. We want to improve things incrementally (and backwards-compatibly) for now, but it will be dramatically simpler for us to make changes to the server now that it's written in python.
Ultimately, we have skeletons in the closet, just like the rest of you - I'm sure all of you have some bad code in production somewhere. Now we're saying "Look, we're getting rid of our skeletons!" and you guys are like "OMG WHY YOU HAVE SKELETONS" instead of "sweet, no more skeletons".
I agree that not having Erlang competence in your company IS a good reason to change language.
Finally, we use a few stateful, global data structures to
track incoming requests and funnel them off to the right
backend queues. In Erlang, the right way to do this is to
spawn off a separate set of actors to manage each data
structure and message pass with them to save and retrieve
data. Our code was not set up this way at all, and it was
clearly crippled by being haphazardly implemented in a
Because of these performance requirements, we originally wrote the
server in Erlang (with MochiWeb) two years ago. After two years of
iteration, the code has become difficult to maintain. No one on
our team is an Erlang expert, and we have had trouble debugging
downtime and performance problems. So, we decided to rewrite it
in Python, the de-facto language at Mixpanel.
This sounds a little strange. How is this possible? High turnover on the team?
So, the lack of iteration wasn't due to high turnover, it was due to having something that worked and other problems to solve.
(I'm an intern at Mixpanel)
Anyone claiming to have mastered it in a couple of months like most fanatics here are (pardon my language) full of crap.
I learned it on the tube to and from work in a relatively short space of time, writing a comet server for streaming updates to browsers with a colleague. After the initial hump it was easy and fun. I don't claim to be a genius programmer, just an interested dabbler.
To be fair though, the more average developer does struggle with it, and that was a reason my company didn't take it up widely.
But competence in erlang is certainly achievable in a couple of months.
In my example, I started having never once written anything in a functional language. I started by reading Armstrongs book with pragmatic programmers. After a couple of months I could write anything I wanted to in erlang, without too much trouble, and much more importantly:
I was in love with erlang
The idea of writing something in erlang was very exciting and pleasurable to do.
I think that's quite an accomplishment for a language with such a different syntax and with me only putting a couple months into it.
Now, obviously, I won't say I'd mastered it. But prior to that I had the fear of the unknown when it came to functional languages, and a distinct repulsion at erlangs syntax. The only reason I learned it was that I believed it to be the only language that had done concurrency right.
And I had a project to write in it that required concurrency. That might be a big help. I imagine it is hard to learn any language if you don't have a goal that it is well positioned to achieve to work towards while learning.
I suspect people who think erlang is hard to learn, haven't spent any time learning erlang. But in truth, no language, except the very first one, was hard for me to learn.
So unless we're talking about thousands of lines of code, it really doesn't matter what library or language you'll choose for something like this. If this would be your only use of Erlang, it's probably not worth it. Erlang is pretty great at building distributed, high-concurrency applications that are good at coping with errors. For one out of those three, you have plenty of other options…
Nope, that’s not the right way. The way you were doing it ended up making all calls sequential and bound to single processes that could lose state. That’s not right.
The best way to do it would have been to use ETS tables (which can be optimized either for parallel reads or writes), which also allows destructive updates, in order to have the best performance and memory usage possible. Note that you could then have had memory-only Mnesia table (adding transactions, sharding and distribution on top of ETS) to do it.
As for string performances, I’m wondering if you used lists-as-strings, binary strings or io-lists to do your things. This can have significant impact in performance and memory use.
Then again, if you had a bunch of Python and no Erlang experts, I can’t really say anything truly convincing against a language switch. Go for what your team feels good with.
Could you expand upon this some more? How do you know the server works 99.99% of the time (or 99.9%)? Do you run regression tests using actual past requests?
Bob's software sucks, let's switch to Bob's software.
Bob Ippolito wrote mochiweb (the erlang web server) and looks to be involved in eventlet http://eventlet.net/doc/history.html
Really simple + rarely changing + needs to scale to really high req/sec = perfect candidate for being written in C. Maybe as an nginx module?
I mean, good enough is good enough, and local culture counts, no problem there, just curious.
The actor model is for concurrency, which is when your threads are communicating with one another, right? What about the task that the API server does requires inter-thread communication?
Here is my proof: http://j2labs.tumblr.com/post/7305664569/python-vs-javascrip...
$ python json_bench.py history.log.1
After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems.
Erlang is historically bad at string processing, and it turns out that string processing is very frequently the limiting factor in networked systems because you have to serialize data every time you want to transfer it. There’s not a lot of documentation online about mochijson’s performance, but switching to Python I knew that simplejson is written in C, and performs roughly 10x better than the default json library.
I was able to provide some important operations in constant time along with other optimizations that were cripplingly slow in the Erlang version.
The [Python] community is extremely active, so many of my questions were already answered on Stack Overflow and in eventlet’s documentation.
In particular, redundant string concatenation and flattening tends to be CPU hog, but IO-Lists automatically flatten all string types during transmission and have already been thoroughly optimized.
Of course, if you ignore the serialization solutions Erlang provides, there is a performance hit.
I once met an intern that is very good at abstraction and writing "OK"-designed OOP code (OK because it looks and sound good minus the ability to unit-test, but other than that it was simple enough for other people to understand and quite flexible). On the flip side, he's not that good when it comes to networking code (pretty much system programming stuff). He could be good, but at that time, software design (in OOP environment) was his forte.
You might have your own pluses. Besides, we don't know what the code looks like or whether what this intern wrote is the truth. If you've been in this industry long enough you'll start to take a lot of things with a lot of grain of salt.
Does this mean that Erlang and node.js are mostly compelling because of the prevalence of async versions of common libraries? Or are they not that compelling in web contexts in the first place?
I always found Haskell's take on parallelism interesting, and maybe it is faster. In Haskell you create a unit of work called a 'spark'. You can have billions of these, they get mapped to lightweight Haskell threads (powered by epoll) and those get mapped onto OS threads.
H. Thoreau once said "In proportion as he simplifies his life, the laws of the universe will appear less complex.." (Walden, Princeton University Press, 1971, p.323-324)
The most sophisticated solutions are often very simple, because they were written from a sophisticated perspective, not a naive perspective.
erlang has stood the test of time, and produced great results because, really, it is very simple.
It just comes from a sophisticated perspective.
1. How does the performance of the new system compare to the old system?
2. What exactly were those maintenance issues with the Erlang server? Did just no-one in your team find the time to learn Erlang well enough? I know Erlang isn't the prettiest of languages, but async I/O isn't the only advantage of Erlang. A battle-tested concurrent runtime and built-in support for fault-tolerance are two obvious examples.
quick_sort([H | T]) ->
quick_sort([X || X <- T, X < H]) ++ [H] ++ quick_sort([X || X <- T, X >= H]).
Still, very cool, congrats ankrgyl, it's awesome to be able to write stuff like that as an intern!
"No one on our team is an Erlang expert"
Also I would love to get a comparison between 2-years old (probably badly written) Erlang server and a new Python/eventlet server.
At the same time, I have gone and looked at code I wrote back when I was first looking at erlang, that does much less and is much more verbose, confusing and sprawling.
I don't think erlang lacks maintainability. I think it just requires some discipline- like any language.
It sounds like your company has a culture of python hackers and erlang was chosen because you felt you needed to choose something "serious" for this bit of work, rather than because you loved erlang and would use erlang even if you needed to write something trivial. There's nothing wrong with that, but I don't see this article as revealing any hidden weaknesses in erlang.
Regarding the JSON parsing issue, erlang has excellent support for code written in other languages, specifically C, and you could wrap any C based JSON parser and use it, though I bet someone has already done this for you. I believed I was watching such a project on GitHub but can't for the life of me find it now.