Hacker News new | past | comments | ask | show | jobs | submit login
Why and How I Switched from Python to Erlang (ourway.ir)
118 points by rodmena on July 6, 2016 | hide | past | favorite | 65 comments

I use both Python and Erlang. Python is to get small stuff done quickly language for me. And "small" doesn't have to be just a demo or hello_world example, it can be a whole full back-end of a business.

(And btw when I say Erlang I also mean Elixir, they both share the same VM so most things apply to both).

But Erlang is a the secret sauce (so to speak) for a high concurrency fault tolerant backend. "Fault tolerant" should probably go first. The reason is the observation that the higher the concurrency or complexity in the system, the higher the need for fault tolerance. If you only serve 10 connections and the backend segfaults, 10 clients have lost connectivity, you get 10 angry phone calls. If you serve 1M connections and your backend segfaults, you get 1M phone calls. Exaggerating here of course to get the idea across. But this is not just a marketing gimmick, this translates to money in the pocket in operational costs. Some subsystem crashes and restarts? Fine, let it do that if it is 4am, no need to wake people up, will fix in the morning. I've heard of stories of subsystems crashing and restarting for years with the main service staying up.

The lack of centralized state might seem minor but unless you have been debugging a shared memory systems with locks, threads and manual memory allocations, with classes that have pages of attributes defined, and trying to understand why it crashes on Wednesdays only at customer A's site, it is easy to miss the benefit. This comes through using small lightweight processes with an isolated heap and also using functional constructs.

Then in general, the ability to reason about concurrency using OTP constructs (Erlang's standard library) and processes is like going from Assembly to C in terms of distributed and concurrent systems. Can express ideas at a higher level. This means having less code to look through and maintain.

There are other niceties like good garbage collection performance, awesome tracing and hot code loading capability (used this a few times, so started to appreciate it more now).

Now, individually all of those features can probably be found in other systems and frameworks, but they are just not integrated or not quite there -- Java has code loading but it is not the same. Can always spawn an OS process to get fault tolerance, but can only spawn so many before system falls down, Go has goroutines but they also share memory, so fault tolerance is not there. Other languages have garbage collection, but most still have to stop the world sometimes and so on.

You seem to be familiar with Elixir. May I ask you some questions?

. What kind of web applications are you building with it? I'm asking what kind of web apps or scenarios do you think Elixir is particularly well suited for?

I saw a thread on Elixir a couple of days ago and it piqued my interest and I saw a couple of videos that were posted there, one from some Ruby Conf that claimed that Elixir was giving better results (in request time) than rails. He never explained how that was posible or what would have been the results if he would have been using a cache (it's always faster to hit an in memory cache than hitting a db that has to touch disks, so if he speeded things up without a cache he would speed things even more with a cache). Then I watched another videos from some conference in Oslo or something, and from what I could understand he was doing away with the db completely.

. So I have another question, how do you architect your application in Elixir to keep application state though multiple requests (sessions) on multiple boxes without using something like memcached or redis (or a network disk)?

. Even if it's running on only one Box? Where does data reside if you are not using a db?

I have a basic understanding of Erlang processes (what's explained here: http://stackoverflow.com/questions/2708033/technically-why-a...) and how it's particularly well suited for concurrency. My questions are about Web Apps and Elixir and scaling.

This is a benchmark I did with the versions of Rails and Phoenix that were current in October 2015.

select * from visits, plus conversion to JSON and delivery to a client on local loop. About 5000 records.

* Phoenix 140 ms

* Rails 248 ms

* Ruby without AR 219 ms

* PostgreSQL 2.97 ms, with no JSON generation and no delivery

select started_at, duration from visits -- JSON and delivery

* Phoenix 74 ms

* Rails 116 ms

* Ruby without AR 88 ms

* PostgreSQL 3.47 ms, no JSON no delivery

Single process, so maybe Phonix could get a larger advantage as the number of processes/requests increase. For the typical none to low traffic site there is little difference, the tool the programmer is more familiar with wins.

Edit: improved formatting.

Thank you for your reply. I can't edit my comment any more but where it says:

>Ruby Conf that claimed that Elixir was giving better results (in request time) than rails

I meant:

Ruby Conf that claimed that Elixir was giving better results (in request time) without a cache than rails with a cache

IIRC, Rails app mentioned in the talk was very old. Not a fair comparison, IMHO.

A couple of answers : 1) anything that is not just a PoC with limited amount of people connecting to it. For 10 clients connected at all, sure use the thing you know well. It will be easier to use and write, and your time to market will be better. Otherwise, use the real tool.

2) db is rarely your problem. Interpretation can be, and tons of other things. Also erlang schedules things far better and use all the core of the machine, so it will block far less, etc It also means that the thourgput and latency will stay stable despite the amount of users growing. (up to a limit, but it will far higher)

3) Store it in memory. And link other node to yours. Tadaaa

4) Where data live depends of your use case but why use a db if you do not need it anyway?

The thing in general is that web framework tend to use db to hide concurrency.

Also why no cache : because cache is complex, cost you another dependency and another program, make debugging harder, make your app less deterministic, etc. If you need cache use it. But caching is a really hard thing.

> You seem to be familiar with Elixir. May I ask you some questions?

Sure, np. I am mainly an Erlang developer but the same properties apply to Elixir, because both share the same battle tested VM -- BEAM.

> I'm asking what kind of web apps or scenarios do you think Elixir is particularly well suited for?

I am not currently working on building web app directly. But in general Elixir/Erlang would be good for a scenario where they'd be multiple connected clients at the same time. Think maybe something like a car sharing service where map updates in realtime as cars move through the city. Lots of users chatting together, or playing a game. Maybe they are bidding on ads, or tagging posts with "likes" and so on.

But there might be a significant improvemtent even in the plain old request-goes-to-database scenario, simply because the VM (BEAM) is better equipped at handling multiple connected sockets, all streaming data in and out. For example, it knows how to take advantage of multiple CPU cores, it handles GC better and so on. Not sure how deeply you want this explanation to go, so will stop here.

But, yes, if it all goes to a single MySQL instance running in another datacenter, that might be the bottleneck so there might not be a speedup seen by using something else. So you have to measure. In that case maybe a cache in front of it will work just as well to speed things up.

> he was doing away with the db completely.

Haven't seen the video. But with Erlang can think of those processes as in-memory storage of state. Can spawn hundreds of thousands of them, and they can live as long as the node stays up. Then can also connect multiple BEAM VM instances on different machines, create a cluster and so can refer to these processes as if they are local, in a rather transparrent manner. Or maybe they mean they used Erlang's built-in database (Mnesia), that had some limitations, but recently in a new release (19.0) it has the ability to scale much better as it can handle a pluggable storage such as LevelDB for example. The advantage there is the database is integrated into the langauge. So queries are not in a different query language, via a driver, so some other process, but queries looks like list comprehensions and transactions are just function closures. That can simplify things immensly in some cases.

> how do you architect your application in Elixir to keep application state though multiple requests (sessions) on multiple boxes without using something like memcached or redis (or a network disk)?

Can create a cluster so can keep state on another node that doesn't go down? (as mentioned), or can still use a database (say Postgres), so it really depends. Hard to give a general advice here.

Thank you for your answer. It has been very enlightening.

Part of the problem I had with the videos had to do with the examples they were using (a shopping cart that stores data on process memory rather than an external db seems a little risky to me).

I will investigate Erlang Clusters. Again, thank you for your answer.

The idea with this is that you were only storing the state (shopping cart contents) up until the point that the user was ready to place the order (at which point it was written to disk). Prior to this point, state is stored in an in-memory database that is clustered among all of the running nodes. So yes, there is a risk that you could lose state, but only if all nodes died at the same time. Otherwise, the nodes jsut recover and re-synch the state.

I don't program Erlang, but discovering communicating sequential processes for multithreading was a revelation to me.

I use it in Java/Kotlin almost every time I do something involving more than one thread.

Add a blocking-queue, define incoming messages, define outgoing messages and forget about it.

Yes, for certain programming jobs (i.e. work) Erlang is much better. Let's just assume we can mostly agree on that.

But "I switched" means you switched your main language, right? I wouldn't switch my main tool to something with a specific usecase. Your main language should be something general purpose, because most of your life's problems have a wide range of usecases to consider.

Therefore I would have expected an argument for why Erlang may be a beter general purpose language or a headline like "Why <project/company/service x> switched from Python to Erlang".

Yes, the article was pretty badly written I thought. It was mainly just complaints about Python than any real exploration of why Erlang was better.

If you're going to set pixel based margins on the left and right side of your page, you need a media query to disable it or at least lower it to 5 pixels on small screens. This is unreadable on mobile :/

I'm always surprised why more people don't just do F12, ctrl+shift+M (on Chrome) while writing their websites. Just four keystrokes :)

Awesome! Had no idea you could do this! For those wondering, it renders a mobile-like view of the current site. Thanks ;)


If you're on iPhone you can use the reader button. Android may have something similar. Obviously I agree with the advice though.

With JavaScript disabled, it's just a long bar of text about 18 letters wide.

I'd be curious as to why exactly the SQLAlchemy query was slower than the raw one. I get that there's gonna be some overhead, but did you figure out exactly what was slowing it down?

I have been comparing the 2, at third glance they don't even appear the be the same query.

Second for performance as much as possible from the SQL building should be outside the function. Also if the query is that static, you should probably use the plain SQL anyway ;) Or maybe even build the SQL query string by using SQL alchemy outside the function, and then using the SQL generated by SQLAlchemy inside the function (if you don't like pure SQL, like for example compatibility).

Two comments:

- It looks like the SQL queries produced by pure_python() and simple_sql() are structurally different, which would make any comparison worthless.

- How does the author solve the long running tasks (> 500 ms in the article) problem with Erlang? In Python, it looks like he was using Celery with Redis then RabbitMQ. What does he use in Erlang? (Erlang message queues have no built-in persistence.)

Erlang can have long living processes (green threads) and that's usually how we solve problems. There is absolutely no problem (it's encouraged) to have thousand of processes living minutes or hours if we need so.

Yes, but how do you manage fault tolerance at the machine level? If the machine is rebooted for some reason, the pending messages will be lost, and will never be processed. A persistent queue is necessary for this reason. This is the reason why people use tools like Celery or RabbitMQ (which is written in Erlang by the way). I don't see how replacing Python with Erlang changes anything in this regard.

You handle that with multiple nodes that sync their state.

Can't comment on what the author used, but Mnesia can be used to build a primitive, durable MQ: http://learnyousomeerlang.com/mnesia

There is also RabbitMQ, which itself is written in Erlang.

Well, Mnesia could be used to build that, but what for in the first place? You just spawn a worker process under a supervisor and leave it to finish its job. Really, no need for a queue (unless you want to postpone some tasks until some others finish, but this is more synchronization problem and can be solved without fully-blown database).

Too bad the author doesn't show specific examples. I can sympathize with the sentiment. The problem with scaling languages like python and ruby in my experience is the number of moving parts and jump in deep knowledge required to scale when your application takes off. One day you're happily writing compact code in your favorite language and suddenly you're rewriting core parts in C/Go/Rust while learning a new language and bolting on a variety of moving parts like redis, memcached, etc. to keep your business from becoming a victim of its own success.

A lot of the problems are more or less solved problems with well known workarounds but it's still a pain and probably why there's so much buzz around finding the "next (insert your favorite language/framework)".

Erlang generated a lot of buzz several years ago but aside from the recent success of WhatsApp it never quite stuck. I'm curious to know how this author thinks differently.

> Erlang generated a lot of buzz several years ago but aside from the recent success of WhatsApp it never quite stuck.

Erlang is quietly running a lot of stuff out there. It's almost certainly running some bits of infrastructure that you use daily, or in a supporting role for it. People who are not old Erlang hackers are picking it for new projects.

Erlang hasn't been and probably will never be quite mainstream, but to say it "hasn't stuck" might be missing the point: it's a mature and performant language that has some very nice standard features in OTP that are lacking or less mature in other languages. If your problem domain is smack in the middle of Erlang's strong points then it's worth considering. It's not going away any time soon.

I see your point and yes I know that Erlang has decades of production use and shines in lots of infrastructure applications, game servers, etc. I was mainly referring to the buzz generated quite a few years ago when the first Pragmatic Programmer books on Erlang came out. There were a lot of startups testing the waters with Erlang but I just got the impression that the language didn't quite take off despite the merits touted about Erlang's strength as a functional language and the built-in handling of concurrency.

This is all of course compared to the current buzz surrounding Elixir/Phoenix. I didn't intend to dismiss Erlang's amazing track record and applications to the domains where it continues to shine.

This is roughly the time span I was referring to when I said "buzz":


Now is a fantastic time for languages. Python and Ruby get you where you need to be fast, and if you get far enough and need to scale, there are a huge number of options, from stalwart holdouts like Java and C# (now on Linux!) to newer languages on rock-solid run times like Clojure and Elixir, to whole new languages like Go and Rust.

All these languages have their advantages and followers, but the thing they all have in common is first-class concurrency, which everyone now realizes they need (and old Erlang devs will tell you "duh").

Honestly, while I write Go almost exclusively anymore, I think Elixr is the most exciting. The more I learn about Beam the more magical it becomes in my eyes, considering just how old and stable it is. Phoenix is a game changer in many ways.

> if you get far enough and need to scale

Alright -- I'll bite.

Rarely anyone "needs to scale." Plenty of $1B businesses can succeed on a fistful of c4.large's running Ruby on Rails. The same Ruby on Rails that ran their MVP. Scaling via language change isn't the path to victory, unless you picked a language that was poorly suited to your domain to begin with.

Too many companies end up following this cargo-cult advice, and spend more time grappling with their tools than innovating. Because "concurrency." Someday.

Optimizing your choice of language for anything other than a linear relationship between Real Complexity and Implementation Time is a fool's errand and a fiduciary travesty.

In all fairness, Elixir/Phoenix could become as well-learned as Ruby on Rails. We'll end up in a best-of-both-worlds scenario. And at that point, I'll happily eat my words.

But in the meantime, solve your "scaling" problems by measuring and optimizing, instead of re-writing your app in the flavor-of-the-week.

I can't think of a single billion dollar Corp that can run Rails on a few AWS boxes. Time and again companies have rewritten systems from MVP to handle data safer, better, and faster, from Dropbox to Twitch to CloudFlare to Uber to Mail.ru...

Then there are plenty of 7 figure companies that have also had scale issues. Game companies come to mind first and foremost but they're hardly the only ones.

You want to run your marketing website on Rails? Yeah of course. But billion dollar companies on Rails?

For the sake of discovering a middle ground between our arguments, let's look at an example of a >$1B company running Rails: Github.

Most of Github's stack is Ruby on Rails, with specialized components/sub-systems written in C. This is a common theme amongst companies that use Rails or Python at scale.

There's a reason why people keep those languages around. Its the same reason why they tend to be used for MVPs: the tool gets out of your way so you can focus on solving the hard problems of your domain. The longer you can preserve those ergonomics, the better.

My guess is that Github uses a few more cpus than "a fistful of c4.larges" suggests.

I wonder about the valuation of Basecamp, they may almost qualify.

> Rarely anyone "needs to scale."

Agree, and I say it as an Erlang full-time dev. Chances are you don't need 9 9s reliability and super scalability.

I've heard and seen cases of companies rewriting stuff unnecessarily due to some incompetent people following latest fads. "Oh I read Python has the GIL so we need to switch to Go. Saw on HN everyone is switching" -- cue years wasted in rewriting and destabilizing a solid usable code base.

Elixir Introduces many new concepts to Erlang. I think it adds to the complexity of language. Erlang is enable to do all with a bit more functional code. Overall I agree that Erlang based frameworks are game changer.

> I think it adds to the complexity of language.

How so? What makes Erlang more functional? And how is Erlang simpler for you?

A big part of the renaissance in programming languages come from developers leveraging rock solid VMs such as BEAM and JVM. It enables developers to have the expressiveness of dynamic languages and without sacrificing performance or scalability later down the road.

In the case of Elixir, many feel that it adds some innovative language features like the pipe ("|>") operator and metaprogramming facilities that ease many of the pain points with Erlang. I'm not an expert in Erlang but there are quite a few notable Erlang developers that enthusiastically support Elixir (including some nice words from Joe Armstrong).

I'm not arguing with you but curious about your thoughts. Looking forward to the next installments in your series.

> feel that it adds some innovative language features like the pipe ("|>") operator and metaprogramming facilities that ease many of the pain points with Erlang

Well both points are valid. Elixir does add some great concepts like the pipe, usable macros and others. But with it comes some complexity -- it is a trade-off. I don't think one is a superset of another. Both will appeal to some people and that's fine. I prefer Erlang, but I hear a lot of people put off by Erlang's syntax, so well maybe Elixir will look better for them.

One thing I am impressed about Elixir is the community -- Jose and the team, did a fantastic job building not just the language but making a friendly and welcoming community, that is hard overstate.

Could you speak a little more to the point about how you feel Elixir actually adds to the complexity of Erlang? Was that comment mostly referring to the addition of those new concepts you just mentioned? Or were you referring to something else that the language does under the hood that adds to the general complexity?

>Python and Ruby get you where you need to be fast, and if you get far enough and need to scale, there are a huge number of options, from stalwart holdouts like Java and C# (now on Linux!) to newer languages on rock-solid run times like Clojure and Elixir, to whole new languages like Go and Rust.

To be clear, are you suggesting that's reasonable to do a rewrite (or at least for certain parts) from Python/Ruby when you need to scale? Is that going to be reasonably accomplishable in many cases? Honest question. Just wondering what you have in mind.

Rewriting systems in another language can certainly be accomplished. If possible, doing it in bits and pieces will help it actually happen; if you do it as a single big switch, you have to work a lot harder to avoid the second-system effect.

In this case, you probably first start by switching your message queuing to Erlang (RabbitMQ), since you had a problem with that anyway; then you can expand outwards from there if it makes sense or makes you happy: job running in Erlang, webservices in Erlang, putting the data in mnesia instead of postgres, etc. You can do the same with any language, of course.

> All these languages have their advantages and followers, but the thing they all have in common is first-class concurrency

Among the languages you cited, C# has async/await, Clojure has core.async, Elixir has processes and messages, Go has goroutines and channels, but the other languages (Python, Ruby, Java, Rust) have nothing built-in specifically related to concurrency, do they?

Python 3 has async coroutines built-in, and there are a number of libraries for Python 2 (e.g. twisted, tornado). Python 2 & 3 also have threading and multiprocessing for concurrency.

Java, Rust & Ruby have built-in support for concurrency, but I haven't used those languages that much.

Maybe I read "first-class" a bit too literally :-)

> Python 3 has async coroutines built-in

Yes, the asyncio library was introduced in Python 3.4.

> there are a number of libraries for Python 2 (e.g. twisted, tornado)

Yes, but they are not part of the language per se.

> Python 2 & 3 also have threading and multiprocessing for concurrency.

Threading and multiprocessing are really not a good fit when the number of concurrent processes increases, because each thread allocates its own stack, which can be quite expensive in terms of memory. This is the reason why Go (and maybe Erlang too but I'm not sure about the implementation) map "green" threads to OS threads, and implement a resizable stack. Libraries like gevent offer something similar for Python, with some low-level trickery.

> Java, Rust & Ruby have built-in support for concurrency

In the language and/or the standard library?

In Java, there is Quasar and Akka, for example, but they are not part of the language/standard library.

  > In the language and/or the standard library?
In Rust, we have extremely strong concurrency support in the language/standard library in the form of our type system, which disallows data races at compile time. We only provide 1:1 threading, but libraries can use this type system to extend it: there's N:M threading libraries, as well as more exotic concurrency stuff than we provide directly. Rust also generally takes a "no batteries included" approach, generally speaking.

In Ruby, there's Thread and Fiber. MRI's GIL hurts it here, but other implementations don't have this problem.

Can you provide links to the Rust N:M threading libraries you recommend? How is the stack managed in these libraries (does it start small and is resized at runtime)?

Same question for Ruby :-)

>Python and Ruby get you where you need to be fast,

My experience with Go has been that it is almost as easy to get it up and running just as fast as with Python.

Has go got a decent web framework that can match features in Django or Rails. I don't find Python any faster to write than other scripting languages, but I do find Django gives me a ton of productivity.

The general consensus in the Go community is that the default http.Handler is sufficient for most things. You can leverage additional middle ware or gorilla toolkit for more complex stuff. Most stuff works nicely with net/http.

You might notice difference in performance when your sqlalchemy query is different to your raw query. If you don't understand sqlalchemy then you are probably best to not use it. How about going to mongo?

An SQL injection issue right there.

Edit: to be fair the variable proj_id sounds like it's not taken from untrusted input, but nevertheless a bad idea.

The title should be: Mentioning some scaling problems I heard about in Python and might have brushed up against in my own projects (likely not production ones). Oh and their's this thing call Erlang and it's fast!

I sincerely hope nobody is making changes to a working production stack based on posts like this.

didn't really explain how.... لطفا بیشترتو ضيح بدید

It's quite interesting to see a Persian comment on HN. And somewhat inspirational - maybe HN can become multi-lingual down the line through the equivalent of 'subreddits' or simply threads being marked as another language, with translate buttons available for those who don't speak the language. One question though - how do you effectively type both latin letters and farsi letters on a single keyboard?

لطفا بیشترتو ضيح بدید ? I tried Google for translation. It wasn't very helpful.

Edit: ok, close enough --

"Please explain me Byshtrtv"

Edit2: replier and op were much more helpful than google, thank you!

In Persian. He means: "Explain More"

don't forget please :)

Well, It was version 1! I will try to explain more and more in coming days. I wrote these articles for mind rest and between my coding sessions.

It was interesting, but I think a word is missing in

"Well, you are trying to add Python code some Erlang features."

and it wasn't obvious to me what you meant.

Looking forward to the follow up!

of course

Wish I could answer and say you're welcome, but I agree with you. It seems to be some dubious why and no how. Dubious in that he gave an arbitrary and unsubstantiated 500ms benchmark for Python.

I updated and clarified it. 500ms is a web2.0 rule. You should give users a status about long process requests. Thats it.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact