Have you re-written any of your applications in Go and experienced significantly higher performance?
Similar throughput, our bottleneck at this point is moving data around.
We’ve abandoned channels for most of this. The next major improvement would be to rebuild the http stack & that’s just not worth it.
And also pointed me to nats.io, a messaging system that handles 10M messages per second on a $50 server.
See the comment at: https://www.indiehackers.com/forum/how-we-handle-25m-api-cal...
What I really meant is we'd need to take things down to the bare network stack. Lots of our memory use/bottlenecks are fairly deep into the std lib.
Also, the default torrent client in Ubuntu is Deluge, which is in Python too. I download 20 torrents at 20M/s without any issue with it.
But I get that using threads, downloading from 1000 clients is annoying to code, and indeed you are right, today with asyncio, the story is different.
And I definitely understand the appeal of Go for network concurrency for a lot of projects.
I just don't get how Python didn't fit the bill for this particular one. It's just a client, after all.
Maybe I'm missing something.
(I like Python a lot; I've written tens of thousands of lines of it. But its code packaging systems leave much to be desired).
But the thread is about highly scalable things, isn't it ?
Lib packaging is a solved problem. For installing, people are just using pipenv. For creating, it's just an 2 lines setup.py file and a ini file to fill in now (http://setuptools.readthedocs.io/en/latest/setuptools.html#c...)
Python can handle a torrent client with appropriate tools, but you just have to be extra careful about algorithms etc.
I used to split them up with around 3000 seeding / client which worked fine.
I wrote another thing in Go that determined the backend latency of an anti-abuse system within Google. That prober made about ten million requests per second. Again I chose Go (over C++) not for its performance but for the ease of giving that thing a fancy interactive status page.
Go also has built-in support for blurting out any structure as JSON so it’s dead simple to write jsonp handlers.
We typically expect around a 20x improvement in throughput when we rewrite a service. Granted this depends on the nature of the service.
As much as reduced server costs and greater performance are awesome, one of my favorite parts is the increased maintainability of the services. Perl's AnyEvent and Python's Twisted (aptly named, btw) were much harder to reason about. Go's concurrency and simplicity make it a win for us.
With the exception of C (or perhaps Node.js for single-threaded programs), I can't imagine we would be running as efficiently on our AWS compute resources if we'd written our code in a different language.
Building systems that scale is not an easy task. Go lends itself very well to this task but at the same time there's more required than just the language. The communities belief in libraries rather than frameworks actually hinders this progress for others. Hopefully other tools like my own will emerge that sway people towards the framework approach.
On python we use Django, DB postgres&citus, rabbitmq, redis.
Our main cluster have more than 25mln API requests daily.
For us Golang is >70 times faster than Django.
Well it depends. It does handle clickjacking, XSS, i18n, l18n, identification, authentification, database handling, CRSF and much more out of the box.
Things that you eventually code yourself, or that you should disable if you don't need it.
What usually happens is that people delegate this complexity more and more to the front end, which becomes, indeed, bloated. Or install more libs. And code glue for them because they are not integrated.
All in all, Django is slow compared to go, but it's not bloated if you take it in the context of what necessary work it saves you. It's opinionated. It has sane default. It's ready to use.
Unless you plan to have static website or an unsafe one.
If you're in the 10% yeah it can be a pain in the butt. However it's not impossible or even particularly difficult to migrate to something else.
[Edit: one of them had some impressive stats on reducing servers while increasing demand]
The hardest problem I've bumped into is getting the model right for making something distributed at all.
Saved us a ton over the pre-built queues we'd been using.
Mind sharing needs not met by "standard" message queues (rabbitMQ, Apollo, NSQ, etc)
>>>> optimized XFS volume
Is this just some specific mount options or is it a more complex setup ?
Thanks. Asking out of curiosity.
Probably not what you're looking for, but I improved the runtime of my shell prompt by one order of magnitude by porting from Python to Go. https://blog.bethselamin.de/posts/latency-matters.html and discussed at https://news.ycombinator.com/item?id=15059795
Where is Python your bottleneck in your old architecture ?
>how ipify.org scaled to 30 billion API calls
I was thinking "an hour" and thinking "damn that's impressive" (it would be 8.3 million per second), or obv per minute or per second would be even more impressive. (And someone is using that really heavily). Instead the end of that sentence is "per month", so 11,415 per second.
You don't need to "scale" for that :) you just need 1 good server.
Go is like a web-safe C. :)
By the way here is the article is talking about: