I've just read this article by Randall Degges on how ipify.org scaled to 30 billion API calls a month on a few Heroku dynamos after the app was re-written in Go.
Have you re-written any of your applications in Go and experienced significantly higher performance?
We wrote our bidder (in app advertising) in Go. It is globally distributed (close to the exchanges) and handles 1.5-2M requests/s (OpenRTB,~50-90k/s per instance) with a p99 of 10-20ms (without network latency). Really happy with Go, especially the GC improvements done by the Go team in the last few releases. For a previous similar project we used Ruby which was quite a bit slower.
Like every engineering decision it is a risk/reward spectrum. But unless you have profiled to know that net/http is your bottleneck, no, you should almost certainly not use fasthttp.
I was thinking "an hour" and thinking "damn that's impressive" (it would be 8.3 million per second), or obv per minute or per second would be even more impressive. (And someone is using that really heavily). Instead the end of that sentence is "per month", so 11,415 per second.
You don't need to "scale" for that :) you just need 1 good server.
Go Cassandra... Elasticsearch is a pain in the arse. I have yet to meet someone who has confidence in it at as primary store of data except with the lightest of workloads.
A BitTorrent client https://github.com/anacrolix/torrent, and several projects using it. The original idea started in Python which just didn't cope. Things are probably better now in Python with green-threading being a standard concept, but you just can't easily get the throughput you need without minimal overhead, and that overhead is just too high in Python.
Oh yeah, no argument there. Even now with the fantastic nuitka that compiles Python seamlessly to a stand alone executable, you still don't have the cross compilation story go has. And you have to be careful with libc.
But the thread is about highly scalable things, isn't it ?
Yeah in it's early days, the torrent network was small, and peer counts were limited. The original, and many later Python implementations used event loops, which side step concurrency implementation overheads (like threads), and very often do a lot of heavy lifting in C. I'm not sure that Deluge is Python for the torrent part. The standard is to use libtorrent these days.
Python can handle a torrent client with appropriate tools, but you just have to be extra careful about algorithms etc.
Deluge is really bad with more torrents. After 1000 it's basically unusable. Most people who do long-term seeding use multiple instances of Transmission or rTorrent. There are some manager tools to help with that, these usually also do the balancing.
I used to split them up with around 3000 seeding / client which worked fine.
Ah ok. That's a very specific use case though. Regular users won't do that. But i can understand that non asyncio python is not good for that specific usage.
You're associating the word scalable with the typical web-scale definition. Don't do this. The OP didn't specify what kind of scaling, and a torrent client has a lot of scalability (vertical?) in the way it manages connections and DHT.
It can scale to your all 4 cores and burn your battery.. kidding aside most bittorent clients handle hundreds/thosuands of connections and sparse files as the splits and number of running clients increase, so scalability is a concern here
I think it is not the definition of scalability. It just means it is resource efficient. Scalability usually means that a service can handle the load beyond a single node capacity, at least for me.
I once wrote a thing that went all over gmail figuring out where all the pending mail was supposed to go and presenting it as an interactive dashboard. It was easy to do in Go because baking in the html templates, static assets (like d3) and backend logic is pretty simple with Go’s standard libraries and build system.
I wrote another thing in Go that determined the backend latency of an anti-abuse system within Google. That prober made about ten million requests per second. Again I chose Go (over C++) not for its performance but for the ease of giving that thing a fancy interactive status page.
At SendGrid, much of our stack is Go but started off as Perl and Python. Our incoming clusters are geographically distributed to reduce latency, but a handful of nodes do just fine processing 40k rps. We could dramatically reduce cluster size, but choose not to for reasons around availability. These incoming requests generally create four to eight logging events that are processed and emitted for stats, tracking, and/or published to customer webhooks. Additionally, our MTA is now in Go, and each incoming request usually has some multiplier for the number of recipients.
We typically expect around a 20x improvement in throughput when we rewrite a service. Granted this depends on the nature of the service.
As much as reduced server costs and greater performance are awesome, one of my favorite parts is the increased maintainability of the services. Perl's AnyEvent and Python's Twisted (aptly named, btw) were much harder to reason about. Go's concurrency and simplicity make it a win for us.
Correct, for the concurrency model. For simplicity, while the concurrency solution fits there, I intended "the simplicity of the language." There is usually one clear way to do something and jumping into a new code base is usually pretty easy.
A significant proportion of Segment's event-handling pipeline and processing code is written in Go. This includes our "Centrifuge" system for ensuring reliable event delivery to HTTP destinations, which we recently blogged about: https://segment.com/blog/introducing-centrifuge/
With the exception of C (or perhaps Node.js for single-threaded programs), I can't imagine we would be running as efficiently on our AWS compute resources if we'd written our code in a different language.
I've written many distributed systems in Go for scalability reasons and more recently have been working on micro, an open source toolkit to help others do this https://github.com/micro/micro. The core of which starts with go-micro, an RPC framework for building cloud-native applications https://github.com/micro/go-micro.
Building systems that scale is not an easy task. Go lends itself very well to this task but at the same time there's more required than just the language. The communities belief in libraries rather than frameworks actually hinders this progress for others. Hopefully other tools like my own will emerge that sway people towards the framework approach.
> I've always thought Django to be terribly bloated.
Well it depends. It does handle clickjacking, XSS, i18n, l18n, identification, authentification, database handling, CRSF and much more out of the box.
Things that you eventually code yourself, or that you should disable if you don't need it.
What usually happens is that people delegate this complexity more and more to the front end, which becomes, indeed, bloated. Or install more libs. And code glue for them because they are not integrated.
All in all, Django is slow compared to go, but it's not bloated if you take it in the context of what necessary work it saves you. It's opinionated. It has sane default. It's ready to use.
Unless you plan to have static website or an unsafe one.
We have built a small geo redirection server using Golang and Redis that handles around 50M requests per day. We have optimized our stack to cut TTFB and Golang make it easier to achieve this.
high performance != highly scalable. I was really hoping for some discussion on highly scalable systems and not necessarily optimizing database queries.
High scalability is somewhat subjective, and often not that interesting. It's relatively easy to design an inefficient solution and "scale" it by throwing more machines at it. It's hard to reduce the footprint of something that's big and/or slow.
The hardest problem I've bumped into is getting the model right for making something distributed at all.
Average of 8 seconds request down to around 80ms. Main issue being the original component in the application was making heavy use of the ORM and honestly, had I really taken a step back and hand crafted queries and used plain old PHP objects I likely might not have needed to rewrite in Go. Doing so however is a really great exercise and leaves perhaps the most complex part of the application nicely separated from the CRUD side of things so I'm still thinking it was a pretty good move.
Go-Jek and Grab are both very strong in SE-Asia in ride sharing. Both their backends are (re)written in Go. Each had a presentation on it during Gopher Con 2018 in Singapore. They are also big sponsors of that event: https://2018.gophercon.sg/
[Edit: one of them had some impressive stats on reducing servers while increasing demand]
Noticed I missed the XFS question. Honestly I’m not entirely sure, DevOPs spent a fair deal of time fiddling with it to get best results. We switched to XFS from EXT4 because we were running out of inodes under heavy traffic.
In our case very large messages without the need for a separate store, autoscaled queue managers, instant recovery, and zero chance of double processing messages.
YTBmp3 https://www.ytbmp3.com is built completely with Go. Autoscaling cloud instances based on load with a custom scaling solution is also written in Go. It handles very long running requests on streaming transcoding and compressing. Go's net/http is exposed to the internet with great results which adds to infrastructure simplifications, graceful reloading (important for long running requests). Go allows so much infrastructure simplifications that there is not even a single container involved. :)
The DNS server (https://github.com/abh/geodns) for the NTP Pool (https://www.ntppool.org/en/) does close to a hundred billion queries a month across a bunch of tiny virtual machines around the world. The steady state load is about 30k qps, but with frequent brief spikes to many times that.
I migrated a image upload service from Python that was using 10’s of server to golang using only 3 servers handlings 500% more traffic than the old version
I never seen any Python website handling upload from Python. You always delegate that to a front-facing server like nginx, then you do stuff in a task queue that calls image magic and co.
Where is Python your bottleneck in your old architecture ?
I rewrote a network TCP scanner in Go (about a year ago) and it performed much better. It scans a /12 for 50 common server ports in roughly 40 minutes. https://github.com/w8rbt/netscan
We wrote a simple fcm/apns gateway in go. Not sure what its throughput is these days, but it must be hundreds of millions or billions of requests per day. (Life360)
