I think shuffle sharding is beneficial for read-only replica cases, not for writing scenarios like this. You'll have to write to the primary and not to a "virtual node". Right? Or am I understand it incorrectly? I just read that article now.
I think you'll have to pay a team millions to figure that out, it is unlikely to be a static rate but rather decided based on multiple traits like time of year, time of flight, distance of flight, cost of ticket, etc.
They probably just have an SSRS report that prints out in a few dozen offices automatically on some schedule.
I'm not trying to be pedantic but this is table stakes stuff. I know we're supposed to shy away from saying things like this but compared to the other engineering that airlines have to do, this is easy. It costs - at most, including wages - a few tens of thousands of dollars yearly to come up with these figures. It's a fraction of the salary of one United Airlines BA.[0] This cost might go up if one of the senior developers convinces their boss that this needs to be a machine learning model but unless they're resume pumping it's going to be at most PCA and a regression.
This is not a team of people working for months on this one thing.
I am a bit perplexed though as to why they have implemented fan-out in a way that each "page" is blocking fetching further pages, they would not have been affected by the high tail latencies if they had not done this,
"In the case of timelines, each “page” of followers is 10,000 users large and each “page” must be fanned out before we fetch the next page. This means that our slowest writes will hold up the fetching and Fanout of the next page."
Basically means that they block on each page, process all the items on the page, and then move on to the next page. Why wouldn't you rather decouple page fetcher and the processing of the pages?
A page fetching activity should be able to continuously keep fetching further set of followers one after another and should not wait for each of the items in the page to be updated to continue.
Something that comes to mind would be to have a fetcher component that fetches pages, stores each page in S3 and publishes the metadata (content) and the S3 location to a queue (SQS) that can be consumed by timeline publishers which can scale independently based on load. You can control the concurrency in this system much better, and you could also partition based on the shards with another system like Kafka by utilizing the shards as keys in the queue to even "slow down" the work without having to effectively drop tweets from timelines (timelines are eventually consistent regardless).
I feel like I'm missing something and there's a valid reason to do it this way.
I interpreted this as a batch write, e.g. "write these 10k entries and then come back". The benefit of that is way less overhead versus 10k concurrent background routines each writing individual rows to the DB. The downside is, as you've noted, that you can't "stream" new writes in as older ones finish.
There's a tradeoff here between batch size and concurrency, but perhaps they've already benchmarked it and "single-threaded" batches of 10k writes performed best.
reply