I always wish more metaphors were built using conveyor belts in these discussions. It helps me mentally underscore that you have to pay attention to what queue/belt you load and why you need to give that a lot of thought.
Granted, I'm probably mostly afraid of diving into factorio again. :D
This article is weird because it conflates queues with pub/sub systems. Even to the point of comparing Amazon SQS with Google Pub/Sub rather than the closer analogue, Cloud Tasks.
I've pushed queues from Amazon and Google to thousands of qps and never found their limits. Flow control is unnecessary; the queue either grows or shrinks, and you adjust processing to compensate (ideally by autoscaling). If the queue depth grows continuously and you can't keep up, you work on the architecture (or budget) problems.
If you're storing so much data in queues that you're hitting the storage limits of something like SQS, you are using the wrong tool for the job. It's not a database.
I’ve worked with on prem and now AWS. Even being on AWS I think the first part of the article is great. Nice visual and description. When on prem, scaling was an issue during event storms. But by then we at least had enough to know what the problem was.
Scaling the number of messages is no longer an issue for us using SQS but size is. Basically there are still constraints. We then need to pass references. “Details created etc find it here”. On prem we could dump around 25 MB into message without issue and could go to 50 MB but it wasn’t safe.
I get your point on the queue and pub/sub not being the same. Just to note we do a lot of hybrid and expect others do to. Publish to a topic and bridge to queues. Fan out is easy and gives choice. I don’t know google’s version, with TIBCO EMS this was easy to manage and was clear to everyone. If you wanted to listen to everything on prod.alerts.* you could or if you want to process prod.alerts.devices.headend you could just queue it and process all.
We use queues like storage for long outages so that senders don’t have to change anything. Not a great use but people were sure happy to know we could help by holding all the events while they could deal with their mess. Never got close to any limit when doing this on prem.
Never used it but isn’t the idea of Kafka to hold all events like a database? Love the idea. Seems so lazy and useful at the same time. Now that I write this I can see the danger too. You become a transaction system of record. Ugh. That’s someone else’s problem ;).
I do not know, if SNS is actually based on Kafka, but Kafka has LogRetention, which is imho enabled by default. So your events disappear after some time. So no, holding events like a database, i would not agree. Database'ish :) Especially with all the layers on top, like SQL.
Now I wonder if Athena can actually run on SNS topics, hmmm.
It also assumes the producer and consumer are monolith-bound data streams. Almost all AMQP back-ends include traffic message-pattern smart-message-routing to pre-sort the data as kind of an implicit sharding operation. This pattern is also particularly performant using static route definitions.
Unless one is using Elixir/Erlang channels, than eventually one must glue on the equivalent queue structure to handle concurrency spikes. Note, this is less to do bandwidth peaks, but rather the OS ISO/OSI limitations will become the bottleneck for concurrency.
If your data traffic is truly separable, than it is trivial load-balancing across multiple pattern-matching routing consumers. Thus, you get pre-sharded localized data, and simply avoid the stores state-syncing cost on back-channel links.
If one has over 40k users, than they should already have re-discovered this phenomena. Simply repeat the mantra "concurrent consumers should never share state". =3
> Flow control is unnecessary
> compensate (ideally by autoscaling)
Isn't that flow control of a different form? It's a new form enabled by the cloud moreso than any previous technology, but it really does seem like a version of "egress based flow control" to me; except, we're widening the output rather than narrowing it.
Queue failures are just a destructive form of back pressure.
Eh, maybe if you generalize the term flow control beyond useful meaning. The author has specific ideas (Backpressure, TTL, and Tail Drop) all of which might be narrowly tailored to his/her specific scenario but aren't things people generally need, as per the title of TFA.
The article states that "Other event sources like SNS use so-called Event Source Mapping", but (at least to my knowledge) SNS directly triggers the Lambda. SQS and few other "stream and queue-based services" use Event Source Mapping.
Granted, I'm probably mostly afraid of diving into factorio again. :D