One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.
If you’re message consumer isn’t idempotent then no MQ can help you. Exactly once delivery is impossible other than with at least once delivery and an idempotent consumer.
> Exactly once delivery is impossible other than with at least once delivery
Can you explain this? Don't many applications deliver once and only once via locking? It's obviously easier as an application developer to say "I will only get this once" and accept losing messages than dealing with idempotence particularly in distributed services.
Locking is no longer 100% reliable as soon as you have horizontal distribution of the same data over multiple nodes (for redundancy, so you can guarantee delivery) instead of sourcing from e.g. a monolithic rdbms. Eventual consistency is the model for a whole lot of distributed systems, e.g. S3 or Mongo. The CAP theorem applies to more than just databases, so MQs tend to use eventual consistency as well, which looks like at-least-once and not guaranteed exactly-once delivery.
While dealing with at-most-once delivery is easier in isolation as an application developer on the consumer side, dealing with lost messages is in practice MUCH MUCH harder on the producer side than idempotent handling where required on the consumer side. You end up building elaborate mechanisms for locking and retries and receipt validation which can all fail.
Just think about emails, there are a lot of situations where you can't be 100% certain whether the other side received the message or not. For something low-priority it may be better to ignore partial failures, but if it's important it may be better to send a second message to guarantee delivery. If you're modeled on never sending the same message twice AND the messages matter, you're in trouble.
No. By default SQS uses `at-least-once delivery` system. It's distributed and when you connect to API one SQS node that you working with could have different state comparing to another, so you may fetch the same message twice by 2 separate workers even with big visibility timeout.
Amazon's hand an answer to that with FIFO queues for the better part of years, but even without a queue based system you probably still need some notion of idempotence in the actions your system takes.
Systems "without queues" have just eschewed one queue for N queues, where N is the number of clients buffering actions for retries at any given moment.
FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated
And it comes with the trade off of 3000 (batches) messages per second (and I bet some reduced potential availability). Sure, it’s a lot of messages for most people but non-FIFO SQS is one of the truly limitless services offered by AWS. No limits to raise, no soft maximums AWS staff ask you not to exceed. Just limitless scaling.
Implementing a queue that does exactly once delivery is a very hard problem and comes with its own trade-offs. Mostly around scalability and reliability.
