Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Exactly-once delivery counter using Watermill messaging library (github.com/threedotslabs)
18 points by roblaszczak 9 months ago | hide | past | favorite | 9 comments

> 1. you need to use a Pub/Sub implementation that does support exactly-once delivery (only MySQL/PostgreSQL for now),

To implement exactly-once delivery you first need exactly once delivery.

But this project doesn't implement exactly once delivery.

> It can be triggered by calling the http://localhost:8080/count/{counterUUID} endpoint.

Oh, but what if my network goes down right after I sent that request? Did it succeed? There is no way of knowing. So now I have to pick between sending again (at least once) or not (at most once).

You can get exactly once delivery. But you need to involve request IDs which you haven't done here. Imagine something like this:

  PUT http://localhost:8080/count/{counterUUID}?request-id={requestUUID}.
The requestUUID must be unique per counter. It should be generated when the action occurred (like when the user clicks a button) and should be reused for retries. (Of course this will still be foiled by a user clicking a button multiple times)

Now the server does:

  INSERT INTO increments(counter_id, request_id, counted)
  VALUES (:counterUUID, :requestUUID, FALSE)
Now instead of deleting the record you just mark `counted = TRUE`.

In theory you can never delete records from this table. In practice you can add a timeout after which double-counting is acceptable.

Exactly-once delivery is covering processing in this scenario. Because of that it is not covered in tests and schemas. REST endpoint was just added to create entire use case. But you are right, if in this part of the system something will go wrong you will have duplicate message. But it’s rather at least once publish rather at least once delivery. Message will be delivered once, but published twice :)

But that seems pointless. Exactly-once covering a subset of the pipeline doesn't guarantee exactly-once. At best it improves your chance of exactly-once delivery.

If you really need exactly-once delivery you need to make it end-to-end. If you don't support end-to-end exactly-once than you are really just an at-least-once delivery system.

So you are making a guarantee that provides little if any benefits because it can not be tied into the whole pipeline.

This seems like another odd definition of "exactly once". The underlying assumption appears to be that all you work is happening insides the sql transaction, so all the work the consumer does is safe and can be rolled back if a failure happens. Many systems are going to be doing at least some work that can't be easily rolled back by a transaction (e.g. calling an external API). In this more interesting world, you really can't get exactly once since we already did something outside of our transaction.

In my experience, it's easier to reason about and build systems when idempotency is an application level concern. For example take a bank that has some messaging system to update account balances. While a exactly once system, if designed perfectly, might achieve this, you could also achieve this by building an idempotent "update balance" system. With application level idempotency, you have more flexibility to later add different paths or technologies without as many re-write headaches.

Also -- the messages per day stat seems irrelevant. I've yet to encounter many real world systems that don't have irregular bursty patterns. With this slow processing rate, you could basically have a single large burst and then normal traffic, but be unable to return to realtime latency for hours/days.

> In my experience, it's easier to reason about and build systems when idempotency is an application level concern.

This is exactly what we are doing in my company :) But it’s always good to have an option and know that we can do it in specific cases without external calls. In this scenario it can be nice simplification.

I think this ACID-based approach is a reasonable way to achieve exactly-once message processing, as long as one is happy to limit that guarantee to only DB operations.

If anyone is interested in a failed attempt to implement this you can see my write up [1]. It contains some learnings from the experimental implementation I created for Lightbus.

The implementation was based on this presentation [2]

[1]: https://github.com/adamcharnock/lightbus/issues/4

[2]: https://vimeo.com/111998645

Exactly-once delivery in a distributed environment is impossible. Close-to-exactly-once is possible but there is always a way to have a situation where exactly-once fails once you get a network involved.

Changing the definition of exactly-once delivery won’t help your users.

Seems like any key value store would be better for this. Redis, DynamoDB, etc.. could get a lot higher throughput on messages.

It's not about the throughput though, but about consistency. You can't do the same thing on Redis if it's not your main storage. You need to have transactions on level of the database engine your application uses.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact