
Our Stripe Billing implementation and the one webhook to rule them all - tnolet
https://blog.checklyhq.com/our-stripe-billing-implementation-the-one-webhook-to-rule-them-all/
======
DonHopkins
The problem I had with implementing subscriptions with Stripe is that the
batches of messages from the webhook could come back in clumps in random
order. So they would have references to objects that I hadn't been notified
about their creation yet.

Theoretically and rarely a webhook could fail and be retransmitted arbitrarily
later due to bad weather on the internet, so you have to be able to tolerate
that, but practically and often it sent bunches of messages all at effectively
the same time, which caused them to be processed by my web server in random
order.

I finally gave up trying to structure the code so it could create objects in
any order, and deal with objects it hadn't heard about yet, and just treated
the webhook callback as a sign that I should soon make a request back to
Stripe and ask them for ALL the events that had transpired.

So I'd log all the information in the webhook just for chuckles, then schedule
a task that polled Stripe for batches of events, and dealt with them all at
once without anything slipping through the cracks because of random reordering
in transit.

~~~
WestCoastJustin
I have run into this also using subscriptions + webhooks. For example, I am
using the "charge.succeeded" webhook event to send a custom email receipt on
my backend. But, for a few new customers there seems to be a race condition
where the "charge.succeeded" webhook event will arrive, before the API call in
my code returns a success, so there is now an event but I have no idea how to
tie this back to a user. I am using the API to create a new customer and
updating their user DB record with the customer token from stripe. So, I get
into a situation where I do not know what customer to send the email too
(because I don't have the customer token yet). I ended up just adding a HTTP
503 error (Service is Unavailable), for this specific webhook event when I
cannot find the user, so the Stripe API will retry that event. This is a hack
but it works. This just started happening a few weeks ago. There are a few of
these things popping up here and there that I need to deal with but generally
it works really well.

~~~
danpalmer
We just use a database lock for this. Lock on the charge/order/user/whatever,
then the stripe webhook delivery is blocked until the transaction commits.

This might seem like a pain, but Stripe have no idea how fast their clients
are, there’s not much point in waiting, say, 1 second minimum before
delivering the webhook, as that only alleviates the problem for clients fast
enough to “finish” within 1 second, whatever their definition of finish is.

For this reason, I think Stripe are doing the right thing, and I’m not sure
there’s much they can do to make it much easier. Once you know it needs to be
done, a solution is pretty straightforward.

~~~
DonHopkins
Yikes!

You really should respond to the webhook handler immediately, by queuing it,
and handling it asynchronously in another process.

It's bad enough to perform slow operations like creating lots of objects in
the database, calling other web apis, sending email, before you return a
success to stripe's webhook.

But for one webhook to lock out all other webhooks while it did all that work
would only compound the problem.

Stripe tends to send a whole flurry of events related to the same transaction,
when there's actually only one thing for you to do in one swoop (create a user
and start their subscription), so to processes them one by one is very
inefficient, especially if you have to call back to stripe for each event.

~~~
danpalmer
Sorry, maybe should have clarified, webhooks are typically processed in a
queue, we reply to Stripe immediately, but we block the processing of the
webhook on that lock. As we're processing in a queue with multiple workers
this typically doesn't block much work from happening.

There are some cases where we specifically want to propagate the error to a
webhook provider (not in the case of Stripe, so we work inline, but that's
rare). There are also some cases where we want to process webhooks on a serial
queue, one at a time, to ensure in-order delivery (again not in the case of
Stripe).

------
DonHopkins
One thing about the Stripe API that I loved was that I could intertwingle it
with my own admin interface, by pushing descriptions and metadata properties
into stripe objects that had url links back into corresponding objects in my
admin interface (like users, products, coupons, subscriptions, transactions,
etc).

Though it may not be explicitly documented, Stripe's web site is smart enough
to make the urls in metadata be clickable links, which is a godsend.

So I didn't have to duplicate stuff you could do on stripe's site in my own
admin interface, I could just link back and forth between them!

------
politician
(Yikes) I feel compelled to remind the folks complaining about webhooks
arriving out of order of the fallacies of distributed computing.

[https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing)

For goodness sakes, log webhooks to a queue and rebuild the object graph later
by probing their API. I would be absolutely blown away if Stripe recommended
that folks attempt to process these inline (simple tutorial-level example code
notwithstanding).

EDIT: And for your own sanity, assume 50% of the webhooks you expected to
arrive didn't. Schedule a periodic task to scrape transactions from their API.

~~~
dbbk
> EDIT: And for your own sanity, assume 50% of the webhooks you expected to
> arrive didn't. Schedule a periodic task to scrape transactions from their
> API.

I'm pretty sure the Webhooks keep retrying until they get a success response
back from your server.

~~~
quelltext
They do that but not forever: [https://stripe.com/docs/webhooks/best-
practices#retry-logic](https://stripe.com/docs/webhooks/best-practices#retry-
logic)

~~~
dbbk
They do it for 3 days though which seems more than enough... and you can even
manually trigger them from the dashboard. So I'm not really seeing why you
can't just rely on Webhooks?

~~~
quelltext
You can rely on them. I just wanted to correct what was said in that comment.

------
deepwell
You just leaked your customers email addresses by improperly obfuscating them.

See
[https://web.archive.org/web/20190409072021im_/https://blog.c...](https://web.archive.org/web/20190409072021im_/https://blog.checklyhq.com/content/images/2019/04/image-3.png)
With knowledge of the font used (which is very easy to figure out as its the
Stripe dashboard) everyone can reconstruct these email addresses.

You should now notify these customers as well as your supervisory authority.

~~~
tnolet
Thanks for bringing this to my attention. I will have my legal help check it
and take the appropriate steps.

------
eemax
Cool post! I don't have a ton of experience using Stripe, but shouldn't you at
least be handling some sort of payment_failed webhook?

It looks like you call _createSubscription and set the initial value for
currentPeriodEnds before you know the payment actually succeeded, and since
you don't ever check or listen for failed payments, anyone could get a free
month (or year) of Checkly by using a bad card, or if the payment just
randomly fails.

Maybe this isn't a huge deal in the early days, but you and your customer
might not even notice the failed payment for quite a while unless you happen
to check your Stripe dashboard!

~~~
tnolet
This is a great comment. And it's stupid I left this out of the post, as I
made a conscious decision to not deal with that now. I should at some stage.

I actually had one failing credit card already, but my customer base in in the
30+ under 100 range, so I easily caught it. Also, it was totally benign from
an early customer that just had bodged renewal for their card.

------
tnolet
OP here. I'm super curious about other folks using Stripe Billing and their
experiences. My SaaS is fairly young, so I probably have missed some things.

~~~
kwindla
Great article!

Here was a related discussion, last week:
[https://news.ycombinator.com/item?id=19556579](https://news.ycombinator.com/item?id=19556579)

~~~
tnolet
Oh wow that's a cool post. The comments kinda echo the spirit of my post. Yes,
Stripe has a good API and SDK. The actual code is not the problem.

The problem is the whole workflow, how it interacts with your customer and
with your business backend processes.

------
bschwindHN
I wrote a Stripe subscription integration a couple years ago in Scala. The
product never worked out and it's running in free mode now.

For fun, here's the main controller code that processes the webhook events
(obviously there is more logic in other files)

I had never worked with a payment service before so there may be mistakes, but
it worked well for us for the time it was in use :)

[https://gist.github.com/bschwind/1371a196920981c1dea2b0e4f02...](https://gist.github.com/bschwind/1371a196920981c1dea2b0e4f028e442)

------
mike1o1
Great post! The number of subscriptions that Stripe sends out is really
overwhelming at first, and especially the order they can be received in. I
ended up coming to the same conclusion that you did, and only end up listening
to the invoice payment succeeded and invoice payment failed webhooks.

Doing so greatly simplified the process, rather than listening for customer or
subscription updates and changes.

Great to read others are taking a similar approach!

------
blackdogie
Thanks for sharing. Very timely for my team.

------
dplgk
Why does this one webhook "rule them all"? Because you can scrape by by
ignoring Stripe's other webhooks?

