Hacker News new | past | comments | ask | show | jobs | submit login
Engineers' billing nightmares (github.com/getlago)
204 points by AnhTho_FR on Nov 28, 2022 | hide | past | favorite | 120 comments



Eh, I've worked on billing, payments and salary systems, some of these problems are not that awful. It sounds like the source of much of it is just classic underestimation of the amount of work & complexity to implement something. If someone gets away with saying "shouldn't be hard right?" to the point it dictates the size of the team (2 engineers? That's barely a team), then you're going to have problems.

Some of it comes down to picking the right tool for the job. The article doesn't touch on technology choice at all, but it matters. When I built a salary system, I deliberately chose a tech stack that had good numeric calculation support (so not JavaScript). These choices matter and have lasting impacts on your project. Same thing for handling dates, for example here .NET and Java have excellent standard library support for everything date related. Other languages, not so much.

Another point the article talks about is change: companies change how they bill over time; customers change their plans. Again there's nothing specific to billing systems here. All systems have to deal with change. All systems beyond a trivial size end up with migration scripts, and sometimes those scripts end up being productized and becoming buttons in dashboards.

I guess my main point here is don't underestimate what might seem to be an "ancillary" product when it's an essential service that makes sure cash keeps flowing. Treat every important project with the care and respect necessary. Allocate resources properly. Don't underestimate. Oh, and don't underestimate.


My boss told me to estimate the effort it would take to make an app that integrates with a bunch of server systems. He estimated 'x' units of time, thinking only of the core functionality of what the app actually does. I told him it's more like 10x, because the core functionality is only a small part of a product. Billing, testing, the dev environment, customer-facing documentation, the product web site, etc... all add up.


What did it end up being? I’m guessing closer to your order of magnitude? Excluding scope creep, of course.


He abandoned the project before it even started, even though IMHO it was a good idea. Similar products that ended up with a wider scope include the likes of Okta, so clearly there was a large addressable market!

Something I've found is that there are two types of people when it comes to planning / forecasting:

SALES -- Wants to hear only positive things. Likes to ignore predictable issues, pretend unlikely risks will never happen, just go-go-go. Often sounds unreasonably optimistic to other people, which can be off-putting.

ENGINEERING -- Wants to hear all the bad things before committing. Likes to mull over all predictable issues to determine if they're show-stoppers and/or if workarounds are available. Often sounds unreasonably pessimistic to other people, which can be off-putting.

Both types can be in agreement that a project is a "good idea" while one type is listing only the upsides and the other type is listing only the downsides.

This can result in rather poor communication and missed opportunities.


> It sounds like the source of much of it is just classic underestimation of the amount of work & complexity to implement something

Yeah, but this doesn't help, because the problem is that billing is perennially underestimated. Everyone thinks it looks easy, and offers "solutions" like those in this thread. And mostly they work, until one day you wake up and you no longer know how your bowl of spaghetti code works.


A lot of things are perennially unterestimated, and making it work anyway is simply an engineer's job and can be quite fun. If it literally gives you nightmares, e.g. because a bad estimate gets turned into a deadline, chances are your job just sucks (and likely your company, too).


Don't forget the accounts receivable and reconciliation side too. I built one of those fresh out of college for an independent pharmacy chain. My and my boss spent years working on it and every time we thought we had it perfect another wrinkle would show up.


Yes, these are great additions. Reconciliation is a doozy, and there are so many reconciliation points - payments, usage data, supplier invoices.

And then there is revenue assurance…


I’ve written a subscription billing app on PHP and MySQL. Just the database queries to figure out what has been paid or not (x in-full/partially/over x in-time/late) are absolutely horrendous to get right even without the complexity of VAT, multiple subscription types, usage billing etc.


yeah that part is strangely hard. I did an rx pricing system for a pharmacy chain early in my career and the weirdest things would happen. The chain would just get random checks from insurance carriers with no EOB or anything to know what invoice to tie the check to. Then, when the insurance company realized their mistake, they wouldn't call us or cancel the check or anything they'd just short a invoice for the amount of the check or sometimes even divide it over multiple invoices. This would happen like 6 month later.

we had an account that was literally called "magic money" where these checks were deposited. Then when an insurance carrier shorted their invoice and we could find no rational reason why we would use the "magic money" account to cover it.


.NET timezones are not really practically useful. I suggest using noda times with its IANA timezone implementations. The EF Core pg adapter has 1st class support for noda times thanks to Shay Rojansky.


Why not javascript specifically? Using integers or floats is wrong in any language. You want a decimal representation library when dealing with money.


No. You want a parametrized "Money" type, one that can do calculations in varying numbers of decimals, and configurable rounding. There are legal requirements that make this necessary.

One (from memory, not exact) calculation I've had to implement for a customer: Multiply prices by whole numbers. Prices are given in cents subdivided to 5 digit precision, result should have 10cent precision, rounded down. Then, for the tax calculation, get taxes from some tables, 3 different taxes for each line item. Apply the first two taxes, Euros with 10 digit subdivisions, then round "according to trade custom" (which is the usual "up from .5 cents, down from below .5"). Then add the third tax, Cents with 5 digit subdivisions, round down. Then add all the line items and their taxes, separately as well as together.

And if there is a Skonto, take care to properly unravel the tax stuff again.

I've had to do it in plain Typescript, it is possible, but it really sucks. Those days when you yearn for COBOL... ;)


> Why not javascript specifically?

Javascript has one number type; float.


It's fair to say most JS code is only handling floats especially if you're just relying on JS's automatic casting but JS does have BigInt as well, and it has a whole bunch of numerical typed arrays that only accept specifically typed numbers like Uint8 or Int32 etc. You can do math in JS quite safely if you delve into the less well-known types a bit.



1) Wow, I didn't know that existed.

2) Wow, those downsides are insane. It's not constant time, thus can't be used in cryptography...? Has huge performance impacts and can't be casted to a "number"- how is this implemented? it can't be native INTs on the CPU?


Any cryptography is surely using Uint8Array.


>Any cryptography is surely using Uint8Array.

Is this to optimise memory operations over a large plaintext/cipher text or is there some other advantage to it that is less obvious (e.g. more resistant to timing attacks?)


No, it’s because Number is internally floating point and cannot represent all integers.


"Integer number of cents" is a fine way to deal with money unless you want sub-cent units. Otherwise, you can divide it up smaller: a few stock exchanges use [price in dollars] * 10^8 as their price unit, and store a 64 bit int.


> "Integer number of cents" is a fine way to deal with money unless you want sub-cent units

I use this approach to minimize the amount of time I have to waste on trivial money operations such as storage and api's, but this approach (number of cents) falls apart quickly the moment you need to do something as simple as calculate tax or a commission. So, still likely no getting around proper float math even if you may not really be all that interested in sub-cent units.


You really want to use a decimal library for any serious financial math. And if you’re doing the math on tiny units of processing you’re going to want the precision and accuracy.

You don’t need to work in decimal cents. Many work in thousandths of a cent or smaller. But you want an infinite upper and lower bound. Even 64 bit integers only has so much upper range depending on how far you subdivide a penny. You also want the precision guarantees when you take those numbers and do more than just add or subtract them.


Why do you need an infinite upper bound for money? I agree that you need to do saturating arithmetic, but 2^64/10^8 dollars is about 10^10 dollars, which is the around the GDP of the US. A number that big is not going to show up in your billing system if you do everything correctly.

With all numerical calculations, you need to be careful about losing precision on mathematical operations, so it's a good idea to think about things like this even if you use a decimal number system with a very wide dynamic range - it won't be very precise all the time.


Exactly, and BigInt fits this role perfectly.


I was thinking the same thing. It’s like “(computer thing) is hard” is the new thing


Only if you can use modern Java.

But yeah, also VAT calculations should be outsourced how ever possible. In the US, sales tax is at the postcode level so it gets really crazy.

It sucks you have to pay VAT even on digital goods and services nowadays. Ideally it'd be abolished entirely.


If you think that government will give up (should they?) on all that tax money, you're probably fooling yourself...


Ha yes I almost edited my comment earlier to mention that calculating tax is probably something most people should just outsource to a third party.

I also have experience with wage tax calculation systems, and the solution we went with was browser automation that used a third party wage tax calculator. That stuff can be impossibly complicated -- and with many governments it changes every year too!


> it sucks you have to pay VAT even on digital goods and services nowadays. Ideally it'd be abolished entirely.

what do you propose as replacement? It's not as if the governments are going to let go part of their tax revenues


+1 on the abolishment!

What would you suggest to outsource VAT calculations? We've yet to find a tool / platform we really like, that we could partner with.


AFAIK, Avalara is the defacto player on this kind of stuff


Yes,it is! but I'm wondering if there is a player that is a bit more: - Developer-friendly - SMB-friendly?

Not sure how much it costs but doesn't look very SMB/startup-friendly...


One should keep in mind that this is an article from a startup that wants to sell you billing software, so take some of the points with a grain of salt. Definitely some truths in there but having built inhouse billing solutions I can say that some of the issues pictured here are quite exaggerated. If you integrate with a payment provider like Stripe they e.g. already provide some support for tax codes, and even if not the rules aren't rocket science, every online store that sells internationally has faced the same issue.

In the end, if you let a third-party provider run your billing you'll be dependent on their solution, and if something is missing that they don't care to provide you're just out of luck, or back to building it yourself, which is why these third-party solutions aren't terribly popular. It's very difficult to build a standardized solution that just works, so relying on outside software can make your problems worse.


I worked on a financial product in a startup. The billing and collection features were so closely tied to the product that it would be quite risky to outsource that when you're still trying to find out what your product is exactly.

In fact, we outsourced part of the software for the late payments collection process to a reputable SaaS and the results were not great.

My take is that if your needs are fairly standard, you can benefit from such service. Especially your time to market will be shorter. But you won't face any disaster should you decide to do it yourself. If you want to innovate how you bill people (most Fintechs do), you should develop in house.


Sure, until you find out Stripe is killing accounts wrongfully[0], something that has been happening since a long while. Even the first comment there hints it[1].

> It's very difficult to build a standardized solution that just works, so relying on outside software can make your problems worse.

I want to see you doing billing without software.

Also, Stripe is not available for every country under the Sun because it's U.S. based. Not that Lago isn't, but at least Lago is open source so you just use it locally.

--

[0]: https://news.ycombinator.com/item?id=33743750

[1]: https://news.ycombinator.com/item?id=33744053


Our payment/billing backend is built so that it can support multiple payment providers, as some larger customers e.g. pay by bank transfer. Stripe is just there to store sensitive billing data like credit card numbers and actually charge customers, everything else happens in our backend. Stripe supports that use case well, they e.g. provide web hooks and an event log which you can replay locally to integrate with them, which makes it easy to run your own billing/invoicing logic on top.


Some of this complexity is self-imposed. It's flailing around and trying many different things on a slow meandering path to success. Often the trick is to simply copy what a bigger org has done after a decade of experience.

As a random example, Azure generally bills for resources on an hourly basis.

Suddenly, months, years, calendars, etc... cease to matter. Hours are hours. You can use UTC for everything, ignore time zones and daylight savings, and aggregate however you like, or how the customer likes.

Similarly, consumption-based plans are relatively straightforward. Simply multiply unit costs with the per-unit costs. You can even blend this into the hourly model above, where each hour has a "units consumed this hour" and a "unit price per hour". This way you can handle things like changes to the unit price fairly accurately.

With cut-overs between plans, the trick is to just bill for them as separate line items instead of one thing going through transformations. E.g.: 220 hours of Standard SKU and 500 hours of Premium SKU this month. It doesn't matter how many times the customer changed their mind this month that way.

A nice trick is that data like the above superficially sounds like it'll be huge in a database, but column-based compression works wonders on it. Any analytics database engine, columnar format, or reporting engine should reduce it by at least 30:1 or more. Also, you now have a ready-made reporting system that allows slicing and dicing! In some database engines like SQL Server, this is as simple as making the main billing table use a Clustered ColumnStore index. In more cloud-native land you could use Parquet files in S3 buckets, or whatever.

I'm not saying that complexity doesn't remain, it's just that complexity shouldn't be voluntary. Copy a working model that matches your industry, and then only the essential complexity will remain.

An example of a hard-to-solve essential complexity is currency conversions and taxation when dealing with international customers. That's an externally imposed set of problems that can't be hand-waved away.


After seeing billing done right, I cringe every time I see billing done wrong.

For example, people building foreign keys to prices and receipt items. In what cases do you expect to retroactively change what you charged all customers? Copy that data to the receipt item, don’t give your accountant a headache and heart attack when you change pricing. I’ve also seen people using those same kinds of rows to determine what features a user gets access to. Don’t do that, use capabilities/roles. When a payment succeeds, update the user’s capabilities. When a payment fails, remove capabilities. This also allows support to just give a customer capabilities without a line item or changing their plan, because some customers are just annoying and it is less expensive to give them a feature. It also allows more easily selling custom-tailored solutions to bigger clients since the features aren’t linked to line items.

This article is more about usage-based billing, which is probably harder to get right, but feature-based billing is something I’m more familiar with and it’s so annoying to see all the ways people will fuck it up.


Your first point about foreign keys was the first contract I ever did. Someone built a billing system in perl+mysql for an ecommerce site. When prices changed so did historical invoices. I was hired to work this out because the existing developers couldn't. They only realised realised that something was wrong after a customer sued them after being paid the wrong amount in a refund.

I rewrote the entire system to snapshot the committed values at the time of purchase and fixing all the old invoices from database backups.


The way you'd assume they'd do it if they wanted relational values would be to never be able to edit prices, only create new ones.


> never be able to edit prices

I can't think of any database that makes a row strictly immutable, as in carved onto the platter immutable. People will find ways to undo any soft-guards you put into place, all in the name of being helpful.


OP here, totally agree!

Feature-based billing is super hard too, and people underestimate the complexity even more.

We were genuinely surprised that some companies wanted to use Lago even if they ""only"" had a "feature-based/subscription-based" pricing, that someone new would consider as "trivial".


(I'm a billing engineering team leader with more than 25 years experience.)

This is a wonderful example of how engineers underestimate the complexity of billing.

> Suddenly, months, years, calendars, etc... cease to matter

No they don't. You don't send a bill every hour, or even every day. Typically you invoice monthly. And if you invoice monthly, what date do you aggregate the invoice? Setting aside the obvious problem that months are not homogenous, do you start on the anniversary date of signup, or the start of the month? Well, anniversary date is generally better, because it spreads out payments, collections and cash flow over the whole month, which reduces a lot of effort in collections and finance, and reduces the impact of any issues - but it's also significantly more difficult to engineer; how do you bill February if they started billing on the 31st of January? Monthly billing is far easier from an engineering perspective, but is not generally the ideal method from a business operations perspective. And get this, if you start with monthly billing and have significant income, and then you want to change to anniversary billing, you get all kinds of income recognition (P&L) problems that you have to deal with; it's really complex.

And what do we mean by "signup", anyway? Do we bill services individually based on when they are created, or do we bill based on the account creation date? What if there is an account hierarchy?

Hourly or unit based billing does literally nothing to solve these problems, because they are endemic to the billing domain.

> Similarly, consumption-based plans are relatively straightforward. Simply multiply unit costs with the per-unit costs.

Right - and what happens when you raise or lower your prices? That has to happen at a particular point in time, which is typically in the future. What if we give tiered discounts (first 1000 minutes at 2c, next 1000 minutes at 1c)? In both cases, we no longer can keep a single per-unit cost, but rather we end up with a cost matrix, which can become really complex when you add other factors (such as discounting).

Also, what happens when you are charging, say, 5c per megabyte... and your incoming CDR feed gives you bytes? Do you configure this as 0.00000005 cents per byte? When do you do the rounding? On an hourly basis? Daily? Monthly? What do you display as the unit price to your users? What if another plan is charging 5c per Gigabyte?

> An example of a hard-to-solve essential complexity is currency conversions and taxation when dealing with international customers. That's an externally imposed set of problems that can't be hand-waved away.

Currency stuff should be very easy in practice, but I agree that tax is hard. That's why you need a multi-stage pipeline for invoice generation. But that's perhaps a story for another post...


> (I'm a billing engineering team leader with more than 25 years experience.)

As a software engineer with >25 years of experience, now running my own SaaS business and dealing with subscription billing, invoicing and tax issues:

1. I agree with everything you wrote. And more. Taxation introduces an entirely new level of complexity on top, and lawmakers design laws for humans, rarely thinking about automation. If you don't believe me, look into EU VAT rules, and enjoy the mess with GB/XI prefixes, the case of AX, and parts of ES that aren't really in the EU VAT zone, but look like they are.

2. The responses to your comment are typically dismissive and are to be expected from this forum. Usually the replies are from people who do not do the thing you do, but think that the thing you do is simple and have all kinds of opinions about it. I used to care more about replies, these days I just smile and move on.


1. Thanks! We were lucky that we dealt only with the (non-US) Asia/Pacific region, where all the tax regimes we encountered in the countries we operated were relatively simple, mostly straight percentages. But we did have to deal with tax system transitions in a couple of countries, and it was obvious to us that tax is one of those things that was quite arbitrary, could be changed overnight (which happened in Malaysia after an election), and needed to be factored out as its own domain. As I’ve been trying to argue, there are no “perfectly spherical cows” when it comes to billing… simplifying assumptions end up killing you slowly.

2. I think it’s often young people who haven’t encountered externally-imposed, non-negotiable complexity yet. I try to take it easy on them because I suspect I might have been like that once :)


EU VAT aren’t too bad compared to the North American ones. But I grew up in the EU so it clicks more with me and dealt with foreign customers


I agree. Billing is fearsomely complex and not ever going to get any simpler. I feel the old rule about not rolling your own crypto should equally well be applied to not rolling your own billing, tax or finance systems. That said, the alternatives all suck. If you are a F500, sure go ahead and spend $$$ on a SAP implementation. But whats a small 10 person company turning over a mill or two to do?


It's interesting that you consider what are solved problems in the world of RDBMS to be difficult, but currency conversions simple.

Out of curiosity, does your system use a NoSQL database?

> Right - and what happens when you raise or lower your prices?

The billing table has "CustomerID, ResourceID, HourID, Units, SKU_ID" columns. You raise or lower your prices on the hour boundaries. The new rows inserted after the price change will have the new SKU ID with the new costs.

Irrespective of when you send an invoice to the customer, the report simply does a "GROUP BY" over the (CustomerID, ResorceID, SKU_ID) columns and some range of the HourID column. You sum up the Units * SKU_ID.UnitCost value, or whatever.

This isn't exactly rocket science for any normal database engine.

I suspect the error people are making is not recording the change-of-billing-model as a "slowly changing dimension", and instead trying to do complicated rules-engines in code...


> Out of curiosity, does your system use a NoSQL database?

No, the last system I built was using PL/PGSQL. It's brilliant for this application.

> The billing table has "CustomerID, ResourceID, HourID, Units, SKU_ID" columns. You raise or lower your prices on the hour boundaries. The new rows inserted after the price change will have the new SKU ID with the new costs.

Like I said, this is a very simplistic way of viewing how to do billing. Sure, today you only have the hour dimension when choosing a price. But then sales asks you to give a discount to a specific, very large, customer. For only a single resource. For only the first three months. Suddenly, and very predictably, your billing is not so simple any more.

The problem is not that simple billing is difficult. The problem is that billing is not simple.


> The problem is not that simple billing is difficult. The problem is that billing is not simple.

Love this quote!


> Like I said, this is a very simplistic way of viewing how to do billing. Sure, today you only have the hour dimension when choosing a price. But then sales asks you to give a discount to a specific, very large, customer. For only a single resource. For only the first three months. Suddenly, and very predictably, your billing is not so simple any more.

I'm not sure if I'm missng some part of the complexity here, but with the solution jiggawatts proposed, wouldn't the above scenario be fairly trivial to solve?

Just implement a new SKU for this special one-time (more like first time...) discount that the account manager applies to the account.

If you want to be fancy, you could code in (or leverage one of the dozen cloud tools, depending on where you're deployed) to enable the SKU for the customer at period X and remove it at period Y and replace with the previous SKU.

To ease reporting, in your main SKU table you could have a parent_sku column and a sku_type column explaining that this row is a special discount child SKU of the parent_sku. Add basic databse restrictions and validations.

Billing is complex and requires lots of domain knowledge, but IMO it's not really THAT hard from my experience.

EDIT:

IMO a harder scenario for any billing system to deal with is account manager A made a sweetheart deal to make customer B less angry, removing one SKU completely from last months bill, giving them a 25% discount for the next invoice, and giving them some freebie.

These types of requests usually end up with a lot of issues unless the business logic promised to the customer already exists.


Well, the problem with the first scenario is that if the discount is percentage based (say, 10% off), then it's going to require code to calculate. It's no longer just an SKU - it's an SKU which does a calculation, which is presumably not part of the platform already. And so we have a new "discount" SKU - but which line items should the SKU apply to? How do we know when to stop using that SKU when the discount runs out? And anyway, why should an SKU have a discount % field on it? Why do we get different behaviours based on some magic SKU field? It's a breakdown in cohesion... and so the spaghetti begins...

You can also argue that we can just "get someone" to apply discounts manually, but isn't that the job of the billing system? Manual labour doesn't scale too well, and a lot of mistakes get made - asking a customer to return an accidentally applied discount is a very, very difficult conversation, and I've worked with companies who simply had to absorb years of incorrect discounts because doing so was the only way to keep the customer.


Also, here is an example of 'looks simple' but 'raises many question'

1/ Should it be a discount on future invoices, or a refund (an angry customer will ask for a real refund: aka $$ on their bank account)?

2/ Do you want to keep track of how much has been discounted/refunded to which customer (so that you can identify deeper trends: if a segment is being discounted many times: maybe you can adjust things, or maybe it's fraud?)? In that case, just erasing a line doesn't help

3/ If you refund, how do you deal with taxes that have been paid already?

Btw, we've written a quick primer on Refunds, Coupons, and Credit Notes: to lay out the pros/cons

https://github.com/getlago/lago/wiki/A-Primer-on-Refunds,-Co...


#3 is an absolutely classic example of unanticipated complexity. Tax might differ depending on individual line items on an invoice, so if you’re refunding $50 on a $100 invoice, how do you compute the tax payable for the invoice?

If memory serves, we ended up solving this by pro-rating the tax against the refund amount, on a per-item basis. And, I think, some smart accounting.

Oh and then you have the problem of when tax is becomes payable. Is it when the invoice is raised (accrual based)? or when it is paid (cash based)? And if memory serves, it depends on the size of your company. :head asplode:


Dont forget co-selling discounts. First 100 messages free with a monthly resource. And is that per monthly resource, or each monthly resource?


Where is the SKU (equivalent) applied? IME the metering data is sourced from or near to the service. We cant move these per-customer-per-period variants there. So instead you have a fixed usage type/metering and use arbitrary “pricing plans” that have differential pricing to be applied post hoc.


The problem, apparently, is, that even in the largest software installations there are a lot of snowflakes to be taken into account (your "very large customer").


Yeah, as engineers we want to say "well tell them we can't do it". But in reality, if the customer represents 25% of your income and they are dead set on getting their way, what you are really saying is that we should lose the customer - which translates to, "we have to sack a quarter of our team". (I had a CEO who used to use exactly this analogy with us).


> "well tell them we can't do it".

hah yeah i've tried that a couple times over the years, it never worked.

Another aggravating thing i ran into is in the industry i was working in (pharmacy) prices are calculated using a contracted formula. The pharmacy (my system) calculates a price and then sends the rx to the insurance company. They calculate using the same formula (hopefully) and if their result matches or comes out greater than what we calculated then they pay the invoice. However, if their calculation came up less than what we sent they'd pay their amount. It was a cat and mouse game constantly.

Well one very large insurance carrier had a bug in their system that under the right conditions would come up a dollar short for a very common rx. I understood what was happening and called and called and bugged them to fix the issue. Finally, they just ended up saying "we're not going to change the system. end of story.". It blew my mind, I was right out of college and too naïve to consider they'd just flat out refuse to fix the bug.


I actually saw something similar in telco, where a carrier was rounding charges to the cent, but charging in fractions of a cent. So for example a CDR might be worth $0.001 but my customer was being charged $0.01.

My customer complained, but in this case they prevailed because the contract said nothing about rounding.

This can be really significant. They ended up having to compute the charges and send the carrier a bill for the difference. It was worth tens of thousands of dollars per month.


I suspect doctor_eval and yourself are talking past each other a bit. You seem to be conflating metering and billing in your model:

> You raise or lower your prices on the hour boundaries. The new rows inserted after the price change will have the new SKU ID with the new costs.

In the big systems Im familiar with metering, pricing, and billing are discrete. Yes, usage will be metered to a SKU. But there are arbitrary pricing plans that denote price/tier/discounts per SKU. So the price is only joined to the SKU usage, per customer, at bill generation time.


Those are all problems which have been "solved" by many companies. I'm sorry but none of this screams complex to me, all your problems have 2-3 "correct" choices, or you could just copy what almost all big companies do.


The whole point of TFA is that billing systems are way more complex than they appear to be. If you're trying to build a billing system, how do you "copy" what these big companies do, when most of the effort is neither obvious nor visible?


OP here, that's exactly the point.

It's easy in theory: 'just do it', 'just copy', BUT the challenges are in - the tiny details (that can have a huge impact: we're talking 'billing' and 'money' here) - the maintenance: it's a living organism: you will inevitably change your pricing, your payment processor, the way you bill, your back-end/front-end etc


> The whole point of TFA is that billing systems are way more complex than they appear to be.

Technically the point of TFA is to market billing software. Billing systems are complex in that they can consist of many different moving parts, but billing system are also simple in that they are easily understood. Billing has to be easily understood, else you'll quickly scare your customers away. Complexity and simplicity are not necessarily at odds with each other. The meat of TFA is mostly about how billing systems can quickly become a time consuming endeavour which again isn't at odds with simplicity.


> Billing has to be easily understood

I disagree. Billing has to be correct - that’s billing rule #1. You can have an easily understood pricing plan, but implementing that plan correctly and consistently in a way that you can defend and audit can be very complicated.

Just take the simple example of implementing anniversary billing for a customer who signs up on the 30th of the month. What date do you bill them when the month has less than 30 days? What about when it has more than 30 days? What about when your customers are spread through 4 time zones? What about when daylight savings starts and ends? Do most engineers understand time and time zones well enough to get this right?

These questions are fundamental to correct billing at any kind of scale, and the answers are not simple, or at least I don’t think they are.


> These questions are fundamental to correct billing at any kind of scale, and the answers are not simple

As someone who also works in billing and subscription management, I 100% agree with everything you said and want to add an extra thing:

Even when the answers ARE simple, there's so many of them that you end up realising you want to pay someone else to make it their problem instead.


> but implementing that plan correctly and consistently in a way that you can defend and audit can be very complicated.

We established earlier that billing is likely to be complicated, but that doesn't mean it isn't simple. Simple things can be complex. Simple refers to ability to understand, while complexity refers to scale of interconnected parts. They are not at odds with each other.

> the answers are not simple, or at least I don’t think they are.

I think you can make a strong case that business isn't simple, but billing systems only need to serve the needs of business after business has already figured things out. You cannot bill at all, even without a formal system, if you haven't answered these questions. A billing system merely needs to adopt the answers that are already understood.


> Simple things can be complex.

I’m sorry but I consider complexity and simplicity to be antonyms - and so does the thesaurus! https://www.thesaurus.com/browse/complex

> billing systems only need to serve the needs of business after business has already figured things out

Billing is a fundamental part of your product and your GTM. Getting it right gives you an advantage, especially in terms of trying different approaches. Getting it wrong can be a burden on the business, especially if you find out that you need to re-price.


...or you could just copy what almost all big companies do.

This is a dangerous strategy for a startup. A big company can afford to make a mistake and fix it even if it costs tens of millions. If you copy their mistake before anyone has realised it's a mistake you probably won't be able to afford to fix it.

It's also very likely you'll only be able to copy the externally parts of what the big company does, without really understanding why or what's going on behind the scenes. Really, you're advocating cargo culting a solution. That's a bad idea.


Right! The decisions and complexity are all hidden. Billing has to look simple otherwise people wouldn’t pay for stuff.

Take anniversary billing, it seems so simple, just store the billing day-of-month in the database and run the billing using the sql:

   select do_billing(customer_id) from customer where billing_day=extract(date from current_date)
And this works great in December and January, but if you don’t have good financial controls you might miss the fact that you’ve dropped an entire week’s worth of billing by the end of the following year.


> you could just copy what almost all big companies do.

What almost all big companies do is pay someone to make the problem go away.


I think you've possibly simplified by solving a different problem to what's described in the article. The problem space described is subscription plan & pay per use (with what sounds like might be percentage pricing as well). This is quite different from per unit billing.

> Some of this complexity is self-imposed

Having spent years working on invoicing & payments systems, the business will impose complexity (which you can only say no to a certain amount of) and then legal will pay you a visit and explain all the many rules that apply to billing & invoicing that vary across each country. Internationalization isn't just a tax complexity.


Good grief this was informative!

Thank you for taking the time to share that.


Of course! Thank you for the kind words!


I can add a few more from my own experience:

Billing really is complicated, but the people who set the policies don't understand that. They want to be able to say "Hey, let's give people a 10% discount if they sign up for a yearly bill, but also don't have this service over here, and for governments and students let's cut that down by another 15%" and so on, basically "billing by English spec", and because they think this is trivial and easy it gets handed as a spec written in stone to the billing team. Even in engineering-type organizations, treating this as an engineering problem is often a foreign concept to the org, so the idea of having the eng team estimate effort and then go back to the original requestor and make sure they really want it 1000 engineer-hour's worth, and that the rest of the org is willing to dedicate that much time to it, is generally not done. (Or whether or not there's an alternative thing that could be done at 1/10th the price that achieves the original goals, e.g. "we don't have annual discounts built into the system but we have this other fairly-close concept already built in, can we just slightly tweak the ask to use that, rather than having to implement a whole new complicated concept?") Even a company that has learned the hard way that "Let's just change the button to blue and move it down an inch" is more complicated than it looks thinks that management can just write whatever billing policy they want and there are no costs associated with it.

On a more pure-engineering front, engineers have the tendency to want to implement billing discounts and such by lying to the usage database. "Ah, we will implement this 10% off policy by telling our record-keeping system that they used 10% less stuff." I've banged on how important it is to never lie to your database a few times, and the billing database is what really brought this home to me. This way lies madness, as you start to stack lies on lies on other lies in the database and then counter the lies with other heuristics and then counter the heuristics with other lies and before you know it your database contains no truth anymore, which is a problem because neither does anything else. The database was supposed to be the source of truth.


> Billing really is complicated, but the people who set the policies don't understand that.

The very first startup I joined in the late '90s tackled this problem. The company tackled billing and all of its complexities by building a SaaS product before SaaS was a widely used term (even the company didn't call it SaaS for a few years). The level of complexity was very high. Creating rules, calculating charges, dealing with taxes, ensuring things were auditable, and making it all performant were all huge challenges.

Basic billing of hours x rate is easy, but (as you've noted) it's a rare person that is content with that. I cringe every time someone says "We'll build it ourselves" when it comes to billing. Even for simple things like the 10% discount (new customers only, plans > $9.99/month, discount expires after 3 months) get complicated very quickly.


Rather than becoming complicated it much better to assume discounts are already complex. Rebates (final bill - X$) aren’t the same as exclusive discounts (X% for seniors or Y% for veterans) which are different from additive discounts ( X% + Y% = Z%) as separate from multiplicative ( (1 - X%) * (1 - Y%) = (1 - Z%)) etc.

Unfortunately, it’s really easy to assume everything is simple until the system gets complicated enough nobody understands it.


Billing is complicated because getting money in exchange for services is complicated. The company wants to increase pricing complexity, as to better reflect their own costs and to better price discriminate, and this runs directly into the problem of trying to express those constraints in the real world.

As noted, trying to fit billing into our stupid calendar system is tricky, because the calendar is stupid. Then, if you want plan upgrades/downgrades, you need to do pro-rata adjustments, and the interactions between that and tax/billing periods/etc makes the entire thing a headache.

And then you often need to add some way for the sales team to change things on an ad-hoc basis for sales/special negotiations/etc because of course big customers are going to negotiate something special, and that's all custom logic. After that, now the marketing wants referral credits, and then they want promotional periods that interact with customer loyalty, and on and on it goes.


Whatever your billing requirements may be, use double-entry bookkeeping as the core data structure.

Every time I didn’t, I’ve wished I had.


Interesting. You mean you use double-entry bookkeeping instead of a database? or mixed with it?


Something has to be the store of data. The double-entry model is a paradigm, not a serialisation, of accounting and financial data. I’ve written double-entry code using text files, key-value stores, and relational databases as the underlying storage. They all work, at varying levels of scale and manageability.


I would be very interested in a call-out guide - say I am going to handover billing to this SaaS API, Instill need to know what events I should identify and call out - it seems to me this is 50%+ of the engineering challenge anyway and I have to do it even if I am not doing in house billing

Simple things like your user just did X, in this country and at this time. This has a tax implication


This is a great idea, we’ll work on it!


This is great, thank you. Makes me think of all those people saying ‘you could run T_____ with 8 people’


I worked on billing for a small UK telco about 18 years ago, and mostly it came down to reasonable-sounding business logic actually being a complete shitemare to make sense of.

We got in Call Data Records (CDRs) every day from BT, which we then had to calculate billing for based on our business rules, which had to be somewhat aligned with BT's pricing so we'd actually not lose our shirts.

It was pretty straightforward with one price between 8am and 1pm, then a cheaper one between 1pm and 6pm, then calls were free from 6pm until 10pm as long as they were less than one hour then billed at an even lower rate, with an hour of free any time minutes that reset at midnight. So, if you never made more than one hour total of calls between 8am and 6pm, and kept your calls to less than one hour the rest of the time, the actual call were free, right?

You would genuinely be surprised how many times the billing system they were using would toss its cookies because someone made a phone call at 1740 (billable), continued right through 1800 (not billable) to 1900 (billable after 1 hour but there's still 40 minutes on "anytime" left so logged but free), right past 1940 (billable again) and on past midnight (free for another hour until anytime runs out).

No, you can't just kick someone's call off after an hour.


Yeah telco billing can be crazy hard, it took me a long time to come up with a reliable model for it.

> No, you can't just kick someone's call off after an hour.

Agreed... but I worked with a telco that did exactly this!


I too remember in some 00s that calls would disconnect at exactly 1 hour mark.


>> These systems were no fun

They are indeed no fun. And payment APIs make me want to shoot myself. Whoever writes them seems to enjoy making end developers bleed from the eyeballs. But I've written my own in-house billing software three times over since 2002 when I had to write my own front-end API to a raw CC gateway and guess what the outputs meant ... it's much simpler now to deal with asynchronous payments and it's not the end of the world to manage them.

The much, much bigger problem is getting your clients to pay on time. Knowing who usually pays and who's okay being late (versus who is habitually delinquent) and keeping tabs on that before you bill them again so you don't have to suddenly confront them about canceling their services. The smaller the business, the more significant a portion of time this can take up.

When I started working for an ad agency in 1995, the owner had me stand outside a guy's factory next to her with a picket signs because he owed her $5000 for an ad that ran but was a month overdue. I have written backdoors into most large pieces of software I've built, in case of nonpayment; but never had to use them. I've pulled the rug out from hosting and left people's websites saying "Joe X doesn't pay his bills" after being treated badly by clients who were 6 months late paying. These things all give me a certain sense of glee but never get me paid.

The best strategy I've found is calling your clients on a rotating basis and checking in to see what they need next and gently reminding them that their bill is due. I try to make this round about once a month.


Yeah - collections is a whole problem on its own. But calling customers on the regular doesn't scale too well.


Do these guys really say "We ran a neobank and figured out that billing is surprisingly complex."? Like, seriously? That's what you think was complex in your domain. If I had an account at that bank, I'd seriously consider closing it ASAP, because that smells like no one had an appreciation of the complexities of finance.


This is an ad that gets posted around a lot of places.

> To solve this problem at scale, we've adopted a radical approach: we're building an open-source billing API for product-led SaaS. Our API and architecture are open, so that you can embed, fork and customize them as much as your pricing and internal processes require.


Honestly, billing is really not that hard. I've built some billing systems myself, and although it's easy to make mistakes if you read up on the known pitfalls rolling your own billing system is not a big deal.

One key thing to remember is that if you want your billing to make sense to your customer you can't get too clever about it. A few rules of thumb: when you have to round anything, round in favor of the customer. Split the logic that calculates what should happen and the billing part of the billing system. For example, if you store a next_billing_date for each customer you can trivially adjust the date as needed, but if you have a crobjob that calculates which customers should get billed and then immediately bills them manual overrides are harder.

Giving a user what amounts to a few free days of service when they upgrade/downgrade is no big deal. Especially because you should see more upgrades than downgrades in any given month. In some cases the easiest way to credit people is to move their next_billing_date forward. For usage based pricing (e.g. number of active users in a month) do the easy and fair thing where you take the minimum of active users during the month (store num_active_users each night for each customer, and use that for billing and for user facing billing dashboard). Otherwise you get edge cases where you overcharge customers when they add and remove many users because then the max of unique user ids is >> the number of active users at any given time. The example query in the blogpost gets this wrong.

The SQL query that aggregates the number of API calls for a user is wrong for the same reason because it confuses API calls with billable API calls. You should have a cron that counts the API calls for each user and stores them separately in a billing table. Then when a good customer has a junior programmer make a ton of API calls by mistake you can easily go into the billing table for that customer and manually adjust the usage, with total confidence that future invoices will be correct. If you compute the usage at the time of billing then everything gets needlessly complicated.


That was very informative - it certainly does not give me the motivation to build my own billing system, but it does shed light on the scope of the task.

Side note: Is "I don't want to write/use a billing system" a compelling reason to release software for free?


It sounds compelling, but it's very difficult to convince free users they need to pay now if your software gets popular and you actually want to monetize it.


Lago is currently free to use and we already have paying users.

I think indeed some users will use us for free and never pay, whereas larger companies actually want to pay to get the hosted product, custom features, SLA, etc.

We are still in the early days, but here is how we think about our own pricing : https://www.getlago.com/blog/how-we-think-about-our-own-pric...


The project should include a high level Domain Model diagram to see how things are connected.


Thanks for the feedback, fantastic idea, will work on this indeed!


Section #6 seems to have a paragraph or image missing.

The first paragraph ends with „your tax decision tree should look like this:“ and the second paragraph starts with „Now, imagine that you sell different types of goods/services to.“


Thanks! Let us fix that!


Fixed, added other images, I posted that quite late last night!


I worked on telco OSS and billing systems at the beginning of my career. The industry benchmark is that 2/3 of all billing systems fail. Fail as in fail completely to deliver, not as in fail to deliver a 100% satisfactory result. A big part of this is legacy and marketers inability to discipline themselves and KISS, but legal and accounting rules also play their part. Never assume billing is going to be easy.


In the spirit of DDD I'd like to offer a few domain boundaries and vocabulary terms that my team and I used when building our (multiple) billing platforms.

"chargeable items" are things we want to charge for. In my experience there are exactly three kinds of chargeable item, which operate largely in their own domain:

- "usage charges" are charges based on usage of some resource (eg, bytes, time or phone calls).

- "recurring charges" are charges that are billed on a regular basis, e.g. a monthly fee. Recurring charges get complex due to the need for pro-rata, e.g. if your customer is on a monthly plan but signs up in the middle of the month, or changes plan in the middle of a month. (you can get away without this, but that just moves the complexity somewhere else - and computing refunds can become a huge pain at scale, especially if you want to build a system that is 100% self-service).

- "one-time charges" are charges that are raised due to actions taken during the billing cycle; you might also call these "event charges". Things like connection/disconnection fees etc that are generated by a process or external system. These are the simplest, just an SKU and a value, maybe a qty and rate if you need it.

Arguably, the most challenging domain for billing in telco circles is "rating", which is the process of computing the usage charges for individual charge detail records (CDRs) and aggregating them so that your database doesn't get overwhelmed. You might get a billion CDRs in a month. Although not all billing applications require rating, anything that does usage billing is doing some kind of rating. In Telco we also talk about "mediation" and "guidance" but that's another story...

What's important about the above is the knowledge that there are different charging domains which need to be handled differently. There's a huge world of difference between a recurring charge and a usage charge.

"billing" is the process of collecting these chargeable items together, to prepare them for sending to a customer. The resulting database rows might include the raw pricing along with tax, discounts etc. The important thing here is that we have a means of separating what we want to bill, from what we actually send to the customer, because the timing of an invoice is different from the timing of the charges.

"invoicing" is the collection of the outstanding chargeable items into a final invoice, which can actually be sent to the customer. Interestingly, once you've issued an invoice to a customer, modifying the amount owed is probably illegal in most countries, and most certainly a bad idea.

"payments" is the process of receiving money from credit cards.

"collections" is the process of asking people to pay (It's also called "dunning", after the famous debt collectors, Dunn and Bradstreet). It arguably also includes policy actions like restricting access for non-payment, raising late fees, etc. Another can of worms...

Anyway ... you can see a kind of billing pipeline here, where you have an ingress of chargeable events, which get rated, aggregated, held for invoicing, and then an invoice is raised, payment is requested, and non-payment is processed.

This is just based on my experience, the industry seems to vary somewhat on the specifics. And ... it seems easy enough when I write it like this, but there are many devils in the details. Nevertheless, I hope this helps someone.


First off, I appreciate your related comments. Two questions Im curious about:

You mention usage and one time charges as being distinct. Ive generally seen them treated as the same thing; “instant”, “action”, or “usage.” Effectively atomic units of metered usage that are aggregated later, and the corollary to recurring usage. Is there some reason to not model “one time charges” as just a another (infrequent) variant of usage. Eg a “service connection” usage with value = 1?

I didnt see you discuss pricing in your list. Do you normally treat pricing as part of usage? Ive always see it as a separate value thats joined with usage per customer-billing period.


So as I see it, one time charges are basically SKUs that just pass through the billing system unmodified. The billing process might compute the price (based off a price list, see below), and it will probably apply discounts and tax, but otherwise they pop out the other end - onto an invoice - 1:1 with the input. They aren’t aggregated or otherwise processed. This means that can be generated by some external black box that we don’t know anything about (eg, decline fees from payment processing).

For usage, as you say, there is a complex process that’s needed to compute the final value for an invoice - in telco the formal names are mediation, guidance and rating - which isn’t needed for one time charges. A usage charge may be the aggregate of thousands of usage samples (CDRs, charge detail records), processed through a bunch of rules. The CDRs can come from many sources and be formatted differently. Importantly, you don’t want to treat these CDRs as full charges that appear on an invoice, because doing so will overwhelm your billing database for no benefit. Similarly, while you could in theory send a one time charge through the rating process, doing so is probably wasteful, as the processing functionality doesn’t significantly overlap between usage and one time charges.

Pricing is very complex! You are absolutely correct, it should be separate, and it should apply to all three types of charge (consider that the actual decline fee charged to an account might be different depending on what plan they are on, or some other factors). So yes, pricing ideally needs to be separate, or at least decoupled from the billing process. That adds another layer of complexity for sure.

I hope this makes sense, It’s getting late where I live :)


Thanks! Your details of one time charges passing through to invoice/billing, probably bypassing aggregation, totally makes sense. I’m thinking of that case as something like “incurred line items” now. I haven’t seen that much as I’m more on the Service side of the *aaS space. And yes the variable pricing still makes sense there. I like your example of variable decline fee based on plan as an analogue to my related SKU discounts I mentioned in another comment.

Edit: belated additions to your great list of “basics most people miss some of”: pro rating & amendments, especially retrospective prorata! And the classic retrospective amendments after you’ve crossed a billing period. Aka “we’re sorry and here are some fungible credits!”


The interesting thing about one time charges is that they can come from anywhere. Maybe you need to charge for sending an SMS late payment reminder, or even a late fee. In some countries you can even charge a credit card success fee, which needs to appear on the next invoice. One time charges solve all these problems.

Pro-rating is a real spanner in the works! If you’re half way through a month and you’ve already charged $100 for the “silver” plan, and the customer wants to upgrade to “gold”, you need to refund the unused half of the month on “silver”, or somehow take that already invoiced amount into account during the upgrade process. The only alternative is to give “gold” away for half a month (which opens a path for gaming) or to wait until the end of the month to change the plan (pisses the customer off).

Calculating all this accurately is possible, but very hairy, and the solutions we came up with after many years in the space were non obvious. It’s far from rocket science, of course, but it’s far from straight forward.


Finally, im glad I’m not alone. When we were building SuperCMMS, we already had a few companies ready to pay us. So we needed a billing system from the get go. While we use Stripe, we still had to wrangle a lot of stuff (like taxes), since we wanted to integrating other payment gateways in future. To save time we decided to have “only a single plan - all features included”.

Now comes the unexpected part. People kept asking us why we have only a single plan. Maybe all of us are so used to multiple pricing plans on saas products. So we had to explain ourselves in the pricing FAQ’s.

https://supercmms.com/pricing


>USD 50/- Per User Per Month Enterprise Support (Optional) Starts at USD 1000/- Month

Are you Indian, perchance? How you write prices in India doesn't work in other places; this looks strange to someone who works in nearly any other currency.

Any of the following would look a lot less out of place:

$50 50 USD $50 USD 50.00 USD $50.00

Of these, I recommend the first one; dollars default to USD worldwide unless specified otherwise.


I think currency notation is more of a regional thing. Had we just mentioned "$", our Singapore customers might think its SGD (Singapore Dollars). We'd be better off using "USD $50.00". Thanks for the perspective.


It was clear to me. Are you in the US perchance? :)


Once a wise senior dev told me, never underestimate the complexity of an e-commerce shopping cart.


Thanks for sharing. This had made me really think about how I can tackle billing in future.


Awesome, keep us posted!


Thats why SAP is worth as much as it is

If there are invoices involved then it is going to be fun :)


So this is why we use Stripe, right?


I'm a billing engineer with more than 25 years experience, and I used to run a specialist billing company.

I think that this article is good in that it provides a significant amount of detail about many of the complex problems of billing. I don't agree that everything in the article is a big deal, but it certainly calls out a bunch of the things you have to think about, and it's a long, long list. And it's also just scratching the surface - some of the questions lead to many more questions, and significant complexity.

So, if you don't architect your billing platform in a way that properly separates the high level moving parts, you'll eventually end up with a big pile of spaghetti. Which is bad form for something that's so critically important.

That's not to say that you have to implement every single possible feature, or even that separating things is particularly hard. It's possible to build a simple billing system with just a few SQL tables - but they are probably not the exact tables that you are thinking of. But it does means that your billing platform needs to make as few assumptions as possible, and be structured in a way that leaves room for change in the future.

That's because billing is critical to a business' success, and it therefore requires lots of flexibility. One way to find product/market fit is to do your billing in a way that better resonates with your customers. I mean ultimately that's what I think AWS did to colocation. So if you can't keep up with the demands of your sales team or your customers, you're going to be in a bit of trouble.

As engineers, we don't get to set the rules for billing, because that's not how customers work. The billing system needs to support the requirements of the business; not dictate to them how they need to do things.

Anyway I wanted to add a couple more things to TFA, just to illustrate that there is even more to it than they've listed.

Policy management - what do you do when a user spends too much before the invoice is raised? Do you send an email / sms / restrict the service? how do you avoid doing this multiple times in a month? how do you integrate the billing policies with other policy systems such as usage policies? This can get really deep...

Invoice rendering - very few people seem to understand that invoicing is the one time every month that you have the legal right to send unsolicited emails to customers. But in my experience, invoices almost always look like crap. What a waste!

There is heaps more to this: how do you quickly fix billing problems, such as an incorrect plan? how do you backdate changes? what's your pro-rata policy on fixed charges? what do you need to integrate with the finance platform? ... the list goes on.

So please forgive my shameless plug - I'm available for consulting on billing systems any time. I really know what I'm doing. See my profile for contact details.


I'll reach out in DM - thanks for sharing!


Hey Anh - I didn't look deeper into the Lago page on GitHub (I just read the linked page) - I've been kicking around the idea of an open source billing platform for some time. Anyway, I might be interested in helping, and I'm looking forward to hearing from you.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: