Your only power to encourage them to fix this is to do the thing they're begging you not to: dispute the charges.
If a threshold of Twilio customers dispute charges, Twilio loses the ability to process credit cards at a lower risk rate, then with all but high risk processors, then may lose the ability to process them at all.
If enough of their customers are getting burned, and enough dispute, Twilio would no longer be able to accept credit cards. They are terrified of that, so begging you not to dispute charges for their lack of fraud prevention.
You accepting anything less than full refund of all fraudulent use they're cascading back on you is a gift to them. You accepting less than a full refund, while not dinging them at all with a chargeback is also a gift to them. If they don't want to give you the full refund for misuse they should be preventing, dispute it, as is your right.
The correct course for Twilio is for Twilio to refund these charges no questions asked while fixing the problem.
There was an online service subscription I had several years back (not Twilio, but something somewhat similar) that I stopped needing. I forget if there either wasn't any way to this on their website or if it was just broken, but after repeatedly trying and failing, I ended up just disputing the charge the next time the monthly payment came up, and my bank stopped the transactions from that point on, so I forgot about it. A while later (probably around 6-12 months), I got an email from that company asking me to go back and tell my bank that the transactions were fine. Given that it had been so long and I only disputed the charge due to their website not giving me the ability to cancel through them, I didn't think it was worth trying to talk to someone at the bank to figure out how that would even work. It left an impression on me because it was the first time that I felt like I actually had any power in my relationship with a company as a customer rather than just having to hope that the company would chose to do right by me with nothing forcing them to.
>Your only power to encourage them to fix this is to do the thing they're begging you not to: dispute the charges.
I'd check their TOS to see if they offer some kind of arbitration option. As noted in other threads, triggering that process can be a surprisingly effective way to make someone from the company actually engage with the issue. Disputing the charges is always a nuclear option. They may never do business with you after that.
> Disputing the charges is always a nuclear option. They may never do business with you after that.
This is something that I think needs to be regulated. I'm not saying that this should be the case for a company the size of Twilio, but I definitely think that a company the size of Apple/Google/Samsung should not be able to ruin your life because you had temerity to stand up to them and dispute a charge.
As I see it, the problem is that those companies are effectively monopolies in fields most of us depend on. If Apple or Google refuses service to someone previously reliant on their services, they could be locked out of accounts on a plethora of other services, have their payments to third parties disabled, lose access to major means of communication, and more.
The last generation of big tech monopoly never had that kind of power. Microsoft couldn't even do much to block someone from using its products as most of them were sold through third parties and didn't require network services to operate.
Shouldn't have "bought" them in the first place. Buy physical books, or DRM-free books on sites like gumroad, or just pirate. Don't give in to the rent seeking business of pretending to sell you what you can't own. If you like your Kindle device, you can use KOReader to read epubs and reduce dependence on Amazon. If I sue the bookstore, they can't just take all the books away, but if you dispute a charge with Amazon, they'll do it because the ToS says they can. At the very least, try downloading and de-DRMing all the books you received from them: https://github.com/noDRM/DeDRM_tools
Twilio has a neat platform, but their standards are very low or nonexistent when you experience jitter, large audio buffers or routing issues.
Providing PCAPs and reproducing routing issues doesn't result in support addressing these issues. Many other IPES and CLECs will actually fix these issues when documented.
Is there anything in the arbitration clauses forbidding them from cancelling your account if you invoke arbitration (regardless of whether you prevail or fail?).
At the individual dispute level but in the long run arbitration means you don't lose your risk level which will almost always cost you a lot more than whatever the actual arbitration/credit disputes cost.
I suspect (but may be wrong, I don't know how trigger-happy the risk level changes are) the absolute number of events needed to trigger a risk level loss at the scale of Twilio would also represent a catastrophic number of arbitration cases.
That Twillio doesn't protect you is bad. However, would a court agree you don't owe them the money? This recommendation seems like abuse of disputing a charge and will just get you banned from Twillio.
The court doesn't have to agree, only the card provider does.
The customer has the right to dispute credit card charges thanks to the agreements between customer and card provider and between card provider and merchant.
Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
This is why you always pay for sketchy merchants with a card, it's one of the few consumer powers you have.
>The court doesn't have to agree, only the card provider does.
If it's a sufficiently large amount, Twilio will collect the money from you via other channels. Them losing a credit card dispute does not release you from liability.
>Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
This is simply not true. Chargeback blacklists are standard and not prohibited by merchant agreements.
Why would the card providers be in the customer's favor. The customer paid for a text to be delivered to a phone number and Twilio did that and then charged the customer for it.
If you pay someone to mow your lawn, then they mow your lawn and charge you. You can't just chargeback after the fact to get that service for free.
In your analogy it would be more like paying someone to mow your lawn because your neighbour got it done for $10, then being charged $100 because your house number is even.
It might be in the terms and conditions, but it’s bad faith to not give any warnings or controls before the services are rendered.
The people adjudicating disputes at card companies are akin to content moderators hired by social media companies, they aren't experts and do not spend much time reading up on the dispute. The decisions are more or less random, with a heavy bias towards the customer.
And then you switch to another SMS provider, which may be costly from an engineering perspective, but clearly worth it if you’re getting slammed with botnets and Twilio doesn’t care.
Further, the customer has the right to dispute credit card charges thanks to the agreement between customer and card provider and between card provider and merchant.
Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
> Twilio will get in trouble with Visa/Mastercard if customers say Twilio is dropping them for disputes the card provider finds in the customers' favor.
This isn't true. Visa/Mastercard care about your chargeback rate. You can block a customer who's done a chargeback. I'm sure the card networks have rules around what you cannot do as a result of a chargeback but you can stop providing services to a customer who has done a chargeback.
Disclosure: I engineered and delivered a high risk payment processing gateway to a firm specializing in being a cc processor of last resort, also working with merchant banks of last resort.
To be clear, my views on this are not legal counsel, they are simply from having worked in this area for a decade before becoming CTO at a global bank.
To lose the ability to process MasterCard or Visa credit cards takes a few months. You can rack up big fines during that time though. If they get put into the probation period that would raise red flags with some execs, assuming people are communicating these things.
From both a user and provider perspective I hate that individual companies are implementing 2FA at all. I don't want another database with my phone and a password in it. I wish Mozilla Persona had took off, or any other auth standard.
You chose to require an SMS OTP for your customers. It is not straightforward at all that the burden of filtering your customers would fall on your provider and not on you -- actually, if the provider you chose does explicitly not provide that filtering, it's effectively on you.
(I have to say that if I were Twilio, I would not have added the "fraud prevention" toggle, because now they can be deemed to be providing that service.)
We've been hit by this at work as well. We had to add CAPTCHA and a several other techniques to defend against this.
How it works:
1. Attacker leases 1 or more premium rate numbers in an international country.
- Attacker can lease a premium rate number for as little as $10/month
- Typically, the attacker gets to keep 70% of the money generated by the premium rate number.
2. Attacker then finds companies with OTP (One-Time Passcodes) or 2FA (Two-Factor Authentication) endpoints that require no validation and writes a script to automate the webpage or call the API endpoint
- Attacker will typically obtain a new IP address per API call using a VPN or a rented botnet from the dark web.
3. If the premium rate number costs 10 cents, then each successful text message they can send to the number generates 7 cents for them.
4. The attacker then just needs to send 150 SMS to the premium rate number to break-even on their $10 investment, not counting the cost of the VPN or rented botnet.
There is a lot of money to be made here by an attacker unfortunately. :(
I agree with other comments here. $0 is the minimum amount people should be willing to pay if they're not disputing charges or reporting fraud to the credit card networks / regulators.
No idea. I think there's a real problem with the whole design of premium numbers because I'm not sure how one is even supposed to know when payment is required or meaningfully accept it, though at least the API apparently allows this.
FWIW, I do think $0 might make a sane default, but you do understand that the user would have to change it from $0 before they could use the account, right? The whole point of using Twilio to send an SMS is because you wanted the SMS to actually be sent, which means you are going to have to pay for the SMS, and SMS is always stupidly expensive.
I would imagine there are rules/regulations about a SMS provider blocking communications before fraudulent behavior is determined? Not saying it shouldn't/couldn't be done, but probably one of those things with a simple tech fix but a complicating social/business aspect.
It could be an option in the API call with a default in account settings. I bet most people who are trying to reduce spam accounts by requiring a phone number would actually prefer to exclude these numbers anyways.
surely not if the customer _explicitly requests_ that the communications are blocked? iirc in Aus it was possible to have your provider block messages to premium rate numbers back in the days when it was popular to buy ringtones.
It isn't just as simple as 'premium rate numbers'.
Some of the criminals behind these attacks will have access to the phone network. They'll pick an expensive route, like a range of phone numbers in Georgia (the country) from the USA, and offer a cheaper route to it. The system will start using their route for those calls. They'll accept all calls to that route, get paid, and never actually connect any calls.
That gives them a range of "normal" phone numbers which helps them avoid throttling on just one number. But they can be just as expensive as premium numbers to call.
At least, this is how it was explained to me as my team fought these attacks a couple years ago. We'd see calls to a large range of a few thousand numbers. Couldn't throttle on a single number.
I think you're conflating toll bypass fraud with IRSF. A grey route that never delivered any calls or only a fraction of them would have bad ACD numbers and people would not use that route. With hacked Asterisk/FreePBX boxes people usually call the international numbers described in OP and split the termination fee with some corrupt carrier/intermediary. There is a related fraud where people use the hacked Asterisk/FreePBX boxes to terminate calls, which from what I understand these actually have pretty good quality until the unwitting owner gets a $40,000 phone bill and shuts everything off. Traditional toll bypass fraud is when countries are expensive to call internationally but have cheap local calls, so people in those countries buy a bunch of sim cards, put them in a box with a bunch of gsm modems, and use those to basically "convert" an expensive international call to a cheap local call (and profit the difference between the two rates).
Edit: Oh, you're talking about number hijacking. I think they usually aren't offering termination services though, usually it goes hand in hand with the kind of fraud described in the OP.
> Traditional toll bypass fraud is when countries are expensive to call internationally but have cheap local calls, so people in those countries buy a bunch of sim cards, put them in a box with a bunch of gsm modems, and use those to basically "convert" an expensive international call to a cheap local call (and profit the difference between the two rates).
Is this really fraud? Is it fraud to offer any VOIP service, or only when it can connect to the phone network, like Skype?
I guess I could see how it might be against the T&C's of the telecom company, to offer a service that undercuts them, but hardly a criminal act of deception.
I consider it to be relatively harmless but how it is classified depends on the country. India is pretty cheap to call even absent simboxes but they still crack down on the practice for “national security reasons” because it makes tracking people more difficult. The UK (Ofcom) banned them outright for some reason a long time ago but that’s being appealed. In some African countries the laws are pretty vague and do not outright ban them, usually they charge people with “unregistered telecommunications business” or something like that.
Fraud is what the government decides it is, the governments have deemed this to be fraud.
Telecom companies don’t necessarily care about this, it’s often the governments who want to tax incoming international calls as an easy revenue source.
How is this a workable system!? Why would anyone pay them. This seems like fraud on the part of the phone networks for billing for service that was never provided or should have been provided cheaper.
while I don't agree with sanctions, this seems like the kind of time where you just block off a country/exchange entirely if you cannot have the confidence of what things cost to send there.
I don't understand why twilio cannot simply set a flag on their phone company account saying "under no circumstances will we pay for these shenanigans", and why the phone company billing stuff cannot simply block sms messages to such scam accounts.
In particular, email (smtp) to sms gateways exist. Why doesn't twilio just use one of those (and maybe pre-arrange a flat monthly payment to avoid being blocked for going over quota).
> I don't understand why twilio cannot simply set a flag on their phone company account saying "under no circumstances will we pay for these shenanigans", and why the phone company billing stuff cannot simply block sms messages to such scam accounts.
Everyone has the idea "just don't pay for fraud" but in practice it is difficult because there are many different carriers in the typical international call chain, which means to dispute charges you need everyone to agree. Also carriers have long term agreements with eachother about billing and it is not as easy to just dispute the charges like you can with a credit card.
You’re talking about the fee your carrier charges for normal texts. These are “premium” charges, meaning the user is charged an extra fee on their bill regardless of their SMS billing plan.
In Germany the standard price for an SMS is 9 cent, of which somewhere around 2-3 cents are paid to the recipient's carrier. Unlimited plans are common, but only because nobody texts anymore (same applies for phone calls).
Maybe on prepaid plans? Been a while since I've heard of SMS costing anything on subscription plan, outside of roaming charges. Mobile Internet effectively cannibalized that income stream for the phone companies.
And to top it off, disabling auto recharge doesn't prevent Twilio from charging your account. They won't charge your card but they won't stop processing requests when your balance reaches 0. We were just hit with toll fraud and even though auto recharge was disabled, they continued processing requests until our balance reached NEGATIVE 4,000 USD and then suspended the account. We received to emails in total:
1. Your balance is running low at -65 USD
2. (30 seconds later) your account is suspended, I checked the account an hour later when I saw this email and the balance was -4,000 USD
I asked support why they continued charging our account even with auto recharge disabled, but they just ignore the question.
Support says it's our fault, asks us not to dispute the charge (although there has been no charge yet as we disabled auto recharge), and said it will take 10 days for finance to issue a partial refund (that was 24 days ago).
When I was working in this space, all of my providers gave me a pricing feed which was essentially phone number prefix, price (and some text that my system didn't care about, I don't need a name, just a price, thanks). It looks like twilio doesn't offer that publicly for SMS, but you can see their voice price list by clicking "download voice prices (.csv)" on their voice pricing page [1]. The SMS page has a similar feed, but it only gives you price by carrier name, which isn't very helpful --- you'd need to pay to do a carrier lookup before you could use the price list; these lists don't feel complete either anyway, but it's an idea.
Wow, ok. I've been hit with this issues twice and both times, the Twilio reps failed to let me know about this. I had previously resorted to just turning off any counties. Looks like mu international app users might get functionality back.... if this feature is still live.
Do you happen to know which MaxPrice number would make sense for simple SMS? Figuring it out a reasonable value to use from Twilio's pricing pages is proving quite tricky, because of the amount of dimensions to their pricing.
You can download phone prefixes with prices from them, and also set max price for an SMS (won't work if they charge just a little amount and you send thousand requests of course).
It gets better. An Ethereum address can receive any tokens, including NFTs, and has no power to reject them. A famous celebrity can have a lot of NFT spam publicly visible in their accounts, and people have no idea if they bought them or not!
I'm surprised/confused: Why is it hard to detect premium rate numbers, or at least set a flag to not allow sending to them? Like, I can't think of a time when twillo should ever be sending to a premium rate number; why is this even possible?
There are 200+ jurisdictions in the phone network and everybody has their own conventions on what a "premium" number is.
For comparison, imagine if each domain in the world could set its own rates for much doing a DNS query would cost you, and governments regulated this only by designating a few second level domains as "premium". That's pretty much the scale of the problem.
Edit: To be clear, this is a very well known problem and Twilio should be doing much better at it. But it's by no means an easy problem, and all the other side needs is one (1) number to exploit.
Depends what you mean by "premium rate." Every number costs money to call in Twilio. Some numbers cost more, in lots of these frauds numbers in ordinary ranges are used (Is a rural number in Chile that costs $0.20/minute to call premium rate/fraud? Because that's what it looks like a lot of the time. How about $0.05 a minute in Austria?). IRSF, the industry term for this kind of fraud causes billions in losses a year and there is no easy answer but Twilio should probably have more infrastructure in place to reduce massive surprise bills.
Block any number that costs more than 25th percentile would be a start and so on... I can come up with plenty of heuristics that would be better than nothing.
I don't mean to do Twilio's work of defending them, but in my experience it's possible they actually don't know how much to bill the customer. What they may know is the generalized per-minute or per-session rate they've agreed with another operator alongside a general "premium rate numbers will be settled at a later date" kind of clause.
My employer got bit by this several years ago, purely on calls within the +1 country code. Before this practice was largely banned, some small carriers were allowed to designate certain rate centers as higher cost. So our VoIP carrier would say that a call to a given area code was $0.003/minute but the calls would later settle out at $0.25/minute because of a 1,000s block of numbers being (unknowing to us our our carrier) as higher cost and being settlement billed back at the higher rate.
Twilio could agree to carry some or all of this risk for its customers as part of their value-add and fees. That way, Twilio has the incentive to make the proper changes for its customers and would have the experience of looking at all of the return billed rates for all of the calls or messages across its entire customer base to help prevent toll fraud.
This is the case, the telephone billing system is perhaps the most complicated pile of softwareshit you have ever seen in your LIFE - and some of it is insane.
When I was an intern, I was working for a big company on a project to optimize some call center management. Basically put mainframe reports on an intranet.
The company had made a change of some sort where whatever EDI connection between the telco and the company stopped working. I learned this when and angry facilities guy came up looking for my boss’s boss to sign for a delivery. I was the only person there, so I did. 15 minutes later, two pallets of bankers boxes came up - thousands of single sided pages of itemized call details.
My favourite was getting charged for an sms my iPhone sent which was a phone home to an Apple headquarters short code for iMessage. iOS hides these from the user. Most providers don’t charge for this, but some do.
Really sucks when you carefully load 10 EUR of credit to buy a 10 EUR prepaid plan for the month and see 0,05 deducted despite being incredibly careful to not do anything that would incur a charge before buying the plan.
Apple DOES say that, when you set up FaceTime and iMessage!
There is a pop up that says "Your carrier may charge for the messages used to activate iMessage and Facetime" You can choose to not activate and do it later.
That warning actually depends on the “carrier profile”, a configuration file the phone silently fetches (or has cached in firmware builds) based on certain attributes of the SIM like the ICCID or MCC/MNC.
There’s a field in there that configured whether that warning should be shown.
Correct, and it didn't appear for carriers which were whitelisted (who zero-rated the iMessage activation SMS).
My memory, which may be wrong, is telling me that the first major version of iOS which included iMessage did not include the warning at all, and that it was added for non-whitelisted carriers (aka those which did not sell the iPhone) to prepare the user for the possibility that they will be billed, based on user feedback precisely like the comment to which I was replying.
Fun fact: +1 is not a country, but all of North America. For a long time it was entirely possible to dial a perfectly ordinary looking +1 258 xxxxxxx number and get charged up the wazoo because (258) is Antigua and Barbuda, not New Jersey.
Pissed me off how American Airlines wouldn't send me SMS updates to my Canadian number, even though it should cost only slightly more than than a USA number. I guess they got burned sending updates to some caribbean island number in the past.
Surely they can aggregate this across all customers though.
If Twilio cops an unexpectedly high settlement for sending an SMS to +1234567890 in January, can they assume that a separate customer sending an SMS to that number in February will end up in the same boat?
I'd be very surprised if the toll fraudsters weren't using the same numbers to hit multiple Twilio accounts.
Twilio works with phone companies across the globe; this is not something that would be that difficult for a company of their size to implement (even if it means one employee whose job it is to keep this up to date). Consider that the timezone database (a similar problem) is administered by one person (a volunteer no less)
Twilio knows which numbers will charge customers, THEY HAVE THE DATA. They can make a list of numbers that charge customers, and then have a flag that disallows SMSes to those numbers.
They also have relationships with phone providers in every market that they are in, and those providers can provide that same information and then allow a blacklist to those numbers or whatever format the premium numbers occur in.
It's not hard at all. It's a nice value-add feature and I'm sure if a competitor like MessageBird implemented something like this, it would be an easy differentiator if Twilio doesn't want to provide this.
Twilio should help their customers with this (and it looks like they do have something, but maybe not enough)... but it's also something you can do a first pass through libphonenumber metadata[1], which was pretty reasonable at my last job.
Google "international premium rate number" and you can get one in any country for free. Up to $0.7 a minute on some satellite and Albanian numbers if you opt for net 30/45 day payment in some places! This stuff happens everyday - I'm surprised no one has talked about it here till now. It is a very difficult problem to address if you want to accept international users and limit false positives.
I remember reading an article where a guy in the UK set one of these up with a bot like Lenny to make money off the scam calls. It was kind of shocking to me you could just setup any number like that, not just 900 numbers.
In fact, I built my own little personal telco around their services and API.
It is all slowly falling apart, however, as their failure to build out their infrastructure offerings (as opposed to their "customer engagement" offerings) and the regulatory SNAFUs[1][2] that are emerging as a result of their behavior erode all of my use-cases ...
All I wanted was more, and more useful, twiml verbs ... instead I got "customer engagement workflows".
Their support for registering 800 numbers and verifying your identity is absolute 100% trash. I've filled out their verify form many times, every time to be prompted to do it all over again months later. Over and over again.
I was a HUGE Twilio booster back in the day. I encouraged its use heavily in my very large org because it was so much better than our entrenched phone provider. I was (maybe still am) featured on their website. But things have just gotten more..business-y, less developer focused, just..generally worse.
I don't know what to do with these guys anymore. I understand they're trying to grapple with the mess of regulations that got dumped on them, but their messaging and support around all this is BAD. There's no answers, just walls of canned text.
They start out insanely great when riding high on VC cash, but over time they become the gorilla in the room and their service and support and quality begins to diminish until they're more annoying than they are helpful.
Hey! I'd naturally recommend SignalWire (as one of the founders over there.)
We have a full messaging + voice + video APIs, including a Twilio-compatible API just for people who need to switch. We're backed by companies like Deutsche Telekom, T-Mobile, and Samsung so we know how to make telecom infra!
We're also the folks behind the open-source FreeSWITCH framework that powers companies like Bandwidth, Five9, Dialpad, Zoom Voice... maybe even your own company's PBX!
I mean, seriously. Twilio could allow (verified-ownership emails only) for the simplest possible email integration into twiml for the sole purpose of alerting and paging, etc.
No spam possible since it's only verified account-controlled emails. Basically sending email to yourself from within twiml.
But no. Instead they bought sendgrid and email integration is a complete abomination of a two-company, two platform, two accounts workflow that is fragile and fails all the time.
Unfortunately, I haven't found any. There are some hacky solutions that I've bookmarked over the years, but nothing reliable enough for a production service(s). At least that I've found.
Most "alternative" SMS services a simply a façade built on top of Twilio, with the markup to prove it.
We faced this at my last company and this is actually a super mild case. In our case, we were dealing with call toll fraud. We ended up with tens of thousands of dollars in charges in less than 24 hours.
In our case, Twilio reached out to us to tell us they were detecting toll fraud. Before that, we actually had no idea what toll fraud was.
We quickly tried to address it with distributed rate limiting and that worked, for all of a couple of hours. The fraudsters quickly figured out the rate limit and worked around it by spacing out the calls and using more IPs.
Eventually, we had to disable a set of countries known for toll fraud and change our product to not connect calls in a variety of scenarios.
Twilio managed to convince everyone that SMS based auth was a good idea but it's always been a bad idea. Drop twilio and go back to using passwords and use a different 2fa method.
Were falling through the computer literacy gap between SMS MFA and authenticator/Yubikey MFA at the moment. While an IT person can do password managers (with secure backups) authenticators, passkeys, and biometrics, all with half-decent opsec, the average user can barely do more than a couple of passwords for everything, and SMS MFA.
It's absolutely critical that the companies we support vastly improve their security, but there's no way to get there from here with their staff, lack of any established processes, and zero training infrastructure.
Yep, I remember getting pinged by a coworker asking why is our Twilio bill so high all of sudden. It turns out to be Toll Fraud through 2FA messages. Malicious actors sign up new accounts and setup 2FA number and just keep requesting 2FA through SMS to profit.
When I recently wrote Twilio code the first thing I did was add in as much stuff as I could to prevent this sort of thing happening. I think I put in captcha and also IP address throttling and request counting.
At the time I wondered if I was overengineering or gold plating but apparently not.
I do seem to recall that Twilio writes about this issue quite alot and includes strategies in its best practices for avoiding the issue.
We've been hit by this exact issue, especially over the last month.
We tried to mitigate as cleanly as possible for our users, adding one-time nounces to signup requests, adding rate-limiting rules, locking down regions, but we still faced an onslaught of tens of thousands of fraudulent signups per day. On our tier we don't have the ability to set block rules ourselves - it requires a support request that takes 2-3 days to get a response on. Our choices are to eat thousands of dollars per day in toll fraud, or disable sign-ups until we can add more fraud prevention on top of what Twilio enables. The problem is the fraudsters are using real browsers across thousands of IPs located in dozens of different countries.
Similar to the OP, Twilio tries to say this is our fault and leaves it up to us to both pay for the issue and to try and fix it.
They should be better equipped to detect and prevent the abuse. It's an order of magnitude higher request volume for phone #s located in remote regions of the world. Twilio knows full-well where those numbers go, and can see them being abused simultaneously across many customers. I don't possess the same ability to know this... unless I use Twilio to run a reverse-lookup, which would of course still incur a cost.
Yes, it would be helpful if they helped fight abuse, but that is not necessary of them. Having the capability is a competitive advantage so it would be in their interest to invest in it.
We were trying to avoid the use of a captcha; originally believing that our API infrastructure was the target. A captcha did end up being the solution, but is not particularly user friendly, and I was also trying to avoid pulling developers out of bed on Christmas to implement - but we're protected now!
I've read through a lot of the responses and I am still kind of confused how the fraud actually works:
1) Scammer leases a "premium phone number" from a provider. From doing some reading, premium numbers are where the caller/texter pays extra for interacting with the service at this number. So like a 1-900-phone-sex line from back in the day, where if you call, you get charged like $5.00 a minute. The provider leased the number to the phone sex operator for $1 per minute. The phone sex operator runs the service and charges access via your telco at $5 a minute, and ends up netting $4. The telco bills you $5 for your 1 minute call.
2) In Twilio's case, they get a request to send a text to a premium phone number leased by the scammer. This text is actually initiated by the scammer, via something like requesting a new one-time password. Twilio sends the text.
3) Twilio then determines that the destination number is a premium phone number. Twilio charges you extra for sending the text because of this. Twilio then remits a payment to someone, either the scammer or the premium phone number provider.
4) Scammer repeats step 3 a very large amount of times and collects. Twilio bills you for all of those texts they sent, on your behalf, to the scammer's premium number.
Step 3 is where I am confused. How do the payment flows work. Is Twilio remitting the money to the scammer, who then needs to pay for the leased number? Or are they remitting the payment to the premium phone number provider, who then pays some portion of that to the scammer?
And come to think of it, how does the phone sex line example work? Which entity actually contracts with the telco to set the cost/toll?
The carrier offers the pay-per-call/sms service to a business (like your phone sex operator). The carrier charges the fee and some percentage is given to their customer.
So, rent one of their numbers with a fee attached, get a bunch of CAPTCHA texts sent to your number. Your carrier charges Twilio some amount for each call, then sends you a check for some percentage of that.
We had this exact thing happen to us. I think the most offensive part is that Twilio still makes money as their customers get scammed so are financially incentivized to keep letting it happen. And on top of that they disrespect their customers by forcing you to prepay for your usage but will still happily let some scammer run your balance into the deep negatives even if you turn off auto-refill.
I used to work on toll fraud as part of IT for a large tech organization. Not FAANG but several thousand employees.
My impression at the time was the fraud detection heuristics helped, but the root cause was scammers working with telcos overseas. Many of these companies have extremely high rates in the first place, and turn a blind eye to the practice because it makes them much of their money.
Some VoIP providers like Twilio may be the same. As a middleman / carrier, they will always profit from traffic on their network, even if they disavow it as fraudulent and wag their finger at you about it.
> Many of these companies have extremely high rates in the first place, and turn a blind eye to the practice because it makes them much of their money
Not a telco but loosely related... I will never forget the day I was working for MediaFire (~2013) and complained directly to the CFO about advertisements on the website that directly violated the advertisement policy. CFO stated that "they paid us a lot of money" and that he's late for a ping pong match then walked out of my office. Well, they might have paid the company a lot of money but I was directly not paid a lot of money and was very much burned from friends for working with scum.
Suffice to say that people in positions to do things about it should be much less forgiving about it.
I spent a lot of time playing cat and mouse with this type of toll fraud in 2022.
1. Rate limited SMS by number/ip: bypassed by large number of proxies/vpn.
2. Added captcha: bypassed by attacker manually signing up thousands of accounts (mechanical turks?) over months and then iterating over them for login OTP.
3. Identifying what carriers/operators are involved and blocking them asap (usually obscure ones).
4. Careful monitoring of SMS send rates and alerting of anomalies to investigate.
Good advice. By the way, the reason captcha didn't stop it is because Recaptcha is $2 per 1000 solves on 2captcha.com (or any other solving service), at $0.02/SMS this only lowers their profitability by 10%.
These numbers are usually not premium in the 1-900 sense of the word. It's more like they are international numbers and there are various intermediaries who work with mobile/landline operators in a bunch of countries to set up these kind of numbers and split the revenue from incoming calls/texts if they can deliver lots of minutes to them. One way of doing so is by getting a bunch of 2fa texts sent.
I trust those with more expertise on the matter understand they nuances and difficulties here better than most or any of us do. This is Twilio we’re talking about - they turned down a ton of money in 2022 political advertising to protect their customers from campaigns that were overly aggressive and had ignored customer requests to opt out. They’re not money grabbing thieves. That’s all to say I imagine this issue is more complex than most of us understand.
I will say though this isn’t a great look and hopefully Twilio addresses it. Perhaps there are significant trade-offs to enabling this option by default, but it does _seem_ (from an admittedly naive perspective) like perhaps it’s better to start folks with the training wheels on and make sure they know how to ride the bike before you let them go in the street.
If anyone's facing this in their auth flows, we're happy to help at https://clerk.dev
We're in the same cat-and-mouse game with the attackers as everyone else, but since we're an auth company, we have full-time folks monitoring for issues and resolving when they come up.
It's worth mentioning that Twilio is in an understandably tough position here. They only receive API requests from your server, and real requests look the same as attack requests except for the phone number.
Clerk is in a better position to help because our API accepts traffic directly from the attacker (e.g. POST /verify-phone-number). We know their IP, user agent, whether they're connecting from AWS, etc, etc. We very much rely on this data to help stop them.
I find it particularly frustrating that they force you to upgrade to Verify to solve the problem unless you want to build out a lot of your own internal risk detection (which we ended up doing instead)
With the rise of AI APIs, I expect we'll see similar attack vectors for apps that integrate APIs from OpenAI or Stability. There won't be a colluding telecomm, but the API output (a completed task) is relatively fungible and far more valuable in itself than a SMS API response. Something to keep in mind if you're building an AI application: https://stytch.com/blog/securing-ai-against-bot-attacks/
Also, a tip for anyone that feels like the low hanging fruit prevention methods aren't working (e.g. CAPTCHA, rate limits, etc.)
Consider installing a device fingerprinting system -- this has be the single most effective solution we've seen our customers integrate for more sophisticated bot problems: https://stytch.com/docs/fraud#device-fingerprinting. I'd recommend against the off-the-shelf solutions (e.g. open source ones) because many of them are easily reverse engineered, so they work well for low-level threats but not for persistent ones. In addition to our solution, Arkose and Fingerprint Pro are a couple ones I'm aware of
Twilio and their users are the victim here... Twilio having KYC would not solve the problem, they do not the own the numbers the expensive texts are being sent to. Twilio should just enable their anti fraud systems by default (by the way this is no panacea, like every other anti fraud this is a cat and mouse game, there is often no clear way of telling that a number is premium rate, and many carriers are in on it too and use normal mobile number ranges).
As a programmatic telephone company, they're a possible (but not really probable) base for fraudulent spam calls. With KYC, and the fact that Twilio requires you call from a number you control, fraudulent calls would be easy to trace back to a person who could be charged for the calls. Much better than status quo, where it's very difficult to get to the originating phone account, and if you could, it's probably not really connected to a person.
Surely the overheads of any useful KYC are way too high for this to work? And basically nobody in this industry does KYC, so how do you propose that would meaningfully affect their interconnects?
We’ve had the same problems. We use Twilio for SMS based OTP login, lost lots of money to toll fraud, and spent lots of time putting up various mitigation strategies to reduce it. Now we only lose a bit of money to toll fraud, but if was lots of engineering effort and $$ down the drain.
My main suggestion would be to avoid any sort of flow, like SMS OTP login, that allows triggering SMS messages without being logged in. Just do a more traditional login, SMS OTP isn’t worth the headaches.
Haven’t tried Twilio Verify, didn’t exist when we were solving these problems ourselves. But like most fraud prevention, it’s probably far from perfect, better to just avoid fraud-prone workflows if you can.
Same thing happened to us, same day. We're now on the hook for $3,500.
When Twilio PROFITS from known fraud, they have no incentive to stop it even when it's obvious. (In our case, tens of thousands of premium SMSs to the same number within the span of a few hours).
They aren't issuing a refund for us. As an unfunded startup, this is incredibly deflating. We feel robbed, and Twilio should feel ashamed for pocketing a juicy kick-back from the robber for letting them continue their robbery without making a peep.
We faced the same problem at Zenly and had to build our own anti-spam strategies to prevent it since Twilio was not taking care of the problem for us.
We used multiple providers to improve our conversion rate and reduce cost.
We are now building this as a service https://www.ding.live/ and are seeing huge improvements for our first customers in term of cost savings and conversion rate.
Feel free to reach out if it could be any interest to you hello@ding.live.
Is it not possible to ban pay Toll numbers in 2FA applications? Why doesn't Twilio do this by default? I would absolutely dispute the charge. Or better yet use only virtual credit cards for these services like Twilio that cannot be trusted, with fixed spending limits and monitor them closely.
There’s no way to set up an account such that it isn’t permitted to text premium numbers? Throttling to prevent the same number being messaged more than a certain number of times in a given window? Or throttling to prevent charges accumulating faster than a set rate?
The fact the Twilio is allowing toll numbers at all is clearly their fault, not that of their customers. Turning around and claiming that customers should be paying for twilio’s bad choices is BS
This is the first I'm hearing of this, so I might be missing some information, bit I don't understand how this is Twilio's fault or responsibility.
Your service got hit with a ddos-style attack that translated into you using twilio to send lots of texts. This cost you a lot of money.
I don't see how this is categorically different than your kid "accidentally" buying movies on Amazon prime or something like that. No way a credit card company would accept a chargeback in that scenario.
Ultimately, you used their product in the intended way. Of course you're on the hook for the bill.
> I don't see how this is categorically different than your kid "accidentally" buying movies on Amazon prime or something like that. No way a credit card company would accept a chargeback in that scenario.
The issue isn't the scale or volume, per se, it's that a bad actor has set up premium numbers (that cost $$$ to message) and is systematically wracking up fraudulent charges via websites sending 2FA codes. Twilio is seemingly aware of the fraud campaign targeting its users, but is not doing a great job protecting them and forcing them to bear the costs.
A better analogy, I think, would be a crime ring skimming credit cards at a gas station and wracking up charges that should be obvious fraud (different country, large amounts, etc.); and when a victim contacts their CC company they go "oh yeah that Shell station is notorious for fraud we've had lots of complaints recently" but refuse to chargeback.
Isn't this something Elon Musk brought up a few weeks ago when Twitter SMS 2FA stopped working in some countries? (India? I think?). On a Twitter spaces he said they were losing millions to SMS fraud for years and found out that some Telecom companies were complicit so they just cut off all SMS traffic to those companies until they re-negotiated terms.
Last time I tried to sign up for Twitter it demanded I verify my account with text messages. Actually, virtually all services do this now when creating an account. The worst (Microsoft for example) let you sign up and use the account for a bit (possibly purchasing some items tied to the account) and then extort the phone number out of you later to maintain access.
It is sort of amusing that these companies hitched their wagon to the now scam laden telephone network to track users and ended up getting scammed themselves.
TBH I would consider this type of fraud of a more Robin Hood variety. Companies that still encourage weak security practices like sms 2fa (or even worse, just hoover your PII under the guise of it) should be defrauded of their money as much as possible.
This is illegal under GDPR. After someone has signed up for a service you can't then demand additional personal information as a condition of continuing to supply the service.
I spent a lot of time working on this exact problem at a "Big Tech Retailer". 6 different teams had worked on it before we did, and all had given up. This is actually a very difficult problem that is at the intersection of two other very big and familiar problems... spam phone calls and bots on the internet.
Spam phone calls... the global phone system is a network of relays. No telecom provider connects everyone on the planet together. To call our grandmother in Russia, we may have to go through Verizon, Deutsche Telecom, MTS, and ~five different smaller, regional telecom providers. The first telecom provider will request the second to complete the call, will trust they do this, and will accept the price they charge upon which they'll add their own costs. This occurs recursively until the phone call has been connected and completed. This implicit trust enables fraudulent actors to get into the circle of trust. Verizon may trust Deutsche, Deutsche may trust MTS, and MTS may trust a smaller telecom provider who in turn trusts a spam caller. This enables you to get spam calls. Telecom providers themselves don't know all the callers on the global telecom network and don't really know how much people will be charged. There is no global government to legislate across all telecoms.
Bots on the internet... the internet as a whole doesn't have a firm sense of identity. It's just a network protocol routing packets to ip addresses. In the past, these ip addresses were mostly human beings. In the current time, the majority of the participants on the internet are bots/computer programs. A website like "Big Tech Retailer" has >90% of all traffic from computer programs. Elon Musk was probably right that Twitter is full of bots, because the entire internet is swimming with bots. They can be incredibly difficult to detect because AI blurs humans with bots.
This toll fraud problem is that bots we struggle to detect place phone messages to phone numbers we struggle to identify. This ends up costing a huge and growing amount of money. You cannot truly solve the problem without solving the two underlying problems of bots on the internet and spam calls. Solutions to those problems may require rethinking and rebuilding the entire communication system we've built our lives around.
Nonetheless, we can greatly reduce the effect of this problem. At "Big Tech Retailer", myself and two others we were able to reduce the cost to a small percentage of what it was. After that point, the business sort of stopped caring because the fraud cost less than the staff. There were perhaps five techniques that were most helpful, all of which were contemporary fraud fighting/bot fighting/security techniques.
If you're a startup facing this problem, I can help give you some guidance. Twilio will probably see this post and start working on a solution, but that may take a long time. There are easy things you can do to mitigate the problem right now. You can contact me at manrajt@gmail.com.
If a threshold of Twilio customers dispute charges, Twilio loses the ability to process credit cards at a lower risk rate, then with all but high risk processors, then may lose the ability to process them at all.
If enough of their customers are getting burned, and enough dispute, Twilio would no longer be able to accept credit cards. They are terrified of that, so begging you not to dispute charges for their lack of fraud prevention.
You accepting anything less than full refund of all fraudulent use they're cascading back on you is a gift to them. You accepting less than a full refund, while not dinging them at all with a chargeback is also a gift to them. If they don't want to give you the full refund for misuse they should be preventing, dispute it, as is your right.
The correct course for Twilio is for Twilio to refund these charges no questions asked while fixing the problem.