Hacker News new | past | comments | ask | show | jobs | submit login
Twitter ‘smytes’ customers (techcrunch.com)
1422 points by panarky on June 22, 2018 | hide | past | web | favorite | 551 comments

Does anyone inside Twitter know who the executive was who pulled the trigger on this decision? There's no way this person knew the gravity of accepting the risk for blowing this many high-profile customers out of the water all at once.

I'm pretty accustomed to executives making dreadful decisions without the approval/acceptance of other stakeholders within said person's firm. I'd rather know who made this error and not interact with that specific person's department rather than stop doing business (e.g. large ad-buys) with Twitter as a whole.

I don't want this to be a witch-hunt so much as I want the person to just come forward and own the decision, because unless they have an exceptionally good reason for it, it comes off as absurdly high-risk to both Twitter-the-business as well as to all the clients who've likely written serious penalties into their contracts for events like this, which again brings that business risk back full-circle to... Twitter.

I work at twitter and wasn’t able to find out immediately, but have been collecting responses like these + the article and communicating with someone who knows what to do. I only found out about this over twitter and I am personally deeply frustrated and working to understand whatever led to this.

Sorry to hear that, can imagine the frustration.

As someone down thread said - this smells like the long arm of GDPR unintending some consequences.

I don't really think there's any polite way to put this: Your employers are simply assholes.

This is the sort of informationless pile-on comment that damages HN regardless of what you're talking about. Please don't post those here.


To be honest, this reads less like an attempt to integrate anti-harassment measures and more like an attempt to completely destroy a business trying to offer them. It further solidifies Twitter management as complicit in aiding and abetting the harassers by using its capital to eliminate a threat from the market. After the speed with which Twitter went after the ICE employees list, the evidence is clear that Twitter only has a problem tackling harassment when it's against marginalized populations.

That's a pretty extreme leap when virtually no information is available yet. The HN guidelines ask you to "assume good faith" for a reason: people are all too ready to take such charges as givens and then pile more on top of them. It's a behavior that harms the container here, and it's not hard to wait until actual evidence appears.

That's fair, I will suspend judgement until further evidence appears, but the well of beneficial doubt is running low. I think a long past history of suspicious behavior is a good reason to update our assumptions.

I don't disagree. But the real reason for having an "assume good faith" rule is what it does to this community when people don't. Therefore it needs practicing even when it feels undeserved.

Here's the whole guideline you're talking about-

> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

It's talking about how we all talk to each other. I think it's main purpose is not to lessen criticisms for corporations so much as to keep the discussion constructive.

You're probably not going to win an argument with a moderator by explaining to them the subtle logical error in the interpretation and purpose of their own guidelines.

Yet now we're failing to assume good faith in our moderators, by assuming that they will fail to recognize a conflict of interest arising from questioning the extent and source of their authority, which they have reason to see maximized.

The parent comment actually seemed spot on to me. The rule is to assume good faith of your interlocutors. The parent failed to assume good faith of a billion dollar corporation. I think it's reasonable to push back against being asked to tone down the latter, while honoring the rule for the former.

I think dang explained it pretty clearly in the comment. It is bad for the forum when, given some very limited set of facts, you offer the most heated, ragefest pile-on inducing explanation or theory. It doesn't require some talmudic reading of the guidelines which are, after all, guidelines and not every single thing is explicitly spelled out. 'Avoid taking discussions into flamey or ragey directions' is advice the mods dole out to some person or another nearly daily.

the well of beneficial doubt is running low

That is a metaphor made with an industrial grade cement mixer.


>Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way.

I don't like the implication here (i.e. white men don't care about harassment/abuse and they are also stupid.) I would wager that most anyone of any color or gender or religion in any high position of authority at any company cares first and foremost about the financial viability of the company they have been charged with running.

Many (most) "white men" do in fact care about this stuff and aren't bigots/sexists/racists. You're naive and part of the problem if you think this is a "white male" issue and not a much more complex problem. In fact, you're exhibiting the bias you attribute to these managers.

I think it's fair to say that people who are not white males (such as myself) are more likely to be targeted by a wide variety of harassment, and that without their views it may be much easier to miss the issue. As a white male I don't get targeted by misogyny or racism (at least not anywhere near the scale that others have). Having people who have experienced that around to help guide the decisions for how to deal with it- and how many resources to devote to it, since most business decisions come down to money.

Also, we can all have biases without it meaning people are being racist. My parents like rock music, so I'm biased towards listening to that over country. I went to certain schools and that colored my world view. I once heard bias described as a heuristic the brain uses to make quick decisions- something that was extremely useful for primitive humans but which needs to be actively balanced in our complex society. A huge part of that for decision makers is exposing themselves to other viewpoints as a way to collect information they themselves wouldn't have gotten. There's a reason why companies have used things like focus groups and surveys to understand their customers- having more people who understand those customers in leadership at companies is an even better way to do it.

Yea me neither. Generalizations like this aren't helpful, or least bit fair.

The abuse is disproportionately targeted at people who aren't white men, and is is disproportionately performed by white men.

This is nothing particularly new. It's the history of America, which was founded by and for white men, and whose story can be told as white men slowly and reluctantly giving up their power.

If "many (most) 'white men' do in fact care", history doesn't really demonstrate that. Not, at least, in terms of doing anything about it. This is not particularly shocking; people care more about problems they experience, and less about problems that happen to benefit them.

We need only look at how the majority of white men supported a blatant racist and sexist for president. And who continue to support him despite (or because) that has only become more clear.

Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way.

To the folks inclined to keep piling on to this comment, I think a more charitable reading is that "The leadership consists of very privileged people who likely haven't dealt firsthand with the kind of harassment to which some demographics are routinely subjected. They appear to just not really get it at times."

I think that's a very realistic assessment of how some things remain intractable. People in power have blind spots because it isn't something they have to personally deal with. So they end up doing things that are counterproductive or seriously problematic without really meaning to.

That doesn't make it okay. That's just reality.

>The leadership consists of very privileged people

Same problem. Who's to say they are "privileged" as opposed to being hard working / talented people? You're pushing the same narrative; white men in positions of power are privileged and don't really deserve what they have. Nonsense. Sometimes that's true, sometimes it's not. Either way it's a useless, tribal way of thinking that does more harm than good.

It's completely fair to comment on this because it is the basis of the GP's argument.

I don't read the parent comment as claiming that the people in power are not hard-working or talented, nor that they don't care. Rather, I read it as asserting that many white men have had the privilege of not having to deal with discrimination and harassment, and that it doesn't come to mind as easily as it does for underrepresented individuals.

Just like a developer is going to spot defects in applications more than a non-technical user because they are exposed to defects more often and understand where things will fail, people who have lived discrimination will be more likely to see it where it happens, and understand where to look to find it.

I think it's asserting specifically that the management of Twitter has had the privilege of not having to deal with discrimination and harassment.

I imagine the leadership of a company like Twitter make really good money, have substantial power and don't have the problems of your average working stiff, regardless of skin color, gender etc.

It's sort of tautalogical to say "People in power don't live like the rest of the world." And this fact has been a problem in terms of creating policies that work well for the masses since time immemorial.

It's one thing to say that people in positions of power, on average, have not directly experienced the harassment or abuse that minorities or people belonging to marginalized groups have. That's fair and may account for some level of ignorance on their part. It is another to say that they _do not care about these things_ specifically because they are a member of a certain class/race/gender/whatever.

It's an important distinction as it sets the parameters by which we engage in conversation and debate. If you begin the conversation by accusing the other side of being inherently bigoted then there is no civil or useful exchange of ideas to be had. It puts the other side in a defensive posture from the get go and only leads to more tribal group-think.

And you refusing to accept my reframing as valid intentionally digs this hole deeper. This is a perpetual problem where one group feels unheard, unserved, left out, etc and then they complain about that and then are told their complaints aren't valid because they said them in a not nice enough fashion or something.

Both sides need to give a little here if we aren't going to remain stuck on such topics. And expecting all that giving to come from women, people of color, and other underprivileged groups is merely entrenching white male privilege in a really ugly, toxic manner.

>And you refusing to accept my reframing as valid intentionally digs this hole deeper.

Because I don't believe it is valid. Class/color/gender are not the only influencers on someone's decision making process, yet you two seem to think that's all we need to consider.

Where is the evidence that these people don't care about abuse and harassment because of their class and color? I see a pretty damning statement which includes zero evidence, just what someone believes to be true for <reasons>.

So no, I won't accept your premise or the GP's until I see evidence. If there is no evidence then fine, you and anyone else are free to believe whatever you like, but don't pretend it's a logically defensible position.

>This is a perpetual problem where one group feels unheard, unserved, left out, etc and then they complain about that and then are told their complaints aren't valid because they said them in a not nice enough fashion or something. Both sides need to give a little here if we aren't going to remain stuck on such topics.

Not every complaint _is_ valid, and we have no idea what the GP's motivations were. I'm not going to give any ground. You and the GP are taking a complex issue and boiling it down to something absurdly simple. All that I'm saying is that it's not simple and, if we're ever to have a reasoned debate, it needs to be predicated on facts, not "feelings". If you're going to attribute someone's actions to their gender and race then you had better be able to back it up.

It's telling that you believe _I'm_ the one doing harm when I have made no accusations at all and the GP was inarguably expressing a racist view.

It was not in fact a racist view. It's a view about power dynamics in a system that is racist and sexist.

If you really can't get over your feelings of personal insult and your need to defend white men, then please go back and substitute "part of the socially dominant racial group" for "white" and "part of the socially dominant gender group" for "men". In US history, those terms are equivalent. If you don't like that, talk to the founding fathers who set it up that way.

The reason it's material here is that a large portion of Twitter abuse is related to this. You can look at famous incidents, for example. Leslie Jones received a giant wave of racist, sexist abuse: https://www.thecut.com/2016/08/a-timeline-of-leslie-joness-h...

And the company admitted they had failed massively: https://people.com/movies/twitter-ceo-jack-dorsey-speaks-out...

You could also look at the differential experience of white men versus others on Twitter. E.g.: https://www.theroot.com/twitter-has-a-serious-harassment-and...

Note also your standard here. You, a white man, refuse to accept that race or gender could be relevant to a social problem until other people take the time to spoon-feed you. Why isn't it your job to know something about American history? Why isn't it your job to learn about the problem of online abuse before deciding that the privilege afforded to white men couldn't possibly be the relevant? Answer: because you were raised to believe that whiteness and maleness were the default and were good. So you choose that as your supposedly neutral stance, and require others to prove you wrong.

The true neutral stance would either be "I don't know about this problem and would like to learn" or "let's assume that online culture has some relationship to the existing offline social dynamic".

Wealth is very nice to have, but it does not insulate someone from suffering racism and sexism.

it's not either or. privileged AND hardworking is a possibility.

> Who's to say they are "privileged"

The employee who is familiar with them.

Yes, exactly.

The problem is not the genetic characteristics of whiteness or maleness. It's that they grew up in a society that favors whiteness and maleness, and is much more accepting of the abuse of people who aren't so favored. If Twitter had started in China, where the Han are favored over ethnic minority groups like Tibetans and the Uyghur, it would be the same dynamic but with different ethnic groups.

Of course, tech is disproportionately white and male, so it's entirely unsurprising that HN readers have risen up en masse to defend the honor of white men. It exactly proves the point. White men will say that fighting abuse is important. But when they have to show their real priorities by how they allocate time and effort, actually understanding the abuse problem and how it disproportionately effects minorities comes well after defending their own interests.

(And: Yes yes, not all white men. Generalizations are general statements that may or may not apply to an individual.)

It's late. I'm tired. I left a couple of replies earlier, then deleted them. This seems like a just no win situation to me.

I'm not a feminist. I'm a former homemaker that self proclaimed feminists routinely crap on. I come to Hacker News because I need a place where I can engage in meaty intellectual discussion so I don't lose my marbles. That is my only agenda for being here. However, dealing with sexual politics is not something I can entirely avoid what with posting as openly female.

Suffice it to say that I don't really want to be associated with this toxic attitude and I am kind of wishing I had not said anything at all. I am starting to feel like "The only winning move is not to play" and I failed to make that move.

While that's true, you could at least tip your hat to Jack Dorsey's $3bn net worth, the insanely high executive and engineering salaries in San Francisco, and the effect that being so insanely rich must have on the complacency and sheltered lives of everyone at Twitter. It's a little bit worrying that you think race and sex are greater assets than class (= "network" and "culture fit") and money.

There is also an interesting 'get off my property' aspect to this acquisition. It may indicate that there is a desire within Twitter the company for complete control over Twitter the social media platform. Because they can, Twitter will always reign over the mirage of an ecosystem built on their social media platform.

I think to a great extent that can be (just about) assumed about anyone in a leadership role at a company the size of Twitter. Regardless of anything else.

Implying that white males don't care is not helpful and is both racist and sexist. I'm a white male and I care about abuse and all kinds of other things. Maybe you should do some reading on unconscious bias.

White males who don't show compassion move up ranks very fast in this society.

Not judging the individuals, but the history of this culture is very well covered.

I don't think Hanlon's Razor is an acceptable excuse above a certain pay-grade.

100% agreed. And I definitely don't mean it as an excuse. Just an explanation for why conspiracy is unnecessary.

>mainly white and mainly men. I don't believe they really care about abuse in a meaningful way.

That's an awfully biased viewpoint you've got there, good grief.

> Their leadership is mainly white and mainly men.

Wow. Racist much?

Another plausible scenario is the acquisition terms required Smyte to wind down the service over the past 6 months or build a replacement to fulfill contracts. Smyte didn't for whatever reason (either through negligence or negligence+hope to continue operations in a new capacity) and the deadline came to hand over the servers in the agreed condition (no externally accessible APIs available).

Twitter could have stepped in and halted things, but that would have required Smyte to have acknowledged breaching the contract and forfeiting $$.

This is all guess of what seems to me more likely than any Twitter exec pulling any triggers like this. Of course, it's still a screwup by Twitter to have not been tuned in and aware that fingers would point to them.

But that's a very different kind of mistake than "I have an idea, let's hit the power button at 6:30. Team: you have 30 minutes to let all our customers know."

You never, ever agree to a change in your business before the acquisition is complete. The chances that an acquisition won't go through are just too high, and it could leave you completely f'ed over.

Yeah, but if you're less experienced, or if you're more devoted to your potential profit than to your customers, you might agree to the term, and then realize what you just said, leaving you with the (immature but perhaps "seemed reasonable") situation we see here - doing nothing until the acquisition is complete and then immediately abiding by the terms.

To a large degree, the responsibility is also on Smyte, right? Selling to a big corp is one thing, and once you are no longer owner of a company you've washed your hands of future decisions. But at the time when they agreed to the sale they were still owners of the company and their hands had not been washed and they could have and arguably should have stipulated some conditions for how the service would be wound down following the sale.

Litigation aside, the founders of Smyte seem pretty culpable here. I would be wary of using another service that they created, given a demonstrated willingness to turn it off without any notice.

Only if the acquisition hasn't closed yet. If it has, who's there to go after?

But you're actually right -- if the acquisition hasn't closed yet, they're independent and independently responsible.

Twitter’s VP of engineering has posted some additional information: https://twitter.com/michaelmontano/status/101024630798476902...

"We couldn't operate their business and continue collecting data from their customers, while continuing to meet our own high standards as a global company."

My hunch as an outsider is that Smyte wasn't GDPR compliant. Their leadership knew it, they knew they couldn't easily become so (for instance, they may have been using Kafka in a way where compaction wouldn't help, and didn't want to build an encryption-based monstrosity [0]), realized that they wouldn't increase in value as a business due to that risk, took an acquihire for cheap in order to give their employees a decent landing and give a return to their investors, and couldn't tell anyone about these plans in advance because it might jeopardize the transaction.

EDIT: They were indeed using Kafka per [1], and due to the strict latency requirements on their business, that may have ruled out the type of encryption scheme in [0].

[0] https://danlebrero.com/2018/04/11/kafka-gdpr-event-sourcing/

[1] https://www.youtube.com/watch?v=6ByXQfIq5uU

I've seen lawyers advice that [0] doesnt do the trick anyway, ie deleting the key is not enough to comply.

GDPR does seem like the most likely culprit here, if they've been planning an aquihire for a while it wouldn't have been worth implementing GDPR and Twitter likely made it a condition that the service is shut down before the acquisition completes

I'm quite surprised none of the cloud vendors were interested/could offer more than Twitter for this team though, seems like a logical service to add.

That's a good thought. The only good reason to shut them down so abruptly is if they posed an existential threat to Twitter itself. The threat of GDPR fines or sanctions might have done it.

I still don't get why they couldn't just come out and say that. Was incompetence and unprofessionalism really a better look then a understandable engineering challenge? Or was this another case of "being honest and doing the right thing would have created legal liability so we opted not to do it"?

Generally your lawyers will advise you that the less you say, the better.

Maybe, but then why would Twitter acquire Smyte in that case?

Presumably because Twitter was interested in the project, but didn't need the actual implementation immediately. Get the team on the cheap for now, and then just have them rebuild it in-house in a compliant way.

This is one of those instances in which I think employees (developers) and companies working with Twitter need to have a "never forget" attitude. I actually know someone personally at Twitter corp dev (the group that usually manages M&A activity) and I hope that individual was not involved here, as it would be pretty out of character.

Why would Twitter care? They make money from ads, right? APIs and legacy deals are going to be time consuming and boring for them. Stuff like filtering hate speech etc is something which can be done globally and in a way which doesn't require highly paid employees, APIs etc. Hasn't Twitter repeatedly show that third parties aren't really that interesting to them?

All companies have suppliers and partners, including Twitter. Getting a reputation for this kind of toxic behaviour harms their ability to build relationships with them, as nobody will feel they can trust them.

It could also hamper their ability to make further acquisitions. In future, a startup being linked with a possible Twitter acquisition is going to cause its customers to panic and start moving elsewhere. That'll make startups more reluctant to engage, and may turn some off entirely, if they're not comfortable with the prospect of totally screwing over the customers who've supported them during their early stages.

A startup's customers panicking? Not sure that's on twitter's radar.

I think potential for reputation damage, on an activity they don't do very often, and where they will justify it, will be weighed up against the financial benefit now, and disregarded.

I think it's possible to overstate the relevance and importance of people typing furiously on internet forums about things like this. Twitter is what it is - most people aren't using APIs, third party clients etc and will never notice this.

I never said Twitter cares about startups' customers panicking, but it cares about being able to make strategically useful acquisitions, like Smyte. And acting like this makes it harder.

This isn't about people typing furiously on internet forums. This is about business risk, as weighed up by people whose job it is to care. Acquisitions take time and negotiation. Just because a startup is talking to Twitter, doesn't mean the acquisition will definitely happen. But if customers get wind of it, then they may start looking to move elsewhere, anyway, instead of waking up to find services they rely on have been shutdown without warning. If you're the customer of a startup Twitter is negotiating to acquire, then you're negligent not to reduce your reliance on them. And if you're the CEO of that startup, then you have to consider the risk of customer flight (and your own comfort with the thought of screwing them over) vs. the likelihood of the acquisition happening.

if the Smyte shutdown limits the opportunity of current and future Twitter associated startups, whose business model is to help companies safely navigate the social media landscape, then that sounds like a win-win for Twitter

But the very behavior you're talking about... didn't make Smyte not sell. So even they didn't care.

Why would anyone else?

They can shrug it off, still those companies that had contracts with Smyte are not one person businesses on the other side of the world. Those Twitter people are going to have lunch or dinner with other people and get to talk about this issue for a while. Not very pleasant, even for employees totally unrelated with this case like the one who answered the very first comment in this page (sorry.)

> A startup's customers panicking? Not sure that's on twitter's radar.

Let's not forget that those customers are also likely candidates to advertise on Twitter

Twitter sells data through an API. Unsure how profitable it is. See: https://developer.twitter.com/en/enterprise.html

We had 20 minutes notice, and then everyone was kicked out of the Slack support channel and API responses simply died. What the actual fuck. They have mobile client SDKs out in the wild that are now just eating up battery life as they retry an impossible query forever.

That sucks. We (Sift Science) have been building something similar for 7 years and aren’t going anywhere. If we can help, please ping me - jason at siftscience dot com

> We (Sift Science) have been building something similar for 7 years and aren’t going anywhere.

But that doesn't come with any actual guarantees does it?

> and aren’t going anywhere

Yet... that you know of.

Nothing specific against Sift Science whatsoever in my comment, but many, many times this sentiment has been conveyed by companies and it rings hollow IMO. There are a lot of different ways a company can change or disappear at some unforeseen point in the future, and claims that "you can trust us to be here forever" do not carry much weight for the large group of experienced users, devs, management, etc who have been burned multiple times.

So are you implying that you should never enter into a business relationship with another company?

Clearly, you need to always be prepared for your partner to go away, but you still have to work with other companies.

I think the parent commenter is just saying that it's not something you can actually believe, just like a company saying they'll never sell your data.

Sure, that's the intention but when someone comes around waving a checkbook, they can in turn buy you and shut you down, sell your customers' info, etc. Sure, its not YOU allowing that to happen but by ceding control you essentially allowed that to happen.

"You can trust us to be here (until enough money comes our way)" is just how it is for most businesses like this. An unfortunate, and in my opinion largely unresolvable, side effect of an ecosystem where specific useful tools are hard enough to create and manage that full companies have to be formed around them.

What kind of guarantee would be satisfying?

A third party code escrow account with instructions on how to build the service and a contract that specifies conditions when a customer gets access to that escrow.

I've been on both the customer and service end of such agreements.

Is there a popular provider for doing this? It sounds like a lot of work to find a solution in which both parties trust (both technically versed and reliability wise). A common notary probably wouldn't cut it, right?

Both companies I’ve worked for - as a client and a customer - used Iron Mountain.


There's a bunch of companies doing it, some dedicated, some as part of larger related offerings. E.g. I know NCC Group (large IT security company) does offer it, TÜV (which does all kinds of certification and compliance work), ...

The contract should specify. The contract probably was sided on smytes favor. Screwing even paying customers.

You can have a contract where you specify indemnification of damage in case the business ceases operations, goes bankrupt, get acquired, etc...

A guarantee is only as good as the guarantor. A guarantor can be a company providing a service, as well as all sorts of assorted guarantees about said service.

In time, the situation can change. The product can get sold. Cash can be spent. The guarantor can no longer make good on its guarantees.

You're back to square one where the guarantees are as worthless as the original service.

You need 3rd party backing (insurance) in such situations, but that costs money. This money is a cost which makes competing against unbacked entities tougher.

In most cases, you can not have 100% foolproof guarantees of anything. The closest I can think of is governments standing behind their banking institutions. Even there though, governments have defaulted on their guarantees.

The world is not a stable, perfect, and cut-and-dry place as many would like to believe. It is dynamic, and ultimately backed by trust.

It's much harder to mitigate this kind of risk in the world of services as against purchased software.

The problem has always been present especially when larger slower moving companies buy from smaller, riskier, companies.

In the days of software, code escrow was possible to mitigate some of this kind of risk. That's still got it's costs but can be an effective hedge against a supplier going bust.

The parent comment is justified.

While you can have a contract that indemnifies that isn’t going to help your going concerns when the other party just pulls the plug.

I think company bylaws that prevent a 'triangle merger' like this might be useful.

what's stopping the company owners -who are generally the ones executing such a merger- from changing the bylaws ?

Do you have a high level overview of what I’d need to do to get started? Are there any specific areas you specialise in as opposed to general users and what information will I need to share with you?

Shockingly bad. I wonder if they shitcanned the whole technical team and they turned the lights off on the way out. Not that it’s really much better but at least it wouldn’t make sense.

And there will be absolutely no consequence to these actions. Completely messed up!

I hope you think over that design the next time. Put it in the cloud they said ...

I actually ripped out Smyte's Android SDK because of how bad it was and replaced it with my own implementation against their API which has a hard limit on retries. But I imagine that other customers were using their SDK and still have live apps out there using it... not good :/

> Indiegogo, GoFundMe, npm, Musical.ly, TaskRabbit, Meetup, OLX, ThredUp, YouNow, 99 Designs, Carousell, and Zendesk

Half an hour of warning, to screw over all of these?

They broke npm's user sign-ups, and publishing of packages, with half an hour of warning.

I can't imagine the havoc over at Zendesk either.

That is not 'winding down'. That's ghosting.

It's worse for companies that have shipped mobile apps with the API embedded in the client. Web apps can at least try an alternative or disable the affected features until they work around this mess..

This confirms my belief that I don't want to ship anything that uses directly a third party API. Instead always go through your own server which then calls the API so at least you have a chance to replace that.

That works for a lot of cases, but for many tracking APIs you want the request to come from the end user, so the tracking service can get their IP and network metadata. That would certainly apply to a fraud detection API.

Tbh that has always frustrated me though. I should be able to pipe those requests and just add an X-Forwarded-For header.

This is not my area of experience, but is it feasible to capture all raw tracking data from the client devices and then just pass it on to the API?

That's basically a proxy or MITM, no?

Yeah, but it sounds like it would satisfy the concern of being able to replace the 3rd party API internally without the mobile apps having to change.

But it will have the cost of 2x API bandwidth (vs 0) and another point of failure that you are responsible for. A DNS CNAME probably wouldn't work if you had to go over SSL. Maybe a 30x redirector (still have the SPoF, but simpler and much less bandwidth)? Except in the past, I've found that most clients react poorly to redirects for anything other than GETs.

Is there any indirection that would avoid having to walk the traffic over my own network?

The way I would implement it is as a “proxy” on my own network, eg trackingservice.myapp.com, that is just a web server listening and forwarding requests (technically a MITM). Override the request method of the client SDK to hit my endpoint instead of the tracking service.

Obviously you then pay the bandwidth, but it’s likely negligible compared to your app traffic. As a bonus you’ll probably circumvent adblockers.

If you want to avoid traffic forwarding, but keep flexibility over the endpoint, you can override the init of the sdk to first query your server for which endpoint to use. That way if the third party service goes down, you just need to change the config on the server.

Damn right I'm the man in the middle if I'm the SSL-offloader. It actually strengthens security because there's one less party to trust for the client.

When I was developing mobile apps, this is what we did. Too many times something changes and trying to update the code in the mobile app is way more of a pain than updating the server.

> Clients had multi-year contracts in some cases.

I wonder how much Twitter is going to pay on top of the acquisition cost of what I can only imagine constitutes breach of contract with all of these companies. (If it doesn't, why even bother having a contract with a duration?)

Kind of depends on the terms of the deal, namely whether it's an asset deal as opposed to a stock deal. In the latter Twitter would assume liability for contracts. In the former they just acquire IP and hire away the employees.

Acquirers usually prefer asset deals for this reason as it allows them to leave unpleasant liabilities behind. There's generally some sort of shell left that goes through wind down but it has few/no assets attached. In this case you can try to go after the original investors including founders or other shareholders (the IRS may do this if there tax issues) but since there's nobody home you are less likely to get much out of a favorable judgement.

This seems like the kind of thing a legal system should be designed to help defend the market against. It's a net negative to the economy as a whole to let torpedoes like this fly without consequence; and evidently, a totally Laissez-faire market will engineer ways to insulate the perpetrators against the damages rather than the insulate economy as a whole against the splash.

On the flip side asset deals also let acquisitions proceed that would otherwise be blocked due to uncertainty about liability, which is another way of saying that the acquirer can't properly assess value of the investment. I have been through this process and have some experience with the type of issues that can arise--in our case they had nothing to do with screwing customers. It's possible Smyte would have just disappeared anyway.

Moreover recovering damages in this kind of case is time-consuming, doubtful, and too late to fix anything. (Like closing the door after the horse is gone.)

What's left is reputation--the names of the founders of Smyte are listed at https://www.crunchbase.com/organization/smyte. I would have some pretty hard questions if they ever showed up in a deal or looking for work.

p.s., According to the Techcrunch page "Smyte stops bad actors on marketplaces and social networks." You can't make this stuff up.

I'm pretty sure their contracts covered their ass. Hence why lawyers write up contracts in the first place.

You can write just about any words you like into a contract. But that doesn't mean they are enforceable and doesn't mean a legislature or court can't add or modify clauses. Then there's the whole doctrine of equity thing over the top of it. In law there are always a whole bunch of fine-grained questions that can change the whole outcome.

I expect one or more lawsuits to emerge, maybe as a class action. It might not come to trial and simply wash up as a settlement.

I am not now, nor have I ever been, a lawyer.

Both parties presumably had lawyers writing ass-covering contracts, so who loses? This isn't the usual "Wells Fargo screws you over then forces arbitration where you lose again" imbalance, there's no particular reason to think Smyte could have swung a favorable contract with e.g. ZenDesk.

It really depends on who was negotiating, whose paper was used to write the contract, and how much each side wanted the deal. Legacy enterprises like Disney often force vendors to use the legacy company's paper, which case substantial protections would be written in or the deal would not happen.

Self-service web services on the other hand generally come with take-it-or-leave-it conditions of use. When's the last time you redlined Github terms of use when starting a project? ;)

More companies than that, and large ones. These are just the ones that went public... some of them don't want you to know their trust/safety/moderation features just went dark. Smyte just ruined a lot of people's day(s).

Twitter. Twitter just ruined a lot of people’s days.

No. Smyte had a responsibility to its customers, and should not have entered into any deal that did not include some reasonable wind-down plan.


Twitter owns Smyte. When you buy a company, you acquire all of its responsibilities. There is no longer a Smyte to point the finger at. It's all Twitter.

This is pedantic. "Smyte" means "the people who formerly made up Smyte, especially the founders".

Legally, the onus is on Twitter for pulling the plug on an API that was being used actively without much notice.

I’m actually surprised that it was done in this manner, as many startups have a “business continuity” section in whatever contracts are drawn up with customers, detailing the specific steps and timeframe of retiring a particular service. I would have thought this is standard practice for a SaaS company.

My point was that we shouldn't point the finger at Smyte because that lets the blame drop on the floor when the "Smyte" brand evaporates.

Twitter owns this now and the way to ensure it's their reputation that is affected over the long term is by correctly calling this a "Twitter" failure.

Have we read the contract? Do we know what happened?

Twitter is the one responsible now.

What about blaming the companies who decided to depend upon a service that they have no control over? I find arguing against using third party APIs to save a few days of development a constantly losing battle, with it being presented as a benefit with no costs or risks. Do those pushing/agreeing with such a view not share some blame?

> What about blaming the companies who decided to depend upon a service that they have no control over?

Unless you're advocating against any kind of *aaS, this is kind of a silly argument.

A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.

Obviously when you make your product/company depend upon these things, you're taking a risk that they could go away, but you protect yourself against that by having robust contracts.

Outages are different, but for a significant outage you expect to see some kind of RFO and explanation of how they're going to mitigate it.

Deliberately withdrawing service for all customers with 30 minutes notice? That's entirely the fault of whomever is in charge at Smyte.

Even 30 days, while it'd probably ruin some existing plans, would at least allow people to have a chance to migrate without things just breaking.

>A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.

For the cases I was involved in, there were options of buying the service but hosting it internally with some update pipeline. So even if the updates were suddenly turned off, you have a far longer time to swap to a different system. Generally deploying these aren't to much time, but what is really the issue to me isn't that the options that were chosen were the ones which were chosen, but that the process of choosing them did not evaluate the risks of creating such a strong external dependency at all. Its one thing if it is a risk taken with full knowledge of the potential costs and benefits, it is another when people do it because they think there is no possible downside.

Are you saying that:

* No website or app should ever call out to any third party API

* Every single piece of functionality a website or app has should be developed in-house

* The functionality third party APIs provide would often take "a few days" to reproduce in-house

No website or app should ever call out to any third party API

Definitely. Your website and app should only call your APIs. Your API can then call the third party API from the server. It's a lot easier to change something on the server than having to update the app.

I wanted to argue with you for posting this, but you're right, this is a smart way of handling API calls.

It's not really addressing the whole issue of depending on third party APIs if your API is calling on them, but you are technically correct, which I've heard is the best kind of correct.

> What about blaming the companies who decided to depend upon a service that they have no control over?

No. Delegation is the cornerstone of civilization. Trying to build everything yourself is a terrible idea.

> that they have no control over

Per the article, they did at least have long-term contracts locked in with Smyte. That's not the same as control over the feature, obviously, but plenty of companies do business on the strength of contracts instead of vertically integrating their entire product.

Obviously we don't have many details yet, but I think there's a real difference between simply using a third party API that works for the moment, and buying a service from a third party. If Twitter/Smyte simply broke contract with all their existing customers, that's very much their responsibility.

This isn’t really a leftpad issue. This was for fraud prevention.

Twitter has ruined a lot of people's days for a long time now. Especially in the Trump era. Nothing new here!

Not sure why you are being downvoted since Twitter has a well-established history of pulling the rug out from under people who depend on it.

What would be these businesses backup plans if Smyte went bankrupt instead being acquired?

A company going bankrupt usually doesn't give 30 minutes warning that they are shutting down services though. They give more warning than that.

Presumably you would get more than half an hour to migrate.

Your supplier going bankrupt is a well known risk that you cover by checking its credit rating and by paying on or after delivery.

That's why often startups can't do business with big companies that hold to rules like that.

Would Smyte's rating have cut it for people?

Your supplier getting acquired is also a well known risk that you cover by reading/negotiating the contract.

The problem is, if all suppliers use the same contract clauses you may not have a choice.

How do you check the credit rating of a startup?

and how would their credit rating be affected if they pay all their bills but just reduce the number of bills gradually over time?

One way that's sometimes done is that you require them to submit an independent respectable audit of their financials, or in case of confidentiality issues, not submit the actual financials but a statement from a respectable auditor that they have verified the suppliers financials and their cashflow/"runway"/etc satisfies the criteria that you required.

I'm not denying that what they've done is really bad, but who puts a 3rd party API in a critical path without having any fallback (alternative service, reasonable timeouts, etc.) for when it goes down (as almost any service will eventually do)?

Everyone that does online transaction of money, and many more. Like all those who use AWS services on AWS.

People who have multi-year contracts with those API providers, probably with an SLA detailed in the contract. Especially since this API call would have to be on the critical path; you don't want people signing up when the anti-spam service isn't running.

1. You have an SLA guaranteeing 100% uptime?

2. If it's that critical, you have redundancy (an alternative provider, or a naive implementation that maybe causes more false positives but that will cover you during an outage)

I might be wrong, but I'm getting the sense that you don't have much actual experience.

I've been a senior developer in an organization and known that a solution isn't perfectly robust. Clearly I want to fix it because I (like most other developers) like building robust solutions, but I have six weeks of tasks to finish in two weeks, and that isn't one of them. What do I do?

You might say "do it anyways" and, if so, that's a junior developer attitude. I'd strongly urge you not to do that as shipping big projects is really more about choreography than engineering.

Or, you might say "fight for permission" and that's not a bad idea, as long as you accept that you'll likely lose.

You'll likely lose for business reasons. On some level, we all know that a SAAS on a critical path is a bad idea, but we do math. Our SLAs are not 100%, but we can accept a few hours of downtime a year. And, the probability of a service you've seemingly vetted being acquired and shutting down this quickly is low. Is the probability of a shut down like this high enough to justify adding and testing redundancy?

The math usually works out in favour of bug fixes and new features > redundancy.

In the first place, it was an awful business decision to take the route of trusting a SaaS provider with such an important task.

It is extremely irresponsible to forward sensitive user data and site interactions to a third party, even if it is in the name of spam/abuse/scam filtering. That its implementation brings down production websites in the case of service failure only adds insult to injury, originally caused by plain-awful product design.

If there's one thing that you should be doing in-house when you run a large online community of privacy-conscious users, it's the kind of service that Smyte provided. If sound business reasoning had prevailed, npm would have suffered no downtime.

It's easy to say that now that everything has gone to hell, but in business, we usually have to make decisions with imperfect knowledge.

I don't agree that it's irresponsible to forward sensitive data to third parties. Rather, I believe that intelligent companies have intelligent policies around sharing data with third parties. Some third parties provide an amazing service that would be extremely expensive for companies to mimic in-house. And, you cannot convince me that the probability of a well vetted service shutting down as fast as Smyte did is high enough to justify bringing this in-house when you have other things to work on.

No, but if I were npm, I'd have an SLA guaranteeing three-and-a-half or four nines of uptime (which is far from unreasonable) and just accept that there might be times that signup is not accessible because that'd only cost me a couple hours of signups a year.

And no, I wouldn't have an alternative provider. It would not make sense to, if I'm paying a company to provide a service. Why should I pay another to do nothing?

I know for a fact that you've done worse in your own life. Spare us the moral posturing, please.

What you just did is called a “Tu quoque” argument, and it is a type of Ad Hominem attack; i.e. a fallacy.


Good effort, buddy, but no.

This whole thread is moral posturing about how unprofessional Twitter/Smyte have been, but if we want to talk about professionalism we should look at ourselves.

If you upload to AWS S3 you put in a fallback to another service? And any other AWS services you use? That's a lot of fall backs. Is the second fallback good enough? Should we add a third? In any practical sense development would grind to a halt. Twitter/Smyte are totally in the wrong for not giving a reasonable notice. I think whoever made this decision should actually be removed from their post and the servers should powered back on again with an apology to all of their clients. If they still want to shut it down later then they should provide a reasonable time frame.

Maybe I'm in the minority, but I take an interest in the latest software/tech/ideas for dealing with the hard problem of moderation, and I had never heard of Smyte, until now (which made me think TC was being literal with its "Twitter 'smytes' customers" headline).

Apparently, they were a dominant player in this space, such that its shutdown impacted such a wide array of significant-sized companies, from ZenDesk to TaskRabbit to Meetup. Even more surprising is that they were part of YC (W15), and yet from what I can tell, have only had one significant mention on HN from 3 years ago (65 upvotes, 11 comments)


I would've guessed that Smyte would've been the kind of Silicon Valley business that was made for media hype (what with all the attention/criticism given to fake news and Google/FB/Twitter moderation). Then again, it does seem that content moderation -- important as it may be -- isn't a field as sexy as AI/deep-learning and other forms of automation.

They were also noteworthy for their founder Pete Hunt, who is largely credited with instigating the open sourcing of React. (As I recall, it was internal Facebook tech that he extracted into a separate library when he moved from FB to Instagram after that merger.)

Which is funny because they were actually dabbling in some AI/ML.

Given that these businesses had contracts with Smyte, how is this not a flagrant violation of the contracts? Surely Twitter took on Smyte's contractual obligations when they acquired the startup?

There are many ways to accomplish this, for example, doing it as an asset purchase instead of acquiring all the stock.

In the former, you can take the assets (including employee contracts) without the liabilities.

There's actually a reasonable quora answer (for once) on this, saving me from writing it up for you :)


In a case like this, does that mean that Smyte (the company, even if they no longer own the trademark) still exists and is liable for all the penalties? Would they usually ask for the cash to pay for them as part of the asset purchase?

In a case like this what's left other than an empty sack? Unless you're able to pierce the corporate veil there isn't anyone/anything with assets left to be made whole by (e.g. sue).

One could argue the transfer was fraudulent. And go after both parties. Twitter and the founders.

Don't know, not a lawyer.

IANAL but I was a plaintiff in a similar case. We were made whole in a settlement but before that, the judge tore the defendants a new one over this tactic leading to a bitter dispute between the acquierer and shareholders (or the board of officers, don't remember) over who was liable for breach of contract. Eventually it came out that the auditors had done the math and included cost of litigation/liabilities on these contracts into their valuations, basically admitting that they knew about the obligations beforehand, so the acquierer ended up settling.

At the end of the day, judges (and appellate panels) have to interpret the law and they don't function like automatons. They take into account the spirit of laws/contracts as well as the letter and most have enough common sense to get missed off at these naive tactics.

The argument that penalties should be paid out of the purchase price would probably hold water. When Smyte entered into those contracts they took on some duty to honor them and to suddenly dishonor them and divert all the assets out of the company so that penalties wouldn't ve avoided would be constructive fraud.

in that case smyte would still exist and now have lots of money from the acquisition you could sue for

Well, usually they are using the money to wind down.

(All of this is a good reason i hate LLC's. They aren't necessary anymore to actually do risky innovation in 90%+ of cases. In a non-LLC, they shareholders would be liable, and then you'd still have someone to go after)

Yes, it does mean the former.

Usually, they would use the cash to wind down.

This is the thing that occurred to me. Surely those contracts don't have a clause that says "Smyte can terminate all services with no notice"? If so, surely someone spotted it by now? Or is this normal in service contracts in the US?

I guess the liability is covered by the usual "no liability outside provision of the service" clause - so you can't claim that you were damaged by their lack of service. But I can see a lawyer making a convincing case that the liability should rest with Smyte. IANAL, obviously, but it seems like the sort of thing they love to fight about ;)

gets popcorn this one will be interesting...

It is in fact a possibility that Smyte's terms were extremely one-sided.

Perhaps the default terms were Smyte-friendly, but I can't imagine that some of these big companies (with savvy lawyers) would sign an agreement with a change-of-control provision that would permit this.

There might be a carveout for consequential damages (which would be huge here), but this seems like it will trigger a number of sizable lawsuits against Smyte's new owner.

Does anyone think they companies that just got the shaft have legal recourse?

If nothing than just getting a pound of flesh from twitter.

Incredible. Shutting of API access to paying customers after a 30 minute heads up at 6AM.

Astoundingly unprofessional.

"trust and safety as a service"

So... what was the Service Level Agreement? and the Terms Of Service?

People get all hyped by -aaS stuff, but services do go down, and if you aren't the one controlling when and how they will go up, then you better have a contract with the Service Provider on some terms you like.

Even then, prepare for when they go down, because they will sooner or later, for shorter or longer periods of time.

I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.

I find it silly to rely on some service, any external service

This is always trotted out and it's, by this point, a completely content-free thing to say. Short of removing yourself from the economy and society in general to live off potatoes grown in your own dung, you're going to rely on some external services and the fact that, for the most part, the parties you do business with will behave responsibly. When they behave very irresponsibly, unpleasant things happen and people quite reasonably complain.

Exactly. You want reliable CAPTCHAs? CDN service? Malware detection? Customer support software? Website uptime monitoring? Accounting software? Analytics? Email delivery? Good luck building and running all that yourself.

All of those can be bought and then self hosted except for CDN.

I don't know why you're getting downvoted because you're absolutely right. Perhaps most of the people in this thread haven't managed servers for more than a decade - because it used to be (and still is in some organisations) the norm to self-host half the stuff that was listed.

  * Malware detection?
Isn't that pretty must how most malware detection works? Sure they use the cloud to pull periodic updates but the software itself is self hosted

  * Customer support software?
I've lost count of the number of solutions available to self host. Both free and commercial.

  * Website uptime monitoring?
While there is a strong argument for hosting that in the cloud, there are umpteen solutions for website monitoring.

  * Accounting software?
I didn't even realise running accounting software as a service was now a thing!

  * Email delivery?
Fair enough, self hosting email can be a massive headache. Less so if you run Microsoft Exchange (one of the few things Windows actually makes easier than it's Linux / UNIX counterparts). You can even go for a hybrid approach and self host the mail server but have a mail proxy between your server and the outside world - so you benefit from spam detection as a service while still hosting your own mail.

I'm not going to argue that one is better than the other when it comes to SaaS vs self hosting. But people in here seem to have very short memories when arguing that self-hosting isn't at all viable. In fact I have personally supported all of the above in self hosted versions in the last couple of years.

I don't think anyone is disputing that it's possible, but rather how cost-effective and time-efficient it is [1].

I also have the experience of managing many of these services, and I therefore know just how much more productive you can be when you don't need to do it yourself.

Of course there are trade-offs; business is nothing if not a game of tradeoffs. We're just talking about what is optimal.

[1] Not to mention how much more stable and secure; Self-hosting can involve huge security and reliability risks.

> We're just talking about what is optimal.

I don't believe what is optimal can be generalised like that. "Optimal" is going to be specific to the business in question. Which is why I'm a firm believer of leaving all the options on the table.

> Self-hosting can involve huge security and reliability risks.

In fairness so can the cloud. I personally don't see difference as 'risk' but more 'responsibility'. Some of the places I've worked have been PCI DSS and Gambling Commission audited so we had that responsibility already. Self hosting a few other resources didn't really add much extra in terms of our security responsibilities. But the case would be very different of lots of other different types of businesses.

I think we're agreed that one shouldn't be absolutist about these things and that different approaches will be optimal depending on the circumstances.

It's worth reminding ourselves that the root comment was an absolutist assertion about the folly of outsourcing.

So the discussion is easily resolved: don't be absolutist :)

> It's worth reminding ourselves that the root comment was an absolutist assertion about the folly of outsourcing.

Even that strikes me as a mischaracterization, or, at best, taking just one statement out of context.

It was an assertion about the folly of relinquishing control, not being prepared for that lack of control, and thereby subsequently suffering a prod outage.

To be fair a couple of those don't qualify as "if it goes down it would cause a prod outage".

hosted where? On a VPS, in a data center, or in your closet? All of these depend on third party services and good luck keeping them up and running as well as someone who makes just that ONE little thing their entire reason for existing.

> On a VPS, in a data center, or in your closet?

Yes, any of those sounds like a good enough plan B. You don't need it to be perfect, just good enough to avoid getting stranded in a pinch. Then you can start shopping around for someone who does that one little thing much better than you do. That way it no longer is a show stopper and rather just an optimization problem.

I don't agree with the parent either (I'd rather use a SaaS if it makes sense) its also not as easy as you make it sound.

For starters, most smallish companies I know of use housing for their servers. So, they actually do own the hard metal servers and have full access to them. Both over SSH and on the console for debugging purposes. The housing provider generally replaces HDDs in case of disk failures. naturally, you're still dependent on electricity and internet, but even that can be mitigated by spreading your servers across several locations.

All of that is obviously possible with AWS, GCP or other VPS hosts, but doing it yourself is not that much harder if you've got an experienced ops person that wants to bother with all of that.

but i repeat... Its often not worth the time investment to get that system properly set up. Just renting VMs or going straight into dockerized microservices is an increasingly better alternative

That definitely solves the building part.

but real-time promiscuity where you spread your users info everywhere and have no backup plan is much more FUN. (/s)

No. Whether it's potatoes or servers, having a single point of failure with no plan B in place, is silly.

Your local restaurant going out of business is no reason for you to starve to death, you just go to another one, or to a grocery store, or indeed grow your own food if you're in some post-apocalyptic world or got no money... but somehow a service provider going out of business is reason enough for your production to stop? No, just no.

You've constructed a straw man, by removing an important part of that sentence and indeed the rest of what you are replying to.

Indeed, it's spurred a discussion of an over-simplification of the issues.

I don't think anyone is suggesting never to outsource, or even that the answer to "build vs. buy" is always "build".

The point is not having a critical dependency on a single vendor.

I don't think anyone is even saying "never", either.

The question is one of risk.

The risk of your tax accountant "going down" (or being crooked or something) is sufficiently low, and the cost of making that redundant is sufficiently high, that I don't think anybody does that. This could even apply to general accounting software, though one might argue that's not critical in the same way and/or more easily replaceable.

There's a sibling sub-thread that essentially implies, if not outright states that there's no choice but to "buy" in "build vs. buy". Although it may be true that building is no longer a practical choice due to time/cost, under certain circustances, it by no means eliminates the risk if there's only one vendor.

It's also a decision that can end up having tremendous costs at scale, if not re-evaluated, especially with even a soft version of vendor lock-in, like AWS.

That's not what the comment I replied to said, though. It just said it's 'silly'. If they had said something specific about, let's say, npm and made some reasonable and knowledgable comment or even speculation about how in this particular case they'd mis-assessed risk, that would be a pretty interesting comment. As it is, it's just a clone of the same tedious comment we get in droves in every thread about some multi-party outage. They're generic and free of insight.

> It just said it's 'silly'.

you did not include "so much that if it goes down it would cause a prod outage" in your quote, which is why the strawman claim was put forth.

It doesn't really change the quality of the comment. There is nothing inherently and outright wrong with such a dependence, whether it causes an outage in prod or not. In some cases it matters (maybe you're making a nuclear reactor) in many, it's a completely sensible tradeoff. Reflexively saying 'zomg you depend on something else and this is bad' every single time something is affected by a problem in another thing is 100% uninteresting. Including or not including 'in prod' or 'in bed' doesn't make it any more interesting or clever. It's just lazy grousing.

> There is nothing inherently and outright wrong with such a dependence

This is yet another strawman. The original commenter found it "silly", which, I agree, was a poor choice of wording.

Perhaps "overly risky" would have been better. I don't know. I just take the most charitable reading, per the guidelines.

It also comes after an exhortation to have a contingency plan, so it's at least implied that the "silliness" isn't inherent merely to the dependence but to the dependence without such a plan.

> in many, it's a completely sensible tradeoff

Is the tradeoff actually considered, though? Or is there just an automatic "we're not a nuclear reactor" decision process?

> is 100% uninteresting

Were that true, I doubt there would be replies. This is not one of those traditionally emotional/political issues.

> Including or not including 'in prod'

I (again, charitably) read "in prod" as a metaphor for "business critical". I'm sure we can all come up with examples, even in Internet companies, where one does not necessarily mean the other (or vice-versa). In the instant case, it seems to apply, so it made sense as shorthand.

> It's just lazy grousing.

I would agree if there were no built-in suggestion on how to avoid the problem at all, but there were.

Just because the point has been made before doesn't make it any less valid, until it has been refuted.

This is yet another strawman.

Only if 'strawman' means 'thing I disagree with'. The comment says such a dependence is bad and how you have to prepare for it with SLAs or whatever. This isn't universally true at all.

I doubt there would be replies.

A lot of really boring, trite things generate piles of replies. That's why they should be avoided.

Just because the point has been made before doesn't make it any less valid

The stated goal of the forum is not 'an exhaustive, if repetitive collection of generically valid things'. Remember that time you wanted to print something and the thing just wouldn't print? That's bad and annoying. It's valid that it's bad and annoying. We probably don't need to talk about it in the general sense every time it happens, though.

> This is always trotted out and it's, by this point, a completely content-free thing to say.

Only if you pretend there are no companies that do in fact responsibly nail this stuff down. There's plenty. Your argument seems to be that nobody does it because it's hard.

Welcome to the world of third-party Twitter clients. Not content with screwing over businesses relying on Twitter's API, it seems, they have now set out to screw over any company relying on any other third-party API as well.

> I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.

There are pretty solid SLA's for that. We provide some services as kind-of-saas but the SLAs are watertight.

Usually in those kind of sla's you can't sell the company (or move assets) without guarantee of delivery of the contract. In some cases they (the contract holders) even own you if you screw up and go under.

> I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.

I'm curious; what would you have these companies do in such a scenario? If 100% uptime is impossible then what you seem to be suggesting is that people should never use third party services or if they do use them they must not be in a critical path.

Are you suggesting everyone should do everything in house?

Use free (FLOSS) software.

If you have to use non-free software, buy software, not services.

If you have to buy services, always have a well-tested fallback path for when that service is not available.

Care to point out the FLOSS equivalent of what Smyte did?

You missed two thirds of my comment.

This was in order of preference. If you can, use free software that you can fix and support yourself if need be. If not, buy the software and run it yourself, so you can at least control its operation. The last option is to by SaaS, and if you do, you need to have contingency plans in case it goes away. You should take this risk seriously, and factor it in to your decision when buying SaaS, and invest the engineering resources to make sure you aren't overly dependent on it.

For any critical service, they should have an alternate provider ready at all times so that they could instantly switch to it. Whether it's another external provider or in house, is not really relevant, although having a bare bones in house alternative for the critical parts is a good idea.

What I'm suggesting is that people should use external services for their lower costs, better performance, or extra features, not to 100% depend on any of them.

No. Smyte wasn't so business critical that they should consider a failure in the service a reason to stop the site. Netflix had an example where if thier authentication service was down, they didn't stop people from watching videos.

There is a popular C# package called Polly that has all sorts of fault tolerant strategies.


In this case, use a Fallback strategy that responds to an API failure with some type of generic success response.

>Clients had multi-year contracts in some cases.

I'm calling it now: B2B startups will soon have to sign poison pill contracts that specifically detail what happens if they're bought out. This might ruin the "buy them for their people and discard the business" method of hiring - or at least it will if the customers have any sense.

I'm baffled if this wasn't already the case. Surely the entire point of signing a long-term contract (as opposed to just paying by the 'unit' used) is to protect against exactly this kind of outcome? I know it doesn't always happen, but it seems like most rapid "hire and shutter" moves hit either consumer-facing services or B2B stuff like analytics that isn't production critical.

I had assumed Twitter was going to be paying out a bunch of breach-of-contract penalties, because this is basically what contracts are for. Is there a standard way around this?

I don't know how it works in Silicon valley, but finance companies I have worked for have pretty much always required that source code be turned over if the vendor folds when doing business with small vendors.

Yes I worked for the company doing the .coop registry and ICANN had very strict rules about code escrow we had to follow.

It's amateur hour if your contracts do not already contain such clauses. For anything business critical that's the norm. Change of control clauses are super common and tend to trigger penalties, escrow arrangements, contract voidance and other niceties.

I feel like there must be more to this story. Maybe they immediately saw security problems, or legal issues. Not saying it's going to exonerate Twitter (maybe the truth is even worse than it sounds), but this is such an obvious PR mess that I can't see them doing it for no reason.

The @HelloSmite Twitter account stopped abruptly on June 4th after daily posts in May.

I postulate more than few roles were halted at that time.

That they didn't notice during due diligence?

Perhaps they still wanted the company for some reason? Someone knows. Time will tell most likely.

YC W15, I wonder if this will have any impact on the ecosystem of startups like Smyte getting established YC companies as early adopters.

Also I wonder how many customers are staying quiet, because "our fraud protection provider just stopped working" isn't something they want to reveal.

This is incredibly fragile ecosystem if one obscure API getting shutdown causes so much damage. This could be prevented by a simple periodic check that would determine if API is available and some fail-safe alternative to remain online. Instead this is like a house of cards that breaks with any of the cards fold. I'm not excusing the abrupt shutdown, its obviously a wrong way to end service, but being prepared for it is much better.

This is a hard call when the system is question is your anti-abuse provider. If you "fail open" when they're down, you risk allowing a flood of bad users during outages. Depending on your business, that may be much worse than having an outage yourself.

(Disclaimer: I work for a smyte competitor).

You can turn on something like manual verification of new users, an alternative security service, or just temporary halt new account registration. All of which don't result in system wide failure. Even a few abusive users is a small and temporary cost to absorb vs complete outage.

Ah, yes, just the casual manual verification of checks notes a fire hose of data.

> just temporary halt new account registration

I'm confused; how do you know these companies did not do that? In my experience even small companies will typically abstract away a third party service so that if it does fail it fails in an expected way.

What you seem to be suggesting is that all of the companies failed in expected ways. I didn't see that in any of the articles. Did you have a source? I'm curious if it hit some harder than others.

>just temporary halt new account registration.

That's an outage...

>manual verification of new users

So is that, unless you have tons of customer support reps sitting around doing nothing when shit hits the fan.

I can see cases where you want an anti-abuse system to "fail closed".

What you're suggesting is something for large companies. Most of these companies affected probably don't have time and resources to develop and pay for alternatives to every single third party service they require. And why would they? They have contracts you'd expect to be honored.

Seems unrealistic especially since the majority of times when these services go down it's likely just a short time so they probably have abstracted around them to avoid unexpected failure and just have routine, down time failures.

When I work on systems, I ask, "how does it scale and how does it fail?" Everything I work on considers handling of failure cases, especially 3rd party APIs. I'm surprised by how many people assume that the network will just work and their provider will be available.

I'm surprised that npm was affected so much since they provide service to so many people.

I think everyone is missing that this is the canary in the coalmine for how the tech giants are going to behave going forward. Hell, Twitter isn't even that big!

Every tech company big enough probably sees the same writing on the wall the rest of us do that it's time to press their advantages. That means shutting everyone else out.

We probably all have mission-critical parts of our infrastructure that aren't core competencies. We probably can't afford for them to go away tomorrow, but now it's time to expect and plan for it.

Get ready to ride the consolidation wave.

but why? If they have contracts with these external companies, and revenue, why shut it down. Even if it isn't for a huge amount of $.

The goodwill itself has to be worth a TINY bit.

Goodwill works for good companies proud of what they do.

Companies like Twitter whose core business is to shit on their users and customers don’t care about goodwill.

While this should be upsetting, and I'm sure it was to the people who were affected, I'm actually happy to see it. I'm of the mindset that this will have an overall positive effect on the state of the Web. Why? Because users and devs both need to remember that companies are not friends, and contracts are only as enforceable as the courts will allow.

Imagine if Google decided that tomorrow at 6AM PT they were pulling the plug on YouTube embedding, or GMail / GDrive integrations. The court's going to say, "They put in the contract that they can change the terms of service at any time, and they're not liable for outages or lack of service. And you signed it. Sucks for you."

Then again, I'm just anti-corp in general and this gives me better reason to avoid Twitter.

There's a difference between Google turning off a feature and Google cutting off a whole service. This is the kind of thing negotiated with contracts. Lawyers on both sides. If you build your business around another company, you protect yourself. Twitter with surely get sued for this, if for no reason other than failure to provide a service that they promised to customers in writing. Thankfully, your flavor of cynicism generally does not apply in situations like this where many tens (or hundreds) of thousands of dollars are at stake.

> Thankfully, your flavor of cynicism generally does not apply in situations like this where many tens (or hundreds) of thousands of dollars are at stake.

This will be how it turns out for a few big players with deep enough pockets. Surely though this is hurting somebody's business that can't afford the lawsuit -- potentially to the point of putting them out of business. That's good for Twitter (and especially Twitter, as precarious as they are).

We've all joked for years about how FAANG pay their engineers so much mostly to keep other companies from being able to hire great engineers. We've gotten the big three slapped for colluding against their employees on wages. Why do you think they would engage in some anti-competitive business practices and not others?

Couldn't you file a amicus curiae, advice the court and state the company has harmed multiple people which are unable to defend due to cost, wait for the case to settle with the big guys and then sue with reference to the original case?

Not a lawyer, just interested.

Google does pull the plug on things, but generally give the customers a little time. Google gave 6 months notice for its ITA travel data.


I'm not sure how good apple is, but when they buy software available on Windows that version usually gets killed. (See Logic music production, killed 3 months after purchasing, but presumably the old installs still worked.).

Even Halo was demoed on mac before MS bought it (The mac version never came out)

> Because users and devs both need to remember that companies are not friends, and contracts are only as enforceable as the courts will allow.

I bet their term of services mentioned that they can cease operations on no notice.

Doubtful. For most long-term corporate contracts there is no "we might cease operations at any point and leave you hanging before end of the contract".

No company lawyer I know would willingly put their signature on such a contract unless the other side is using a free tier. If someone actually signed 3 year contract with such a clause, congrats, you managed to throw a lot of money into a black hole.

It'd be good if some customers share the terms of the contract. That'll resolve a part of the discussion.

For a free product, sure. But when you're signing a multi-year contract where money is changing hands, no way in hell would any customer accept that kind of clause in the contract.

What other unpleasant things would you wish on people as a way to educate the public about the realities of contract law? Your internet service just deciding to cut you off with no notice? Water? Electricity? Ambulance or firefighters not showing up when you need them? That might help cull the hopelessly naive from the herd and have an overall positive effect on the state of the Web.

I’m aware of your sarcasm but will reply at face value:

1) If your business depends on Internet access you shouldn’t pay for a residential plan. You need an SLA.

2) Natural disasters happen. Not having emergency drinking water is very naïve.

3) If your business depends on electricity you should also get business rate with SLA and maybe have a backup generator or two. Heck, I have a UPS just to not lose the 15 minutes of work that’s not automatically backed up.

4) Yes, get a weapon (preferably a gun) and at least know enough first-aid to be able to stop a bleeding. Keep an aspirin on hand in case of a heart attack. Know your emergency exits and evacuation plan. Make sure you have the mindset to leave all your possessions behind in a burning inferno.

All of the above used to be common sense before technology made us think we are immortal and entertainment distracted us from real world.

I'm not sure that's a reply to something I said. The poster thinks a crappy thing happening to someone is good for the web because, I dunno, it's some sort of teachable moment. That's pretty silly, to put it politely. I'm not wishing you an earthquake so that you'll learn basic disaster preparedness. And if you did, fates forbid, end up in an earthquake and were short on water, I don't think I or any sane person is going to be happy about it.

I wouldn't be happy about any loss of life, you're right. However, has the Smyte situation led to anyone coming to physical harm or put in mortal danger? If it has, I'm sorry, I didn't see anything about that.

What I've seen was a select group of companies and their users who've been inconvenienced by the abrupt shutdown of a service; and whose unpleasant experience should be used to teach every user - this can happen to you. I would not be at all happy if, say, the servers for medical tracking software used in hospitals around the world went down the same way; or if suddenly air traffic control software that every airport uses goes offline.

This particular situation doesn't cause potential physical harm to a human being. It's not actually an emergency situation. That, to me, is a major difference.

What if it’s a little earthquake? It will make you think twice about building your home on a bad foundation.

I think the issue here, other than earthquakes being potentially very deadly, which makes this comparison a little absurd, is that they are in a sense unavoidable.

On the other hand you can avoid using SaaS. For example, there’s enough market demand for Github to provide on-premises secure hosting. Clearly then, if SaaS as a whole is seen as fundamentally unreliable it will create more market demand for similar on-premises, or open source, or otherwise “buy not rent” model services, which I see as the motivation for OP’s comment.

I mean, we can stretch these analogies like pieces of tasteless gum all day. I don't think being happy at the misfortune of others is a particularly constrictive attitude nor do I think Twitter buying and then apparently shutting down some service with 20 minutes notice is good for the web, teaches anyone anything or is in any way a desirable outcome. I don't understand how any reasonable person would.

I don't think I or any sane person is going to be happy about it.

You don't spend a lot of time on Facebook, do you?

If you're looking for a Smyte replacement, check out Koko. It's AI for content moderation and classification, developed at the MIT MediaLa.


Another SaaS that may suddenly stop functioning tomorrow.

That's really amazing - I am having great difficulty imagining a situation in which doing this would make sense. Just for starters, even if you completely discount goodwill, ethics or even PR, I'm guessing Twitter just bought a bunch of lawsuits. This just seems nuts.

Twitter seem to have an institutional blindess towards the importance of APIs. Their execs need to be sat down and have it drummed into them that breeding this much bad “karma” (for lack of a better term) is going to come back and bite them in the ass.

> Twitter seem to have an institutional blindness towards the importance of APIs.

Which is especially weird for Twitter. Their official iOS client was a third-party API client they bought. Third-parties created @replies, #hashtags, etc. If anyone should recognize the important of third-parties using APIs, it should be Twitter... yet they've been probably the most abusive of the large APIs.

This is the same company with NIH Syndrome severe enough to in-house their own woefully-under-performing message queue. Twice. And they blamed Rails after the first go of it.

I'm not sure they know what interop means.

Yup. They started the "rails don't scale" narrative.

Something something, a good craftsman doesn't blame his tools.


Twitter will never change how they do business, replacement is the best strategy.

This is not an appropriate place for your Mastodon spam.

Seems like it is, considering its a twitter replacement and the op is kinda anti-twitter

TFA has nothing to do with Twitter the product, which is what Mastodon purports to replace.

Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact