Run a database and a UI to put and show things from it: 0 engineers to run after installation. A full stack hacker can build it.
Run the same thing but for number of people larger than a single server can handle: A an engineer or two for bug catching and resource allocations and a few engineers to build it using software tools for caching and load balancing.
The same thing but tens of millions around the globe: A few engineers to run the thing, A few to keep an eye on the network around the globe and access quality, a few to mitigate hostile behaviour. To build the thing you need a few people with knowledge on distributed systems and specialised people on 3rd party products for CDN and its management. Of course you need the people to build the thing and its UI too.
Oh, so you want to make money from it? You need people to build tools which will interface with your thing to deliver things like ads and special functions. Now you need to replicate all of the previous stuff but this time for your money making software.
Europe says data collection should be like X, UK says should be like Y and the USA should be like Z. Now the people who built the thing can't go home and leave it to the maintainers because they need to modify how things work all the time.
Israel wants to know some info about someone, Turkey wants the some specific data collected and some tweets must not be visible in Turkey, Saudis say they don't want some type of tweets to exist etc. Now you need people to build these things when still don't mess up with the EU, UK, USA practices. Once again you replicate a lot of people but with special duties for special requirements.
Oh crap, some pedos are posting very illegal stuff and others are running massive spam or political opinion manufacturing operations and everyone blames you. You need to create tools to interface with your product to tackle these things. Once again a lot of engineers are needed because you don't simply run an SQL query from your local database, so you need people who understand how to make the thing work on your scale.
Something can no longer scale and you need an alternative solution, you build that alternative solution with a team of engineers but the old stuff can't just go away because some other systems depend on it and it will take years to change their thing so maybe you simply keep the old stuff around together with their engineers?
Before you know it, hundreds or even thousands of people are working on it. Elon firing content safety team is probably about untangling the mess and not about love for sexual predators or fascist or something.
I don't know about the reality of Twitter but having seen how sausage is made in some large companies, I will speculate that it's a build up over time as the nature of the business evolved. Knowing how things work today it's probably possible to simplify it with a complete re-write but how many re-writes go well?
The cost factor of international complications along with the impossibility of meeting every country's speech restrictions simultaneously may be why Elon is reducing Twitter's offshore presence dramatically. It could easily become a US-only platform. Which may be the only way for it to survive financially while meeting free speech goals.
Exactly. It's remarkable that people on HackerNews, of all places, output drivel like "it's just a 140-character messaging app". Twitter probably has dozens of office-facing apps to maintain before you even get to the massively-scaled user-facing system.
This trite argument made some sense when there were 8000 engineers, but when the 4th most popular social media platform has only 550 engineers working on it, you don’t need to dig deep to figure out how so many engineers are needed.
Even the famously nimble WhatsApp which has fewer features and complexity needs this order of magnitude engineers so why would you expect Twitter to be leaner?
WhatsApp also had no need for moderation or ads handling (IIRC, they used small yearly payments plus contracts with network operators).
Just handling the ads, both backend and user interfaces for buyers, would quickly net you several more developers.
It's easy to have only 45 engineers when you constrain the scope of what they have to do very well, but the scope was much, much, much smaller than Twitter's.
Yes, 55 employees iirc. I don't think it was 'surprisingly small', though. It's just that cash-rich startups have had the habit of hiring as much as they could instead of hiring the minimum required, which is actually the sensible way to manage a company.
I imagine the team responsible for maintaining the direct messaging feature of twitter is also around 50 or so people.
In case you haven't noticed, twitter has many, many, many other features beyond direct messaging, each of which requires its own team to maintain, and many of which are far more complex than a messaging client.
Still; numbers. I do not use Twitter, but the web app is relatively simple; I am not hearing much good about the mobile apps. So we got what; maybe 30 people working on the frontends?
I do not think Twitter had ever 8000 engineers - my understanding is that there were that many employees. It's easy to imagine a lot of non-tech roles, b2b sales, management, support etc.
WhatsApp always comes up in comparison to Twitter, but they are fundamentally different. WhatsApp is mostly one on one interaction, while Twitter potentially allows every user to interact with every other user, it's unbelievably more complex to do that at scale compared to WhatsApp, not even talking about all the extra features Twitter has. Search probably being best example of complexity differences.
Out of curiosity what makes e2e encryption impact ordering guarantees? With feed things like ranking (bunch of ml pipelines) etc make it pretty stochastic?
On Whatsapp the ordering guarantees are limited to within the thread and server side timestamps are good enough for the most part to enforce it. Whatsapp/messaging is a very horizontally scalable problem. With feed (assuming) you have a fully personal feed the storage alone is a pain to manage and optimize.
i'd say key exchange. You need to get key information and message in the correct order orelse you won't be able to decode the info.
Now of course you can reimplement an ordering mechanism on top of an unordered one, with lost messages detection etc on the client side, and let the server be loose.
but that means reimplementing that logic on every client codebase.
Twitter has a direct messaging feature. It also has many, many other features. Whatsapp is direct messaging, full stop. The two are not at all comparable.
Dunno, ever since Musk started purges and especially after infamous "I'll switch off a bunch of servers" thing, I observe more and more bugs on Twitter.
Mind you, I already had searches that triggered bugs in mobile app where first search would always fail, but things like significant drop in tweets I see in my feed from people I follow (and no, none of them were bots), random outages like Lists or the whole search feature just failing, pervasive feeling that even searches return maybe 1/10th of what I was getting before...
I'll continue to take whatever Musk says with a grain of sand.
Thank you.
For real though: He has proven time again that he lies on trivial matters.
Regrettably my trust in him telling the truth is at rock bottom now. It used to be at the same level as other marketing departments/generic politicians but dropped over time.
"Lies" implies deliberate rather than in error. This is much harder to demonstrate even when it is correct.
In either case, still a good reason to take with a grain of salt, though one should do that for everyone anyway unless they're talking about their specialist subject.
Elon didn't have a verbal oopsies when he said that an emergency worker was a pedophile during a deadly rescue event. He didn't have an accident when he doubled and then tripled-down on the claim, by further clarifying that that by pedo he meant sex with young Thai boys. Elon didn't accidentally hire an investigator to dig up dirt on the rescue worker.
When Elon very recently reassured stockholders that he wouldn't sell, he didn't have an accident when he had two selloff events worth billions. It's no surprise that someone would lie for billions, but Elon will risk his reputation by lying just to destroy a rescue worker for offending him over social media. Lying is not merely some accidental or defensive maneuver that Elon employs when his back is against the wall; dishonesty is very much part of his tactical repertoire for public speech.
I'm willing to trust him more over time (as I would most other people) but he's got to earn it. E.g. Bill Gates managed to do that to a degree.
Not that the (sometimes) richest person in the world cares, but he has squandered any up front good will in unforced lie after unforced lie over trivial matters. I've come to the conclusion that he's a particular self centered person (including his interest of the day) with a strong tendency toward pettiness and a severe lack of skill to shrug things of.
Since he's still rich and I don't have to deal with him outside the internet we'll both be fine I think.
WhatsApp is a very different product: it’s built around one to one communication so you don’t get all of the fan out problems of having to distribute messages everyone’s followers, or the search / advertising / timeline filtering. Consider that there are single people who can post something which will be watched within seconds by millions of people, except those who have blocked that person or filtered that topic and ask how hard that looks behind the scenes.
It's a tip of the iceberg. The part you see - the UI - is a small part, the complexity is in the backend (advertising/marketing tools, content moderation, metrics, devops etc).
Musk being the archetypal Mr First Principles Guy, he probably thought just the same: "it's a simple webapp, how hard can it be?"
Probably exactly what both Elon and George Hotz thought, thankfully these people saying that don’t have 50 billion to spare to bet on that unsubstantiated argument.
Yet months later, Twitter is still running despite deranged predictions [0] of its immediate collapse and clearly the other 7,000 employees were just not needed.
Again, the media really knew how to fool and create pure hysteria and manipulation to get almost everyone believing everything they write.
The media definitely knew they are exaggerating nonsense, but they wanted to convince people and advertisers to leave Twitter, and all this scaremongering helped.
This is news because later this year they are probably going to have problems. You have obviously never worked at a large internet service, but when you spend some time backstage at such a venture you learn that there is a great deal of inertia within various projects and pieces of infrastructure. If 80-90% of the team for component X departs the system is unlikely to fail quickly (a frequently repeated mistaken belief when Twitter was shedding engineers so fast) but it is going to start having problems that will not be visible to the outside world. This is not a problem for a while because the former employees who built and maintained the system were fairly smart people and built something that was relatively reliable; they wanted to be able to sleep through the night without a page or go on vacation too.
Bugs exposed by other teams using that component will go unfixed and will need to be patched around by the caller. New features that might have used component X will be sidelined because no one wants to put more load on it or risk their feature on a component that people have started whispering about in the corridors. What is worse is that the people frantically spending their days patching and repairing this Titanic below decks do not see the iceberg and do not have enough manpower to turn the ship even if they did, the people who could help give warning or helped prevent the problem left a while ago. Eventually this component is going to get stressed, some lingering bug will get triggered, or some other system elsewhere in the architecture diagram will fail and shed load to our component X that is barely keeping up and it too will fail. The danger for Twitter is that the failure could cascade through multiple poorly-maintained systems and lead to a much bigger problem.
You should visit big telcos. Things chug along not because there is a great competent army fixing anything. Institutional knowledge is regularly lost thanks to changes in tech, ever changing fortunes of the biz, divisions getting sold off etc etc. They chug along because people make do with what ever works. It sort of like your OS crashing. You keep backups, reload, reboot. You dont debug and fix. The older systems get that's pretty much the norm.
The app staying up is a testament to the robust engineering that the team did in previous years. If it'd fallen over immediately that would be a sign that the engineers did a bad job.
The fact Twitter hasn't died shows that Elon fired a good engineering team.
Yeah, thinking about this as an extended vacation seems right to me. Things don't usually go down over the weekend or winter break. That doesn't mean "hey that means we only need 0 employees".
Certainly you can make a microblog service with less than 500 engineers, but there is a lot of work invested in their datacenters (from physically managing the machines to their network / orchestration control plane). That can even be outsourced as many companies do, but it ain't free.
the "story" that the employees were all terrible and the code a disaster would have been proven if the site crashed day 1
I am actually a fan of cycles where "no new features" is a priority, making systems less fragile and tollerate the weekend / holiday helps you sleep at night, and sleeping at night is nice.
550 engineers to maintain and develop the code base is still a rather large number. Is it too few? I don't know. There is little doubt that Twitter was bloated, the question is by how much.
Ok I want you to write a system that can write hundreds of thousands of messages per second, and read millions of messages per second, globally and with low latency, with every message being permanently stored and available for query. Till need you design and implement redundancy and fault tolerance. You’ll need a strategy for backing up everything. You’ll need to be able to deploy code updates to your global infrastructure which is self managed because running this in a cloud would be ridiculously expensive. You’ll need video ingestion systems, monitoring and alerting. You’ll need advertising systems, billing systems, content moderation systems, anti spam and anti bot systems. You’ll need to implement an API for developers. You’ll need applications for mobile and web. You’ll need developers to do this, testers to test it, people to organize all this stuff.
To think you necessarily need thousands of them, though, is a stretch.
I think people have drunk the kool aid over the past decade of 'growth-driven' startup where the game was to hire as many engineers as possible as fast as possible. This is what is partially fueling the current job reduction cycle in the industry. There are simply more engineers on the payroll than needed to deliver a profitable product.
It's not going to disappear overnight, but you can see the cracks. The 2fa went down in multiple countries for hours, Oz and NZ lost access for almost a day, people reported "looping" timelines of the same 7 tweets, notification issues, and some other things I already forgot. Sure, they're up, but things are not 100% ok. The number of broken things is not even close to what we've experienced in the last few years - this was an extremely reliable service before.
it's pretty funny how stupid rich people like elon musk try to convince the poor that "we can do more with less engineers". "hey slave, you have to do more work so we don't have thousands of employees. i don't want to pay thousands of employees, you understand slave?"
