Speaking from very painful, personal experience, few things are more agitating than being forced to execute on something you fully know is a horrible idea, especially when you tried and failed to communicate this fact to the individual pushing you to go against your best judgement.
Even more so when that person later loudly proclaims that they never made such a request, even when provided with written proof.
I can of course not say whether the people currently working at Twitter did warn that the recent measures could have such major side effects, but I would not be surprised in the slightest, considering their leadership's mode of operation.
Even as someone who very much detests what Twitter has become over the last few months and in fact did not like Twitter before the acquisition, partly due to short format making nuance impossible, but mostly for the effect Tweets easy embeddability had on reporting (3 Tweets from random people should not serve as the main basis for an article in my opinion), I must say, I feel very sorry for the people forced to work at that company under that management.
The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at least one of them was fast.
I worked in the games industry for a while, and came to understand how they could spend so much money and so much time, and yet release a game where even basic functionality was broken. It's exactly this sort of extreme schedule pressure that, ironically, makes a huge morass where changing one thing breaks 10 other things, so progress grinds to a halt.
>The people at Twitter who understood the system and could predict the side effects were all fired or left
Not necessarily. I’ve predicted bad outcomes for decisions in a few cases and been ignored but stuck around regardless. Mostly because I like my job and the goals of my organization even if it makes bad decisions.
Of course to remain productive and improve my influence in future decision making it is absolutely critical that when predictions come true, I do not go anywhere near an “I told you so mentality.”
Instead I do what I can to clean up the mess with a “how can I help?” attitude. And increasingly over time people take my opinions and analysis much much more seriously.
I wouldn’t say that’s the path everyone should take, especially because some work environments are just too toxic for any progress at all (I ran away, fast, from two jobs like that). And some people cherish having an entirely new type of challenge every few years instead of shepherding something through longer periods of time. All valid paths.
People didn’t leave because of the bad decisions. They left because Musk said he wanted “hardcore” people who’d work 80 hour weeks. Turns out masochism isn’t correlated with great engineering.
I think there may be a disconnect between what Musk says and the reality on the ground. Enough so that I, given such a situation, would wait around a little bit to see how things played out before jumping ship. But as I said in my original comment, I’ve sprinted away from toxicity before, and will do so again if it becomes obvious.
If it was only that Elon had said it, I might agree. But he also had multiple rounds of layoffs (after lying every time and saying "this was the last!"). He had multiple horrible mandatory meetings forcing people to drop everything and fly over the country. He forced people to print out their code so he could review their work.
If you really want I can qualify my earlier statement, I thought it was obvious:
GP wrote:
> Enough so that I, given such a situation, would wait around a little bit to see how things played out before jumping ship.
That means "nobody" is talking about the group of people who have the option of "waiting around a little bit to see how things played out before jumping ship". H1B visa holders aren't included in that earlier group, so I didn't think a further qualification is necessary.
> I think there may be a disconnect between what Musk says and the reality on the ground.
Which runs against every good piece of advice that has ever been uttered about leadership. Musk far overpaid for twitter because he wanted to be the center of attention and what better way to do that than to buy the network which gets the most attention from "important" people?
He then took the Michael Jordan trope of "I never asked anyone to do anything I was unwilling to do" and tried to turn that into reality by sleeping in his office every once in a while. The problem with this sentiment is that the only employees who are going to stick around long-term in such a ridiculous working arrangement are those who either can't find jobs elsewhere or are terrified that they won't be able to find jobs elsewhere.
So now you've got a highly toxic work environment full of people who are unconfident in their own abilities to get the work done, and Elon constantly pretend like he's some sort of business genius from the movies who just walks into a meeting, throws a bunch of turds on top of the agenda without having a firm grasp of anything, and storms off to light the next fire.
> Enough so that I, given such a situation, would wait around a little bit to see how things played out before jumping ship.
The twitter engineers were presented with an opportunity to jump ship and also get 3 months of severance. I think the rational ones, who had a choice took it, leaving employees who didn't consider it rationally, as well as employees on H1Bs who didn't have the luxury to quit without something else lined up
Individually yes. It's less clear what happens if Twitter is shedding hundreds of engineers at the same time many other companies are freezing hiring or letting people go.
The parent is technically correct, you're not deported for losing your job on a H1-B. Your permission to remain in the US ends, and presuming that you leave by the deadline, you are not deported.
Deportation is a different legal event. It's a forceful expulsion which occurs because you did something seriously negative like break the law. Deportations are a big deal and a bad thing to have on your record in any country when it comes to your future prospects with that country.
To use a super rough analogy it's kind of like an honorable vs dishonorable discharge from the military.
> Mostly because I like my job and the goals of my organization even if it makes bad decisions.
Keep in mind that Musk intentionally turned Twitter completely upside down. Anything that people there liked about it before Musk is likely gone—coworkers, WFH, perks.
Tell me you're in your 4th year at a BigCo without telling me you're in your 4th year at a BigCo.
The goals of the organization are mostly a facade. The people running the organization, and their actions, are what the goals of the organization actually are.
No, I’m not at BigCo, at least not in anything at all close to the scale of of FAANG.
But I do work in an industry where even the C-level people usually (not always) have at least a little interest in truly pursuing mission <X>
I know this for a fact because even though I am not at all C-level or even the manager of a large team, I often have a seat of the table in the meetings where such people come together. Those meeting can be ugly, they can reveal how the sausage is made, to borrow that analogy. And I’ve seen how many (not all) truly are trying to get <X> done but doing so may require a bit of ugly sausage making to get there.
And I’m not a wide eyed 4th year either. I’m a grizzled and usually cynical veteran in my field. My job is often to put out fires, or produce analytical tools or output of strategic importance, and also to sometimes to plug a major gap in operational capabilities. I’m not really a manger but I’ve earned a seat at the table when the highest people get together as well as when they interface with counterparts at other organizations.
Don’t take that that to mean too much though: I may have a voice, but it is by far, very far, the smallest voice in the room.
> The people running the organization, and their actions, are what the goals of the organization actually are.
Obviously. Are you suggesting that one can’t appreciate those? Or that this is some secret? Maybe the communication in the orgs you’ve been with has been poor?
The organization’s stated mission, inherent not just to it but to all places of its sort, is <X>, even though, unfortunately, we often end up chasing <Y> instead.
It’s a difficult balance: <Y> is in fact necessary to continually achieve <X> but there are times where decisions focus exclusively, or at least too much, on <Y>. But we live in the real world, and sometimes that’s necessary. (<Y> is not money though it has an impact on our financials) And also sometime people with a broader view see further than I do and those choices that seem wrong come around a few years later and it turns out <X> is actually better off for it. It keeps me humble, skeptical of my own certainty even when it seems faultless.
But I’m also at a point now where people who pop up and start shouting <r>! or <f>! or something completely random like “Well how about <~€€€~>?” I can easily deal with: I go back, do a bit of the work I do, show it to the right people, and those shouts -disappear. Sometimes one gets through and it’s annoying, but whatever, nothing is perfects.
Of course the above vastly oversimplifies things. There are many, many more variables to juggle along the way. But I hope it gives a reasonable sense of things.
And I’m sure the “<X>” style notation of things in my explanation makes it harder understand what I mean, but I value my privacy, hence the abstractions of the factors involved.
I guess the way you abstracted will probably reduce your privacy (a tiny bit). At least I have not seen anybody write "<X>" instead of just X in this usecase.
So either this is very specific to you, or is very common in your circles so you do it too. Both of which reduce the number of potential candidate if somebody tries to doxx you.
Additional you (or your editor) uses “” over " which also reduces the number of candidates.
(Not trying to attack you here, just thought it was notable)
<*> was my own ad hoc convention in the moment. My
“.” style is the default for the mobile phone I’m on, which is the US, and I don’t mind sharing that since I’ve mentioned it in plenty of prior posts. But I do honesty appreciate the analysis, those aren’t things I’d specifically considered when posting now or in the past, and it’s always useful to know what subtle signals can be picked up in that sort of detail. Heck someone observant could probably infer broad geo region (time zone) just from the fact that I am making these comments at this time.
Then you'll probably appreciate the following info. You have commented excessively on HN over the years. If you have done any publicly accessible writing with your real name attached, then it is quite easy to find out who you are by correlating writing styles. (If you don't have done so publicly then at least your employer will be able to do that using all internal writing on one hand and all your HN contributions on the other.)
Sounds far fetched, but it's really not that hard. Quite recently somebody hacked this for correlating HN accounts with each other and found alt accounts of people with high accuracy. Which people confirmed. And that wasn't even a serious attempt, just a little hack on a sunday night.
In a sense, it's all too late now since all your writing is already out there. But could be good to know for the future.
All true, but that’s a level personal detail I’m prepared to live with having it out in the world. There’s a limit to how much mental energy I’m prepared to put into worrying about doxing, and it doesn’t go this far. But it is a remarkable aspect of just how much we can be fingerprinted by so many different things we do in life.
One option is to change HN accounts often. Every few posts you need to make a new one. Successfully correlating writing styles needs some data and if there are only a few sentences to go off then that's not enough.
I know that the site guidelines discourage that, but what can you do.
I guess you can try to develop two very distinct writing styles.
Or, as I do, consider everything posted to HN to be linked to me. My handle is actually an abbreviation of my full name. I consider HN to be "professional" correspondence.
Writing style is also a lot more individual than people recognize.
Iirc word histograms almost uniquely identify authors. Of course this is on larger amounts of text, but I guess you could identify users over seperate platforms this way.
E.g. Intend to use ellipsis (...) to separate thoughts in online conversation a lot. But I try to not do that in reddit, where I try to stay somewhat anonymous.
Still, I assume that it would be possible to correlate my reddit and HN account just by comparing the word histograms (ie which words I use and how often).
Yes, some of my academic work in comp-ling (massively outdated by today’s advances) explored things like the perplexity scores of different Shakespeare plays to explore the controversial claim that some the work attributed to him was actually done by Marlowe.
(As a complete aside, that program of study also included Forensic Linguistics which truly fascinating. And of course the work of Claude Shannon and information theory, though not in any great depth)
I can share a post-mortem in a non-confrontational way and have a discussion about it. Even if it often results in an “Oh well, I guess it won’t work”, sometimes it’s a highly productive conversation on how things could have been done better, and the person or team goes off to execute some new variation to great success. And sometimes the variation fails as well, but such is life. I try to engage with people in a way where even if failure is highly likely, we go in eyes open knowing the risk is worth it. Sometime we’re wrong (or I’m wrong, it’s my fault) but such is life. Not all shiny ideas keep their shine once they’re unpacked into an attempted implementation.
Avoiding the mentality isn't the same as referring to the past. With most people, It's perfectly fine to bring up the past in a matter of fact and dispassionate way, so long as it's constructive and not meant to shove "see I was right" in their faces.
You might be right; the person you're replying to might have zero idea.
Now me... I know someone personally who was a senior exec for Twitter's software team, who left after Elon's purge.
He left because all the people who understood the system and could predict the side effects were fired or left. He'd been with companies going through death spirals before, and had no interest in being involved with another one.
So, while the person you're replying to might not know, my friend DOES know.
About 80% of Twitter was laid off or quit. I think it's a reasonable supposition that a good number of those were critical personnel who felt they could get a better deal somewhere else.
This rhetoric is well past its peak. When the firings happened, people said twitter would crash in a week. It's been a long time since then, and twitter, for my very generic uses and purposes, has just gotten better.
Demonizing past hard decisions at every unrelated point of difficulty has to be the worst kind of toxicity there is.
There were some people who predicted that in the heat of the moment, but the rational people at the time were predicting that Twitter would limp along, gradually start to show cracks, and eventually become a husk of what it was as the systems slowly degraded. I'd say we're well on track for that prediction.
This isn't an unrelated difficulty—this kind of bug is the direct result of losing (or ignoring) the people who knew better. Institutional knowledge is a tech company's lifeblood, and Musk gleefully discarded most of Twitter's when he came in.
It's a big holiday weekend in America, and having an outage seems like a minimal inconvenience, especially for those of us without an account.
There are entire communities of people who relied on the ability to simply read Twitter without an account, took the time to write code of their own, and now are reacting with much more maturity than HN seems to be. The petty personal attacks are simply astonishing.
I really don't expect this to be permanent. For this very second though, I do get your point, there are quite a few services I only visit occasionally through links and even after making an account I was a lurker for the longest time on Twitter.
However talking in a way that takes the current critical temporary state as the default forever isn't very fair
It really depends I suppose. If you consider the accused to to be in an unearned place of power, it might feel like an acceptable thing to do (albeit still quite icky and pedesterian). But if the situation is very different and/or you are not sure if your PoV is fully justified, this can absolutely be the most toxic low-effort thing one can do when thrown at someone who's already fighting an uphill battle. Without going to specifics this is something I've experienced myself, and have also seen happen close to me in a very toxic calculated way. So these kinds of comments are IMHO overally very unconstructive.
On the community side, it now better reflects its mission as the public market square, where before it was so unbalanced to the left (also making what was to the right way more polarized) that it just wasn't very much fun or developing to stay on for long. Before I was only a lurker but now I sometimes chime in to tech threads without fearing someone will try to cancel me over a way I name a fruit or something.
On the tech side, it has retained everything that made it good (didn't implode!), and the tweet length / "show more" logic fits my style of writing perfectly. Spaces are also a kind of thing that I didn't use before but became immediately accessible as it was added right to twitter itself (and things like the 24h wagner coup space with 6M visitors isn't something I have seen in the past). And other simple things, like long videos sometimes fit a need, even while most of the time a youtube link also works.
Some things like crypto spam also seem to be in a bit better state, though can't obviously ever be completely removed
It's interesting to me how deeply politics gets into everything. You like Twitter more because "it was unbalanced to the left" and now reflects your political views better. My experience is that the first 100 replies to every semi-political post are now right-leaning blue checkmark holders and conspiracy theorists, and now I find it "not very much fun".
Honestly, I don't think the "public market square" has ever worked all that well, not even in a physical market square. You get 2 groups with sufficiently different views and before long it's devolved into shouting, if not a brawl.
I actually don't like politics being so prevalent in the space but it's quite unavoidable to give free speech any chance in the current environment. I really hope it evens out.
It's just the sad state of the world that the most aggressive, but voice-defining leftists would like to live in a situation where politics is talked about less but take a speech-impeding dictatorial rule as the precondition to allow for something like that to happen. Meaning, as long as every person in the thread or platform has somehow been "vetted" to not be conservative or even moderate, they'll act "normal". (and that ignores purity spiraling in such echo chambers making even that a stretch)
Maybe in one of the futures of this planet people can go back to not being as polarized and twitchy about talking with people with even the opposite viewpoints. Increased amount of mutual respect in a conversation plus all sides having more mental robustness reduces escalation, reducing the speech and experience of having the kind of speech you probably are talking about to a very manageable level and is absolutely best for everyone.
I've learnt to personally take a lot of pains to maintain communication lines with even some quite extreme leftists and actually managed to retain a level of mutual respect with people some of my peers don't even dare to talk to. The end result makes otherwise impossible things greater than individuals could achieve, possible. But it's not very fair feeling like the human in the "pigeon vs human" chess match at times.
Regardless of everything, as long as we're not in some kind of shittyfuture war scenario, I will not stop believing in the concept of a "public market square" of free speech. I don't believe there is any other value that can keep an intellectually diverse human society together.
No matter how good your documentation or comments are, that doesn't mean whoever is left holding the bag will understand it. Or at least not in the amount of time it would take to keep it from exploding
I understand the abstracted theory of how a nuclear power plant works (uranium heats water -> makes steam -> drives turbines etc) but if you sat me down at the control console and asked me to restart a reactor? Yeah I'd have no idea where to even begin. Even if I had a manual as thick as a fridge to (slowly) flick through
I have the impression that the comment you are replying to was sarcastic. Of course you're right that experience can't be replaced with documentation. The role of documentation IMHO is making it easier for people to gain experience by smoothing out the process -> it aids the process of gaining experience, does not replace it
Seriously ? How did you end up in that line of work?
HN has a tendancy to find something like "nominative determinism" - "comment determinism" where a comment about a job produces a (contradictory) reply from someone doing that job.
Can’t speak for them, but being poor, intelligent, and in need of steady income when you graduate High School is a very good way to end up in the Navy’s nuclear program.
Source: Was kicked out, poor, and intelligent. Ended up a submarine reactor operator.
The person I know who is a reactor operator was really good at math, joined the Navy and got trained to operate a nuclear sub. Then in the private sector I am sure that skillset is somewhat rare, so it was easy to get a job there.
This is such a disingenuous take. Twitter isn’t just a web page that displays the last thing saved to a database. It curates a different version of that for the millions of users reading it. That’s before you think about Spaces, the whole advertising platform that supports it, the abuse systems, and the infrastructure required to do all of that at a scale far beyond some bootcamp clone.
Just to give one example, I (and many others) have tens of thousands of blocked accounts. How do you implement that efficiently, at scale? Are you gonna do an extra database check for every single tweet that comes my way? Hopefully not.
And if you think about how Twitter works for five seconds, just consider everything that has to happen when an account with 100M followers posts a tweet. It's an absolute nightmare.
A bit off topic, but any book/article recommendations for building things at Twitter/IG scale? I know for example IG switched to a "fan-out" approach early on to handle accounts with a lot of users. Would love to learn more about this.
In some previous HN thread, I dialed in on a recommendation for "Designing Data-Intensive Applications" by Martin Kleppmann, which I honestly still haven't gotten around to reading so....YMMV. Maybe if you search HN for that book title, it might lead you to other useful discussions?
Out of curiosity, what do you mean by tens of thousands of blocked accounts? Are those your bot accounts that get blocked or it is you blocking people from your feed?
back when I was doing personal projects or working at small early-stage startups I used to think the same, but then I've worked at later stage scaleups and even large publicly traded companies and realised that once a project becomes large enough, that prototyping phase that you just described is like 1% or 2% of the total workload. There is so much more to the SDLC that is not just writing code (some examples: liaising with stakeholders, requirements gathering, clients acceptance testing, QA, integration testing, certifications, regulatory frameworks etc. etc. etc. -> all things that take much of your time and involve nearly 0 lines of code written).
Some of that sounds like bloat to me. The fact that Twitter is still here and building features faster with 20% of the workforce may indicate that a lot of what was thought needed really wasn't. You might point to the current outage or other bugs but you have to ask yourself, how serious is it really and is it worth 400% more employees to mitigate them?
Long form video, Twitter blue at the very least, which seems like more than Twitter managed to do per year before then with 4x the manpower.
Many of those things you listed are what people who have been in the institutionalized employees find important, essentially process for its own sake.
Before the layoffs Twitter had about 7500 employees, while SpaceX at 12000 builds frickin' rockets, have rocket launches every week, designing and building the next level launch platform that will revolutionize space travel, all the while revolutionizing satellite internet. Musk knows a thing or two about what smart, focused people are capable of.
So you're saying that at SpaceX or Tesla people don't work on any of those things (requirements gathering, clients acceptance testing, QA, integration testing, certifications, regulatory frameworks) and they just happily code away?
> thousands of people in bootcamps are making twitter clones
Heh Heh Heh
Those twitter clones will tend to be of the "looks pretty on the front end, but a fucking nightmare on the backend" type. ;)
Likely with part of the backend nightmare being security issues as well. From what I've seen of bootcamp output, they seem to rarely explain much about security to the people doing the learning. :(
That's not even the problem. The Stack Overflow guy wrote a very informative response to people claiming they could clone SO in a weekend (I can't find it now)
The core of the problem is not even the quality of the code but rather the thousands of things the site does besides letting you post a tweet and displaying it to other people.
Spam protection and protection from people falsely claiming spam on posts they don't like is already enough for me to laugh off the "clone in a weekend" camp.
Apparently you do, given that we're posting in a thread where Twitter is self-imploding due to amateurish technical decisions foisted down by a God King.
The current state of Twitter seems to indicate there's a significant problem with the code, which implies the people who would have either spotted that before it was deployed or fixed it quickly afterwards, are unable to do so. Given the number of layoffs and resignations Occam's Razor would suggest the reason is because those people aren't available to do that work.
We don't kmow it's true but it is a likely explanation.
But these are "significant problems with the code" which evidently can't be fixed by simply reverting relevant changes. In that case, it seems more likely to me that these are due to long-standing issues which would have caught up to the company sooner or later.
which evidently can't be fixed by simply reverting relevant changes
Rumors suggest that part of the change is moving from GCP to something else. Something like can't be reverted without signing a new contract with Google (and paying the bill..).
Ah - I hadn't heard that and it does make some sense. But I'm not sure that the alternative version of events (where the engineers who were fired or left were still around) would look much better in that case - perhaps delay/better planning might have helped (though of course we don't know how much planning was done) but it's also entirely credible that this was essentially inevitable with such a move.
I think most experts were fascinated by the whole experiment as it is essentially gathering data on a subject we had theories on but no experiments.
I used to work at a company where both the main data center and the main dev office were in the flight path of a major airport. We joked that if the data center had a plane hit it we’d go down quickly but recover but if the office building got hit we be fine for a while but long term in trouble.
> The people at Twitter who understood the system and could predict the side effects were all fired or left.
This is like a case study in what happens when you fire everyone except the sycophants and yes-men.
I only feel sorry for remaining non-yes-men twitter employees who might still be there because for whatever personal reasons they're in a precarious economic situation where they can't quit (H1B?) or are tied to the company for healthcare coverage (Thanks, America, for being the greatest country in the world) because they can't afford any other health insurance option.
I don't see how that would be any better than what we have now. If employer provided healthcare was removed, that would leave us with a bunch of private healthcare companies. These private companies would drive up the cost even more.
If an affordable or free healthcare option was offered on top of making employer provided healthcare illegal, then I completely am behind your idea.
Well let me enlighten you, as someone who doesn't live in america. I pay a bit less than 150 euros a months in health insurance in the Netherlands. It is not tied to my employer in any way. If I was poor I could ask for those payments to be subsidized by the state.
If I am sick I can just get an appointment with my GP within the day and not pay a thing, they can refer me to specialists or blood tests if needed, which are also fast and free. The remaining healthcare costs for medications or dentistry are so low I don't even notice them.
Hope this will shed some light to you about what's happening in other countries.
The problem with the above account is that the 150 euros you claim to pay per month for health "insurance" is transparently a fantasy number, and in reality you probably pay closer to an order of magnitude more into the Dutch health system.
Nope. Look at how much the Netherlands actually spends on Healthcare, it's about 11.2% of GDP in 2021 [1]. Per capita GDP in 2021 was ~53k€.
((53k€*11.2%)/12) ≈ 495€
Only problem is, like most developed countries, close to a majority of people are net recipients (around 40%). Someone will have to pay their share too. Chances are, if you're posting on HN, that's you, as you'll be somewhere in the top 5% income bracket. I think if the OP does the math based on their actual numbers, they'd be more likely to find themselves in the ~1000€/month ballpark than the 150€/month they seem to think they are paying.
> But even then, counting all payers and not just the residents' sticker price, the USA is the high-priced outlier.
The Netherlands (11% of GDP) is not quite as extreme as the US (17%), but it's certainly nothing to write home about, especially as I don't get the impression that either health care expenditure as percentage of GDP or demographics are moving in a favorable direction.
My second link is pretty much the same as the numbers you're giving.
What's the difference between my first and my second? I don't know. If you force me to guess, post-retirement and/or terminal care, possibly?
> like most developed countries, close to a majority of people are net recipients (around 40%)
Yes, and? Isn't much the same also true for private insurance?
You've got the potential for arguing about what "fair" looks like; I'm fine with it being funded like a progressive tax, based on income rather than risk factors, but that's not hugely important.
> I think if the OP does the math based on their actual numbers, they'd be more likely to find themselves in the ~1000€/month ballpark than the 150€/month they seem to think they are paying.
I would assume that zer0tonin pays whatever they say they pay. They're likely to have better insight into their own finances than random internet strangers like thee and me.
> The Netherlands (11% of GDP) is not quite as extreme as the US (17%), but it's certainly nothing to write home about, especially as I don't get the impression that either health care expenditure as percentage of GDP or demographics are moving in a favorable direction.
The direction of movement may or may not be favourable (given the pandemic I assume "not"), but the USA is kinda the outlier in developed nations for spending a lot without delivering particularly good outcomes:
I'm not trying to make any statements about fairness or the superiority of the US healthcare system, I'm saying that for the purposes of comparing the cost impacts of different putative health policies in the US (which was the context of the thread zer0tonin was replying to) the €150 you and zer0tonin think zer0tonin is paying a month is nonsense, because it very obviously is not an accurate reflection of zer0tonin's actual monetary contribution to the Dutch health care system.
That's less than half of what the typical UK taxpayer pays for healthcare. Surely the Dutch health system is also partially funded by government revenue?
The Swiss healthcare system is privatized to a higher degree than the US one (no Medicare/aid equivalents) yet it seems to be doing mostly fine because of sensible regulation?
While generally true it needs to be noted that health insurance is very expensive in Switzerland.
I'm not complaining since, first, this is a political decision and second, the level of service is outstanding.
For example: psychotherapy is paid for or, if your doc orders an MRI you get an appointment after tomorrow.
There's also no such shit as in network health providers (exceptions apply for some insurance models) or pre-existing conditions for the basic health plan (which is still pretty good and comprehensive).
While I do think that it's an overall good system it is expensive (and subsidized for people who can't afford it).
- It is mandatory to have health insurance, if you don't chose one the state will chose one for you
- On the whole it's very expensive, although this is somewhat offset by the high standard of living
- The insurance companies are legally forced to provide a lowest tier plan
As someone born in the UK, grew up in Australia and now in the US, who knows of paying a seven thousand dollar copay after my “platinum” insurance for “elective” surgery to remove a kidney stone that was too big to pass, versus a nine day stay for gout in Australia that resulted in a $38 out of pocket because I wanted premium TV channels in my room.
> that would leave us with a bunch of private healthcare companies.
It's already a bunch of private companies.
> These private companies would drive up the cost even more.
Other way around - by having to actually directly compete for customers, instead of just having to convince a few large corporation prices would go down, not up.
Although we really should not ignore that insurance companies are not the drivers of higher costs, it's health care providers that do that.
It's enjoyable to blame insurance companies, but the reality is their profits are capped by law - they are not the problem. Dr.'s will have to take a pay cut, and there will have to be mass layoffs, there's no other way to reduce costs.
> It's enjoyable to blame insurance companies, but the reality is their profits are capped by law - they are not the problem.
Health insurance profits are capped only as a percentage of premiums collected, not a fixed dollar amount cap. The rule is you must pay out 80% of premiums collected, everything else is OH&P.
Turns out, if healthcare costs go up, then premiums go up. If premiums go up, then insurer profits go up.
Healthcare providers and health insurers have an aligned perverse incentive to have healthcare cost as much as possible, since that is what increases their profits.
This isn’t a hard relationship to uncover if you are familiar with the insurer profit cap portion of the ACA and also how money gets made.
So if increasing healthcare expenses allow them to earn more profits, then why do you think having capped profits means that insurance companies are 'not the problem'?
Insurance companies are incentivised, under law, to have the highest healthcare expenses possible.
> Insurance companies are incentivised, under law, to have the highest healthcare expenses possible.
Yes, that is true. But it doesn't change the fact that prices will have to change at the healthcare providers. Dr.s will earn less, people will be fired as positions are eliminated. There's no other way to reduce prices.
Where do you think all that "incentivized" money is going? It's going to people in healthcare will either take a pay-cut or will lose their jobs.
perhaps we should sidestep the problem of profitability in healthcare entirely and create a non-profit healthcare system? which even the most free-market loving enthusiast should be in agreement with -- inelastic demand and all
The gigantic UPMC is a non profit. Actually a TON of hospitals are non profits - every religious founded hospital is a non profit (Maimonides Medical Center, or every Mercy Hospital (Wikipedia counts 33 of them)).
Non-profit insurance companies also exist. It still helps nothing.
Are you hoping for non-profit drug and equipment makers as well? How far do you need this "non-profit" thing to go before you acknowledge it doesn't help at all?
The NHS is non-profit. That's a healthcare system, not just a hospital or insurance company. I repeat that he said "system" and you continue to talk about things that are not a healthcare system.
I can acknowledge that it not just helps but that it is far more functional than the US system I've had to suffer through for many years. In addition to years of experience with the US and NHS, I also have many years of experience with Italian national healthcare which is also non-profit and better than the US system.
The US system is better for some diseases but only if you are rich. And an absolute failure if you aren't employed. Even if you can manage to stay employed with a serious illness you better have a healthy family member with a lot of energy who can fight the insurance company that really doesn't want the cost and burden of you and will make that clear in every action.
How far do you need this "for profit" thing to go before you acknowledge its very serious flaws and inadequacies?
NHS is not a combo of "hospitals, doctors, and insurance". It does not involve insurance at all. It is a comprehensive and integrated healthcare system that runs as a whole.
I'm writing this from the UK where I use the NHS. I've also used the US system extensively and the Italian healthcare system extensively.
You need to get basic facts right if you want to be a part of the debate.
If this is aimed at free-market living enthusiasts, I believe the response from them would be that the market is currently not stopping anyone from opening non-profit healthcare providers today.
> Other way around - by having to actually directly compete for customers, instead of just having to convince a few large corporation prices would go down, not up.
This is backwards logic. Those few large corporations have the bargaining power to negotiate lower premiums. Individual consumers have zero bargaining power.
You wish it was backwards logic. In reality the insurance company just needs to convince a single person in a company to pick them. And they are very good at doing that.
On the other hand individual consumers have ALL the bargaining power - they can simply pick a different insurance company, and insurance companies have to work very very very hard to get customers. They would compete on price because that's by far the most important thing to a consumer.
A company on the other hand cares about other stuff, how integrated in the system, how easy can we import members, manage members, how much marketing material do they give? Do we have to educate our employees, or will the insurance company do that for us?
Just tons of other stuff that isn't price. Individuals: It's 99% price.
So, you spend every single day of your existence in the US being absolutely fucked by the private medical sector, and you think that for some reason having no collective bargaining ability will make you better off?
Plus, health care spending is like >10% of GDP. You don't rewrite the rules like that with upsetting a great deal of corporate interests. An appreciable number of people benefit from all the inefficiency in delivering health care.
> An appreciable number of people benefit from all the inefficiency in delivering health care.
Not anything close the the number of people who suffer from the all the inefficiency in delivering health care, but guess which has more money to bribe lawmakers with
Meanwhile, France made mutuelle (half of the healthcare) mandatorily paid by employers. When you leave, you can keep it for a time, but it’s essentially paid by the employer…
> This is like a case study in what happens when you fire everyone except the sycophants and yes-men.
The initial and biggest waves of layoffs last year were of people who hadn’t yet had a chance to demonstrate whether they were or were not sycophants. They were essentially random.
I don’t think that’s true at all. Elon allowed any employee who wanted to quit with three months’ severance. Anyone who wasn’t a true believer and had options could have quit.
It is very possible that they did not predict this specific down stream effect, for one of many reasons. I do not want to discount that, though, as I am not at Twitter and am not on a first name basis with anyone there. Maybe there are some who did see this.
What I maintain though is that most anyone still working on code at Twitter, regardless of their experience or overview of the code base, would strongly argue for testing and staging, which appear to go against current leadership's mode of operation, likely because of the time pressure you mentioned.
Not pushing such changes straight to production is a concept I feel anyone working at Twitter would subscribe to, yet has to painfully go against, lest they be led go.
Except, progress grinding to a halt pretty much described Twitter before Elon Musk too. The story that anyone who knew anything was let go is just that, a story, as the number of employees by year shows Twitter, and other tech companies in the same period, overhired and then went back to pre-2020 levels.
The only difference between then and now is that there is a big personality at the top who now personifies everything Twitter does, especially if things go wrong, whereas before it was mainly just a faceless bureaucracy whose Trust and Safety lead at times had more visibility than the CEO.
The subtext is that Twitter changing hands also involved trimming a lot of the dead weight, particularly hitting the softer managerial/diversity/HR side. Now a lot of people are rooting for the site to fail because it has gotten too "bro-ey", as the era of trust-and-safety and $15k backchannel bluecheck deals has made way for free-speech and monthly subscriptions.
You claim with a straight face that, had Musk not taken over Twitter, it would have had the same rate of outages and level of degradation as it does now? It's completely obvious to anyone familiar with the stability of Twitter pre and post Musk that his takeover was an inflection point. You're ignoring reality in order to support a claim that fired non-coders and "diversity hires" contributed nothing.
What I claim is that:
- OPs story that Twitter was a healthy and productive tech company pre-Elon is complete non-sense. How many years did people pine for an edit button?
- Twitter returned to pre-2020 staffing levels, which is true
- Twitter struggled to push out new features (like an edit button) for years, which is true, wheras post-Elon they pushed out edit buttons, longer tweets, subscriptions, etc.
Nope! From your original post: "The only difference between then and now is that there is a big personality at the top"
Claiming that nothing is different other than Elon's presence implies that Twitter was just as much of a technical dumpster fire as it is now. That is not _at all_ what you're now claiming you said, which is, effectively, "Twitter made slow progress on product priorities". No one disputes that, but it's not what we're discussing here, which is the rapid degradation of service since Musk took over. Maybe those "diversity hires", as you call them, actually contributed to keeping the site running.
Here’s just a few of the people (in these cases journalists) that Elon has banned. It’s not hard to find other examples of censorship either. That’s his right, he owns all of it, but he lied about ideals of free speech. If it’s speech he doesn’t like, he kills it:
Ryan Mac
Drew Harwell
Micah Lee
Matt Binder
Aaron Rupar
Donie O’Sullivan
Tony Webster
EDIT: these bans were related to reporting on the elonjet tracking account that was banned. He didn’t just ban the account he didn’t like, he banned the accounts of journalists who talked about that.
For posterity, those journalists were banned (unbanned a few months later) for just reporting on ElonJet, who used publicly available FAA data to track Elon's jet. Elon called it doxxing. And also temporarily banned links to Mastodon just to prevent people from accessing ElonJet that way.
Good point, they were all related as bans resulting from reporting or tweeting about the elonjet ban. I edited my comment accordingly since it’s important context to know the ones I mentioned were all related to a single event (though he’s silenced others as well for different reasons too)
Elonjet was bait by people with an axe to grind and a chip on their shoulder, and if this isn't blindingly obvious, I have a bridge to sell you.
There is no doubt that, after firing Vijaya Gadde (sp?), the corporate focus has shifted away from censoring every single tweet, and more towards letting people say what they want. This does not mean every single tweet is left up, or that annoying Elon isn't a catastrophically stupid thing for journalists to do.
> There is no doubt that [...] the corporate focus has shifted away from censoring every single tweet
It is absolutely unbelievable that anyone could say this when every Tweet is currently censored to non-members.
I've traditionally considered the Dorsey administration to be the worse steward, but this is an insane take to steelman considering how self-conscious the past few months of banning has gotten.
Unless you have some information I don’t, Twitter is way below its pre-pandemic staffing levels. Obviously hard to tell exactly how low they went as they’re not public anymore, but this is quite different from Meta and such that really did just go back to pre-pandemic level. CNBC say Twitter is at 1300 people, a drop of 80%. It’s mind boggling.
Your statement is an admission that you prefer to judge messages for their implied moral valence instead of their truth content. Personally I find that pathetic.
What a fantastically bad take supported by no facts whatsoever. I like your graphics work but you should really think before you type this kind of a thing up.
Twitter went from a 4.9k headcount in Dec 2019, to 7.5k in Dec 2021. Fact.
I don't care if you "like my graphics work" or not. You seem to be implying I owe you something, which is crazy. I have been publishing stuff online for decades and I can tell you, judgy and entitled people such as yourself have _never_ done anything useful in return.
What's amazing is you chastizing me for a "bad post", even as you dispute something you could've googled in 5 seconds.
It is -- and I cannot stress this enough -- entirely OK to root for the failure of a company that is owned, directed and dominated by an odious person with abhorrent politics.
Compare:
- hoping Ballmer-era Microsoft would fail in their attempts to snuff out Linux
- hoping that USSR would fail in their attempts to snuff out large numbers of their own citizens
- hoping that the Confederates would fail at snuffing out resistance to literal slavery
- (in fiction,) hoping that the Death Star would fail at snuffing out various planets, etc
and so forth.
There is not some weird list of permissible root-reasons. You have no gotcha; you are just gotten.
And speaking of having no gotcha but just being gotten, if you have showdead=true you can see one of Musk's biggest stans trevioustrouble rolling out their very best most deep and thoughtful arguments in support of Musk:
>trevioustrouble 1 hour ago [flagged] [dead] | parent | context | prev | next [–] | on: Twitter Is DDOSing Itself
>It’s just a feed, and needs to be rate-limited for unregistered users. No need to pull out your philosophy-degree. The people that were fired from Twitter were fired for good reason and if you think you’d do a better job than Elon with Twitter, you wouldnt.
>* PS: Dislike my comment fags
And then they sum up their politics and best arguments and what the Twitter they're fighting so hard for and what Musk they worship so much is all about, in just one word:
>trevioustrouble 1 hour ago [flagged] [dead] | parent | context | flag | vouch | favorite | on: Twitter Is DDOSing Itself
>fag
And that's the best they've got.
It really makes Musk's apologists so angry and frustrated to see everyone laughing their asses off at Musk explosively and bloodily sharting himself in public like that, because now they have to follow behind the elephant and wipe up all the mess.
Disclaimer: I'm not American and have only the foggiest notion of who RFK is, but the urge to identify boundary conditions for 'permissible wants' does feel especially America-y to me. I speculate that it has something to do with the veneration given to markets --- the underlying anxiety being that having feelings about an intensely cultural use of forty billion dollars is somehow antithetical to the conduct befitting an ideal rational agent.
None of your examples actually align with the general sentiment here. In your examples, you cite specific goals that should fail, which may be part of a broader approach which includes more noble goals.
For example, you say that Microsoft should fail in attempting to snuff out Linux, not that Microsoft should fail generally.
You say that the USSR should fail to kill their own people, not that the USSR should fail to thrive as a people or a nation.
In this case the equivalent would be to call for Twitter to fail at... what exactly? Free speech?
I think you've been gotten. You don't perceive these examples as equivocations when they are, and it is blinded by dislike for a figure you disagree with rather than a specific bad goal.
One may compare circumstances of the same kind without asserting that they are of the same degree. This is generally referred to as an 'intuition pump' -- it's not the most reliable move, philosophically speaking, but it does have a pretty long history, IIRC Plato's _Republic_ mentions looking at big things from far away in order to understand things that would remain obscure if (as he put it) written small.
And if he didn't make his politics part of how he runs stuff, I probably wouldn't even care about his politics. But you know how celebrities get crap for inserting politics into things, when they know very little about that topic? Elon is going to get the same.
Is it..? All the replies to this thread just mention how he has abhorrent politics but there's no specifics. I'm not sure anyone really knows why they dislike him, and thought critically about it. A lot of it seems assumption-based, fed mostly by the media who is Twitter's biggest competitor. And on Hacker News, of all places. I'd think it the most likely place to find people who understand the rift between tech and trad media, and that character assassination is the biggest tool used by one particular side in this war. Instead the middling tech armchair specialists who partake in and promote the culture that birthed Elon and his multitude of companies have bought the other side's narrative to hate on their own.
I think its like, totally reasonable? to want the Gates Foundation to like implode and everybody who works for it to be unemployed? and all those poor African kids to go unvaccinated?
Because I kinda think Bill Gates has bad politics?
It’s not his politics, it’s him personally. Many find him an arrogant, loathesome baffoon and an asshole, even outside of the political realm. Obviously many people are rooting for him to fail.
Prior to Twitter's acquisition by Musk, they worked quite closely with the State and even hired the former top FBI lawyer as their chief legal counsel.
And now that he's opened his big fat mouth and revealed just what kind of a person he really is, I'm also rooting for unconed to fail because of his politics.
> The story that anyone who knew anything was let go is just that, a story, as the number of employees by year shows Twitter, and other tech companies in the same period, overhired and then went back to pre-2020 levels.
Citation needed. Show me the 2020 Twitter headcount. Show me the 2023 Twitter headcount.
> The only difference between then and now is that there is a big personality at the top who now personifies everything Twitter does, especially if things go wrong, whereas before it was mainly just a faceless bureaucracy whose Trust and Safety lead at times had more visibility than the CEO.
That is not the 'only difference'. There is also the matter of all the hate speech. Which I guess you don't really notice, as you're not one of the targets, but I sure am. This is like Trump going 'boo hoo everyone hates me because I'm Donald Trump,' when in fact there is this small matter of an armed insurrection. You are writing off the valid political concerns of your opponents as being rooted in personality, rather than in odious politics.
> The subtext is that Twitter changing hands also involved trimming a lot of the dead weight, particularly hitting the softer managerial/diversity/HR side.
This is the most reality-defying way of recalling that the entire company basically quit on him overnight, but ok, bro
> Now a lot of people are rooting for the site to fail because it has gotten too "bro-ey", as the era of trust-and-safety and $15k backchannel bluecheck deals has made way for free-speech and monthly subscriptions.
That is not the reason we are anticipating its failure. We are anticipating its failure because we understand how people, platforms, and software interact. 'Hope' has nothing to do with it; we just read it off the verniers.
> So about the only thing they could do was issue multiple, parallel requests and hope that at least one of them was fast.
lol nobody would do this to solve this problem because it doesn't even remotely solve it or give the appearance of solving it, if anything it's guaranteed to make things go slower
Google did this a long time ago. It’s more nuanced than you think. Two requests would be issued and the first acknowledged one cause the second to be cancelled. There was a public paper on this from Dean iirc and the method is a decade old.
I think the fundamental difference is that Google was using request hedging within their own network, sending the same request to two different internal servers in case one was slow, while Twitter appears to be sending the same request to the same server over the public Internet.
That's the first thing that came to mind. It's a pay-to-play move, which is not at all surprising. People seem to forget that fully-featured social media-centric corporations are run for profit, and if they can't get enough advertising to balance the books, then what's left but subscriptions?
Sometimes people buy media corporations because they're interesting in using them to promote their other more lucrative operations (think Jeff Bezos and the Washington Post) so they don't really care about profitability, but I don't think Twitter fits that model, but who knows?
The WeChat model is the obvious answer to that rhetorical question. Build paid services on top of your hellsite with millions of addicts using it daily. For example, Twitter could have easily been the gateway to OnlyFans, or to Patreon. Apple is another example of this kind of value-added ecosystem that leverages a foothold to sell other crap to people.
Or there's the TikTok model: gobble up all the data, sell data to governments, give MBS or Putin admin access to Twitter, build AI on top of the dataset.
Even a private equity chop-shop like Bain could do a better job of extracting value from Twitter than this mess.
This isn't a profit-seeking venture for Musk. This is about politics, about power, and primarily about revenge. Musk is giving Notch stiff competition for the title of most pathetic billionaire.
>This isn't a profit-seeking venture for Musk. This is about politics, about power, and primarily about revenge.
I strongly disagree; this is about Musk buying Twitter accidentally, and then running it as best he could without losing face as "real world iron man". He's fucking up left, right and center because he was completely unprepared to actually run it, and suddenly needs $40B to pay off his debtors.
It's also about politics, power and revenge, but it's primarily about Musk being a fucking idiot and constantly digging himself deeper.
That's a non-sequitir. You can be at the top of your field and not completely understand a complex system. Also, the right people may not have even been involved in the implementation of this "feature".
Bingo. You can know a specific technology inside and out, but easily get lost in a large system built in that technology if you haven't worked with it before.
It's very easy to get caught in assumptions like, "Nobody would ever do things THIS way, so they must have built it THAT way," only to find out that, once upon a time, THIS way was the right way to do things, only for it to over time become less and less optimal, but the costs of changing things were too high to fix it. Once your system is old enough and large enough, you'll have several thousand things just like that.
For sure. I'd add that even super-geniuses get overwhelmed when asked to do too many things at once. Kudos to whoever's still left at Twitter; the current erratic decay is better than I would have expected after this long. But when maintenance and improvement are deferred long enough, eventually you reach the point where on average solving 1 problem creates >1.0 problems.
> You can be at the top of your field and not completely understand a complex system.
The tweet reads: "Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon's latest genius innovation is to block people from being able to read Twitter without logging in."
Does that strike you as complex? I mean, surely they had the context (need to log in) because it was all over the news
I don't know what kind of systems you worked on in your career, but even simpler systems with smaller userbases than twitter are quite complex if you are new to it.
Twitter serves their service to the entire world, with multiple layers of systems working in conjunction in order to make things work smoothly. A new engineer that has not been working on it for no more than a couple months would likely be unaware of how the different systems communicate and interact. A change like this will have have a lot of unintended consequences, and not having a senior engineer with lots of context leading the change will undoubtedly cause these kinds of issues.
> A change like this will have have a lot of unintended consequences, and not having a senior engineer with lots of context leading the change will undoubtedly cause these kinds of issues.
Having a senior engineer with a lot of context is worthless if the work environment does not promote open communication. You don't want to be the senior engineer or leader who shows "poor judgement" by opposing the mercurial owner "for no reason" if you're overridden and the feature launch succeeds without a glitch; no one gets fired for implementing a request that came straight from the top.
This is why non-rushed, scaled roll-outs are essential for large system: had they tried this on 1% / 5% / 10% of random traffic first, they could have caught this. Yet again, if the directive to roll it out to production came from the very top, you set that gate to 100% immediately.
“Trying to make a pigtail out of these unbrushed hair may unscrew the ears”. Sorry, yep, twitter is big. But if preventing tens of doomed requests before a login requires a senior engineer with lots of context, then the program was screwed up long before the layoffs.
I don't disagree, but: Twitter was big and it worked. Then someone created incentives for many people who know the code to leave.
The one's left don't know all the code (how could they?), but were forced to change many things about the site at a "just do it" basis. This error didn't happen because someone was too stupid to remove the code, it did happen because the connection to another thing was removed and the failsafe on the landing page doesn't have exponential backdown built in, not something you can necessarily know or investigate before, when an executive breaths down your neck and wants you to just do it.
This is about the new managment, not about engineers.
Your argument is that it is a non sequitur to say that the right skilled people are not at twitter, using the argument that the right people with the available skills were not involved in this feature.
Twitter could be packed with extremely skillful senior engineers who don't understand the product well enough to predict complex outcomes of planned changes.
Likewise you can have a "poor quality" 1x employee who has knowledge of how everything within the stack is glued together; where there is chewingum and where there are steel beams.
They are potentially more vauable than the 100x engineer who has intimate knowledge of how googles shipping container datacentres work.
Seems Twitter has had a culture of rolling out changes straight into production without incremental testing, at least since well before Musk took over. Mudge was hired there in 2020 and what he found was a complete mess.
They lack knowledge not skills. To my knowledge there is no "intuitively understand how my changes will impact a complex system without studying it" skill (unless that system is under a robust test suite, which Twitter is not)
> knowledge there is no "intuitively understand how my changes will impact a complex system without studying it" skill (unless that system is under a robust test suite, which Twitter is not
then how do they have confidence that anything works before they subject hundreds of millions of people around the world to system updates? that seems disrespectful to the user, if you asked me.
Alternately, if they have the “right skills” why is this happening? Clearly they don’t have some of the needed skills to prevent this. And a self-DDOS might be a good indication of that.
> and needs to be rate-limited for unregistered users
It seemingly needs to be rate-limited for not just registered users, but registered paying premium users as well. And the rate limits for them are enough to be passed in less than an hour of average idle use!
> if you think you’d do a better job than Elon with Twitter, you wouldnt.
Doing nothing and not saddling twitter with unnecessary extra debt would have already been doing better than Elon.
I have no experience managing a company at that scale so I won't challenge the assertion that Elon is doing a better job than I would but... the feed has not been rate limited for unregistered users for the history of the site as far as I know, and its been pretty damn stable and snappy!
Beware of bugs in the above code; I have only proved it correct, not tried it.
– Donald Knuth
It's not just hypothetical either. There was a bug in a sorting algorithm a few years back that had been 'proved' correct. I think it was to do with numbers wrapping, and that hadn't been considered in the mathematical proof.
WPA2 also had an exploit (KRACK) while the handshake algorithm itself was "proven to be secure". Formal verification is a powerful tool but it does not guarantee bug-free code: it merely guarantees that the particular bugs you checked for are not possible.
That's exactly why you write tests. You want to be sure future changes don't break present functionality, especially if the future changes are being done by someone who doesn't understand your part of the system.
You write tests for your code as it exists to make sure it functions. You cannot write tests for every conceivable change that someone else might make in the future that either mis-uses your code or adds new code inbetween that didn't previously exist. You need the people who add that new code to write new tests. Something that it seems the remaining engineers at Twitter do not have time for.
But this is Scaling-101 stuff. It's not some super complex or unique system going wrong. At least according to the article, it's a classic case of bad retry logic leading to a death spiral.
Explain why not, if you please. If unresponsiveness causes increased traffic, which causes further unresponsiveness, is that not referred to as a thundering herd problem? Is the stated mitigation of a backoff not fully relevant here?
It's the difference between one customer asking a hundred cooks for a waffle and a hundred customers asking one cook for a waffle. The former is the thundering herd (a bunch of processes trying to do something that only needs to be done once, causing resource contention) and this is akin to the latter (with the "customers" being parallel requests from the frontend).
The thundering herd problem is more like, there's a hundred cooks, one griddle, and only one of them can make an acceptable waffle for the customer.
This specific problem we're discussing, of concurrent client retries effectively launching a self-imposed DDOS attack, isn't exactly the thundering herd problem. It's clients and servers instead of threads, for one thing. But it's a good enough analogy to another type of cascading failure in concurrent computing, IMO.
Hmm, I was thinking it still applies in the sense that the many many duplicate retries are hitting many of Twitter's servers causing unnecessary duplicated load when a single successful response would satisfy the client and reduce the traffic.
In my mind, it is much closer to needlessly asking every server for the same information because the requests are most likely load balanced, but I guess it's true that I don't know the load balancing strategy. Even still, is it not more likely than not that those retries are hitting multiple servers?
Sure, maybe? We (or at least I) know little about the actual problem here, and metaphors only go so far. But to my mind, "too many things trying to handle a request" gets a cool name because it is a fairly narrow and unusual problem, whereas "too many requests" goes by many names (DoS, hammering, flood, etc) because it's depressingly common.
I’ll play the devils advocate here but frontend devs need to smarten up. This is basic error handling that should have been in place for years. Blocking tweets with 403 or whatever they chose shouldn’t trigger endless retries on short intervals.. ever!
You're right about error handling, but consider that the user who posted this was logged in. The screenshot they post is of a specific tweet, but they reference the home feed being down. What kind of API call was it making where it's asking for a list of tweets to serve to a logged in user and that call is not authenticated? It makes me think that the blocking was implemented improperly.
Unless the home feed being down is simply a side effect - the service that fetches tweets being DDOS'd by other views in the app making numerous non authenticated calls.
But I was also thinking about this earlier today. These days, everybody is so quick to say "the software is easy, it's the community that's hard" - I've even said it myself a few times in the past few weeks, but I think that might be overstated.
Building good software is hard. Keeping it good is even harder. What does the codebase look like for Twitter's front-end at this point?
How many frameworks has the base functionality been ported through? How many quick pivots from product adding features, adjusting things squashed down the ability to address technical debt or even have functioning unit and regression testing?
The fact that this 1. Made it to production and 2. Was not noticed and rolled back immediately (like, in under 30 minutes) is extremely concerning (and obviously very embarrassing.) If I had private data stored on Twitter of ANY kind (like DMs that I don't want getting out - a messaging system rich, famous, and powerful people have been using like email for over a decade), at this point I would be trying to get that data removed however I could, or accept that there's a strong possibility there's going to be a huge data breach and all of the data will be leaked.
That would give the server side more control over the retrying logic (when the header is properly interpreted). I'm surprised Elon hasn't implemented this himself.
I’d bet (not that much, but like $20) that someone has a .ifClientError() or if responseStatus === 4xx somewhere.
If you’ve never had to handle authorization in a particular area, it might have been safe to assume that any 4xx error should have been retried when the code was originally written and someone didn’t write them all out
But wouldn't responseStatus === 4xx indicate that the problem is on your end and retrying is unlikely to fix the issue. A 5xx is worth a retry, a 4xx should imho just produce an error message.
And even if you do retry, exponential backoff has been the standard for a long time (and is mentioned by the Twitter API documentation as a good solution to 429 responses)
There are libraries, usually under the general heading of "circuit breaker", that handle 429 and other reasons to retry in a sane manner. I'm not a JS expert but I believe either yammer/circuit-breaker-js or nodeshift/opossum would work in the browser. Even a hand-coded exponential backoff with jitter is simple enough to do for most cases.
..and using exponential back-off, if not limiting the number of retries.
Though it’s hard to know for sure what really went down. Could be a number of things. Including a lack of subject matter experts (Elon recently admitted to laying off some people they shouldn’t have).
Devil's advocate here: did we consider that any such exponential back off goes out the window when users, faced with a non-working site, will just refresh the page therefor reseting the whole process?
Personally I'd just cache HTTP 429 responses for 1 minute, but you could also implement rate-limiting inside the load balancer with an in-memory KV store or bloom filter if you wanted to.
Perhaps the context you're missing is that all large sites use ECMP routing and consistent hashing to ensure that requests from the same IP hit the same load balancer. Twitter only has ~238 million daily active users. 10 requests/second on keepalive TCP+TLS connections can be handled by a couple of nginx servers. The linked "Full-stack Drupal developer" has no idea how any of this works and it's kinda sad how most people in this thread took his post at face value
Just speculating, but the server is probably throwing 429 (Too Many Requests) errors, and yes, the client should respect and back off when it encounters that. According to the post, it seems not to. Strange!
BS. If there's a new limit of 600 tweets or whatever, then they should try hitting that limit on each client to see what happens. This sounds very reproducible.
I don't think its intentional. I think it's a side effect of using a listener pattern.
If you use a listener, useEffect in react, to load data, it will start the request, track it is loading with a boolean, and then store the payload. That passes unit tests and QA.
If the listener doesn't check the error before starting the api request again, you have this infinite loop happen where the loading flag goes off and the payload is still null, so it just starts it again.
It's sloppy code, but its an unintentional side effect.
I have a general rule that I make proactive, diligent efforts to inform and correct, and I do it 3 times to 3 different groups in leadership, and if that doesn't work, I'm out.
I got tasked with essentially project managing a doomed project that the boss had already made all the tech decisions & promises on. Nothing was built but we had customers with firm dates for go-live. It was clearly not going to work, and not be on time. I pointed some of this out.
He then asked me to do weekly RAG status for stakeholders. So I did and it all slowly turned red with no easy remediation.
He did two things that made me realize it was time to go:
First, when I suggested that "if I report red status all year and then it doesn't complete on time, but I kept everyone well informed - am I going to be rewarded end of year".. to which I got nervous laughter response.
Second, he started talking about "what if we change the definition of done", such that we just start marking things amber/green because like.. well some of it is running in QA or hey its like 70% done, so why not mark it done?
Just seemed like he handed me the keys to a sinking boat as he stepped off in the last life boat.
I worked for a guy whose boss was trying to throw him under the bus constantly.
In one scenario after being reminded of something, he said "I don't recall that conversation, you need to put this kind of stuff into email". So in a following scenario where he did send an email, boss man said "I get lots of email, you can't send an email and assume I read it."
This is why I find if you don't already have good relations with management and trust each others judgement, it really doesn't matter.
They will do as they wish, and throw you under the bus as needed.
Talk verbally, and then send an email afterwards with notes. This means that they definitely were informed, because you talked to them, and you have something to point them to if they don't remember, because you send notes.
This is protection in adversarial scenarios, but is also just a great habit In general. Verbal discussion is really good for getting people on the same page, but without notes it's very easy for details and decisions to get lost.
If you are in the kind of adversarial management relationship where this is necessary, you have already lost.
Do you think this kind of guy, when you point to "hey remember the conversation, here's the follow-up mail with the meeting notes" he's gonna be like "oh yeah, I was wrong, you are right." ?
It's good to have meeting agendas and follow up minutes, I just rarely find that they are going to help you litigate anything. More to remind you how a decision came to be.
You don't happen to have a blog or something where you outline, even if it's without tons of detail or being too specific, what these incidents looked like, do you? These feel like (hopefully) rare enough events that I'd love to try to learn from other people's experiences before I run into this kind of situation myself.
No, they're not the kind of thing I like to talk about in more than generalities because they paint me in a negative light, even though most of the work I do is more about positive-sum games.
If you ever have a beverage with me or drop me an email I'm happy to discuss without naming names, but public is unwise, sadly.
One of these years I'm going to retire and start a youtube channel. If you like similar stories, ThePrimagen[1] definitely has a similar flavor. He talks about some situations at Netflix that are eerily familiar, even though I've never worked for them.
> Every person in power is told he's wrong every day, all day, by most everyone.
Wait, what? That's a ridiculous assertion, even taken as hyperbole. A huge percentage of people in power surround themselves with "yes-men", and are allergic to (even constructive) criticism, let alone direct disagreement.
How many people tell Musk he's wrong every day? From reading social media, probably around a million! Heck, just look at the replies to his Twitter posts.
That’s not the same thing at all. The original comment was clearly talking about people who work directly for him, whom he has the power to fire on the spot.
Not in Musk’s defense, I guess there are enough signs that his posture is authoritarian and therefore bound to end up with yes-men, but people in power also sometimes are willing to listen in the proper context. I.e. as long as it is in private or in a trusted forum, where being the Cassandra does not link the warning with their leadership. Of course the really powerful will also react to potentially uncomfortable feedback in a proper forum or in person, rather than in public.
I have usually just produced a disclaimer and asked the relevant party to please read and sign it, indemnifying me against any consequences of the decision they have made.
That usually gets them to sit up and reconsider.
Occasionally they sign the damned thing, and then it’s popcorn “I told you so” time.
I'm going through this right now in my job, having built a tool to manage vulnerability tickets for my team, and having it's first use be deleting every vulnerability ticket, despite my protests. The person executing this is more concerned about looking good on paper than actually improving security.
That's nothing. Instead hopping on the bandwagon hating on the thing we all use and care about, a situation where you would know a solution but are not allowed to act because of fear of even trying anything is so much worse. Especially when you know of it for a very long time and are not even allowed to try, while also being ostracized and being "that guy" for suggesting it repeatedly. Talking of different technical and business approaches to things where the "original" still somehow works but is slowly killing the company. Been there, done that.
But when a higher-up so clearly takes responsibility of something that is risky but deemed necessary, it provides for a lot better space to also respond to any potential problems that pop up, while not needing to suffer snarky comments in hindsight as a dev.
Of course something like this is easier to hate on ...
>Speaking from very painful, personal experience, few things are more agitating than being forced to execute on something you fully know is a horrible idea, especially when you tried and failed to communicate this fact to the individual pushing you to go against your best judgement.
In this case the horrible idea is being forced to push changes to production at a moments notice
Yes, that is very much true. Even if, as some have commented, the people remaining at Twitter lack oversight over the entire codebase to predict that this specific change could have such a knock-on effect, not having testing and staging environments, pushing something straight to production, would give even an inexperienced person pause for thought.
As said, I don't want to speculate whether someone raised their voice on this being a likely outcome specifically, as not having testing prior to release is the much more obvious and significant issue here, driven by leadership.
Yes, but so is Change Failure Rate (and a good team will use feature flagging anyway for gradual rollouts, so you don't roll out a DDoSing broken feature to 100% of users, just a tiny subset).
The push to "make the content less user friendly" and push then to our app/logged in experience is similar to reddit's push and it doesn't really present an insurmountable challenge on the technical side.
These technical hurdles are temporary. Not the end of the world. The "only in our app/only logged in" requirements are bad on a philosophical level of user choice, not on the "this is terrible technically" side.
Attacking it on that note is just perpetuating the narrative where elonmuskmanbad (which is more political than consistent with the treatment others get) above more specific truths.
I'm curious about this: When you were experiencing this, why didn't you quit? I truly feel bummed for the people remaining at Twitter who are miserable but for whatever reason feel they can't just quit.
There are a lot of reasons people stay in positions. Not everyone has the same degree of mobility in terms of a job. Maybe they have little time in their personal life to deal with interview prepping. Maybe they're older and deal with age discrimination. Maybe they're on a work visa and know they're in a leveraged position. Maybe they're not good at negotiation. Maybe they loathe the market. Maybe they have a pile of debt and the current situation is less risky.
As an industry we need to stop perpetuating this idea that everyone is fully mobile and software engineers are some mythical creature that isn't also shackled to the same constraints the rest of the labor force is. It's a job and a lot of people are dependent on a steady income stream. Part of that is putting up with the least worst of the options they have reasonably available to them, or sometimes taking up worse options just because it balances other aspects out. Ultimately during certain times you may have more or less leverage to put up with or reject crap practices. SWEs tend to have a lot more leeway than say a retail worker of course so we can be a bit more demanding but we can't just say crud off, I don't deal with any BS, even in the best of times. Higher salaries give you some negotiation room as well as you can lower your TC expectations in terms of negotiating better working environments (i.e. taking lower rates but demanding things like less pressured schedules, make autonomy, etc.) and as the extreme sof the markets are high enough, even taking TC hits can still let you live comfortably (this is to contrast many other professions who don't have this flexibility).
Unless you've lived modestly and invested significantly or started some side venture to become financially independent of labor based income, you're ultimately giving up some degree of agency to employers. We like to pretend this isn't the case but it is the case for most people, even highly paid labor.
I was asking about this specific person's situation. I do recognize (and despise) the visa issue that indentures immigrants to employers. But I'm curious what other circumstances there may be.
For some people, "going back to your home country" means going back to a place where basic human rights are not protected by law. Further, I'm not sure what having your visa revoked would do to your future immigration prospects--not to mention that, once that visa is revoked, you're technically "in the country illegally", and you may have heard about how badly our legal system treats people, both with and without citizenship.
Have you moved to another country across the world, made it your home, and had to choose between doing stupid things for a corporation bought by a new CEO or uproot your life and go back home? As much as it's sounds like an easy decision, most people won't uproot their life for Twitter.
That is a good question, but on the two specific occasions when something of this sort did not make me leave instantly, there were very specific factors at play. I cannot comment on the first situation as that would require an extensive post with an obscene amount of context on my background, politics in my country, the people involved, weighing up the public good, etc. Simply, would go beyond a HN comment and I feel I couldn't do the situation justice anyway for someone who wasn't following local news at the time these things happened.
The second instance, I feel I can comment on, though allow me to leave out details to spare the people involved public embarrassment.
There, the changes, whilst annoying and born out of a complete misunderstanding of a core part of their alleged competency (Imagine a Botanist telling you leaves are always blue because that is the color of the sky), were not going to break anything, just look silly and create unnecessary, but compensated, work.
In that case, I also viewed the specific project finally launching as vitally important to our user base and wanted the results to go public for their benefit, so the decision was made to document and execute on their requests, so we could go live.
Of course, two weeks before our go live date, they changed their requests again ("leaves are not blue, they are violet because of the wave length of light") and had a hard time understanding that changes can have a knock on effect and some things are a bit more complex than Find and Replace. If I had been forced to make those changes at that point, I'd have packed my bags.
Simply said, when my work has the potential to benefit users and I know that arguing, even though I am correct, will lead to massive delays, I'd rather just put that silly request in writing and deal with these things after the users have received what has been worked on. Try to explain when someone is wrong, but if that doesn't work, finish the project, argue later.
Of course, if after the fact responsibility isn't taken, that finished project gets mentioned in my CV for the next employer.
That makes sense, thanks for sharing. I definitely get the thing about not wanting to let people down. And I do wonder if that's how a lot of the people remaining at Twitter feel.
Well "forced to execute" is somewhat subjective. If you are convinced leadership is doing the "wrong thing," then best to either leave or accept that you're just collecting your paycheck.
In the case of Twitter, the new owner has thoroughly broken the advertising business and is trying to aggressively pursue a new version of the data business Twitter once had-- E.g. Google's Caffeine, which Twitter also eventually lost https://searchengineland.com/google-search-algorithm-change-... ... The statements about "too many scrapers" are almost certainly as illegitimate as the previous pre-acquisition statements about "too many bots."
The nature of business is that there's no judiciary or referee... the purpose of a business is to make money. Tech businesses just happen to hire lots of academically-oriented engineers who developed their skills in a different environment. It's possible to build a culture of "fairness" in a business, but at the end of the day even Google dropped "don't be evil."
> Well "forced to execute" is somewhat subjective. If you are convinced leadership is doing the "wrong thing," then best to either leave or accept that you're just collecting your paycheck.
Я, Unless your visa is sponsored by your employer.
Everybody has an opportunity cost against losing their job. And employees do have Rights but the right to have technical initiatives adjudicated against stupidity is not one of them.
The OP is fundamentally wrestling with an issue of justice, but the bottom line is that in business there is no punishment for stupid. There is only reward for profit.
If your visa is sponsored, you should be especially prepared to execute even if the leadership is doing the wrong thing, at least until you have an alternative. Being hired on visa means having less contractual power, and there's no way around it.
I don't understand why anyone, at any job that is not C-level, has an attitude of anything other than collecting your paycheck. I care about my professional reputation and my career, but I don't care at all about the well-being of the company that has hired me to perform some task. I will advise to the best of my ability, but if they blow themselves up, why should I lose any sleep? That's their problem, not mine.
Detaching myself emotionally from my employer was one of the best things I've ever done for my mental health. When I was young I got upset when "the company" made bad decisions. Now I feel no negative emotion about it, sometimes I laugh at them.
This is healthy but some of us find themselves in a position where caring about the company has tangential benefits like bonuses, stock options, que promotions and other objectives with a well defined incentive for you to personally care about it.
In my experience the difference between the average bonus you get by plodding along and the amazing bonus is either your boss likes you on a personal level or you killed yourself to get it. I have never seen a case where the $ difference between the average bonus and amazing bonus is worth the metaphorical "killing yourself" to earn it.
My recommendation is to recognize the stock options, bonuses, etc. as the emotional manipulations they are. Either perform the tasks or don't, but try to avoid getting emotionally bogged down in it. Instead, lift your head up and look around at the broader economy and make your decision based on that. Most of the time the fastest path to a promotion, higher income, etc. is to leave.
The better place to seek emotional fulfillment and validation is at home with your family.
Wouldn't it be more efficient if Twitter just served as a broker for connecting scrapers directly to bots, so they didn't have to pay the bandwidth charges?
You say that in a tone of contradiction, but sycophants are absolutely trying to earn their pay. They've just decided that their true job is what their context rewards.
Nothing magic about it. When you fire all the people who contradict you, you create yes men and a culture of sycophancy. At that point it's self-perpetuating.
Twitter had many issues as a company, but one of its strengths was a lot of independent thinking and honest speech among staff. That's surely gone, and once gone it's very hard to get back.
> the effect Tweets easy embeddability had on reporting
This is something I don't get. They say that Twitter is now blocking embeds and I've seen an author respond by... including a screenshot of a tweet in their piece.
Except that's obviously better than embedding a tweet in the first place. It's better in every possible way. It's easier to write your article with an included image that you provide than to hotlink content from some other website. And when the tweet vanishes into the mists of history, your captured image of it will still be around, illustrating your article the way you were hoping the embedded tweet would do. There are so many articles out there with dead embedded tweets illustrating... something.
So, all that said... what was the effect that embeddability had on reporting? Why hasn't it been screenshots the whole time? What did easy embeddability enable that wasn't just as easy anyway?
It's just another requirement.
I don't see anything terrible or even bad with the requirement itself. Rate limiting is the new normal, in a post ChatGPT world.
The implementation, of course, is atrocious and should be corrected.
You are putting your opinion over the requirements of the job, and basically guaranteeing a bad outcome and an atrocious implementation.
Maybe you can share your stories about real bad decisions, so we can judge properly, but so far this is not a good overview.
On the other hand, some unions (mine) sign contracts that say they will not strike and do whatever they can to prevent one.
I haven't witnessed what happens if the members were determined to anyway, or read up on the circumstances of past wildcat strikes.
Also, the last time I saw something about my local hospital nurses' union in the news, they were alleging that the recruiting and employment of foreign nurses violated human trafficking laws. Basically, they signed people up from another country with a huge penalty if they quit early as "compensation" for the cost of importation.
To me, that strongly suggested the union was not playing a strong hand, when it comes to "unsafe policies".
Those people often have a better idea how to run "their" company (and are more interested in keeping the company alive) than the latest celebrity CEO who's then replaced with the next one after six months.
Unions are generally predicated on representing their members, but there are countless permutations of nuance where such representation would also benefit or protect the company, at least to some extent.
The board of directors, rich shareholders and the C-Suite team. Incidentally they also employ powerless servants who are interchangeable and disposable.
Modern Western capitalism is pretty similar to the Soviet Union or China's.
"State capitalism" and "monopoly capitalism" (the bad kind according to socialists) are about as different as Coke and Pepsi.
Ultimately the "owners" in practice are people you never heard of that work for (government) pension funds and index funds. The billionaires, founders, and celebrities are mostly a sideshow. That's not a denial of wealth inequality, just as officially socialist countries have.
But nearly all big companies have no owner in a top hat running the show, just faceless committees, responsible for buying everything - that company and all its competitors. Bureaucrats, apparatchiks, public or private, it's much the same.
If the C-suite and directors of huge companies are generally parasites, it's because robber barons and corporate raiders are the exception today. People whose job is to invest trillions of dollars by simply buying everything for sale have the least control imaginable over company managers. The current situation reflects classic criticisms of socialism, but it won anyway.
Nearly every source of information on public company stocks in the US has a figure for "institutional ownership". If a company is of any size, and is a real business, this is usually a high % - this is a reality check if what I'm writing sounds outlandish because nobody talks about anything but Twitter around here.
"As radical theorists like Michael Albert were already pointing out in the 1970s, this is the key flaw of traditional socialism: actual members of the working classes have no immediate hatred for capitalists because they never meet them; in most circumstances, the immediate face of oppression comes in the form of managers, supervisors, bureaucrats, and educated professionals of one sort or another—that is, precisely the people to whom a state socialist regime would give more power, rather than less (Albert and Hahnel 1979; Albert 2003). The decisive victory of capitalism in the 1980s and 1990s, ironically, has had precisely the same effect. It has led to both a continual inflation of what are often purely make-work managerial and administrative positions—“bullshit jobs”—and an endless bureaucratization of daily life, driven, in large part, by the Internet."
I disagree with every part of this quote and the whole "bullshit jobs" thesis.
People think they know what a job is, without having done it.
You cannot imagine even, say, stocking shelves at a retail store, without doing it.
If, for the sake of argument, most jobs are bullshit, the only way that can be maintained is if people are ignorant of what other people do.
But if you're so ignorant, you should consider that you can't imagine jobs you don't do, let alone do them.
One job I had I suppose was bullshit, was sweeping a fairly spotless warehouse, because, I was told, a division boss was coming to inspect, and so they hired a temp to look busy.
But a job is not bullshit, just because you have a flight of fancy that involves massive restructuring of an organization or society. "Writing this CRUD application should be unnecessary because everybody should've used the same database in the beginning". So...make a time machine, or get everyone using one of them to switch. Should be roughly comparable difficulty.
And I'm pretty sure that "the key flaw of traditional socialism" is not that they recognized reality. I think it was Lenin who declared that there would have to be a temporary phase of socialism. It's a sick mind that thinks the "key flaw" of existent socialism was employing educated administrators and not, say, dekulakization.
A key flaw in your thinking is that the "bullshit jobs" label that Graeber talks about was not assigned by outside observers, but by the people doing those jobs themselves. The point was that a huge number of people believe their own job is bullshit, not that some arbiter of truth believes that others' jobs are bullshit.
> a huge number of people believe their own job is bullshit
That's no more likely to be true. In order to correctly understand the context of our jobs, we would have to understand other jobs we don't and never will do.
Note that this paper was written in 2014, three years before the book "Bullshit Jobs" was published. I wasn't really trying to bring out the whole book into discussion (which has noticable flaws and details left out.) I was just trying to point out similarities between your comments and Graeber's, that's all.
But since you've begun a knee-jerk reaction against this, I think there are some flaws with your criticisms (assuming you have read the book):
> People think they know what a job is, without having done it.
> But if you're so ignorant, you should consider that you can't imagine jobs you don't do, let alone do them.
What Graeber has done in the book is to actually do numerous interviews with the people who have actually claimed to have done these jobs, and then categorize them into some noticable patterns to arrive at a conclusion. If you can't experience every job in the universe, the closest you can get is to talk with the people who have done them - and this is what he's precisely did. There are claims that the sample size wasn't enough or it was biased - which I think is totally apt. But it's incredibly dismissive of you to describe this attempt as "ignorant": how are we supposed to do any anthropological / sociological work in a large scale when you claim "no scholar can even try to analyze various types of work without actually doing everything in-person beforehand?"
> But a job is not bullshit, just because you have a flight of fancy that involves massive restructuring of an organization or society. "Writing this CRUD application should be unnecessary because everybody should've used the same database in the beginning". So...make a time machine, or get everyone using one of them to switch. Should be roughly comparable difficulty.
I think the "duck-tapers" Graeber describes in his book are a bit different from what you understand currently. He's mostly talking about the people who are doing tedious cleanup work because of reasons that can obviously and trivially be fixed but the higher-ups in the organization are not doing it for various reasons (mostly politics).
>I think the "duck-tapers" Graeber describes in his book are a bit different from what you understand currently. He's mostly talking about the people who are doing tedious cleanup work because of reasons that can obviously and trivially be fixed but the higher-ups in the organization are not doing it for various reasons (mostly politics).
No, no different - this sounds like every bit the same phenomenon I thought I was addressing.
It's a fake distinction because every job can be framed in a way that puts it on either side of the divide.
And lack of understanding of other peoples' jobs is clearly at the core of the issue.
The "higher ups being stupid because of politics" can never be really definitely false, but it never, ever, is an explanation that shows understanding or justifies calling something trivial.
You know who thought he knew what was a "bullshit job"? Paul Bremer, that's who. That worked out well.
Edit: Maybe the clusterf*ck was Rumsfeld's idea.
"all public sector employees affiliated with the Ba'ath Party were to be removed from their positions and be banned from any future employment in the public sector... When the CPA turned over enforcement of de-Ba'athification to Iraqi politicians, however, these rules were broadly expanded and used to punish political opponents, including nearly 11,000 teachers who were dismissed from the party and removed from government"
Very bold claim. In the history of the world like you say, unions shaped our world in a good way. Certainly not all unions, but it literally saved workers life from corporate abuse. Check the Asbestos Corporation in 1950s. https://en.m.wikipedia.org/wiki/Asbestos_strike.
Unions are the reason we have a minimum wage, limited work week, paid holidays, equal pay for equal work, unemployment benefits, workers' comp, the family medical leave act, and many more things. If not for unions we would all still be doing sixty hour weeks getting paid in company scrip.
"Socialism" brought us Social Security, Medicare, child labor laws and agriculture subsidies, health insurance, food assistance, housing subsidies, energy and utilities subsidies, and education and childcare assistance. And it brings the rest of the modern world free education and healthcare.
You can disagree with workplace organizing or the political positions of socialism, but this type of generalization is anti-intellectual, lazy thinking, and just wrong.
Although yes. I don’t think Twitter unionizing will save the site.
I generally agree with you and do believe that having the ability to just quit a job is a privilege (regardless of what industry you’re in) for the reasons you mentioned (other reasons include your age (harder to get a job if you’re over 40 b/c of ageism), family responsibilities, part of the country you live in) BUT I do want to push back a little bit on current Twitter employees just a bit.
It is absolutely true that for some people, they don’t have a choice in just quitting because of healthcare or H1B stuff. That’s absolutely true. But we're now at the 8 month mark of Elon buying Twitter. If you truly feel like it is a toxic and untenable place to work, you have had opportunities to explore other options. That doesn’t mean I judge or fault people who have chosen to stay, but this wasn’t January when there were tons of layoffs and uncertainty. Things have calmed down and while job offers aren’t falling off trees like they were in 2021, it isn’t the complete end of the world either.
People do have some agency eight months in. Again, I won’t judge anyone for staying for whatever the reason is, but let’s not pretend as if every person remaining (H1B or not) is void of all agency.
Sure, if you are unskilled worker that's struggling from paycheck to paycheck, then you might not be able to easily change work.
But in our line of work, and ESPECIALLY if you are good enough to work at something like twitter, pretending we are force to work where we are is a cope.
Sure, maybe there are some extreme instances when it is the case, there are always exceptions.
But on average, it's cringe when people who earn literally twice what the average population does talk about "job insecurity" and pretend they are in the same boat as trully struggling lower class worker.
I don't agree or disagree (well that's not true, but it's not particularly important), but I sure hope this trend of dismissing everything as a "cope" like some middleschool kid sitting up the back of the classroom doesn't survive '23.
Listened to some of that twitter spaces the other day about the Russian coup and discovered that there are grown adults using that word to label just about anything that doesn't fit with their stance.
Parent comment specifically mentioned H1B visas as the problem. It doesn't have much to do with how good you are. You can't just choose any other job. You need to find an employer who has a H1B visa Program support. While I'm sure that there are employers who offer this support and still have some quotas available, the choice is very limited in comparison with the whole line of work you are referring to, plus in this case the leverage that the employee has is very little so many will choose to stay and endure but not as a preference but just to avoid the problems.
You're confusing "forced" to mean legal slavery and "forced" to mean that the worker would have to make very significant and unpleasant changes to their life that a reasonable person would say that they shouldn't. Nobody is trying to imply slavery.
There's a whole host of behaviors that are legally unacceptable in the workplace because telling people to just get another job isn't really a solution.
No one is claiming Musk is forcing these people to work at Twitter. The claim is some people are forced by the visa system to choose between (a) working at a place they do not like and would not endorse, (b) leaving the country, or (c) taking a huge risk that they might have to do (b) anyway.
The point is not to say that Twitter or Musk are benefitting from forced labor. The point is to say that we shouldn't think everyone working at Twitter is is endorsing Twitter's decisions or enjoying the workplace as is.
H1-Bs are not an "extreme instance", to they are incredibly common. At my previous company I knew a excellent engineer who was treated poorly and paid way less than he deserved. He had no choice though because loosing the job means he's kicked out of the country. Obviously he was grateful for the opportunity, but there's a flip side to it as well.
That engineer could work remotely collecting $400k from a nice and cheap place these days. Imagine working from Costa Rica or somewhere in the Andes. If they were really excellent they would have landed such opportunity at some point.
> But on average, it's cringe when people who earn literally twice what the average population does talk about "job insecurity" and pretend they are in the same boat as trully struggling lower class worker.
Because the Hedonic treadmill exists, expenses are also higher for high-earning people than the average population's: why would you expect well-paid folk not to worry about being unable to afford paying for their car note, mortgage, child-care and other obligations. Your position would be reasonable if all costs were uniform, and people who earn double put all their money in savings and investments.
There's significant paperwork and delays involved in switching one's H1B permit to the new employer. For that reason, the companies you can potentially move to are limited to those that have internal processes for handling H1B-based employment.
I’ve personally (and successfully) brought a H1B employee from a BigCo into my 15 person startup (we were actually 7 people at the time of hiring them).
There’s some paperwork, but it’s not a major hurdle. Maybe a few grand in legal fees and a few hours of work on the employer side.
and your personal experience shows what?
just go to any job anncouncment website and see how many specifically say that Visa Support is not available (thus requiring US Residence) and how many say it is.
the Ratio from a glance is maybe 1/100 has a Visa Program.
It also depends on where they are in H1B process. These days perm and i140 approvals are taking ages(2 years). Basically engineer is stuck while they come out at the other side.
Also a lot of employers who had layoffs aren't sponsoring new Green cards because u know...
partially, but even then the point still stands, altough then we adjust the pros and cons of staying at your workplace.
But still, that's no way near comperable to working class who, if they loose their current job, won't have just the inconvenience of flying back home, but will actually loose their (already below standard) day-to-day nessecities
When you’re on H1-B you’re applying for the green card through your employer, which is a multi-year process. So even if you find an employer who will sponsor it, you’ll be going back to the end of the line.
Only after labor and then I-140 have been approved, which together typically take more than a year. Also, many employers won’t start your green card application until you’ve been there for a year.
I’ve done the paperwork myself for a H1B transfer as an employer. Maybe my memory isn’t serving me well, but I don’t remember it taking more than a few weeks to a month. (This was 5 yrs ago maybe my experience is dated). I definitely remember paying additional fees to prioritize every application which significantly sped up everything.
To be clear: You are saying that if you are working in the US, having already been granted an H1-B visa, it takes more than a year to complete an H1-B transfer?
Or are you talking about green cards, which is a completely different thing?
You replied to my comment, in which I’m explicitly talking about the green card process. Most people on H1-B are trying to get a green card.
I thought you were talking about transferring the green card process, because I was talking about the green card process. Transfer is possible, but you have to be past a certain point that takes probably a couple of years, which is pretty good compared to how long I had to wait way back when.
Thank you for clarifying. You initially responded to my point about H1-B visas with a different issue with a different immigration process. It is very informative but has nothing to do with H1-B holders as a group categorically.
According to some recent articles from mainstream media:
There are about 550 full-time engineers at Twitter. [1]
There are about 625-670 H1-B holders working at Twitter.[2]
This isn't quite the full story since of course H1-B holders might be doing other jobs than engineering; many of those 550 engineers are not US-based; there are other visas for foreign workers in the USA than the H1-B; and Musk has reassigned some workers from his other companies to work at Twitter.
The entire H1B situation is bad for everyone in tech. Because it's significantly harder for H1B holders to change jobs, they have a harder negotiation position and thus reducer upward pressure for comp.
Even worse they can’t disagree as easily with bad ideas from leadership. A healthy tech organization needs to make decisions based on negotiations between what is wished to be and what is; making a significant portion of the people who know the ground truth of things be reluctant to speak up when that ground truth is inconvenient can make for some pretty bad decisions.
Universal healthcare isn't a hard requirement here. At a minimum we having more affordable medical insurance that isn't tied to employment would be a huge help.
Whoever survived multiple internal knockout round of reviews would definitely have high value in job marketplace. Believe it or not many people like working in current twitter like arrangement rather than cushy no impact 9-5 jobs.
I agree about healthcare. Employees in the United States over-perform because we can't risk losing health insurance and ending up in financial ruin.
H1-B, now, as much as I am sympathetic to that (an immigrant coming from extreme poverty myself), I don't understand why HN treats losing a work visa as a death sentence.
The tragedy here is what exactly? That you get to go home to your family and a country with a safety net and free healthcare, as bad as it might be? Before you apply for a new job? Is that really worse than some narcissistic rich douche shaving the years off your life by working you into the ground like this?
If you’re a software engineer anywhere in the US (with exception of H1B/TN/etc), where you work is 100% your choice. There isn’t a shortage of open engineering roles in this country. Nearly 100% of full time positions include fully paid healthcare.
In the age of widespread layoffs at big tech, it's not quite so easy. And you really can't begin to predict life situations for people. For example, those who grew up in poverty and use the majority of their take-home pay to send money to family.
I still don’t understand. The monthly HN who’s hiring thread literally has many hundreds of job openings every month. And if you cast the net wider you’ll find thousands of open & remote engineering positions.
bro i got 3.5 YOE and ive been applying in vain for the past couple of months. in my last job search i applied to ~ 10 positions and got 4 interviews...
Are you applying to HN type startups or "regular" companies?
Do you have a problem with lowering your standards, at least temporarily? Non-SWE positions? Or are you like "I'm overqualified, I won't apply for that..."
Based solely on my linkedin inbox, that's not currently true. A year ago I would get ~10-12 recruiters pitching me jobs a month. For the last three months that's down to exactly 1 each month.
This might be kind of true for a software engineer, but not for most people. Especially since if you quit or get fired, and can't find another job, you have no health insurance.
This is simply not true. Anyone can go onto Healthcare.gov and sign up for insurance. If you are low income because you got laid off, you'll probably get a pretty good tax credit to help pay for it. You pay even qualify for Medicaid.
Last time I checked, layoffs are still happening. You have two choices: 1) stay at this current job even though it is stupid and unpredictable 2) Voluntarily give up this job and compete with thousands if not tens of thousands of very skilled engineers whe have just been layed off for positions. Easy choice, right?
In fact, I'd recommend you do some job hunting right now and get an offer to prove your point.
It is always amazing that some people can be so out of touch and speak to people in this tone.
There are a lot of laws in the US that force me to do something. Twitter cannot force me to do anything. The only thing any of my employers could force me to do was leave the premises if they didn't want me anymore.
As for Visa issues, your complaint should be with the US government, not Twitter.
Consider a situation where I want to invite someone to visit me on my property, but some government also claims jurisdiction over my property and decides that person is not allowed to visit my property. Whose property is it again?
Healthcare expenses are a huge fraction of a person's "income" but it's not usually counted part of their top line compensation number.
So paying COBRA might be 20% of your prior income, at the same time as losing all your income and still having to pay your rent and all other expenses.
> Have you actually been in a position where you needed to rely on COBRA?
Yes. I used it for 18 months, then bought an insurance policy. That lasted until Obamacare forced that insurer to leave the state, and now I am on Obamacare.
> But I can damn well believe there are Twitter employees who have made all the right moves, yet still hesitate to leave their jobs right now.
I can believe it, too.
Long ago, the company I was working on encountered hard times, and did an across the board 10% pay cut. One of my coworkers was furiously grumbling about it, so I asked him why he didn't just quit (there are plenty of tech jobs around Seattle). He said his finances were so tight he couldn't afford to miss even one paycheck. He had a McMansion, his & hers new cars, expensive furniture, and expensive clothes.
He was in a cage he forged for himself, the company didn't do it to him.
Jobs in the private sector are not guaranteed. It's best to save a portion of each paycheck. I bought stock with my first job, and drove beaters for a long time (and still do). My last beater I drove for 30 years, until it finally expired.
Indeed, it was stellar... I worked for a large national telescope facility!
Also, it was a good plan but more than we could carry indefinitely. My health collapsed and we were in the second year of trying to figure out why. So we needed to carry it for a while, but were able to switch to a different arrangement when long-term medical leave came into effect some months later.
A lot of H1-Bs are from India. They can easily move back and get very well paying jobs back home. I would even say that in terms of purchasing power, they’ll be better off in India - salaries of $80-100k are common for pedigreed tech talent, and your expenses won’t be more than $20-25k.
So what is your threshold of classing a behavior as excerting force? Using whiplashes and locks? I suspect that even then you would find an argument to bootlick.
Exactly. It is the THREAT of execution that exerts all of the force. After you are executed you probably don't care anymore. But I am not an after-life expert.
I’m not sure if you are just being pedantic or intentionally obtuse. Of course no one is being physically forced to work at Twitter. But you surely recognize that some people have limited viable choices not to.
While having the choice to quit, uproot the entire family and leave the country you’ve called home for many years is a choice technically, it’s not a choice most people are in a position to realistically make.
Honestly your attitude here and in other comments over the years has turned me off using D forever. No one in power in the Rust community would be able to get away with the sorts of comments you're making.
(The Rust community has its own very serious issues but no one would even try to say "boo hoo if they don't like it they can just uproot their lives and go back to a place they specifically got away from, nbd")
> they can just uproot their lives and go back to a place they specifically got away from, nbd
What I said was they were not forced to stay. I've read a number of history books, and what forced labor is is very clear. Besides, when one comes to the US on a Visa one chooses to accept the terms, and they do not include a guarantee of lifetime employment.
What I actually believe in is freedom and individual rights, and my posts are consistent with that. At least I hope so!
As for D, it is Boost licensed. It is the most free license I could find. You can do whatever you like with the code, as long as the copyright is included. (I would have made it public domain, but that isn't recognized in many countries.)
Each member of the D community has their own reasons for contributing, and many (myself included) do not receive any compensation. It's just our gift to you and everyone else.
It is your right to use it, or not, as you please.
P.S. The D Foundation also discourages politics of any sort in the discussion forums. I don't even know what the politics of the other contributors are. It's not relevant.
Considering that you are so pedantic about definitions, in which dictionary does "forced to stay" equal to "forced labor"?
More generally your interpretation of "forced" seems to be extremely narrow. How would you interpret a statement like "today at work I was forced to attend two completely useless meetings", which IMO is perfectly common use of the word "forced", but certainly does not imply that security was standing in front of the door monitoring who goes in and out.
I'm not being weird about it. Just calling out the idea that Musk is forcing people to work for him is hyperbolic. If you want to call Musk mean, arrogant, uncaring, bad, etc., that's all fine.
Wikipedia's definition is not official anything. It's a commonly accepted (and rather expansive) definition of the phrase. I did not invent my own personal definition of it.
Seems you've decided that instead of adding value to the discussion by saying anything of interest, you're just quibbling about word definitions. Pretty lazy?
Every decision made by anybody in life arises from a mix of choice, circumstances and coercion [1]. The relative proportions of each depend on the person and the decision. What's often described as "privilege" really is the ability to make decisions with less coercion and more choice than other people in similar circumstances.
If someone is making a decision primarily based on coercion—and having to uproot your life absolutely is coercion—saying that "no actually, it's a choice" is unkind. In 2013 Tumblr terms, I implore you to check your privilege.
[1] I've stolen this rubric from the sex work community, which is at the leading edge not just in infosec but also in the philosophy of labor.
Is there any reason why you chose to capitalize 'Visa' (sic) in your comment? I've read a number of policies and documents and this is the first instance I've seen it represented in that way. Given your own reading history, I'm sure that there is a good reason - we wouldn't want to be incorrect and inaccurate, of course. For all we know, you could be talking about the company, or the credit card!
Ooh, good idea. People can go build something against the Twitter API! Oh wait!
I'm absolutely going to keep enjoying Musk's comeuppance. In this, consider me a traditionalist. As the Greek tragedies show, people have been enjoying the just punishment of hubris for thousands of years.
You can relish in the idea of Twitter having technical issues as if it hurts Elon in some tangible way, but the fact is he could pull the plug on it tomorrow… simply shut Twitter down entirely. Take the entire value of the company as a personal loss, and give everyone working there $1m for their service.
And still be the wealthiest individual on the planet.
He could do it another 5 times and still be a billionaire.
Yes, I’m aware his net worth doesn’t mean liquid assets but the point remains.
This is actually not true. His wealth is propped up by trust. If everyone starts to believe that he's incompetent, stock prices will go down and loans will be more expensive. Also, he bought twitter by taking debt against his other assets.
No one can lose 40B and come out unscathed, not even Musk.
And honestly, that's the part that I've enjoyed the most. Musk's faux "Tony Stark" shtick was always PR. But it was widely believed PR, and it gave him an incredibly low cost of capital and oceans of free marketing.
But basically from the moment he tried to buy Twitter, he's been ruining that reputation. It started with the business press, who have more of an incentive to see through the BS, and who quickly pointed out that it was a clown show of a takeover even before the Delaware courts forced him to buy it. That alone put a major dent in Tesla's stock price.
Now everybody has gotten to see how bad he is at running Twitter. Most of us didn't know shit about launching satellites, but a lot more people understood Twitter. By objective measures, things have gone terribly, and there have been waves and waves of stories showing that Musk is not half the genius he was painted as. Which has caused a lot of questions being asked about all of his businesses.
So now we're entering an age of electric cars, with global intent to switch. But Tesla, the nominal leader, has a stock price down 40% from its peak, and with widespread questions about how well it can really do. And part of that is definitely brand damage from Musk; I recently saw a Tesla with a bumper sticker that said, "When we bought it, we didn't know he was an asshole." Surely that has something to do with weakening sales despite previously unheard-of discounts.
Maybe Musk will get it together, and mainly for the sake of others, I hope so. But if I had to bet, it would bet that his launching a bid for Twitter will be seen as a turning point where he began a long slide into irrelevance.
Would Musk really lose $40 billion (or is it $46b), if he dumps Twitter? He could sell it to someone, Microsoft for example, at a loss e.g. $25b or $30b, which covers the loans, and write off the rest.
That said, his historical pattern is not to sell out so quickly, but rather to keep iterating until he and his team get it right.
And that was a few months ago. I'd argue that one of Twitter's most valuable assets, the blue checkmark, has been irrevocably destroyed. And if I were looking at buying it, I'd also be concerned that the technical infrastructure is a significantly impaired asset, possibly one of negative value.
I’m pretty sure those are supposed to be moral lessons, not misery porn. But hey, you seem to openly accept the idea of enjoying another’s suffering, and I admire that kind of honesty. I hope it doesn’t end up causing you misery.
I don't think you're quite getting the moral lesson in telling stories about gods punishing hubris, so let me spell it out: titanic arrogance is bad and comes with consequences, so don't do it, kids. The point is for people to see particular kinds of suffering as both appropriate and necessary.
I agree that people enjoying suffering generally is pretty bad, and I avoid it. But I make an exception for people who cause suffering receiving the just consequences of their terrible actions. If Musk were just some guy who were living a quiet life and then got cancer or something, I'd feel for him. But he's not. He's someone who's been aggressively a jerk for more than a decade.
His deciding to take on being a CEO of yet another company, one he didn't really understand except as a user, was bad enough. But as a former Twitter user and employee, I've had to watch him destroy something that a lot of people, me included, worked very hard on. I've had to watch him harm thousands of people quite directly, and many more indirectly. And all for nothing more than his arrogance and some half-baked clout-chasing notions.
So I am absolutely going to enjoy the fuck out of his serial rake-stepping. Could this be bad for me? Well that's a question I will take seriously from somebody in saffron robes that spends 10 hours a day meditating. But it's not one I'll entertain from the Volunteer Musk Defense Brigade, and especially not from one of its members who was busy performing lack of empathy recently on this very site: https://news.ycombinator.com/item?id=35824824
Yes, pride -> fall, I get it. Although it's the celebration of the fall from the sidelines that I'm questioning. I can feel the spite in your words--towards me, towards Musk, and the "other side", which are a faceless group you call the Volunteer Musk Defense Brigade, of which I'm apparently an unknowing member, I guess because I don't think some technical glitch is that big of a deal. You seem to feel lots of spite for people you've never met. I think there are plenty of tragedies written of people destroyed by their spite--maybe Notes from the Underground qualifies.
And in an irony of ironies, you crawl through my post history and take objection to me expressing my lack of empathy for those that make repeated bad financial decisions, but you're not only defending that very sentiment, but taking it a step further: enjoyment of the suffering. I hope you find peace, brother. I'm going to put my robes back on and get those last few meditation hours in now.
You're not questioning it. You're objecting to it. If you were actually questioning, you would have engaged with the fact that stories about hubris justly punished have been a staple of moral education for literal millennia.
But you're only objecting to it when it's somebody you like. When it's poor people, you're fine with trumpeting your imagined moral superiority. But when it's a famous rich guy, you're out here doing free PR on his behalf. You did not "go build something" then, and you're unable to even reflect on your blatant double standard.
You have no standing to moralize at me. And just to be clear, what I feel for you is not spite, but contempt, because you're not only a hypocrite, but you're using what might be otherwise valid moral points to derail discussion of one of the world's biggest jerks. Motes and beams, buddy.
"Forced" could also mean holding someone's visa hostage, threatening consequences in addition to losing a job, or just in general being in a situation where, while they are not physically or legally required to keep a job, they have no other viable options.
Can we please recognize that this is a holiday weekend and Elon rammed a big release forcing his engineers to come in and iterate multiple patches over the last 12 hours.
The arrogance of the guy is so huge, hubris does not touch him. Thoughts are with the programmers.
What I’m more surprised about is how gum and shoestring the twitter engineering is now a days. They put in no emphasis on doing deep divides into the code base and instead opt to do the simplest shortest fix. And it causes problems.
From experience management really don't care about the state of the code or infrastructure. All they want to know is the soonest something can be done.
Elon strikes me as worse because he likes to think he understands what his engineers know.
> Elon strikes me as worse because he likes to think he understands
> what his engineers know.
Early on in my career I had a name for this. I called it "gamle helter" in Norwegian which roughly translates to "old heroes". An "old hero" is someone who used to be competent in a field, has stopped being competent, doesn't recognize this themselves and is now a nuisance to anyone who actually knows what they are doing, but can't pull rank. One way to become an old hero is typically to end up in management and not practice whatever discipline you think you understand.
To be fair, I highly doubt that Musk was ever a competent software engineer, much less a good engineering manager. He is a PR person. He sells an image that he is a technology person.
I'm not seeing the imposter side of things personally but maybe he does come across like the old addage : dumb people think they know everything, smart people know they know nothing. Or something to that effect.
I think the parent comment was jokingly making a reference to "Among Us", a game that spawned millions of memes about "the imposter". But my assumption could be wrong.
Almost everyone is under (the folk version of) the Dunning-Kruger effect about the Dunning-Kruger effect.
Since you linked Wikipedia, I'll quote it.
> Nevertheless, low performers' self-assessment is lower than that of high performers.
> Among laypeople, the Dunning–Kruger effect is often misunderstood as the claim that people with low intelligence are more confident in their knowledge and skills than people with high intelligence.
Didn’t wanna imply that management care for the state of the code with my original comment. That would be a bit delusional. Sorry if that caused confusion.
What I wanted to say is that if the state of the code is bad, it might make them care a bit more when their own state gets synchronized with that. Not necessarily about the code, but surely about the revenue and the looks.
Now, management being what it is, they of course will try to find a scapegoat. But that’s just part of the game and may be better than burning yourself out by trying to fix their nonsense.
Often the simplest shortest fix is the result of management’s directives. “We need this thing done yesterday” leads to the shortest fix, which compounds over time. Then the same managers wonder why it’s so hard to make changes or implement things in the future, without understanding their insistence on taking the path of least resistance is the direct cause.
Twitter Engineers have had over a year to get out and find another job. At this point the terms of their employment and the expectations from their boss are all crystal clear. Hard to sympathize with those who continue to stick around for whatever reason.
Someone pointed out that it looks like most of the workers that stayed at Twitter appear to be foreign born and are probably reliant on Twitter's sponsorship of their H1-B visa.
How many Twitter engineers are on H-1B? 10%? 50%? 90%? Unless someone can find this number out for real, why is the default argument "they are all being held captive and cannot leave"? Would you apply the same logic to Tesla and SpaceX, who have the same working conditions? Here's a more reasonable take - engineers believe in the product they are building and are happy with the salary they get.
Facebook was reportedly ~15% H1B workers. It doesn't take a lot of skew in who was fired and who didn't leave for other employment for Twitter to be >50% H1B at the moment.
Can sponsored H1-B visa holders look for another sponsorship to try to switch jobs, or would they get deported after resigning from "hardcore Twitter"?
I agree that this is very much a management problem. Incompetent management, to add. I'm just curious why would these engineers stick for so long. Everything I read about Twitter nowadays suggest a highly toxic work environment. And from experience: having an unstable boss is tiring, to say the least.
They can look for other jobs but have to be in a new job within 60 days of leaving twitter with all the paperwork etc, given the market I would expect it is not as easy as say a year back even.
It is easier with other tech companies because unlike Musk they are/were generous severance time so those probably got 90+ days in severance time and the 60 days with say FAANG layoffs.
Also switching jobs can set them back on their path to permanent residency application timelines , depending on the country of citizenship that could be adding some years to continue on H1B and this dependency cycle so people close to their PR may try to stick it out
No one gets deported per se, most undocumented immigrants are people who fly over and overstay their visas, the USCIS will not check anything when they leave (by design so no one worries about leaving) however if they overstay the visa, it will be a tough time entering again , very likely to be rejected at the border or new visa application would be denied.
Without knowing much about the people I opt to approach this with a bit more empathy. Things are shitty for lots of people. Now is not the time to judge people.
...if you flip burgers or change sheets in a hotel, sure. Maybe read about people finding engineering positions, understand that section of the job market and realize how out of touch you are.
Agreed (assuming we’re talking about the US here). I’m not a rockstar dev but I have a lot of years of solid experience. I applied to 60+ programming jobs over the last four months without getting even a single technical interview. Every previous job search I’ve done over the last 20 years went much better so I don’t think it’s (mostly) me.
If they truly cannot get another job despite trying hard for 15 months then really they should be thanking Musk for giving them a steady paycheck when no one else in the industry will.
> If they truly cannot get another job despite trying hard for 15 months then really they should be thanking Musk for giving them a steady paycheck when no one else in the industry will.
That's an asshole take.
Everything I've read is that Twitter is working its remaining engineers to the bone. Job searches can be extremely draining, and I can imagine many Twitter engineers don't have the energy remaining. Add to that mix visa and immigration issues, and I could totally see people getting trapped there.
Devs should be thankful not to Musk but to previous management who sponsored them visas.
This is what I think happened:
- few companies are willing to go through paperwork to sponsor visas. Old Twitter was, Musk Twitter probably isn't
- Musk inherited a bunch of visa-sponsored devs from old Twitter, and they will have trouble finding a new h1b sponsor to transfer
- those are the devs that cannot negotiate better salaries or leave easily
Basically Musk's Twitter's lights are on thanks to those employees. And whether they are good or not doesn't matter, they can't leave either way, job market is hostile
If I understand correctly, the U.S. doesn’t allow them to get another job without getting a new H-1B and there are dramatically more applicants than visas each year. They are unlikely to get a new one so their options are to either stay at Twitter or leave the U.S.
You understand wrong. If you already have an H-1B visa then you don't need to enter the lottery again. You just need another job offer in a similar role, and can start working immediately.
What you said is true - a new lottery is not needed. However before offering a job to a H1B candidate the company should demonstrate that it advertised this job to the general public and no non H1B candidate (a US citizen or permanent resident) applied or was eligible. In the current job market, there are tons of citizens and permanent residents in the job market making it difficult if not impossible for a H1B holder to get a new job.
The original purpose of the H1B system was exactly for those rare individuals that have some X skill(s) that can't be met by the domestic job market. The implication being that practically zero entry to intermediate level jobs would have H1Bs filling them, even in SV.
Though the expected norm nowadays is for it to be a cheaper second-rate labour supply restricted by years long waiting lists and lotteries, and not a smooth pipeline of geniuses and super-geniuses interested in emigrating to the US,
Abso-lutely not. Check out Reddit to look for accounts of how the transfer works and see how easy it really is...
When you transfer, you need a lot of cooperation from the new firm as they have to be ok filing paperwork, they have to do it in a timely fashion, and due to processing delays on the order of weeks (if all goes well) you cannot start work immediately. For many, many employers, this puts you at a strong disadvantage against applicants who are all good to go and ready to show up for work on Monday.
> You understand wrong. If you already have an H-1B visa then you don't need to enter the lottery again. You just need another job offer in a similar role, and can start working immediately.
The other employer still needs to be willing to sponsor you, and that's often not the case.
Can you provide a source on this? It sounds like you know what you're talking about, but this is the internet and I'm not in a position to tell if you're mistaken or not.
The actual insulting part is telling a group of skilled individuals who are willingly working a job and getting a good salary that they are "victims" and helpless and abused. Maybe let them do their thing, and you do yours. There's no need to be outraged on behalf of anyone else.
I mean, that's not an "against Twitter" complaint, that's an "against everything in the way the world works" complaint.
I think it's a pretty wrong complaint too, but for sure that's not on Twitter.
(And I really don't think Twitter employees are the best case study here, they are mostly capable of not literally being homeless if they work somewhere else.)
Yes as in, why is it normalised that you would expect to pushed too hard in many work environments, and it be on you to deal with this? Elon Musk is just a poster child for this kind of institutionalised bullying
As in my OP here, if you've set up your life around working at Twitter and then a new CEO rocks up and demands everyone 'go hard or go home', then sure you can go home and if you don't like his approach then eventually you'll have to. There will be people who find it more difficult to change jobs for whatever reason, and while they need to take responsibility for getting themselves out of a bad situation, it's not their fault that a new bully turned up in a position of authority on their block
The activist mindset is not part of professionalism. If people chose to stick with their jobs, it is their professionalism. And if people chose not to leave, it is a sacrifice they made.
Any professional worth his salt would do the same.
Looking for a new job requires time and energy (especially at the moment) and these are scarce resources when you've been un constant crunch for the past 9 months…
Its the weekend before the 4th of July, the independence day for USA - many employers give employees Monday and Tuesday off, creating a potential four day break during what is typically very good weather - that being said Twitter is presumably (still) an international company with people in all different holiday areas.
I searched "what holiday is it this weekend" on kagi and found nothing. I don't know what country GP is in and I haven't heard about any public holidays this weekend.
The "America is the default" is tiring. The US makes up ~ 5% of the population, and most of us do not default to American thinking.
I generally agree with your sentiment, but Twitter is a US company with most of their employees residing in the US (at least presumably?). So in this case it’s reasonable to assume they meant it’s a holiday in the US.
As someone not from the US, it’s not a surprise at all. But if someone says a foreign company is making their employees work on a holiday weekend, I don’t fault them for not explaining what specific holiday it is.
I guarantee you well over 50% of this website is American or lives currently in America. Twitter is also an American company so it doesn’t take a rocket scientist to connect the dots and understand that the commenter is referring to the US.
I haven't generally used sentence-style queries wth any search engine since AskJeeves. Paitly because AskJeeves was terrible at finding what I was looking for, and partly because it almost always produces poor results on every other search engine I've known.
I just don't expect natural-language queries to work (yet). I've also watched too many people struggle to get "Ok Google, …" to return usable results; seems the majority of the time it fails or would have been faster to type out.
I still don't have a feel for Kagi. I used it occasionally when they had a free tier, but was always afraid my habit of submitting multiple revised queries to search engines would use up the free allotment. The current sample search for headphones was somewhat helpful recently, though "best x" is not a search pattern I tend to use, and even though I tend to search for 'phones with flatter response than most.
> I still don't have a feel for Kagi. I used it occasionally when they had a free tier, but was always afraid my habit of submitting multiple revised queries to search engines would use up the free allotment.
I have the base plan (1400 queries a month I believe) and regularly revise queries or put in things I should use something else for (e.g "current time UTC"). I'm yet to exceed the included query quantity.
not everything is obvious to everyone all the time. what's wrong with people asking questions?
to other parts of the world american customs are weird/foreign, for instance regarding getting a day off.
You seem to be positing that there are two choices:
1. Work for a toxic dictator who makes bad, micro-managed technical decisions and then expects his engineers to work a holiday weekend to clean up his mess.
2. Never ship code, ever.
I'm pleased to report that most software engineers exist happily somewhere in between these two extremes, and I would recommend you find a new job if you don't.
Canada just changed their immigration program such that they'll accept all H1B holders (working or not) into Canada with a similar visa, but without the requirement to find a job immediately.
Something to look into for any Twitter (or other) engineers who are bound to a company due to lack of options.
This bug is very unlikely to be the reason. The rate limiter on the server side is cheap and the frontend bug only gets triggered with the rate limit active.
I have seen similar bugs in the systems I oversee because network libraries love to retry requests without sane limitations by default. But I never saw them make our rate limiters sweat. It's slightly more annoying when they hit an API which actually does some expensive work before returning an error but that's why we have rate limits on all public endpoints.
I also guess that the webapp is the least of Twitters traffic and the native apps probably don't have this problem.
I don’t think it’s necessarily saying the self-inflicted DDoS has caused a technical issue that’s forced them to shut down access. I think it’s possible that shutting down anonymous access caused the DDoS, which led to giant spikes in some metric, which led them (Elon) to conclude that there was an uptick in scraping, so they imposed the 600/tweet/day limit to punish scrapers.
Seems like either my quota reset or they changed the policy because I’m able to access the site again.
This. I'd bet substantial amounts of money that the evil scraper idea is the result of a) another issue + b) paranoia + c) Musk thinking he understands better than anybody else.
This is a really ignorant take to dismiss scrapers. LLMs operate by having petabytes of conversational training data. Scraping is how OpenAI trained GPT. It’s how all their copycats are trying to do the same.
Elon can be a monumental asshat, and he can be self-DDOS’ing, and can be accurate about scraping at the same time. It’s why every single social media platform is heading toward becoming a walled garden.
I'm not denying that scrapers exist, I'm just highly suspicious of this explanation given that:
a) he's proven time and time again how willing he is to say shit just to get attention
b) he doesn't seem to understand software very well
c) if shit was imploding for reasons related to decisions he made, this is precisely the kind of blame externalization I would expect.
Yeah, scrapers have always existed and while their traffic is undoubtedly higher than it has been in the past, it can't possibly be any significant amount of traffic when compared to the rest of the traffic hitting the site.
A real scraper would be stopped by a rate limit set to, like, 100 tweets/minute. 600 tweets/day is a completely pointless, punitive limit.
I'm guessing you've never played an offensive or defensive role in scraping because what you've described is in no way a problem for a serious scraping effort. I agree the rate limits are stupid. They fuck over users, they stop amateur scrapers, and do nothing whatsoever to impede professional scraping.
If you want to stop most scraping, employ device attestation techniques and TLS fingerprinting.
But then you have to contend with this: https://github.com/bogdanfinn/tls-client... Just used this to bypass a Cloudflare check! I've never scraped Twitter but Elon said there was a large scraping operation from Oracle IPs. He could substantially raise the cost of scraping by just banning datacenter IPs. Something like p0f would probably help too. I pay for static residential proxies (basically servers running squid that somehow have IPs belonging to consumer ISPs) and with TCP fingerprinting these would be detected as Linux and expose my Windows or iPhone user-agents as inconsistent but I've never encountered a site that checks this. Although maybe sites are doing so silently but I don't notice because I don't otherwise meet the bot threshold.
for sure, using a custom TLS library like uTLS helps -- need to inject that GREASE cipher selection. I have a suspicion that private residential proxies are out of budget for many outfits, or the IPs are too few and then simple rate limiting kicks in? Who do you use if you're willing to share? I've not been happy with the, uhh, questionable ethics of Luminati/BrightData in the past.
There are definitely more and more sites doing TLS/TCP/etc fingerprinting or device attestation for mobile APIs, but it's still pretty rare. I mean Twitter is trying to limit requests by IP, so definitely amateur hour over there.
I use https://www.pingproxies.com/isp which is like $3/IP/month and unlimited bandwidth (I assume if you used a ridiculous amount they might charge you). Luminati pricing is extortionate. I have no idea how anyone doing anything at scale can afford $10/GB. I haven't investigated but I don't know if Twitter limits are per account or per IP.
I agree, but there are hundreds if not thousands of AI startups trying to make their own relevant LLM, and they're going to be scraping Twitter. The Onion called it many years ago [1]: "400 billion tweets and not one useful bit of data was ever transmitted".
According to Elon, shutting down anonymous access was itself an emergency measure to deal with the DDoS[0]. Twitter did increase the quota significantly from earlier today[1].
[0] "Temporary emergency measure. We were getting data pillaged so much that it was degrading service for normal users!" (in reference to Twitter requiring users to be logged in)
[1] "Now to 10k, 1k & 0.5k" (in reference to rate limits which were originally 6K 0.6K and 0.3K)
> This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping.
> Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data.
> It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation.
One thing about having leadership that is known to lie about anything or everything, for any sort of imagined personal gain, is that the very concept of truth is destroyed.
I agree that this is probably not the bug at the root of it all. But I also don't believe the story that Musk is selling for why he's in effect shutting down the site. But both could be true and I'm still thinking about other potential reasons, a complete waste of my time, but it's a weird mental honeypot.
The book "Nothing is true and everything is possible" describes Putin's use of misinformation to maintain control of the populace and eliminate democratic types of politics, but it really feels like it applies here too. There will always be Musk fanbois who will parrot whatever he wants them to say, but most know it's just self-serving BS. And anybody trying to get to the root of everything gets easily sidetracked into narratives that feel right but have zero data backing them, like this bug.
Anyway, highly recommend this book if you want to see a likely path for the future of the US:
That’s exactly it. Musk a week ago was telling us that there were a record number of user seconds on the site. Now he’s telling us they’re all content-scrapers. The very concept of truth is eroded.
That's what makes it so perfect when he echos conspiracy theories, Russian talking points, and other whacky talking points and elevates untrustworthy sources by tweeting things like "interesting" catapulting them into everyone's feed.
The substantiation is Musk's own words, easily visible to all.
Part of the "nothing is true" aspect is to make people start to doubt their own eyes. Let's not fall in honesty to the point where Musk's own Twitter timeline is not proof of what he is saying.
To assume good faith in the face of a mountain of evidence to the contrary would be intellectually dishonest, something that is still, fortunately, held in disregard on this site.
Same. Up until that moment he was still somewhat laudable, if weird, at least for casual observers. Then he labeled that guy a pedophile and it’s been pretty much all downhill from there.
The media is flawed. Journalists are flawed. The Verge in particular has some atrocious coverage of tech.
But the problem with Elon is that in his world, we must rely on him for true information. As if the direct source is inherently true. He’s just as full of it as anyone else with interests.
Man, the really annoying thing is that Elon-haters are so dishonest (refusing to engage when obviously wrong claims are made, repeating things that have been long disproven, repeating hearsay as fact, etc), that it's hard to actually identify any actual lies Elon has made. Meaning, not just exaggerations or failed predictions, but straight up intentional lies.
My favorite is when things are claimed to be impossible, that Elon's lying because the thing is impossible, but then it happens anyway.
Where Elon gets in trouble is he's wildly over-optimistic on a few things, such as AI. He predicted an AGI would take over in 5 years around 2015 or so (so we're 3 years off), and I think he really believed it. That's why he's always claiming things like self-driving, and doing it without sensors or whatever. His paranoia of AGI and his over-confidence on self-driving have exactly the same root cause (believing AI will conquer all). Elon has had so many instances of overcoming status quo expert predictions (whether on solar energy, battery-electric vehicles, reusable launch vehicles, or whathaveyou), that I think it makes him increasingly unable to very effectively listen to experts.
And he's also incredibly gullible and easily taken in by all sorts of scammers, including rightists and just plain sycophants telling him what they think he wants to hear. Which is increasingly what he's left with as everyone else who is sick of his bull has left.
He has always been a rightists. He just dumped money on some cool stuff like self driving electric cars and rockets. Which is really cool, of course. But he wasn't "left with no other options". That's a lame excuse.
I know this is getting too political so I'll deserve the downvotes, but totally ignoring anything else, this is why I fundamentally could not and cannot stand Donald Trump. This is his playbook too, just constant willful dishonesty and distortion.
> One thing about having leadership that is known to lie about anything or everything, for any sort of imagined persona gain
I will note that the few times I investigated claims of Elon lies they were not proper lies, either being misunderstood, misleading (which IS unethical, don't get me wrong), of indeterminate truth value (he said, she said type stuff), delusional optimism or actually true.
Like journalists, Musk rarely outright knowingly makes literally false statements, but this does not mean you should take what he says at face value.
I think it's an important distinction to make. Because it does means that you can actually infer true things from his statements (or a journalists statements, as news orgs do this kind of "lying" a lot), unlike with a habitual liar.
I don't agree. There is little to learn from Musk's statements. And lumping all of journalism as being as dishonest as Musk is just plain silly. Sure, if you're on the right the media which is biased towards yoi has been serving you dishonesty and lies for a long time. But that's not journalism as a whole.
Saying "he really believes his lies" is not excuse, because most habitual liars are exactly the same. They have prioritized their own narcissism so far above reality, but that doesn't make them any less of a liar. They are just lying to themselves and everyone else.
There's a very well known case of Musk making the same claim multiple years in a row, with it being false every single time.
I prefer to assume the best out of people, but when someone is that obviously wrong that many times with that much personal gain to come from it, I can't believe that man as smart as he is would also be that misled.
This is the guy who still thinks a Martian colony is a good idea and that the woke mind virus is one of the most dangerous threats to society. I'm not sure he's very grounded in reality at all when it comes to the future.
Depends on the scale of the overall system. I have personally seen and attempted to mitigate degenerate cases where these retries overwhelmed the backend so much that the servers were falling behind in simply rejecting the requests.
Infact it got so bad because of all those retries at multiple levels from upstream callers that requests were essentially timing out at the TCP buffer/queue before they could be processed by the application.
Don’t know if the Twitter homepage backend is at similar scale.
It is unlikely that a system with the scale of Twitter implements the API rate limiter in the backend. Usually you'd do this as early as possible together with other WAF stuff.
If IPs or IP ranges get really annoying we block them on the network level.
Big public sites like Twitter obviously need to have this technology. Due to their political content they probably also need sophisticated DDoS protection.
Judging from the screenshot, a huge amount of GET /TweetDetail is generated which triggers some rate limiting, as shown by the 429.
If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream.
Also, this behavior seem to never stop, which isn't what one would expect from an exponential backoff retry.
I don't claim to be a better engineer than the folks working at Twitter, but it is interesting to see something like this in the wild, all Musk-related considerations aside.
While exponential backoff is theoretically optimal, I doubt it's actually used that often in practice. I've seen too many cases where someone decides serving user requests with low latency is so important that they'd rather have a constant randomized backoff than exponential backoff. I've been in many design meetings and seen enough documents where the decision not to use exponential backoff is explicitly made, understanding the tradeoff with overloading and system recovery.
I’ve had to… uhh… eagerly advocate for exponential backoff for weeks of constant uptime issues before someone listened and actually implemented it and solved the problems.
Like several times in different roles.
People do it, exponential backoff is everywhere in your stack, but it doesn’t end up in your application layer until you have enough traffic that you actually have to manage throughout.
I would guess the front end was written under the assumption that the back end would still work without auth. Perhaps the backend changes (mandatory auth + rate limiting) were pushed without sufficient testing of the front + back?
I mean, when spaceman has been bad over and over and over you can’t be all that surprised when people see something go wrong and say “probably spaceman bad”.
While it looks like they never started the move over to AWS, the press release makes it sound like they do use some AWS services.
> In addition, Twitter will continue to use AWS services such as Amazon CloudFront (AWS’s fast content delivery network service that securely delivers data, videos, applications, and APIs with low latency and high transfer speeds to customers globally) and Amazon DynamoDB (AWS’s key-value database that delivers single-digit millisecond performance at any scale).
I worked there. Services running on GCP are a significant part of the internal service infra (ml platform, etc.) and it's not impossible that the abrupt loss of GCP would cause user-facing problems. The GCP spend was many, many times the AWS spend. Unless things changed since last November, AWS is not a meaningful part of the internal or user-facing infra.
Correct, no more recent (or less public) info than that. Like I say, losing GCP could cause problems users notice, but sounds like that’s not going to happen.
And yet they also host with AWS, Google Cloud and Oracle. Cloud people take note: this is what lock-in looks like, and it's coming to a place near you.
> If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream.
The way I understand it, DDoS is not caused by enforced authentication - enforced authentication is just a temporary measure against DDoS.
> If you missed the initial news of the impending showdown, Platformer reported on June 10th that Twitter had been refusing to pay Google for its cloud services ahead of their contract’s June 30th renewal date.
> Twitter’s Google Cloud contract dates back to 2018.
As the article says, and as Bloomberg, etc have also reported. Twitter finally agreed to play nice with Google and the issue was resolved. They are probably trying to get off of GCP, but this isn't because they suddenly lost access to GCP from payment refusal.
The sourcing on that story is a single unnamed person whereas both companies refused to comment. Seems like there could be more to the story than "Twitter started paying again and everything's fine."
It's an interesting theory but the DDoS predates the decision to disable anonymous access. In fact, that decision was taken in order to mitigate an ongoing DDoS[0][1]. So while it's possible that the questionnable web front end retry logic is not helping things, it's not the root cause.
> This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping.
> Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data.
> It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation.
Honestly it probably has more to do with the decision to basically shut down API access without exorbitant fees than it has to do with AI startups or whatever.
Surely there are some colossal archives of all tweets before whatever date floating around that those startups are using instead.
It seems more plausible than the alternative. Let's say that Twitter availability issues were really caused by disabling anonymous access and that Elon wants to back out of his decision due to public backlash, wouldn't he just reverse the decision? That would fix both of his issues at once. He could even claim to have solved the scraping issues through a clever trick that doesn't require disabling anonymous access. The fact he hasn't done so indicates that there is a genuine issue that he can't just fix by restoring anonymous access.
The theory is more that Elon wants to prevent public access to Twitter in hopes of making it more profitable, but he doesn't like the backlash to such cash grabs and has invented this story which makes the Auth requirement and non-paying account view quotas more palatable.
It's important to note that Twitter's desires are fundamentally at odds with the desires of the content creators on Twitter on this point. Twitter, ideally, wants to only show it's tweet to real people who will click on an ad, since it costs money to show tweets and bots or people who don't look at ads are not paying that money back. Creators on Twitter want their tweet to be shown to anyone at all, and will leave if they feel people just aren't seeing their tweets.
So, Twitter must walk a fine line when trying to restrict who can see their tweets, lest they alienate content creators. Inventing a common foe is a tried-and-true tactic for getting people to accept changes that hurt them.
My initial contention was directed towards the claim that Twitter was accidentally DDoSing itself. However, it seems you are proposing another theory - that there is no DDoS at all and Twitter is "faking" its availability issues to introduce quotas and a login requirement. While I personally perceive this as veering into conspiracy theory territory, I concede that it isn't entirely beyond the realm of possibility.
1. Twitter decides to go account only for monetization reasons, and implements this feature with some bugs.
2. People start complaining about the new Twitter policy.
3. Availability issues start being observed, caused by the bugs in the implementation of 1 - possibly a self-DDoS.
4. Elon, in response to 2, lies about 1 being a temporary response to an external pre-existing DDoS attack. People start associating 3 with the claimed DDoS attack.
5. In a hasty attempt to fix 3, whose exact cause they have not yet determined, Twitter starts implementing stringent view quotas. Since 3 was not caused by an external DDoS attack, this actually only makes the problem worse.
I'm not claiming this is definitely what happened, just that it is a plausible time-line of events. The one Elon presented is also plausible, of course.
The best question that could help us distinguish these two cases from the outside is whether Twitter's availability issues started being observed before or after the authentication policy change. I would say that Elon's claim is far less likely to be true if the noticeable availability issues only appeared after the policy change. Conversely, my version of events (which is more or less the same as TFA's, I think) is far less believable if the availability issues happened before the Auth change.
I will note that I didn't use Twitter at all in the last few days and thus don't know which is the case. On HN, I definitely saw the Auth policy change story at least a day before seeing significant complaints about other availability issues, though.
Anybody knows if these requests were happening before the login only change? Because it would be hilarious if huge scrapping operation was a bug in their javascript.
I've noticed the frontend hammering the backend quite often in the past few weeks. It would not surprise me at all to learn that the "influx of scraping" was mostly Twitter's fault.
I can say for sure that a certain flow ("back" from Profile view or similar) would trigger an infinite redirecting loop on Firefox on my Android device, with probably dozens of requests over a couple of seconds until rate limiting kicked in. Maybe there were many of these little bugs which together looked like some kind of DDOS or scraping.
Seems like it might not even be a bug. Elon says they limited it to 600 viewed tweets per day which is an insane limit. Most people would go through that in 5 minutes of scrolling
So many things are useless, but still entertaining in one way or another.
And then Twitter is used for "relevant" information as well, from emergency or transport authorities using it for quick information instead of having their own system to news of any kind or discussions and information exchange in different peer groups.
Considering that they count all tweets being loaded towards the quota (for instance replies to a tweet, if you open the detail page) and they like to shove tweets from Musk and others in the timelines the quota can be reached quite quickly even on considerate use.
>And then Twitter is used for "relevant" information as well, from emergency or transport authorities using it for quick information instead of having their own system to news of any kind or discussions and information exchange in different peer groups.
I really don't think it's wise to be putting so many eggs in one basket, convenient as that might be. This right here is a strong example of that.
5 minutes is maybe an exaggeration. I know that when Tweetbot reported I had >500 in my feed I had enough nonsense to read for a couple of 20 minute tram rides.
I noticed the frontend hammering the backend for the past few weeks, so I suspect that these new rate limits are a response to that, even if Musk wouldn't publicly admit it.
I don't doubt that Twitter saw a massive increase in traffic recently, but I feel at least somewhat confident that it's mostly self-inflicted on Twitter's part.
I have an alternative theory for this. Once Elon was forced by the DE court to buy Twitter, he went to autocrats and dictators with money and told them he'd burn Twitter to the ground on their behalf for 44B.
No Twitter, no Arab Spring. No live updates of disasters, no need to shutdown the internet to prevent dissenting voices from spreading.
Are we doing theories? Mine is that Peter Thiel sold Musk on buying Twitter by convincing him of his ability to make it profitable with his genius, knowing full well he'd sink the thing in 18 months, and another pesky tool would be off the table for awhile.
That theory is pure speculation on the basis of motive with no actual grounding in reality, no offense. There's not a shred of evidence that GlobalAutocracy™ banded together to cough up $50B to kill of Twitter...
>To address extreme levels of data scraping & system manipulation, we’ve applied the following temporary limits:
>
>- Verified accounts are limited to reading 6000 posts/day
>- Unverified accounts to 600 posts/day
>- New unverified accounts to 300/day
I have an idea. We should create a scaping system that scrapes all tweets and makes them availabe so that people are able to see what other people think they are linking to when they link to a broken Twitter page. /s
Either that or people should stop linking to broken pages.
Yeah, between the rate limit, and the algorithmic weighting toward "blue" users, there's very little incentive to interacting with high-follower accounts at the moment.
I hold that as an instrumental belief in trying to use twitter well. Just follow <20k accounts who are near me in-person, or in social-graph. It feels more like a group chat app, and less like the weight of the world, advertising/thread-vertizing hole, and I avoid much of the convenient dunks or quote-hate-fest.
Its funny how the entirety of silicon valley just forgot that Twitter was infamous for their unholy mess of a codebase which at some points looked like it had a secret mission to increase world entropy and terraform the planet.
I still remember watching podcasts, reading blogposts, hearing prominent VC's like Elad talking about etc.. Now everbody silent because Elon is the baba yaga...
The difference is that Twitter was not arrogant about it. Saying they where hardcore engineers with insane working hours and other bollocks. Musk dislike comes from him being extremely dickish about other peoples work while being an absolute clusterfuck of a manager.
Twitter was not bad because of its tech. Musk made it worse.
To go further, its crazy how much cognitive dissonance can penetrate even the smartest among us.. If you had some understanding of how modern systems at twitters scale works, you'd know such a bug is highly unlikely.
You have multiple clients (android, ios, web) and a whole cluster of microservices calling each other. The microservices calling each other will easily use up more requests.
IMO the likely culprit is rate limits being triggered by a huge increase in scraping caused by the new api pricing.
Not quite reading material, but look up George Hotz's videos about it. I think there's one Twitter Space he hosted / recorded where Musk was also in there.
Hotz is very vocal and blunt about the messy codebase situation at Twitter.
Apparently their 5-year cloud contract failed to renew today
So, there may be a much simpler explanation for why their new rate limits on regular users to can-barely-scroll levels, and how that has all sorts of unintended consequences they weren't ready for
Legally no, you are a customer if you have a contract .
same way we are a tenant( and have rights) even if you don’t pay rent.
Disputes are resolved via arbitration and courts , no payment does not give GCP ( or any vendor) unilateral right to terminate (the contract however possibly might ) .
Usually vendor has to give notice of termination and there would be a time window before termination can occur .
Unpaid dues are like unpaid mortgage there are many steps before the vendor can get a court to force the party to make payment or seize assets in lieu
Yep, less about fully shutting off and more about negotiation. This can be Musk doing a direct cut on some of their biggest web traffic to show that they can optimize it down. Or same, but internally, telling internal engineers they only have so much budget.
Putting the personality of the owner aside, they have been running an experiment for the last few months on how to operate a tech company, overworking your engineers and treating them like replaceable automata. Do you want them to succeed and set that as an example for others to follow?
Yep. And it's hard to separate the personality of the owner from this—the more successful Musk is, the more that he's seen as the next Steve Jobs, the higher my odds of ending up working for a Musk wannabe. He is not a good leader, and the sooner his leadership style blows up in his face the better for all of us.
Every social network and large tech company has awful people running it? And I'm yet to see anything beyond "an anonymous source" on this apparent overworking.
If you work for a large US tech company, you are replacable automata. How do people not know this? Am I the crazy one here?
Personally, not just twitter, all social media I’m happy to see it dies, and no, “stop using it” mindset isn’t an argument when it severely affects the majority of the people around you, and start not just influencing the populace opinions but shaping the next generation. Who ever controls that narrative in the media get to decide what’s the next laws to be considered, after all, that’s how democracy works.
Some people signed up for Twitter, years ago, specifically to avoid other people using their name. I think it was a thread on HN.
Once you are signed up, it's tempting to use it. Then you realize it's a bad idea to use it, but it takes self-control.
Now, with the current leadership, who knows if they will purge inactive accounts and/or reissue IDs.
I don't know if this scenario accounts for much of the antagonism, but it is one possibility. It is possible that some people don't like it, but can't let go of it completely, and the new management creates uncertainty and doubt that never ends.
I've got a rule of thumb, if a communication network is so large that governments who control nuclear arms use it to communicate "just ignore it" doesn't really work
The person I replied to claimed that Musk is racist without providing any evidence.
Unless you disagree that everyone holds unintentional racist views because of implicit bias (https://www.bbc.com/news/magazine-40124781) I am curious why Musk is being singled out as being more racist than the average person.
I think people’s actions speak louder than words, and unless he is also intentionally funding institutions or causes that actively perpetuate racism or racist systems, I’m not convinced of the purity of intention behind the accusation.
If he is such a big racist, why would the Biden administration contract SpaceX? What judgment would you pass on them?
So I’ll repeat again, because I want to educate myself: what is the evidence of Musk’s racism?
The major tech companies outsource stuff that matters - I mean, for 20 years at least - to contracted outside companies who pay 1/10th the amount to Americans in "flyover" country and even less to offshore employees.
They do this because SWE prima donnas will not do an adequate job for some things no matter how much you pay them. Like, oh, say, cleaning the office microwave.
So I guarantee you that you are overlooking most of the actual work "at" any given large tech corp.
Source: working for a BPO company with offices in Appalachia, Manila, and Uttar Pradesh, among others.
> Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon's latest genius innovation is to block people from being able to read Twitter without logging in.
It seems an outlandish claim, but then again Muskified Twitter has previous form for this kind of thing with that time when they self-derailed by locking themselves out of their own api, right?[1]
I definitely agree. I do think the major issue at Twitter is more managerial than engineering though. They likely could have run Twitter at the same output with the current team. The problem is a CEO who is learning on the job by altering basically everything at a superfast pace.
The root cause is business & feature experimentation at scale with a tight runway & no executive oversight.
I mean, I'll give you that, I did chuckle. Frankly, this whole thing is a huge debacle and I can only imagine the context switch penalty being paid going from cars, to ISP, to rockets to Twitter. That said, I personally am astounded by Elon's tenacity. I would definitely not bet against it succeeding, but im not sure I'd bet on its success. If I had to choose though, I would bet on success at like 49/51 odds
He may be tenacious, or at least has been in the past. And TBH I hope that Twitter succeeds, but it's becoming pretty clear that my definition of success is different than his.
This could explain why my computer fan goes into overdrive when trying to read Twitter, has been going on for a few months but I don't care enough to figure it what's going on in the bg
This kind of stuff is unavoidable with what's going on at Twitter. Infrastructure changes, platform changes and mass layoffs all at once. I'm actually impressed they haven't experienced more and longer outages.
Its a testament to how well designed and implemented the code base is, if the wheels havent started falling off yet (shockingly). I know we’re all supposed to shit on Twitter, but they had world class engineers working there before the mass layoffs and brain drain.
I can easily visualize a wooden wagon rolling along a rocky slow descent forever and all of the reasons its wheels would eventually break loose.
In the case of a platform like Twitter, aside from random acts of god and weather, aside from money, what are the sort of stuff that slowly breaks its wheels off over time such that, without staff, it inevitably stops working completely?
The ostensible cause for the rate limiting was “excessive scraping”. Battling stuff like this or e.g. spam bots is a constant arms race. So, without staff, the bad actors will eventually win.
True, the rate of improvements has been insane. I use it pretty often and haven't even noticed a second of unavailability even today and right now, not to say I use it 24/7 though. It's pretty obvious some people have a political bend to their hatred of Twitter so any second something isn't perfect they jump on it. It's really a sad state of affairs really.
What I find more ironic than this bug is is that all of these Mastodon evangelists still have Twitter accounts after their Elon meltdown. So much so that they even use it after Twitter stopped public access.
I’ve been off Twitter since naughty ol’ mr car started his war on the journalists (I did return briefly to make a silly joke when he briefly banned mention of the dread term ‘mastodon’). I’ve kept the account, because, well, I’ve had it for over 15 years (back when they had a visible auto-incrementing numeric ID it was six figures), and maybe at some point he’ll lose interest and Twitter will return to semi-competent (it was never great) management. I suspect a lot of people are in the same boat.
But also, yes, I use it to observe the latest meltdown; why shouldn’t I?
Jitter is a little randomness in how long clients wait between retries. It ensures that you don't have a "thundering herd" all retrying at the same time. Imagine if your API used exponential backoff of [1s, 2s, 4s, 8s, ...] and a large group of requests gets a retryable error at t=0. They will all retry at exactly t=1, t=2, etc. If the group is large enough that repeated surge of requests can knock you offline.
There's nasty form of this where the site is offline for a bit and then all the clients rush their requests in when it comes back online. The client requests are all coordinated on the site recovery time and end up overloading the site with their coordinated retries.
I enjoyed your comment a lot. Interesting to think that abstract structures like a load balanced webserver have a simil to the fundamental frequency observer in physical structure
Say you have a bug that caused 100,000 HTTP requests to hang, and you kick the node and make them all fail at once. One second later, 100,000 clients suddenly retry simultaneously, causing a huge spike in load which makes most of their requests fail. They use exponential backoff, so two seconds after that, 99,000 clients retry, causing a huge spike in load that makes most of their requests fail. Four seconds after that, 98,000 clients retry...
If you introduce a bit of randomness into the retry timing (say, multiply by 1.8~2.2 instead of a straight doubling), that thundering herd will spread itself out and be much easier to recover from.
I doubt it ever would become the standard unless everyone was using third party libraries that forced it in some way, most likely opaque by default which would cause plenty of devs headaches, right?
The easiest path will always be the default for the majority of devs, with a simple "timer" type solution being the easiest to implement in pretty much all cases except where otherwise it's literally forced on them.
There may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
I'm astounded it took this long to finally die. All those Twitter engineers who were harassed by Elmo's very online fan club, and were treated with such cruelty and disdain on their way out, clearly did an incredible job. Hats off to them.
The curse of IT Operations: if you do everything right, management grifters never think you did anything at all. Why do I need you? Everything works fine!
Most Internet platforms are put together by popsicle sticks and bubble gum. If Reddit didn't have 2000 full time employees babysitting that steaming POS, it would be offline by the end of the week. But Twitter was like a chicken running around with its head chopped off, and it didn't stop running for 9 months. It's a testament to how incredible those engineers were.
This collapse at Twitter should have happened by January. It's a real shame it didn't. How many thousands of tech workers were laid off because Musk's fellow parasitic oligarchs saw Twitter running "fine" after the lobotomy, and followed him off the cliff like a bunch of lemmings? How many billions of dollars has the economy lost because of this one despicable man?
Elon Musk is a fraud. He is not an engineer. He's a lazy bum who mooched his whole life off his groomer daddy's apartheid emerald money. He would be nothing without that disgusting, incestuous old man. Despite his enormous financial privilege, Elmo was too lazy to enter the US legally. He was in the US for many years as an illegal immigrant. He only bothered to get off his lazy ass and finish his Economics degree because the risk of his deportation was becoming a real problem for X.com. That crooked philistine doesn't know the first thing about hard work, engineering, or finance. He's a damn good conman, he's good at shitposting, and that's about it. He has no other skills. He is not an engineer. He has no STEM degree. His only real accomplishment in life is proving that you can, in fact, spend billions of dollars in one lifetime, if you simply buy a bird app, run it into the ground, and salt the earth behind it.
Further, while I share some criticism of the man, many very technical people from companies such as SpaceX and Tesla have come forward with public comments, praising him for "truly grasping the engineering" and "being involved in every technical design decision". Make of these what you will:
Kevin Watson, Falcon 9 avionics:
Elon is brilliant. He’s involved in just about everything. He understands everything. If he asks you a question, you learn very quickly not to go give him a gut reaction.
He wants answers that get down to the fundamental laws of physics. One thing he understands really well is the physics of the rockets. He understands that like nobody else. The stuff I have seen him do in his head is crazy.
He can get in discussions about flying a satellite and whether we can make the right orbit and deliver Dragon at the same time and solve all these equations in real time. It’s amazing to watch the amount of knowledge he has accumulated over the years.
Tom Mueller, SpaceX founding employee:
We’ll have, you know, a group of people sitting in a room, making a key decision. And everybody in that room will say, you know, basically, “We need to turn left,” and Elon will say “No, we’re gonna turn right.” You know, to put it in a metaphor. And that’s how he thinks. He’s like, “You guys are taking the easy way out; we need to take the hard way.”*
And, uh, I’ve seen that hurt us before, I’ve seen that fail, but I’ve also seen— where nobody thought it would work— it was the right decision. It was the harder way to do it, but in the end, it was the right thing.*
Garrett Reisman, engineer and former NASA astronaut:
What's really remarkable to me is the breadth of his knowledge. I mean I've met a lot of super super smart people but they're usually super super smart on one thing and he's able to have conversations with our top engineers about the software, and the most arcane aspects of that and then he'll turn to our manufacturing engineers and have discussions about some really esoteric welding process for some crazy alloy and he'll just go back and forth and his ability to do that across the different technologies that go into rockets cars and everything else he does.
Josh Boehm, former Head of Software QA at SpaceX:
Elon is both the Chief Executive Officer and Chief Technology Officer of SpaceX, so of course he does more than just ‘some very technical work’. He is integrally involved in the actual design and engineering of the rocket, and at least touches every other aspect of the business (but I would say the former takes up much more of his mental real estate). Elon is an engineer at heart, and that’s where and how he works best.
Instead of judging Elon by quotes from people who might have something to gain by praising a narcissist, how about we judge Elon by the insanely dumb things he says and does on a near daily basis?
And you're right he has a degree, although he repeatedly lied about when he received it, and even lied about where he received it, so please forgive people for their confusion.
I do think this probably isn't the reason for rate-limiting tweets so harshly; I don't know for sure, but I suspect this isn't a technical issue and it is legitimately Elon snapping and just implementing a wildly bad business decision.
But that being said, a loop of users requesting access from Twitter's system is a thing that would burden Twitter's system.
Requesting access and denying access isn't free. You can definitely DDOS your own website by having all of your users repeatedly request access to it in a loop.
So this "unforeseen" loop now eats in the users 600 tweets per whatever, so now when you land and Twitter's homepage you are pass this measly limit with 5 rotations of your scroll wheel.
> There must have been an insane nest of logic that the devs had to tiptoe through to make the service workable as a private only service.
It seems that this statement can lead to different conclusions based on one's biases for or against Musk / freedom of speech:
1) The code base is a mess because the smaller, current team don't know what they're doing and/or management (i.e. Musk) is pushing them too hard and making poor decisions
2) The code base is a mess because the far greater number of engineers there before didn't know what they were doing and/or management weren't pushing them hard enough and were making poor decisions.
Both could be true but I'm going with something the more complex view that it's a little bit of everything, and that Twitter does seem to be moving in a positive direction overall.
Why do you say Twitter is moving in a positive direction? My experience is that I dislike their app and they killed the one I liked (Tweetbot), it's frequently down (I still get the "Something went wrong" error), and any kind of conversation is dead because blue-check replies (usually quite low-quality) are prioritised. There are other reasons to dislike today's Twitter that I needn't go into here, but these are enough to turn me away any time I decide to come back and check it out again.
A few reasons, the first being that I can actually log in again after 2 or 3 years, as the login was broken for me for that long so I can't say I was a fan of what was going on before. I had several frustrations before but that, as you can imagine, pissed me off quite a bit.
The second being free speech, I'm a big supporter of that and there's certainly more of that now. It's not perfect but I'm not going to throw the baby out with the bathwater by arguing against perfection or pointing out some of the hypocrisies in play. It's a move in a positive direction.
They're comparisons against recent Twitter, which before Musk felt like an app with so many broken parts that were never going to get fixed (the login problem being the cherry on that cake for me). Now there's a feeling that things might get fixed, that new features definitely are coming, and that complaints might be listened to.
I also don't like the changes you've highlighted, though I do see them as inevitable at this point in internet history. We see Instagram and others moving towards paid verification, Reddit closing down its API and killing apps (and should we forget that Twitter has done this at least once before in one form or another). So, I don't think it's all rosy in the garden, but overall it's positive, and the negatives (mostly) feel inevitable.
Add to that, alternatives are now getting a bit of a look in from people that wouldn't have have bothered before. I don't think that Mastodon, to choose one example, is a direct replacement, but it's good that people are giving it a try.
It sounds like the login problem and the free speech policy (which I think we won't agree on but that's by the by) are the main issues. But it's interesting we have completely alternate impressions how it was before/after Musk. After the initial growth explosion when Twitter launched (and a few encounters with the fail whale) it stabilised and always felt pretty reliable for me. Whereas now downtime is commonplace again, and it feels a bit self-inflicted. Now that the company has axed staff, stopped paying bills and is going to attempt a rapid migration off AWS, I don't have much hope that this will get better.
But to your last sentence - I ended up trying Mastodon, found a nice server, downloaded a familiar app (Ivory by TapBots, the team who made Tweetbot - RIP) and found a pretty good bunch of people to follow so I'm pretty happy there. I probably wouldn't have otherwise tried it had I been able to continue using Tweetbot. I'd be happier if both existed and I could bounce between them, but if Twitter melts into obscurity I'll be a little sad but it would be fine.
> It sounds like the login problem ... are the main issues.
I had a long list before that, off the top of my head:
- notifications were broken (disappearing or not appearing at all) which included DMs, or appearing late
- the follow/unfollow problem that so many complain(ed) about
- the algorithm dominating the main feed (the For you and Following is at least an improvement on this, if not a fix). Not really a Twitter only problem that one.
- lists was the old fix for the main feed problem, but they would go missing at times and set up was a pain
- clicking on "More replies" (or whatever it is/was) and they disappear/don't appear, sometimes on tweets I've already seen
- Let's not forget when they last killed off 3rd party apps en masse[1]
- changing settings often wouldn't work, and are still a UI mess today
Related to the freedom of speech thing because the ways it was implemented lead directly to them:
- shadow banning (I hate that everywhere, including the hellban here). Even searching a particular account was blocked, which is a pain when you're trying to find something you know exists.
- watching stats drop precipitously on tweets after Twitter had put their thumb on the scale. Shouting into the void isn't a fun experience. Watching interesting accounts wither also was a waste. Still plenty of trolls and bots though.
- trending topics being curated. Possibly Twitter's real USP is instant news, the trends can give an idea of what people are really talking about at a glance and get you straight into breaking news. Most of that was gone. They're worth a look again.
And several forgotten problems, I'm certain of it, I could've spent all day going through them, at the time. What a pile of steaming rubbish!
I often wondered if the first half of this list should go under the second half of this list because most of them appeared after Twitter became blatant in its ways, and some of them occur on other social networks too, but only on contentious topics (it amazes me how little is said about the heavy censorship Facebook engages in, must have a better PR department than Twitter. Maybe that's why Musk fired them all).
Then you find you can't log in often or at all. When I saw that Project Veritas video about Twitter, I believed it, because it made sense that no one was doing any real work.
This makes me realize "the code base is a mess" has been so overused it could be a standard greeting in our profession.
Everyone has their horror story and extreme examples, but for any mildly profitable company that went through a decade or more of operation, I think it's kinda of a meaningless statement.
At least if it's blocking me from viewing the tweet's comments, returns me some decent messages. But no, just a plain "Something went wrong." like a hackathon page.
When people say "Elon is a cult" this video is exactly what they mean. Elon does something incredibly dumb and you immediately have people just making up, whole cloth, fantastical reasons why it's a 5D chess master plan.
I was skeptical, but this was a really good watch. Pity that it’s a video.
For the to;dw crowd: he makes the case that Twitter is by far the most useful social network to government agencies, because the graph is mostly public (unlike Facebook) and all users are both consumers and creators (unlike YouTube). Thus, it is being used by “researchers” and state departments to build what he calls “the Death Star of AI censorship engines,” and these moves to rate limit will cripple those efforts.
Focusing on the technical issue at hand - can anyone tell the exact issue at play? Is this an issue of client retry logic that on a 429 without backoff? Hence, the self-ddos?
I made an account around the time ChatGPT API (GPT-3.5) access was released to follow all the LLM news. The account is now getting aggressively rate limited that I cannot even see new tweets for 2-3 minutes.
Certainly not good, but not all GET requests are equal. If these are responded to cheaply, at the point of connection termination, then it might be the case that no one has bothered to clean it up yet.
Not exactly sure if that's what you recommend. But connection termination is not necessarily a good thing for DDOS mitigation. The reason is that the client might just retry immediatly - and it will do that using a new TLS connection. And the handshake for that connection has a huge cost. If you plan on disconnecting clients *after* a TLS connection had been established, you will also need to implement TLS handshake rate and connection limiting. That's possible, but I've only seen a tiny amount of services every implementing it.
the irony of all the premature celebrations over the past 8 months. If you stick your head in the sand and pretend there are no problems, i guess there really arent any, until maybe there is?
I managed to get in without login few minutes ago and its still sending api request every 130ms and getting 429 Too Many Requests error, how is this possible? :)
wow. how would one even fix this without deliberate downtime? you'd have to deploy and hope that the frontend will make it through CDNs to reduce pressure, right?
At minimum, you revert the commit/deploy to prod that caused the issue. But then that would likely mean reverting the recent policies and would make Elon look weak, so he'd never support it.
You first remove rate limits, then implement and release exponential backoff on frontend, then apply rate limits again (on a small segment of users first, then more). No biggie, you just need to be very careful. And boss needs to chill for that time, which is unlikely to happen.
Yeah frontend retry DDoS is not a great situation to get in. I've tripped it in a test env before with a websocket app (erroneous retries caused certain clients to open the WS over and over and break the server).
First thing I would try is seeing if the front end has a different retry strategy for a different status code (say 503). If so I'd change the status returned for throttling to be that (503).
Barring that, turning off server side throttling or atleast making it less aggressive to slow the retry storm seems the most reasonable.
It's not a self ddos if twitter isn't going down. You can see in the video twitter is properly handling the load and is returning HTTP 429 when the client is sending too many requests. Hitting the rate limiter or requesting a post is light weight. It's not like it's spamming login requests which require a lot of resources due to key stretching.
The theory here is that this JavaScript bug caused the huge increase in requests which is why Twitter introduced strict limits on how many tweets users could access.
So the result wasn't an outage, it was a radical reduction in functionality.
I disagree. Ddos is a type of attack, not the result of an attack. If they're hitting their services way too many times in a distributed fashion, it's a ddos regardless of how it was handled.
Similarly, would you say this[0] wasn't a ddos because it was mitigated? I think not.
Self ddos is more fuzzy because there is no intent. If I wget all pages from a site's site map is that a dos? If it was architecture to handle that kind If load the site would be unaffected and if wasn't it could potentially cause availability problems. If after starting my download script I noticed the server couldn't handle my requests then I would recognize I am dosing them.
I agree. DoS is an attack, so without intent it is fuzzy. But I think it is pretty descriptive, so it's okay. I know exactly what self-ddos means instantly (flooding your own service, without malicious intent).
I think it's kind of a limitation with English or the term ddos. If it really is only used it for intent to attack, it becomes less useful of a word IMO.
I finally did it. I finally got rid of my Twitter account and migrated to Mastodon. I wish I had done it sooner because Mastodon is great. Particularly the Hachyderm.io server for the tech community <3
Also, taking Elon's word at face value for a second... is Twitter really worth scraping for AI training or whatever?
Its a hive of misinformation, disinformation and toxicity. Its succinct I guess, but nothing is eloquent or descriptive because of the character limit. And its full of repetitive "filler" information.
Who wants that in a foundational LLM dataset?
Maybe its OK for finding labeled images... But that still seems kidna iffy.
While there may be huge sections of Twitter content that are like what you describe, I haven't encountered that. Instead I see tons of hyper-focused discussion from very specialized scientists that I wouldn't see otherwise. I see lots of discussion if obscure housing policy, that I wouldn't see otherwise.
Now, this has been severely degraded by the changes that Musk has made. The spam in direct messages is off the charts now, whereas in the past I would get maybe a spam per year. And when one of my areas of interest has a post that gets popular, I have to scroll past all the insipid clout-chasing replies from blue check marks which get floated to the top of replies in an attempt to reward some of the worst people on the internet. Also the long form tweets that need to be expanded are a big deflation of user experience, as reading and replying to those are suboptimal compared to a tweet thread.
But this is also the general internet: 99% spam plus 1% quality. And the quality of the 1% of good Twitter is some of the very best of timer material out there.
And since LLMs have been trained on this same mix... they seem to be mostly good at filtering. But they do lie an awful lot.
As someone who doesnt use twitter, I dont understand how can you have any sort of a real discussion with a 140 character limit.
The best discussion platform is IMHO the older version of reddit / i.reddit with the nested comments + possibility to be indexed by google + possibility to reply to old posts. The super-nesting comments feature is great.
It's a 280 character per message limit, with replies.
This is actually hugely beneficial to discussion as it makes people focus on the most salient point first, and further points go below, and each are easy to address individually.
Longer form material goes to outside links, sometimes, but Twitter threads are also great for long form content. At least for executive summaries that link out to the detailed bits for each primary point. Once the UI for Twitter prioritized threading, it became quite easy to express extremely long chains of evidence.
Twitter threads seem awful for long form content. I have never seen long form content on Twitter that I could be sure I'd seen the way the author intended.
As a light user of Twitter (and not at all at the moment, i don't have an account) the character limit for tweets felt like a good thing.
The tweet threads are not terrible, but are inconvenient enough for people to be succinct as possible. Now there are walls of text from blue check marks that like the sound of their own voice far more than their content is insightful.
Sure I've read interesting long tweets, but I'd rather have a link to another site meant for long form writing than it living on Twitter, doubly so now as what bits of good content there were are behind a login wall.
But i get it, Elon needed something to make the blue check "worth it".
I would scroll through my timeline, but it is now impossible to show you the good content.
Often times the best posters are not the same people publishing the best stuff in their field, but sometimes they are. Aggregators are a different category.
What types of science are you interested in? Some random accounts that I can see right now:
@ShanuMathew93 - renewable energy tech and biz and news
@IdoTheThinking - California housing
@TheStalwart - finance, macroceconomics, microeconomics, etc.
I once got paid $20 as an undergrad to go through hundreds of thousands of tweets and convert slang into plain english for training data. The only thing I took away from the experience, aside from finally getting good with vim macros, is the average tweet is really low effort an uninteresting. I don't recall reading a single thing that I would imagine someone retweeting (think that's what it's called). Maybe I was given only replies. Anyway, not sure if there's value there for LLMs, but I'd be skeptical.
The effectiveness of this sort of lockdown is questionable anyway, because the cat's already out of the bag and there's no getting it back in. Same for Reddit. The bulk of the data's already out there and nothing these companies can do will change that.
It's useful if you want your LLM to be able to generate tweet-like microblogging text. That does have some value.
Or maybe you want to get an aggregate idea of what people are currently talking about in the world, stuff that doesn't rise to the level of capital-n News. There aren't a lot of alternatives for that.
The thing that LLMs bring to the table isn't factual knowledge — we already had that, even some AI projects specifically dedicated to that — but rather the ability to correctly interact with natural language.
Twitter is great for examples of that, and the toxicity and disinformation doesn't get in the way.
Conversely, a training set doesn't need to be up to date to be useful for that.
I don't know if anyone really was trying to scrape it (examples of Musk disagreeing with his own engineers come to mind), but I assume it's possible, and given the quality of code ChatGPT spits out I can easily believe a really bad scraper has been produced by someone who thought they could do without hiring a software developer. If so, they might think they can get hot stock tips or forewarning of a pandemic from which emoji people post or something — not really what an LLM is for, but loads of people (even here!) conflate all the different kinds of AI into one thing.
Maybe someone is trying to make a disinformation bot. (half-serious)
I mean as far as uses for LLMs go that seems to me a pretty realistic one. Mass quick propaganda with little effort. Go for immediate impact, doesn't matter if people look deeper, you're just looking to get a swell of emotional reactions.
It seems the majority of Twitter articles are about the poor quality of engineering solutions or Elon's inability to "pay his Google bill". It strikes me as propaganda when what we are truly witnessing is the rush to wall-garden the internet as AI LLM takes over.
That's not a scalability problem, that's a poor engineering problem, let's not start to find excuses
Talents left twitter already, I wouldn't be surprised if the ones that took over are the ones who come from the intelligence industry, as opposed to the tech industry
People commenting that people who are still at Twitter because of their own volition, just stop. Certainly some are still there because they want to be, but there are absolutely people trapped there economically one way or another, working for a tyrant.
What nonsense. Silicon Valley employees are some of the best paid in the world and have chosen to enter into a contract in a capitalist society. Even if any are "economically trapped" (they're not) and it's not of their own volition (it is), it's been a year since Musk bought the company, plenty of time to find another job.
I don't think you understand what a "tyrant" is. Companies aren't democracies.
At my last start-up, there were 3-4 times the number of engineers and yet they failed to do basic testing and engineers ended up do that job too. Funnily some of them were paid more than engineers
Worked fine with me. It’s a bit annoying seeing a mastodon user shtting on twitter, there was no self own, the post sounded good because it resonates with Elon haters.
Unlike what people imagine, the selfDDOS bug that occurs does not burden requests to the Twitter system, but becomes a loop for users requesting access.
yeh, i tried to mention this, but it got flagged quick. it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"
(I have to email my own admin every few months to ckeck if things are OK.)
And during the October Revolution as hoardes arrived from birdland, things got ssssslllloooowwww globally.
Worked out eventually, but it took a bit.
Individual instances also tend to run into scaling issues, with Jerry Bell's Infosec.Exchange coming to mind. (Mostly because Jerry's discussed this a bit.) And of course individual instances can be shut down or fail in various ways. I've migrated several times myself.
I will say that most of the time things seem fine, and it's exceptionally rare for there to be truly Fediverse-wide issues.
(I've been on Mastodon / the Fediverse since 2017, for the most part quite actively.)
I've been on their waiting list for what seems like months at this point. Huge missed opportunity for them, being unable to leverage Twitter's failings.
A lot of people who’ve wanted to jump ship to Mastodon have had plenty of opportunity to do so in the past year. Bsky is still invite only but had significantly increased its rate of handing out invite codes in the past 2 weeks. Today might be the day that invitees have decided to finally check things out
Mastodon absolutely does not load quickly. I clicked the link and saw the followup tweet/post/whatever for a couple of seconds before it reflowed and showed me the actually linked tweet in a smaller body font. The had to wait longer for the whole thing to finish loading. There is a ton of unnecessary JS bloat on the linked site.
HN traffic is not nearly the DDOS people make it out to be. I made it to spot #2 once and that was a few dozen hits per second at most. Maybe not something to serve with wordpress from a raspberry pi, but hardly apocalyptic levels of traffic.
I hear always that centralising everything is great because efficiencies of scale: but then we have something that works as good or better and the response is; “ah yeah, but the load is so high!”
Why do I care? I don't honestly give a shit about how much load you have, you could be factoring Pi on every page load; it means -nothing- to me. I kindly invite you to give more of a shit about user experience.
This also goes for when “complicated” systems fail, maybe making them so complicated and centralised is not the way.
Mastodon isn't truly decentralized. Every instance is its own feifdom and you have to communicate directly with them to exchange messages. This is unlike how Usenet, FidoNet, and SMTP work. They are truly decentralized as you can exchange data without directly accessing a peer's host server.
I couldn't see a few threads on Mastadon a few weeks ago, the post was up for 24 hours, and the perceived outage never dropped through that entire period, and probably longer after I stopped caring about the post. There were numerous people on HN experiencing the same.
And if some massive org needs bandwidth for posts, can't they host their own public instance? I'm sure many organizations would prefer that over being at Twitter's mercy.
The point of it is that it isn't a company but a standard, so if one instance owner goes crazy and patches ads in you can move to another (or your own) and keep your network
The problem there is that there's nothing stopping the crazy instance owner from retaliating by "defederating" your own instance and cutting you off from your network.
Only if you have a single account. Most Mastodon clients allow multi-accounting quite trivially.
Also probably not the best argument to make in a thread whose main topic of conversation is about how one of the biggest social networks on the internet is disintegrating in real time thanks in part to the management of its owner.
He could defederate from you, but one for his instance. If you haven't pissed off the whole federation and you backed up your data (another thing you can do on mastodon) you'll be fine, and keep all your followers too.
Sorry if this is a dumb question, I’m still new to it. Wouldn’t you have the same problem as with an email address? I mean, your handle or whatever it is called would still point to that instance/owner right? Or is there some kind of “DNS” or registry so you can move your handle to other instances?
Let’s say that I want to move my @ploum@mamot.fr account (my real Mastodon account) to another server, let say "writing.exchange".
1. I create an account @ploum@writing.exchange on writing.exchange.
2. I go to mamot.fr and, in the settings, I enable migration to @ploum@writing.exchange.
3. I go to writing.exchange and, in the settings, I start the migration from @ploum@mamot.fr.
All my followers and following are automatically transfered. For them, it is transparent. They still follow me on my new account without them being even notified.
Of course, you need cooperation from mamot.fr. If mamot.fr decide to close your account, you can’t migrate it.
But it works well, I’ve used it myself. It is really great and allows people to do "server hoping" to join a community that fit better their need.
Assuming the old instance doesn't just totally shut down, you can trigger a "move" process that makes your followers automatically re-follow your new address.
On top of that, you can host your own webfinger alias, as sibling suggests, which lets you have an unchanging address that forwards to your current server. But note that accounts follows URI's not the handles, so you still need the move process to migrate existing followers.
It's not by any means perfect, but it's improving (e.g. the move process is relatively new) and probably will keep improving.
Yes. This IMO is one of the 3 key issues with federation as it is now:
1. Noone understand what "federation" is so they all flock to the big servers hence making the majority of the system totally non-federated in nature
2. Findability (of users, topics, servers) is terrible which pushes people to 1)
3. What you said. Until there's such a thing as federated identity, we're all still tied to one server, thus one server owner can ban / switch off / over-moderate and we're all back to square one
Some of this can be solved with ux and education but I worry that some of it is basically baked in to federation.
Edit: yeh I mean in theory you can move servers but it's apparently not easy...!
Yes, you can use your own domain as a handle even if you're relying on someone else's server for hosting. It's sadly a little more complicated than DNS though:
Apart from Facebook (I mean Meta (I mean Threads)). Let's see how that one plays out, but initially it seems like they'll be blocked by almost every bigger instance anyway.
Even if Facebook starts using ActivityPub in one of their products the protocol will still be just a protocol. And if FB's product goes to shit it won't affect ActivityPub.
Well, it's a FOSS self-hostable server program. Mastodon isn't a service, so it's not susceptible to enshittification per se. A particular Mastodon host, sure. But Mastodon itself is just a codebase.
The biggest problem with Mastodon is that 3 instances comprise 50% of all traffic. That’s not that bad but that creates the conditions for the largest X instances to become Mastodon Inc., use VC money to fund advertising for their site and fund improvements the other instances don’t get, eventually defederate from the other instances, and finally enshittify. I think Mastodon is a bit more insulated from this than fully private companies, but it’s not invulnerable while that many users are on the biggest instances.
Make no mistake though, enshittification follows the population. Mastodon is about as protected as html in that sense. Could you build a nice lean mean performant static site in html? Of course, but hardly anyone does that. Most popular sites that you are able to discover these days are enshittified because the incentives favor that.
> Mastodon is about as protected as html in that sense.
I think you're probably using the term "enshittify" differently than the parent comment. Enshittification, at least as I tend to see it used, doesn't really follow from a particular technology stack, but more about how an organization itself approaches its end users, particularly against over-exploitation/monetization of a given platform. It typically doesn't speak to the underlying technology (i.e. html vs. MB of Javascript vs. WASM), since that is (within reason) somewhat orthogonal to how the organizations running instances treat their users/how end users actually experience the platform.
Also most of the rest of the world doesn't use Twitter (I'm seeing about 240M total users?). The ones that do use Twitter do so because of a significant amount of money going into marketing Twitter's platform.
Conversely, relatively nothing goes into pushing people to use mastodon. It can only take off if it really does prove, not just useful, but more useful than a centralized version that's got money behind it.
I imagine 'none'. With Twitter, the most recognized micro logging site in the world only has 240M of the worlds population, I conclude micro logging isn't popular worldwide.
I'm LGBT and my spouse is trans. I don't believe you should silence anti-trans rhetoric. I believe you should engage it.
The ability for /r/conservative to ban my counter arguments is just as harmful as Mastodon shutting down the anti-trans positions.
Conversation is what moves us forward and is how we find commonality.
I grew up religious and conservative. I changed a lot of my viewpoints through friendly conversations in the internet of 2000-2010, before tumblrism, cancel culture, and censorship took hold.
If I grew up in today's world or internet, I might never have been exposed to different opinions in a non-hostile, no-judgment environment. By trying to segregate, censor, and ban we're only leading to intractable polarization. Never giving folks an opportunity to change. Never accepting that people are capable of growth.
Please let's talk with each other. Even if we disagree. You'd be surprised how effective that can be.
We're all suffering though this world together. Laugh away our differences and find the ways and the things that we share. We all hold more in common than you might think.
Love your enemy, even if they don't love you (yet).
If I could have one lasting impact on this world, it would be this message.
I'm a fifty-something trans lady and I am just fucking tired of trying to change viewpoints. I just want to live my life and talk with my friends without some butthead coming in and telling me I shouldn't exist, I get enough of that by checking the news lately.
If you have the energy to politely engage people who think of you as a child molester who should be shot on sight, great! Go for it! But I have done that, and I am tired, and I do not want to do it any more. I run a Mastodon and I just want it to be a space to talk to my friends and maybe make some new ones, and thus, I block the fuck out of places I do not expect to get anything but hate from.
Common rhetoric among PoC even twenty years ago was essentially, "We're tired of being spokespeople and tutors for your problems."
Not everyone is cut out to be an educator, and I think you should have the option not to be voluntold for the job. Not just because it should be your right, but because insisting that everyone in a group can speak for that group is itself stereotyping. I think once you see that it's really hard to be patient with people who don't.
As a trans woman living in a conservative area, I get where you're coming from but I seriously disagree. The hardest pill for me to swallow through all of this has been realizing that some people will never change no matter how much you engage them. The only way I maintain my sanity is to do whatever I can to reduce my exposure to that kind of thing so I don't end up engaging with it.. because it just never goes well.
I realize it's a complicated issue, and I'm never a fan of banning speech. But not all speech deserves a response.
Mastodon isn't shutting down anti-trans positions, specific instances are choosing not to federate with other instances that harbor those positions. Those instances and the homophobes and transphobes are still there.
And you're free to engage the people who want to put you and your spouse on a train car in conversation all you like. Maybe you'll deprogram one or two, but you'll just help spread their propaganda to exponentially more people than you could ever help.
I have no commonality with such people and don't want to find any. I don't want to share a society with them, and I know they don't want to share one with me. I certainly don't want to debate the Jewish Question or "groomers" or race science with them on my gamedev instance.
>We're all suffering though this world together. Laugh away our differences and find the ways and the things that we share.
You know these people want you dead, right? They don't believe you have a right to exist. You and your spouse. Especially your spouse. We're not talking about a difference in belief about tax laws or support for opposing soccer teams here. "Laugh away our differences?" I'm sorry but with all due respect fuck that.
I agree with this, I think the only way we'll get answers to the most controversial and divisive issues are for people to discuss them. Especially since many viewpoints being shouted down as "anti-trans" are actually very reasonable and need more discussion.
For example, the fairness and safety issues with regards to males competing in women's sports, or the issues of safety and dignity in women's prisons when males are incarcerated there.
And no, there is no such thing as eradicating "transgenderism" without eradicating transgender people anymore than you could eradicate "blackism" without eradicating black people. It is a meaningless distinction invented to provide a paper-thin veneer over what is simply a call for mass murder.
Come on, in a lot coastal discourse it is practically taboo to mention vanilla opinions that are held by 70+% of the population.
Mastodon instances are largely moderated by people from the other 30%. You are free to judge if you want. But don't pretend this is a violation of publicly accepted morals in the 1st world.
I am having trouble imagining what you mean. Can you illustrate your point with an example opinion that is held by 70% of the population but is taboo to discuss?
Nearly 70% of U.S. adults say transgender athletes should be allowed to compete only on sports teams that correspond with the sexes they were assigned at birth
This is obviously not taboo to discuss since every mainstream media source has been discussing it nonstop with virtually no reference to like, actual data, for over a year.
None of those are really taboo to discuss: police funding has only increased in recent years, Joe Biden has been quietly upgrading security on the southern border since the start of his term, and the supreme court recently ruled against affirmative action.
I've noticed than when people on the internet say "I'm not allowed to discuss X", what they actually mean 99% of the time is "I'm not allowed to be an abusive jerk to other participants". (The other 1% of the time, they're posting off-topic in a tightly focused forum.)
It is funny how, in their own bubble, people assume that their opinion is held by 70% of the population while it is often the opposite (fun fact: more people voter for Hillary Clinton that Trump yet Trump voters believe that they are the majority. Same for abortion where polls showed that a clear majority of the US was pro-choice yet a very loud minority has a lot of political power)
That didn't really count, because they all agreed that black people and natives weren't Americans and had no rights they were bound to respect. The real oppression is when relatively wealthy upper-middle class people get criticized.
Trans people in sports? Wanting strong borders? Disagreeing with stuff like drag reading sessions?
Not American or white or whatever, just stating the obviously less widely supported stuff that may sound uncontroversial to the more terminally online.
> Disagreeing with stuff like drag reading sessions?
What does this mean? Drag queens shouldn't be allowed to read? Like what concrete policy are you saying they can't propose which isn't obviously overreach?
To be clear it's not personal beliefs that I have, just contentious issues I could think of that are far more controversial irl versus online. Also I think the issue is the reading sessions for kids.
Drag queens reading at libraries is one of those things that's absolutely more controversial online than IRL. Nobody gave a shit until Facebook and Twitter groups whipped it up as a culture war issue, and so far even conservative analysts and judges are like "wtf, this is nothing, you can't ban it" and they've had to shift to broad obscenity bans instead, which do not have popular support.
Someone is invoking censorship as a reason not to adopt a new platform. No specifics, just rabble rousing. That's manipulation. Pushing back is thinking for yourself.
Or, they are being imprecise and undermining their position, in which case what I said works as advice on further conversations. Either way is thinking.
Or they're assuming good faith in their opponents. To not know what has been censored on twitter, and that the recent interest in mastodon was a reaction to the lifting of some of that censorship, is either to be playing stupid or to actually not have the background to discuss the subject usefully.
We weren't talking about twitter. We were talking about one Mastodon server defederating another for hate speech. That's not 'some topics'. That's hate speech.
If they're talking about some other kind of speech that I might actually care about, they should have mentioned it. Because the next bad one I can think of is even worse than hate speech.
Yes, but the point being made is that you then choose a more welcoming instance and then it's defederated instead because it allowed your post, so then your Mastodon experience sucks anyway and you only gave yourself an illusion of freedom.
I'm not sure how common this issue is but I _can_ say that I've been through a defederation bullshit myself because the large instance did something as egregious as welcoming people regardless alignment to Swedish government party (i.e. any party with over 4% of votes in Sweden). That was far too much for some instances like mastodon.art to handle. The admin got fed up since he had neither will nor moderation resources of that kind and shut down the instance, so everyone had to migrate which is a headache by its own even if supported.
From other stories, I swear the greatest threat to the Fediverse is politics and more or less childish cross-instance strife. I just now checked my Mastodon feed and this very fucking issue was discussed once more so I guess some drama has went down again while I was away. There's been trouble of this kind on Lemmy too already.
People say "it's like e-mail". Yeah, if we have like 20 major e-mail servers in the world and there's drama across them as we bet on the winners via Patreon.
And then I get defederated for having done so. Did you even read my post? It's an illusion freedom that does not exist in practice because this is Mastodon we're talking about.
My issue with this is that this kind of instance hopping can easily kill a network. Many who were on my instance that shut down didn't migrate away in time and now they lost all their posts, follows, and followers. Many of those won't bother to come back from that terrible experience. I still see those people in some users' friend lists over there (months later) and if I click on them, Mastodon just breaks with a cryptic error message.
So. With Mastodon instance admins and this kind of drama, making internal strife about politics, gender equality, whatever, between a few persons affect thousands of users and greatly inconvenience them, and running a personal Mastodon instance taking great technical know-how and basically being a bit of a geek, I still consider these major issues with Mastodon as it stands today.
So, yes, I want to "enforce" people to hear me. If they choose to follow me. Because this means all the above problems are solved. The only problem federation solves today seems to be the scaling problem. Everything else is about friction and this kind of trouble and persistent worry. "Which instance do I pick" where it feels like you want to have an interview over a beer with the admin first. This has thus far been the major demotivator from what I've seen as a Mastodon user. I think I'd like it more like a P2P-based social network.
Freedom doesn't mean everyone gets whatever they want.
Your freedom just doesn't override the freedom of others to avoid you. You can't force others to interact with you and there's nothing wrong with that.
> You can't force others to interact with you and there's nothing wrong with that.
On Mastodon, no one is forced to listen to anyone without a follow. You don't have to worry about that. This is not what I'm talking about, but the infrastructure problem.
I've repeated myself a lot already so I refer you to one of my replies to your sibling comments if you still want to discuss this.
You can, generally, say whatever you want via Activitypub. However, no individual Mastodon instance is obliged to facilitate this, nor is any particular Mastodon user obliged to listen to you.
Personally, I prefer not to listen to certain peoples’ Important Opinions about how people like me shouldn’t have rights (life is too bloody short), so I use a Mastodon instance which doesn’t tolerate that. People with such opinions are of course entitled to use a Mastodon instance which does (and there are plenty of them). I’m struggling to see an issue here. Person A is free to say whatever old nonsense they like, Person B is free not to listen to it.
I am, by the way, genuinely curious; I just don’t get the issue here. If a person with Important Opinions can’t hassle the rest of us with said opinions because we choose to opt out of them, well, so what?
> Personally, I prefer not to listen to certain peoples’ Important Opinions
Of course you can avoid hearing this!
You do so by not following said person. Mastodon works just like Twitter or Facebook in this regard.
This is not the problem that I'm talking about at all. I'm talking about the infrastructure problem. It's about instances shutting down, people not having had time to migrate, and thousands of users losing everything (their entire account and content) because some admin decided he didn't like furries enough or let any political party member join his instance, admin is defederated, admin says fuck this.
Sure, I can make my own instance to guarantee my place on the network if I buy a server and run it on that but how many do you guys expect are enough Mastodon enhtusiasts (of all things, haha) to even bother with learning and doing so. It's a stillborn workaround if this problem keeps resurfacing.
> if i don't like what someone is saying i have the freedom to disassociate from them
Yeah, on Mastodon we usually do so by not following that person?
This is not what I'm talking about but the infrastructure issue of internal strife between admins affecting tens of thousands of users for the most ridiculous of reasons, and this has not just been a hypothetical scenario, unfortunately.
And no, setting up your personal instance isn't a realistic solution for most people who just want to chat.
Note that this still (of course) requires following said person on said instance. So people won't be spammed by views they disagree with. Mastodon is no different than, say, Facebook there. You still need to explicitly follow people. Mastodon doesn't even have an "algorithm".
But I can't see any positives from being able to defederate like this. What is the main benefit to remove the freedom of your users to follow people?
>Though, the only time when I did see that happen, was when someone was transphobic, homophobic, racist and such
“I mean, when all the wrong and bad people are kicked out everything is great!”
No one is complaining about people with Ford vs Chevy comments being banned. It’s the controversial things that need to be refuted, not hidden.
What makes you so absolutely certain you are on the “right side” of any opinion? Because the people in charge of these services are censoring the other side?
How long before you find yourself with “the wrong thoughts”?
I run a Mastodon instance. You don't have to see eye to eye with me or my users to talk to us. However, I have defederated from instances that host:
* Loli porn
* Extreme neo-Nazi content; I'm talking about swastikas, hardcore racial slurs, and the like
* Targeted bullying and harassment
You want to spark a conversation about the relative merits of Republican fiscal policy, let's chat! You want to say that we should still own slaves, Jews eat babies, or gay people shouldn't exist? Go away. I don't owe you a soapbox.
Disconnecting from a server with despicable content doesn't take away that server's right to speak. It just preserves my -- and my users' -- right not to hear it.
That's what happened. People started demanding that Nazis be censored on Myspace. I knew people at the time who were under the impression that being a racist was already illegal in some way, and thought that Myspace not immediately banning all of them made the site an accessory to the crime. The worst part was they seemed to be centering it around me because I was the only black person they knew.
Which instance? Sounds like you joined a niche instance of a few hundred people. I find twitter to be extremely restrictive, you can’t have open discussions, you either get banned or get piled on by abusive blue check accounts.
> On Mastodon simply discussing certain topics will get you banned, and instances that don't ban those users get defederated.
This is why P2P is superior. Federation nodes can be used to strong-arm collective behavior against the will of individual users.
I don't mind being exposed to liberal and conservative thought. I want to consume almost the entire spectrum of human discourse so that I can synthesize ideas for myself and understand more effectively. As long as the signal is reasonably high.
Reddit and fediverse moderators wield absolute power over their fiefdoms. They're intellectual dictatorships. (Not to mention egotistical behaviors some of them have.)
P2P allows the end user to consume what they want, weight discussions how they want, and participate in any number of emergent clusters. It's the real path forward.
You've figured it out entirely. Mastodon is run by the kind of people who are willing to put in a bunch of time, effort, and money into dictating the conversation of others. People who enjoy that kind of power.
> I want to consume almost the entire spectrum of human discourse
...said no one who's ever been a moderator.
You find out quickly that there are some perfectly horrid people out there. You absolutely do not want to hear everything that people say. It seems like you would, but you really don't.
It's a shitty monoculture mostly filled with a particular demographic (like most new and obscure tech things), the people who program computers, and the people who they meet at parties.
This would disappear with more widespread usage. The problem is the software, not the culture. If the software is improved, or the dead ends are pruned and something else is created that learns the lessons from previous tries, the new cultures will bury the old.
If building software required experts on model trains or K-pop, the culture would suck, too. The goal is to make that a stage rather than an endpoint.
edit: I enjoy model trains, but I do not get into political or social discussions with model train guys.
May I suggest 8chan? Almost no moderation at all, but make sure you have a gallon of Clorox by your desk because you will want to pour that into your eyes.
What would be grotesquely offensive stuff to you? You realize that grotesquely offensive to very online Americans is an extremely niche thing? It makes sense to NOT want your online presence to be tied to whatever some Americans think makes perfect sense, right?
I don't think you'd agree that it would be weird to not want your social media and what you see online to be tied to what some, for example, Saudi dudes think is acceptable at the moment.
I'm not sure what you're complaining about. Don't join the Saudi instances, then? That's kind of the whole point of Mastodon, and federated services in general. There will always be instances that match whatever worldview you may have.
What you can't do, and should never expect other people to do, is to be forced to receive what you're posting, or to put it another way, you can't force people to listen to you.
In twitter, people can choose not to follow or mute you on individual basis, that's basically what “not listening” means. Banning somebody means preventing other people from listening to them, so it's not the same.
The fact is that Twitter makes you believe that you are listened to, even if you say shitty stuff.
Mastodon confronts you that if you say shitty stuff, nobody wants to listen to you.
People complaining being banned or being on defederated instances are people other don’t want to listen. They pretend to have a personal opinion while they are only assaulting others.
LGBT is a good example: you cannot have an opinion about it. Those people exist. They have the right to exist. You have the right to not engage in any LGBT activity. But you don’t have the right to talk about a "debate". There’s none. If you do, I you maintain that using "cisgender" should be a banned word, you are simply an asshole and can’t complain that people don’t want to listen to your ramblings. And yes, this will get you banned.
On Mastodon, people can choose to pick an instance that will rarely defederate anyone and follow or mute on an individual basis, or they can choose an instance where moderators will take a firmer line.
The trouble is how to judge them. Automated censorship of others is invisible unless they announce it or you somehow know what you should have been able to see.
Mastodon instances tend to have admins who are if anything pretty loud and argumentative about what their mod policies are. If that matters to you, you can easily find instances that make a point of being open about what they block. A whole lot of people don't care about the how or why as long as they're not subjected to content they don't want to see, and that's their choice.
If you choose to only talk to like-minded people, that is your right, as it should be. And might well be one of the reasons most people aren't on Twitter, and many who are subscribe to extensive block lists.
It is kinda funny if you consider these companies might consider their user data to be useful, especially with recent advances in LLM models. I've been thinking if you just exclude Reddit posts from training youll probably achieve much lower bullshit scores, as that seems to be what most posts on there seem to represent. I think data curation (by sources) could achieve quite a bit.
unlike what others have said, Twitter was very useful during the saturday mutiny in Russia. I follow a lot of people who supplied updates and thoughts.
I think we're about at the point where the people who predicted chaos at twitter after Elon basically fired most of the experienced engineers have been proven correct. The duct tape is all coming apart at the seams now.
It isn't quite as decisive as a submarine imploding, and ceasing to exist, but it has turned into a brightly burning tirefire.
25 years of products being honed for shareholder value, instead of customer or user value. We may be at peak consumer tolerance for anti-pattern, in-app purchase, subscription-model, ad-packed, data-siphoning, dopamine driven, gated experiences.
This is a perfect example of how "shareholder value" is a thought-terminating cliche.
Twitter was previously a public company, which was beholden to shareholders, and aimed to try and increase its stock price (as far as "shareholder value" actually means anything, this is basically it). I wouldn't praise previous management (the company wasn't profitable), but they were not a complete dumpster fire.
Then Twitter was bought out, and taken private, removing the obligation to "shareholder value." The ensuing dumpster fire is one that will be marveled at for years.
I'm not saying public corporations are better than private, or that "shareholder value" is a good slogan. I'm just saying that your comment is every bit as irrelevant as the porn spam that's clogging Twitter these days. (Thanks for fixing the spam problem, y'all!).
> Then Twitter was bought out, and taken private, removing the obligation to "shareholder value."
Does it really though? Private shareholders are still shareholders. It replaces a diffuse duty to keep a bunch of public-shareholders happen with a possibly-more-direct "do what I say or be replaced tomorrow."
> "shareholder value" is a thought-terminating cliche
I think when people use it dismissively, it's not really about shareholders per se, but about one that are focused on short-term growth at the expense of long-term growth or a sustainable business model.
If your point is that both old Twitter and new Twitter have people who have put money into it, and expect to not lose their money, you are correct.
I would still recommend not using the word “shareholder value” for the concept. It’s just…having a business that you don’t want to lose money? Some people do dislike the concept of business, but I don’t think they should talk about “shareholder value”, they should just attack capitalism.
In any case, it’s still irrelevant to a discussion of Twitter. The old management was also expected to turn a profit, but somehow avoided Elon’s string of silly ideas.
> This is a perfect example of how "shareholder value" is a thought-terminating cliche.
I think "shareholder value" is just a distraction and a rationalization.
The driving force is the MBA-ization of management and people looking to juice short-term profitability so that they can cash out or get large bonuses and then job hop away.
It’s well documented that advertisers have been fleeing Twitter because they see the new management as bad for them. While Twitter has engineering and reliability problems, the loss of advertising revenue is the life or death challenge for the company.
I’m pretty ambivalent about advertising, but it was the only reasonable way for Twitter to make money, so I would not have bought Twitter and then chased away all the advertisers.
> We may be at peak consumer tolerance for anti-pattern, in-app purchase, subscription-model, ad-packed, data-siphoning, dopamine driven, gated experiences.
As much as I want this to be true, I think this sentiment is really only popular on tech-savvy forums like HN. Most people don't use ad blockers, and I've had people get mad at me when I suggest that they do (directly in response to something where they are complaining about ads).
Not the op but I have that experience frequently. These are perceived as geeky needy techie things that are not for normal people (sprinkle quotations as needed). It's the same as people getting upset at suggestion to add lock of some sort to their phone (face, fingerprint, whatever) or backup their phone.
After some pondering I think it's peoples' insecurity misfiring. They use these complicated layered and potentially risky and dangerous pieces of technology, aware they don't fully understand them, that they work as magic that could stop any moment. Trying to understand and secure them is a massive rabbit hole. So I think there's kind of a rejection to go down that hole or acknowledge the problem or, most of all, face the vulnerability and exposure.
Ultimately it boils down to "it should just work" but to be more specific one person said "I shouldn't have to do anything different!" and directed their anger towards me instead of the ads they were previously complaining about.
People that aren't tech-savvy don't want to think about tech any more than they already do. Having to understand something new about tech is just another problem to them. I'm not saying that as an insult - just an observation.
AAA video game industry sure seems to be pushing this idea with the past years of broken, unfinished, beta projects being released as complete products.
I have to disagree. If you honestly take the emotion and politics out of this and evaluate on merit alone, what do you think?
Twitter wasn’t healthy before Musk bought it. It wasn’t a thriving business, it was a very old, very large startup still struggling to find market fit and loosing a lot of money.
Also, it wasn’t a thriving product. It was stagnant.
Since Twitter was purchased, the amount of features they have shipped has been impressive. They’ve shipped a lot of features and extended the platform a lot. To your point they have also done this with far less engineers than before.
Regarding any downtime, everyone has downtime. Google, Amazon, Meta… the best of the best still have it regardless of money or manpower.
Considering what that team has done with less resources, I think the achievement still pretty good. What do you think?
I'm starting to think today the word startup means a non profitable company that only exist, because interest rates are low and investors don't have any better options than invest into them.
Totally agree with you. It was only a startup in the sense that it was still struggling to find profitability / solid market fit.
As opposed to something like Amazon which grew and grew for nearly 20 years, always burning more cash than it made to fuel growth, but they understood the business really well and when they decided to optimize for profitability rather than growth, never never gone back.
Amazon's retail side was almost always marginally profitable and that was while they were reinvesting like mad in retail infrastructure (data centers, warehouses, etc).
They had lots of Free Cash Flow. It was always routed directly back into growing the business instead of taking profits. Which is always the right thing to do if you can grow the business without hitting any walls, and you avoid paying any taxes that way.
Companies that behave that way and have good returns on capital employed and have large growth in earnings, free cash flow, etc are good investments. Doesn't matter if they're not showing profits.
The company was not thriving but the product itself was rock solid.
It’s hilarious to think it is at all acceptable to kill public access, and drastically limit authenticated access, because of a few scrapers. There is no way Twitter prior to Musk’s acquisition would have had to do so.
> What do you think?
I think you are not looking at the situation objectively.
Regarding the product, it was solid as in reliable, but stagnant.
I don’t pretend to know all of the motivations behind the policy moves.
That said, I’ve got some experience with scraping; got sued by LinkedIn in 2014. We were using AWS Spot Instances to hit it hard for very little money. It was not uncommon to accidentally take large services down.
Scrapers can and do add very significant load. We also scraped Twitter back in the day as well.
Sites that wish to be ubiquitous must handle scraping in a way that is transparent to valuable users. It’s not like scraping is a new or complex threat to availability. This is table stakes for large services in 2023.
Twitter was very good at this, and their new-found inability is a glaring sign that their engineering is slipping.
Really? I disagree. Running a ubiquitous service that is good for users does not require that you allow every random person to scrape your site and incur that cost.
You simply setup API deals with those who you want to have your data, those that benefit your business, aka Google etc…
Then you close everything else up. This saves cost and complexity and real users, the target of your advertisers, don’t even notice.
This isn’t a sign that engineering is slipping.
It’s a sign that a in a company which struggles to make money, someone is paying attention and trying new things to fix the money problem.
> Running a ubiquitous service that is good for users does not require that you allow every random person to scrape your site and incur that cost.
What I said is that they must handle the problem transparently to their valuable users. That includes (requires, usually) targeted techniques to block high-volume scraping.
> Regarding any downtime, everyone has downtime. Google, Amazon, Meta… the best of the best still have it regardless of money or manpower.
I mean, I would expect Microsoft to do a much better job than Twitter to keep GitHub from going down every single month after acquiring it. The frequency of GitHub going down with 100M+ users using it is much worse than Twitter.
It turns out that GitHub's constant downtime for years is all fine (especially tech folks) here despite the monthly complaints anyway. The latest one here [0] But only with Twitter, the speed-bumps are exaggerated and magnified.
I am not disagreeing with you but self-ddos is not entirely uncommon. When I worked at Amazon this would happen a few times a year. Not on the main amazon.com website but on supporting services often initiated by but not limited to kindle devices. Having something like this slip through the cracks of even experienced engineers isn't uncommon.
Yeah it happened when I worked at Amazon as well. I also more recently worked pretty closely with people at Facebook and knew something about the issues they would occasionally have (which didn't match what the headline speculations were at all). But twitter is repeatedly having these kinds of issues.
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.
> But in such cases surely there's some kind of rate limiter in place?
Not really, our team maintained a reverse-proxy that fronted all requests that came into amazon. And whenever we would have a self-ddos event, we'd get a request from the backend team whose service was getting self-ddos'd to shed traffic before it reached their service hosts to prevent it from browning out. In many case ddos's were coming from kindle devices which were not even easy to update so deploying a "fix" wasn't even always an option.
I dunno. We are several months out after the major layoffs. Maybe some very recent bad decisions were made internally that pared back too far, but I think Musk has long since been proven correct that the core platform could function on a fraction of the workforce it had at the time of takeover.
Do you think if we fired every civil engineer tomorrow, the bridges and the highways they built would fall apart right afterwards, or even in a few months?
Engineering isn't like service positions where the lack of competent personnel is felt immediately; the debt keeps growing until your whole system collapses under it one day, how far the day is in the future depends on what system you're working on.
>I think Musk has long since been proven correct that the core platform could function on a fraction of the workforce it had at the time of takeover.
It REALLY sounds like you don't understand how any of this works.
Tech products don't stop working when you fire most of the staff.
But bugs stop being fixed and problems begin to add up, until a critical point is reached,m where the whole house of cards collapses.
Thinking that "Elon was proven right" simply because Twitter didn't implode the second he announced the layoffs, makes me think you don't understand how tech and software works.
Complex systems like Twitter don't break overnight when you lay off the talented engineers.
They deteriorate piece-by-piece, potentially over the course of many months, until the compounding effects of these problems and the growing technical debt overwhelms the team that they have left.
I currently can't read any tweets from https://twitter.com/elonmusk because it's just said "something went wrong. try reloading" for the last 8 hours or so.
I'd consider that deteriorated service.
also just out of curiosity while trying to find historical outage data I found this article.
Last july (before elon took over), the site was apparently down for 45 minutes and "one of the site’s longest outages for years". Today it's been basically barely usable for most of the day.
Parent is suggesting that gradual deterioration is occurring. I'm trying to figure out if that's what's happening, or if this is simply a bug that hit production (possibly due to the higher rate of product changes, or otherwise.)
It's hard to find nuance and information anymore. It's as if all we have to work with is politics and hatred.
Fair enough, I'm sorry for being rude with my answer. For what it's worth, I don't think any of us outside of twitter will truly know if things have actually deteriorated or it's just a one-off bug. At this point though, I don't think there's much difference since the effect is the same.
I would imagine the checks and balances that a mature engineering organization maintains to prevent the unintended consequences of capricious management decisions.
Agree. Plus it’s easy to crap on Elon but it’s also the poor Twitter architecture and quality of people they have working for them that caused this despite Elon’s desire to require login to read Twitter. He didn’t write the code.
right? it's a shame hacker news doesn't let you reply to old comments, there's a few threads I'd like to follow up on from people who refused to believe Elon was destroying twitter
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.
it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life" ...
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.
I see a giant circle around like 10 sentences, none of which feel like the sum up twitter in any way? After some consideration I guess maybe you're referring to "What's happening?"
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.
Keep in mind that Twitter engineers are under extreme pressure, knowing that their H1B visas are at stake, to implement Elon’s whims as quickly as possible on systems they’re unfamiliar with. I doubt many of us would perform better in their situation.
IIRC the H-1B application period is some time in the first half of April every year (so all the ones that would be filed this year have already been filed), but IDK if renewals need to catch that train.
H-1B approval has two stages, the registration period (March/April) and the petition period (ends 90 days after post-registration selection). AIUI this is counting petitions (i.e. I-129s), so won't be accurate until at least 90 days after selection notices - which would be around now - but then the site itself says it can further trail by a quarter.
Do you think twitter has a lot of marketers, PR, product managers under H1B? I don't have data, but in the tech companies I have worked at it engineers were the H1B visa holders, almost exclusively.
I would imagine the vast majority of those are engineers. It’s pretty hard to get an H1B in a non technical field and I’m not sure Twitter would even have many of those types of jobs.
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.
> Lest anyone doubt that Twitter was idiotic enough to release code that would cause a race condition and result in its own users executing a DDOS attack on it
Given that this is a race condition bug as stated in the original post, it's unlikely something reproducible in a deterministic way. You could probably give it a little bit more attention before leaving this reply...
over what period? I left my dev tools open since my original post (14 minutes) and and it looks like this, so 197 requests / 14 minutes = 14 requests per minutes, which is not a DDOS:
My personal feeling is that a lot of the social troubles of the past decade stem from information overload. I subscribed to News Minimalist and have been feeling much happier lately.
While I visit here a lot I would guess that your claim that "HN is all that's left" is just wrong. There are hundreds if not thousands of forums and various gathering places where people are having conversations, just not in a place you are aware of.
> Where do you all get your info lately? How do you stay informed? I really don't know how to get a wide swath of information anymore with Twitter and Reddit gone.
MSM hasn't gone anywhere. And realistically speaking most news doesn't matter anyway.
But I do think the case of "I need to learn about X and it's not on reddit or twitter anymore" is a realistic concern. Hopefully search engines can fight the tide of LLM created bullshit and help you find what you need to know.
> The two remaining sites that allow regular people to post and have discussions are being wiped off the internet in time for the next election. By the next US presidential election, there won't be ways to access information that don't go through the "wrong" people first. Hacker News is all that's left, and it's not big enough to be significant.
The 2028 election is more worrying IMO, especially since it seems like Musk is angling to run for president and doesn't seem to mind running despite not legally being able to hold the office. It feels like he views kowtowing to the MAGA crowd and controlling large media platform(s) as essential to that plan.
I'm aware. I'm saying that he clearly isn't beyond bribing SCOTUS to pave the way for him to a presidential campaign in 2028 or even initiating a fascist coup.
> Where do you all get your info lately? How do you stay informed? I really don't know how to get a wide swath of information anymore with Twitter and Reddit gone
News sites reddit and twitter source stuff from... Or pick your favorite aggregator.
Sometimes Discord/Matrix and blogs are good for really niche topics.
There are many thousands of forums, but do remember that there was a time (not that long ago) that social networks didn’t exist and things seemed a hell of a lot more rational back then. Both Reddit and Twitter have been highly weaponized by both foreign and domestic entities. If this is a concerted effort to destroy social media (and I’m not so convinced it is), I think it’s not a bad thing. It’s just become extremely toxic and even dangerous for society.
Those that remain, outside national papers, are often thin husks of themselves.
A year or so back, the Chicago Sun-Times effectively paid the local NPR affiliate, WBEZ, to take over the paper. That is negative value.
And the competing city daily, the Chicago Tribune is arguably doing worse.
That's in the 3rd largest city and metro region of the US. Many other cities are in similar shape, or have no traditional newspaper journalism at all. There's some coverage through TV & radio, though often it's the local NPR affiliates which seem to do the heavy lifting.
Is there anywhere in this thread where there is actual info and not just people complaining about and taking shots at Elon Musk because they dont like him or his politics?
Twitter has been severely hampered for 12+ hours, and that’s only the latest issue. I don’t think Musk has demonstrated anything conclusively quite yet.
I don't see any unusual behavior when looking at the debugger in Chrome. People have a political issue with Elon. Which is fine. People are entitled to it.
The post suggests a race condition as the underlying cause of the behavior shown in screen captures. It’s perfectly possible the race condition does exist, and being a race condition you just didn’t happen to trigger it. It’s also perfectly possible to discover such a defect with or without any personal bias towards Musk or Twitter.
Reporting it publicly this way can also be a favor to both, as it gives them a fairly malleable narrative for reversing course, or directing responsibility, or both. Even if it’s simultaneously embarrassing to have it out there. And it’s not like either Musk or Twitter is a stranger to embarrassment, or particularly shy about courting it.
Clearly there is something wrong with the site. So I don't doubt there is a bug. But the OP gives no helpful information that could be used to reproduce the issue and fails to describe it in any helpful detail really. They seem content to just effectively say "Twitter is dumb." Which, again, is fine but not all that technical and interesting.
There’s information in the post that very probably points at the underlying cause. The scrollbar jumping is a dead giveaway: Twitter loads more content based on scroll position. If your viewport is sized such that displaying the error message triggers another load event by its corresponding change in scroll position, it will try again indefinitely absent some additional logic to bail out.
Edit: this of course depends on some other implementation details, like the rendering flow and browser behavior. If showing the error is memoized, it shouldn’t trigger the loop unless they’re also rendering some intermediate loading state asynchronously.
yeh, i tried to mention this, but it got flagged quick. it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"
>There’s information in the post that very probably points at the underlying cause. The scrollbar jumping is a dead giveaway
Maybe that theory is correct. But I feel like pretending this post is getting to the top of HN because it's a technically interesting diagnostic analysis is sort of silly.
It made its way to HN because people here find it interesting enough to discuss. It’s not some anti-Musk conspiracy. And it’s not anyone else’s fault you didn’t recognize the technical implications from the information available.
And after a while out with my pup enjoying sunshine and play, I think I should be a little more friendly than this.
It can take a lot of time and effort to develop instincts like “scrollbar jumping and network calls are related”. They’re so obvious to me that I didn’t even bother inspecting anything, I just “debugged” on my phone by connecting familiar dots and describing the familiarity. But if it’s not immediately obvious to you and if you have an inclination to be more familiar, I’d strongly recommend spending more time manually fuzzing rando sites with dev tools open. You’ll probably get a lot more out of that than dissecting personal motivations on any thread on any site. And you’ll have a much better calibrated bullshit detector too.
The kind of input you have when feeding tweets into an LLM is toxicity. It looks like something you can blackmail LLM for. Look, you have all this toxic trash in your model.
Like: we have the sources and you could detox you model if you pay for it.
No, look closely: the animation does not show the console, it shows the network pane. Each line is a (presumably unsuccessful) request to the Twitter backend. If a lot of people have the Twitter web frontend open and running in this state, it could in fact overload the backend.
Sure, I mis-spoke in saying it was console.log spam, that’s not specifically what’s happening. But let’s be real, if you’re getting a 429 response that is at most a cached hash table lookup per request, for a site has probably top 50 in the world in terms of “scaling to handle heavy request loads” for a decade.
There’s just no chance this specific thing is actually what’s causing issues for Twitter, it’s obviously a consequence of the heavy scraping and steps to stop the heavy scraping. It frustrates me to no end that smart technical software people, who can have intelligent discussions on not just code but a wide range of topics from nuclear powers to superconductors, suddenly lose their ability to have these discussions when a certain person is involved.
THANK YOU! i tried to mention this, but it got flagged quick. it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"
> Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon's latest genius innovation is to block people from being able to read Twitter without logging in
Well, for one, this sounds just like yet another Elon-hater's comment. In _most_ cases, Elon's businesses are fairly successful, so a sarcastic "Elon's latest genius innovation" doesn't belong here.
For second, how do you know the idea came directly from Elon? It could easily be the result of a brainstorming session led by a product team.
Elon is not writing Twitter's code. He is not a Software Engineer at Twitter. Even if the idea came from him, why is he being attacked for a bad deployment?
And lastly, why do people think that the change is bad? There are plenty of web sites that do not allow you to view their content without creating an account first. Is Twitter not allowed to experiment? Why not? I would appreciate hearing well-thought-out arguments that explain the potential negative impact on the business.
A CTO is responsible for failures in the technology org. A CTO who describes himself as responsible for "sysops" is specifically responsible for failures in SRE/ops. Leaders are responsible for work they didn't personally do. If they don't want to be responsible for their reports then they can always go back to being a junior IC.
right, i tried to mention this above, but it got flagged quick. it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"
Why does it matter that he made the call to do that? I get plenty of "do this, do that" requests from my manager. And if I break things, is it my manager's fault? I'm trying to understand your logic. Please, explain.
Edit: here is my take on it + the most likely scenario of what's going to happen next. This was a bad deployment and such things are not unusual in software industry. In this specific case, there are zero reasons to blame the CTO. From what I'm observing right now, Twitter has already fixed the issue - the website loads just fine. Next, they will do a retro, learn from their mistakes and try to not repeat the same mistakes again in the future. There will be more bad deployments and that's normal. However, with time, they'll make things better, the SLA will go up and the overall stability of their services will stabilize, – one of the biggest social networks on the planet will have the least number of engineers running that same social network.
(A) How is a CTO responsible for deployments? CTO oversees the development evolving the company’s strategic technical direction so that's its beneficial for a business. Twitter is not your typical YC startup. Large companies' CTOs should not care about things like "this could DDOS us". These are the responsibilities of SDEs.
(B) No, but I just googled it and I can't see how the term applies here. In my opinion, it doesn't.
it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.
Even more so when that person later loudly proclaims that they never made such a request, even when provided with written proof.
I can of course not say whether the people currently working at Twitter did warn that the recent measures could have such major side effects, but I would not be surprised in the slightest, considering their leadership's mode of operation.
Even as someone who very much detests what Twitter has become over the last few months and in fact did not like Twitter before the acquisition, partly due to short format making nuance impossible, but mostly for the effect Tweets easy embeddability had on reporting (3 Tweets from random people should not serve as the main basis for an article in my opinion), I must say, I feel very sorry for the people forced to work at that company under that management.