Hacker News new | past | comments | ask | show | jobs | submit login

Disclaimer: I worked at AWS from 2012-2020. I am not a current employee.

> I came to think of Amazon and Google as the providers of the very infrastructure of the internet, so embedded in the architecture of the digital world that even their competitors had to rely on their services.

This is the right take. I disagree with the disappointment around this article.

It’s about the fact that tech giants are running the infrastructure of the internet. Its the federal highway program of the 21st century. It’s a critical piece of public infrastructure. I can’t even begin to wrap my head around the death and mayhem that would ensue if AWS and Google simply packed up and turned off the data centers.

When we consider the future and safety of our country we should keep these things in mind.

Imagine an unusually well-attended AWS all-hands was hit by a freak meteor. What do you even do? Take enough engineers off the oncall rotation and things will eventually fall over. I think the government would actually need to take emergency action at a level we don’t understand to prevent the collapse of our financial system beyond god knows what else. It would be utter mayhem. Globally.

That said, no one takes this more seriously than AWS. I really believe this. I sat in meetings regarding how to recover if we lost entire states, like literally what if we lose the Virginia to a catastrophe. (I worked on the Key Management Service or KMS which underlies much of the rest of the infrastructure.) But it is good and correct that our representatives are questioning our tech giants about their role in the future of our civilization. They’re playing perhaps the most critical role, and that should scare you.

I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you?




Companies are way faster, focussed and efficient than any form of non-profit or governmental organization in building infrastructure that supports their cause.

I'm eternally grateful to Google for HTTP/2, Chrome, Web Vitals, Angular and so many other puzzle pieces without which the whole Internet wouldn't be what it is today. It showed foresight, wisdom, excellence, responsibility and stewardship.

The problem starts where growth ends, and all growth needs to end somewhere. There are only so many people on the planet with 24h a day to spend.

When that problem starts, a corporation still has to maximize shareholder value, and that's where the formally great stewardship gets endangered.

And then Google used its web browser, standards and search monopoly in order to

- shove the open-but-really-not-open AMP "standard" down everybody's throat

- eat up any kinds of comparison competitors by stealing their content into the featured snippets

- ...

So going back to your question, I'd still mostly trust in the benevolence of AWS — but I have zero doubts this benevolence will go away as soon as there is an opportunity to crush a competitor by ignoring that benevolence.

To quote Jeff Bezos here: "Your margin is my opportunity".


> Google for HTTP/2, Chrome, Web Vitals, Angular and so many other puzzle pieces without which the whole Internet wouldn't be what it is today. It showed foresight, wisdom, excellence, responsibility and stewardship.

That is a very creative way to describe forcing one-side standards via monopolies. By your logic, you also love IE4, HTTPxml, flashcookies, etc.

And you call microsoft business tactics in the 90s as "foresight, wisdom, excellence, responsibility and stewardship" because that is the text book those companies are following to push those "technologies" that are completely self-serving to show rich ads to you and control which video your browser can play or not.


IE gave us XHRs, which are kind of important to the modern web.

Monopolies have agency to remake the world; ecosystems have competition over a relatively fixed landscape. Each has their pros and cons, but to build a radically new social structure, nothing beats a monopoly.


> in building infrastructure that supports their cause.

Emphasis on this part.


To fully spell it out, the causes of a company don't always align with those of everyone else. Profit motive alone is not sufficient to adequately align companies with the interests of broader society.

Which is why a market that is completely "free", where actors are not beholden to any laws or regulations, is not a good idea.


> Companies are way faster, focussed and efficient than any form of non-profit or governmental organization in building infrastructure that supports their cause.

So what? What does that have to do with anything? I would disagree with this statement wholeheartedly. Private enterprise did not save the midwest from the dust storms, government did. Private enterprise did not save the world from the Nazi's, government did. Private enterprise will not save us from global warming, government will. Indeed, all of these problems are generally started by private monied interest.

Without the government, those companies can't even operate, so your entire premise is totally backwards.


Only technology can save us from global warming and only private enterprises like Tesla and others can develop and - more importantly - cheaply scale that technology. There are billions of Chinese and Indians who will become the equivalent of American middle class in the coming decade and no amount of conservation or carbon tax is going to prevent the acceleration of global temperatures that would cause.


>I can’t even begin to wrap my head around the death and mayhem that would ensue if AWS and Google simply packed up and turned off the data centers.

There's lots of companies out there that would cause mayhem if they simply closed up shop. But there's plenty of alternatives out there and self hosting is still an option. We can do without AWS. It would take more than a day to migrate but we could do it.


Sorry, you seriously have no idea what you are talking about.

Self hosting our company infrastructure would require hundreds of physical servers. That's the easy part. Put those servers into a room, where they can work, they have power, network etc. Yes, we could move to self hosting. Yes, the company is dead before the servers are shipped, if amazon goes down everywhere and not recover. If they lose a region we are fine.

And we are a medium company.


> Yes, the company is dead before the servers are shipped, if amazon goes down everywhere and not recover.

I don't understand why you would allow this situation to arise, though. What are you waiting for ordering those servers right now, while Amazon is still working?

If you require hundreds of physical servers, it's basically guaranteed that on the long term it would be cheaper to host them yourself than let Amazon do the job, with the risks that that entails.

It's fine to take this kind of risk while you're not sure if you're going to operate long term at all, but once that's established and if you're not a small shop, depending absolutely on one company is unforgivable.


> What are you waiting for ordering those servers right now, while Amazon is still working?

The startup costs, ongoing maintenance, and organizational costs of mitigating this kind of risk have to be considered with an evaluation of the likelihood of the event occurring. And other things the company could do with those same resources.

Do you have a fully equipped backup home somewhere far away you keep stocked with consumables and perishables in case where you live is destroyed by a meteorite one day? If not, why not?


> Do you have a fully equipped backup home somewhere far away you keep stocked with consumables and perishables in case where you live is destroyed by a meteorite one day?

Yes. I have some friends and family members who live anywhere between 1 mile and 10,000 miles away, in both same country and other countries, that I can turn to provide me with food and shelter if my home is no longer available for some reason.

Regardless of them, I can take shelter short term in a hotel or other rental facility, as I have sufficient savings that I'll continue to be warm and fed.

The comparison doesn't hold up. A business cannot just reform overnight if critical dependencies disappear. A person can.


Similarly, a lot of companies use systems that they believe could get the up and running on a different cloud vendor should that day come to pass. Until then, they don't maintain a full second or third environment.

This is equivalent to knowing you could get a hotel room or crash with a friend, instead of maintaining a whole second home yourself.


I'm not a company myself so it doesn't really make sense, although I do consider my parent's home as a backup if I need to leave my house for some reason.

But my company does everything we can in order not to have a single point of failure, however unlikely it is to fail. Because low probabilities of something still mean it can happen, and you don't want to allow that if you can avoid it.


I'm a prepper so yeah.


?

I'm not sure why the hostility or what exactly you disagree with. I said that we could migrate off of AWS but it'd take more than a day. You say that your company can migrate off AWS but it would take more than a day. Where's the disagreement?


I'm curious how many companies would survive it at all.

Sure, you could _migrate_ off of AWS if AWS was still alive. Could you restore your data and start your business from scratch if you lost everything? I suspect many rather large businesses could not.


So what is so special about AWS in this regard? I've worked on state contracts with IBM and Northrop Grumman to provide data center services which includes everything from VMs to fully-managed Windows AD/Exchange environments. This has been an ongoing thing since at least the mid-90s and if it wasn't going to be Amazon it would have been someone else.

Ask yourself why Enterprise isn't in the cloud yet. There are significant reasons.


So it's "we have, without necessity, put all our eggs in one basket, so now this basket should be treated as a utility so we don't have to spent a bit more money & time to protect us against unforeseen consequences of that basket breaking"?

What if there's an error and Amazon bans your account, do you just shut down and give up? There's no contingency plan whatsoever?


Did you just make a strong argument for why these services ought to be “nationalized” like public infrastructure?


Well I'm glad you thought my argument was strong!

Yes, that is what I believe. I see a path where after Amazon creates something that becomes a de facto public good they hand it over to a highly competent government agency full of the world's best operational engineers. Those people turn it into a public works project, and it has legendary 25-nines uptime.

Amazon engineers are excited to see their work become part of the fabric of the Galactic Alliance of Good and Kind Beings and no one does oncall like a Quarkian, am I right, 700-hour sleep/wake cycles how cool is that, love working with them.

Do I see a path to get there from here? Uh, not at the moment. We're still working on "hitting each other is bad" so I'm feeling pretty depressed about our outlook.


Yeah, in the real world nationalizing AWS would have the same effect as just tearing it down. Which is the opposite of what you want if you actually think of it as an essential service.


I don't know how to think about the idea that the government is totally useless for technological problems. If we just accept that I think we're probably just doomed to be overtaken by more advanced societies. Agreed that we can't just dump AWS on the USDS or whatever tomorrow, but if we never plan for public digital infrastructure we're screaming towards some kind of Gibsonian cyberpunk future where the tubes are privately controlled.


Once upon a time there were millions of individually-owned windmills in the U.S. There were hundreds of windmill manufacturers. Around the 1920s that went into decline. It didn't take long before all of the U.S. was dependant on power companies instead of harvesting their own energy.

Perhaps you would argue that "if we never plan for public energy infrastructure we're screaming towards some kind of Gibsonian cyberpunk future where energy is privately controlled".

Do you know any Americans who are completely self-sufficient in food production? Should we plan for public infrastructure there? Or is it ok that our food production is privately controlled?

I personally think it is foolish to move a company's data entirely outside the company and place it wholly at the mercy of a single third party outside your control. But I don't see it as needing some plan to put it into the hands of "public digital infrastructure".


I think power is an interesting example because everywhere that it is privatized it tends to be a mess. Shit, PG&E burnt down half of California... what, on behalf of its shareholders, trying to cut corners and make them a buck? Who the fuck did that help? So yeah, I'd really rather see basic internet infrastructure be made into a public utility. Maybe 100 years from now S3 might make sense too.

The problem is that we're so busy arguing about whether we should even HAVE a government it's incredibly difficult to do anything smart with ours.


Nice, you brought up two markets that have plenty of government involvement.

Do you know who own the Hoover dam? Have you ever heard of this thing called National Grid?

Lets adress food production - ever heard of farming subsidies?

Did you never read anything about how these markets function? Because if you do, you will realise that government absolutely does have a hand inevery indutry that's essential to survival of organised civilisation.


If you worked for the Department of Defense having Windows Vista based computers at their oldest you would be a heroic figure in their IT. They earned their bad reputation all on their own. Calling it sclerotic is an insult to those with scoliosis, especially as obstinance and incompetence didn't cause their disease.

The zeal to nationalize has gone from."theorerical improvement" to dogma to outright mental sickness. At this point just accept that partnerships where entities specialize in what they are good it can exist already instead of going full paranoid improbable worst case scenario. Amazon would not do well if they tried to occupy a small nation-state. It would be trivial to the US government. (Whether it would be wise for either to do is another question.)

The treatment of "nationalizing being a bad idea" as a threat is a black and white insanity like equating being "unable to be murder someone without consequences" equates to the target automatically becoming the immortal god king eternal tyrant over all.


The bit of this that I think is easy to overlook is that if this is the status quo indefinitely I assume this is a very long term problem.

Amazon is a baby, it's a new-ish company in the grand scheme of things. It's a huge, huge baby, it has accumulated power incredibly quickly. Where is it gonna be in 200 years? Will the federal government still be more powerful?

No one is saying nationalize Amazon tomorrow. But we haven't even managed to give our citizens good internet service yet, we're way behind here, and we do not appear to be catching up.


American railroads were privately owned from the beginning, until passenger rail started dying off due to competition from airlines, when the US nationalized passenger rail. Freight rail is still private and owns the actual tracks, and passenger rail is even less important today outside of Acela, which would probably do even better than it does today if Amtrak were privatized and allowed to drop the legacy long-distance lines that only lose money in order to reinvest in Acela.

AWS isn’t the highway system. It’s the railroad. Now, railroads aren’t all good. Farmers were once relieved that the railroad would buy all of their produce but ultimately grew resentful that the railroad had become the only cost-effective buyer and shipper and could dictate prices to them—this is the etymology of the term “railroaded”. And eventually with the interstate highway system, the railroads had to compete with trucking.

And from that perspective, AWS isn’t even the railroad. It’s a trucking company. The railroad owns the tracks. AWS doesn’t own the tracks; they operate on the same highway that Azure and GCP use. If someone invents something better than AWS, people will invest in migrating to it; that’s how they ended up on AWS in the first place.

Does that mean that the network connectivity should be a public utility? Maybe at the last mile (municipal broadband?) but the backbones seem to work fine as a competitive market, too.

Did you know that the US was one of the only countries where radio was privatized when it was first invented? Most countries nationalized their radio airwaves and had the government control all the broadcasting. We do have the FCC to allocate spectrum but we also had a remarkably free market in broadcasting and, as a result, led the world in radio technology well into the 20th century. Even Britain was so committed to controlling the radio that as late as the 1960’s, the unmet demand for listening to rock music on the radio could only be met by pirate radio stations broadcasting from ships at sea.


I think very few organisations could run AWS as successfully as Amazon.

Why do you think the U.S. government is one of them? They previously have shown no organizational aptitude for developing software far less complicated than AWS.


Let me be clear, I do not. I think the U.S. government in its state can barely run itself. It appears to be actively working to be as net harmful to the general populous as possible.

I genuinely believe in Amazon and its humans. But at some point in the distant future internet infrastructure will be so ingrained in our lives that it IS our society, and I want to live in a Democracy.

I want our representatives to continue asking powerful companies how they're handling our data, and if they're becoming too big, and all of these things. I see this as a fundamentally good thing.

I don't believe that AWS should be handed to the government, that would kill AWS. I believe that when something becomes critical to the public we should talk about what that means, and how we stay safe.

Amazon holds themselves to an incredibly high bar, and I really applaud them for that. It made me proud to work there when we talked about how critical the work was, and I was awed to see how far we would go to recover from disaster.

But Amazon has this saying about relying on mechanisms and not best intentions, and relying on Amazon's best intentions is foolish as a citizenry; we should continue to question them. And if at some point in the future we realize that AWS is more important than the self-governing society that we live in we should ask whether that is okay.

Tech companies have become some of the most powerful entities in the world in short decades. In generations they could become the world. We shouldn't rely on their best intentions.


> they hand it over to a highly competent government agency full of the world's best operational engineers.

Fantasyland. As soon as the profit motive disappears, you’ll have the DMV.

As an example, the “world’s best operational engineers” — what happens when one of those engineers isn’t world’s best anymore? Can you fire him/her? Because in government, it’s extremely hard to fire mediocrity. It’s hard to even fire for incompetence. You’ll have unions getting involved and it will quickly degrade to a public school level of tomfoolery. You’ll have strikes when those world’s best engineers want more money or more holidays. Just look at how good the French train system is — until there is a widespread transit strike, then the country grinds to a halt. I don’t want critical business processes in the hands of a public union that can shut it down whenever they feel offended. With a profit motive, there is a shared incentive to make things better — you provide the service, you continue to get paid. When a government agency employee gets paid regardless of outcome, there is no incentive or fear when a service gets slowed down or interrupted: their job is secure.

Having experienced public sector strikes in France on multiple occasions, there is no way I’d trust the infrastructure of the internet to be benevolently operated by a government entity.

Imagine the politics! Imagine people operating an e-commerce store selling sodas — sorry, but since we have a government mandate to reduce obesity, We are going to cut off your service. You run a porn site? We don’t like that, so we are going to cut off your service. You are running a website critical of our government/union/political party? We are going to slow you down. You don’t have correct diversity of your workforce? We are going to penalize your service.

As soon as you let politicians control something upon which you absolutely depend, you have given those politicians unlimited power. “Vote for us or your servers will go down.” “Agree to our demands or your business is toast.”

It’s scary to give government that power. If AWS goes down, a person will move their business somewhere else. But when the Government system goes down, there is no other options.

Governments can and should regulate, but they absolutely shouldn’t own. And, the regulation they do employ should have the sole purpose of ensuring competition can thrive. Look at how banking regulations have kneecapped the marijuana industry — now imagine that sort of power controlling the entire internet infrastructure.

We need more competition, not a socialized monopoly.


> Fantasyland

I love that in a comment about fucking space aliens and competent governments the thing that made you call it fantasyland was the latter.

Okay, the French have an incredible train system that goes down sometimes when the union feels the workers are being treated unfairly. That compares poorly to... what? To our total lack of a train system? To our anemic unions and abused workers? I read this and it sounds like a success story for government competence.


It's a natural good like water at this point. And the psycho-social mechanistic designs which go into what you consume are much more dangerous than added fluoride.


That's a great idea! At this point, you could totally say that Larry, Sergey, Jeff, and Mark have all won at capitalism. Let's cash them out and give their shares to the people.


I love this.

Imagine, your company hits $1T. POTUS gets you on stage. They tell you you did SO GOOD. We're really proud of you. We're gonna make you a statue in the Hall of Really Good Capitalists, and we have the best news. You are SO GOOD at this that we're gonna let you do it again. You're poor now, but we're gonna put your name in history books! If you get this rich again on hardcore mode we will make your statue bigger!


I'm not suggesting taking away their money. They built a trillion dollar company that's vital to the national economy. Yay! Now we (the people) buy it from them and it belongs to all of us.


It's like a game of cookie clicker. Once you get all the cookies, you ascend and start clicking again.


No, I don't believe any piece of critical infrastructure works if privately owned. I'm glad the highways aren't privately owned. I'm glad USPS exists so it can deliver mail unprofitably to my rural community. I'm glad water isn't privatized.

The industries that exist privately in such crucial sectors must also undergo heavy regulation.

I have no misgivings that Amazon and Google at their highest levels, pretty much only work out of private interest. It's nice when they align with the public interest but as soon as they don't, I fear what will happen.

Energy companies have no problems with polluting the environment. Between the environmental, political, and military consequences, it seems destructive when a profit motive has leverage over the rest of the country.


> No, I don't believe any piece of critical infrastructure works if privately owned. I'm glad the highways aren't privately owned. I'm glad USPS exists so it can deliver mail unprofitably to my rural community. I'm glad water isn't privatized.

Aren't some highways in the east coast privately run toll roads? And right now Trump and the Republicans are trying to gut the USPS.


I don't support highway tolls b/c they are regressive taxes. Trump and the Republicans have been trying to gut USPS for years. Now they seem extremely urgent to advance their plans.


Absolutely. So long as the employees who run it believe in upholding that benevolence themselves.

Disclaimer: I'm a current TAM with AWS.


As Snowden says, tech workers are complicit in how their companies hurt society: https://www.vice.com/en_us/article/wxqx8q/snowden-tech-worke...


The real problem is "hurting society" is a bullshit charge which lets you call yourself a hero for making Socrates poison himself.

The sophistry is all brain-hacking bullshit best dealt with like a poisoned drink - thrown in the face of the server.


If that's true, then so are those that invest in big tech companies.


Absolutely. The material interests of the shareholders ultimately drive the decisions that these companies make. If you invest in them, you are complicit.


> I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you?

I do not (I spent 6 years at AWS, 2008-2014), but don't believe in "evil" either; to support your point, though, the "alternative" to AWS is certainly worse.

I agree with your point that taking AWS away would cause a lot of problems (and actually, Azure and GCP are the redundancy to AWS; hopefully we don't get any meteor to hit that crowd, but if we do, hopefully there's only one, not 3).


>I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you?

I have no reason to doubt the benevolence of AWS at present, but how much faith do you have in the perpetuity of that benevolence? There is a tremendous power imbalance between Amazon/Google/Microsoft and the companies that depend on them for infrastructure, and the nature of capitalism and the declining rate of profitability means that eventually, the markets will demand for that power imbalance to be exploited economically.

When the Manorial system emerged in Europe, it was a largely benevolent force offering stability, security, and community in the wake of the collapsing Western Roman Empire - within a few generations it morphed into 1000 years of serfdom.


> how much faith do you have in the perpetuity of that benevolence

None! Literally none. This is a huge risk for our country, and we should thinking about what it means for the next 1000 years, not the next 10.

Bezos appears to truly, truly believe in Customer Obsession. The culture definitely softened while I was there though, and I can't imagine how it'll change in 200 years. I can't guarantee his great grandkids won't be dicks. I don't really want to live in the corporatist city state of Seattle By Amazon with its "benevolent" AI overlord and its drone stazi in 2234, ya know?


I didn't realise that KMS underlay AWS' internal infrastructure, literally the same KMS service that's exposed to users? Do you know if AWS' internal infrastructure is actually documented publicly in terms of how services fit together internally anywhere? Or is that all considered secret beyond what's exposed to users?


Yeah, this stuff isn't secret. Note how if you encrypt your S3 bucket it uses a KMS CMK; that's an dependency between S3 and KMS, and yep it's the same KMS. This is true for probably over a hundred AWS services that rely on KMS; it's a "foundational" service. (As is S3.)


Sure, I'll believe it. I've not heard enough to the contrary to not believe it, and you make a good recommendation. But benevolence is transient and brittle. I don't want to depend on it.


> But benevolence is transient and brittle. I don't want to depend on it.

Yup, exactly.


That makes me wonder, how did AWS or other big cloud providers handle the stay at home orders in March? Did they just keep having people go in?


Except where absolutely necessary, we've been WFH since March. We were allowed to expense some things for setting up a home office.

The few physical workspaces that are still active out of necessity have a ton of sanitation measures in place. Regular cleanings, temp sensors, etc. I knew someone was paying attention to detail when those "foot handles" started appearing on the bottoms of the bathroom entry doors.

A lot of my colleagues and myself have been busier it feels than any time prior. There is honestly a real sense that a lot of people are suddenly depending on us more than ever. It does give the sense that we're critical infrastructure, and the leadership communicates and appreciates as such.


> the leadership communicates and appreciates as such.

What does this mean, especially the latter?


The data center operations team is probably still in the building. But the software engineering teams which is what I’m familiar with are all working from home. We don’t have to go to the data centers anyway usually.


I'm pretty sure data center workers were considered essential workers.


not OP, but my AWS buddy was told to work from home back in February if I recall correctly.


> I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you?

As many problems as capitalism has, often monetary goals align perfectly with the 'ethical' action.

In my experience, capitalistic companies that optimize for long term survival (as is characteristic of Amazon) usually act in a less malevolent manner. However, any company that sees a revolving door of CEOs, will be seen to optimize for short term gains at the cost of consumer friendliness.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: