Seems a bit too tailored though - "To even make a bid, a provider must maintain a distance of at least 150 miles between its data centers, a prerequisite that only Amazon can currently meet. JEDI also asks for “32 GB of RAM”—the precise specification of Amazon’s services. (Microsoft, by contrast, offers only 28 GB, and Google provides 30 GB.) In places, JEDI echoes Amazon’s own language: It calls for a “ruggedized” storage system, the same word Amazon uses to tout its Snowball Edge product."
It sounds like the military wants what Amazon can provide, but is allowing other competitors to step up and give it a better deal. There are obvious reasons why a military customer would want distance between data centers and ruggedized storage, that's surely one of the reasons why Amazon has implemented them.
I think the major concern perhaps isn't the particulars of the spec, or even that maybe only Amazon can fully meet some of them, but that they are so clearly based on Amazon's exact capabilities that it raises the question of whether another provider will even get a fair hearing.
As an analogy, imagine that there's a new senior job opening in your organisation, and both you and a colleague are considering applying. You're both pretty skilled, with resumes that are quite similar, albeit not identical. You hear through the grapevine that you co-worker plays golf with a couple of the people who'll be on the interview panel, but you trust them to be impartial.
Then the official job description gets published, and it's identical to your colleague's resume. It's as if somebody has gone through it line by line and translated that into the job's desirables. In some cases, the requirements are verbatim copies of some experience your colleague mentions.
Now, you're convinced that you could still do the job. And perhaps if you really knock it out of the park during the interview, and make a deliberately lowballed salary demand, you've got a chance. But, at this point, would you believe that the process is impartial? Or would you conclude that, at the very least, it is so stacked heavily in favour of your colleague that your chances of swaying the interviewers over to your side are slim, and perhaps nonexistent?
When I was in college, I was a systems administrator for a lab as a part-time student job. After I graduated, they wanted to hire me full-time. But it was a state university, so they had to open up the job to everyone, and they had to have a specific window for applications. So... my boss wrote the job description exactly around my resume. Lo-and-behold, I got the job.
There is a 50-miles requirement for financial companies that came into effect after the 9/11. So I think 150 miles isn't all that unreasonable for military.
I'm going to speculate (Occam's razor) that the majority of their server portfolio are 32GB servers in those 400 datacenters, and so this is what they need, and so they documented it as a requirement. For data center migrations, this is fairly typical.
Google was and likely still is in the running for JEDI, because Eric Schmidt is an advisor for the DoD and nudges them towards Google, and JEDI was the whole point of signing the relatively small Maven contract. Google offers pretrained AI models and hands-on consulting to put AI on drones, and in exchange the DoD gives them a piece of JEDI. But Maven turned into a huge controversy which doesn't make Google leadership look good in the eyes of the DoD, and they probably wanted to go all-in with the market leader anyway (seeing as they're very conservative), so now maybe Amazon will just win the whole thing.
"JEDI also asks for “32 GB of RAM”—the precise specification of Amazon’s services. (Microsoft, by contrast, offers only 28 GB, and Google provides 30 GB.)"
What? Microsoft only provides 28 GB of RAM, and Google only 30 GB? I'm pretty sure neither Azure nor GCP even launched with limits that strict.
Those seem like standard specs to me. 32 GB is a standard number for RAM. Also, I'd be shocked if other major cloud providers have all their data centers within 150 miles of each other - that would eliminate geographic redundancies.
This took about 30 seconds to fact check. Google offers an n1-standard-32 with 32GB RAM, and Azure offers A4MS and B4MS with 32GB of RAM. I can't even find any instances from either provider for 28 or 30 GB.
I skimmed through some of this and I don't see anything obviously rigged for AWS. There are mentions of 32GB VMs, but it doesn't say they have to be exactly 32GB. I found the 150-mile requirement but it also doesn't look AWS-specific. The description of the ruggedized "tactical edge" device doesn't sound identical to the existing Snowball Edge.
Everyone seems focused on the numbers 32GB or 150 miles...Amazon has moved into lobbying in a big way. That is the actual story. Bigger than major investment banks. Bezos owns the only paper in Washington that actually does journalism. People involved with Amazon and the head of the DOD have been instrumental in crafting the proposal for JEDI.
Just because you learned AWS and it's been good for your resume doesn't mean that they aren't being underhanded in D.C.
If you are going to be a fan boy about it, then you should be excited that they seem to be outmaneuvering established players in the federal sector. That's what's happening.
This isn't "they built a better mousetrap", this is "they hired all the people that are ever asked about mousetraps, and they changed the language that you have to use to ask for a mousetrap, and they took over one of the better outlets that runs stories about fraud in the mousetrap business. Oh and their commodity mousetrap business is one of three equally good ones."
I know that you think AWS is WAY better and that it makes a difference. It doesn't. Azure and GCP are just as good for most of the things that people are building these days. Sure there are differences, but it's a commodity market.
More than reasonable. Not awarding the contract to an organization with the requisite scale would be cause for investigating the process.
In my mind, the likelihood is that people would complain no matter who got this contract. If it went to Microsoft, the article would be about all the high ranking contacts between Microsoft and the Pentagon. As a pragmatic matter, any organization with the expertise and scale to realistically execute this contract will have deep connections with the Pentagon. At several levels.
Why would such a massively important backbone of US defense want to rely on only a single provider? It should be using all 3 of the major players (or others) and even some on premise stuff. This seems like a nightmare waiting to happen, and crony capitalism at its finest. The fact that Amazon says in the article that using fail overs to other providers is some hassle...is just amazing.
Because government contracts of this size play by different rules. They're not using Amazon as a customer like normal companies -- they're hiring Amazon the firm to build something specific for them.
A government contract where the sole company that can ever win the contract also says that fail overs to other providers is bad. So if Amazon goes down, the DoD is shit out of luck. I don't see how that's not a national security problem. Not only that, but after the $10 Billion is expended the DoD is vendor locked. How people don't see this as a major problem/cronyism is baffling. The vendor lock itself is bad enough.
Because in this case it relates to national security and if the provider goes down the US is potentially at more risk than it would have been if it had multi-vendor fail overs. I can understand it not being justified for some government programs or companies due to expense or impracticality, but the DoD has a massive budget and a huge pool of resources. Not having multi-vendor fail overs just seems irresponsible given the stakes.
Amazon employee here so I am biased. While multi-cloud might give you more redundancy, it:
a) prevents you from using any of the custom features that many each service great (for instance lambda, aurora, dynamodb or in the case of google cloud, some of their ML tech). Might as well use on-prem if you’re not going to using AWS/Azure/GSuite feature on top of IaaS
b) requires you to set up multiple alarms, logs, metrics, on different providers which can themselves be the source of bugs
c) doesn’t provide you much more availability/redundancy them simply using multiple regions
d)while this may give you long term negotiating power, you will likely just save more money with long term exclusive contracts.
The DOD will also likely never be 100% migrated to AWS and there will probably be more contracts
> d)while this may give you long term negotiating power, you will likely just save more money with long term exclusive contracts.
Uh, This would be true, if one expected prices (and technology) to remain stable, but since that isn't the case, maintaining re-negotiating power is quite a bit more advantageous than you imply.
Yeah not sure. I think it depends on the contracts. Maintaining infrastructure on 3 cloud providers that don’t work the same is costly.
All of these services provide pretty steep discounts (as a percentage of original costs) for customers who are able to provide accurate capacity forecasts and make long term commitments.
And the while the “technology gets cheaper” argument is true, I believe it is generally understood that while AWS/Azure/Google can use this trend at the scale they operate at (and then pass the savings to the customers), most cloud customers would spend more money hiring additional engineers to maintain more complicated infrastructure then they would save trying to take advantage of this trend.
> Maintaining infrastructure on 3 cloud providers that don’t work the same is costly.
Maintaining the ability to move between providers is not at all the same as maintaining your systems on all three simultaneously.
> All of these services provide pretty steep discounts (as a percentage of original costs) for customers who are able to [...] make long term commitments.
Sure. But the question is, at the end of that commitment, are you locked in to that provider by the cost of having to redevelop for a different provider (which gives your existing provider an unearned pricing advantage), or can you switch providers because you had the foresight to make sure your deployments were portable?
> most cloud customers would spend more money hiring additional engineers to maintain more complicated infrastructure then they would save trying to take advantage of this trend.
Sure, because most cloud customers are small and are drawn by the prospect of not maintaining infrastructure at all (in other words, they're actually SaaS customers by preference), and the non-portability of G-Suite's integration with GCP as compared with Office 365's integration with Azure is relatively immarterial compared to the non-portability between G-Suite and Office 365.
But (right now, at least) at the scale where you are for example selecting a cloud orchestration solution, going for the one that provides better portability between different providers with an abstraction layer doesn't really impose additional costs (because you'll need to develop internal expertise in whatever solution you choose).
> To even make a bid, a provider must maintain a distance of at least 150 miles between its data centers, a prerequisite that only Amazon can currently meet
I don't get this one.
Azure's government data centers are in Virginia, Iowa, Arizona, and Texas. Their DoD data centers are Virginia and Iowa. I don't know where they are in those specific states, but the only pair that could be less than 150 miles apart are Texas and Arizona, and that is only if the Arizona one is within a few miles of the SE corner of Arizona and the Texas one is up in the NW sort-of corner of Texas.
FWIW, it's a long-time, standard complaint by the losers that the specs were written in a way that favored the winner. On one hand, it's an obvious tactic for insiders to favor certain outsiders.
On the other, of course the winner has capabilities that better suit the specs - that's the goal of competitive bidding and we should hope that it's true of all winning bidders. So the fact that the winner's capabilities match the specs well doesn't tell us anything; if they didn't match well, it would be signal of corruption in the selection process.
On the other hand doesn't match much with reality, and "regulatory capture" is well documented and it seems that is exactly what has happened here. The big 3, AWS, GSC and Azure should all three be used, at the very least as fail overs.
This is a bad article. I'm a fan and I don't like my favorite defense contractor maligned? Get a grip.
This is Amazon trying to engineer a permanent piece of the public largess for themselves. That's all. They aren't better at this than anyone else. For that matter, the same could be said for all the other top tech companies. I'm just tired of AWS fans saying that it's strictly merit based. That's not how anything works in big business. It's naïve and childish.
Amazon being in line to win a large government contract while the President openly despises Amazon is actually a testament to proper government procurement. The real scandal would be if the President ordered Amazon to not be considered because of his personal beef with the Washington Post (and IMO general jealousy regarding Bezos who is actually as successful as Trump has always dreamed of being).
tl;dr It is not a scandal that the leading cloud provider is in the pole position for a large cloud contract.
More concerning is the fact that the UK government is happy to store UK data in the US with AWS under its "G-cloud" scheme. At least the US guys have picked a "local" vendor.
G-cloud is a pretty broad scheme that covers all sorts of computing services and consulting at all levels of secrecy. It's a marketplace with many different uses.
If you do a search with, for example, a minimum provider staff security clearance of DV (which is pretty high, would apply to many matters of national security), all the major cloud providers disappear from the results, and you get smaller companies based in the UK, who use UK datacenters, and who are seemingly adding their own layers of encryption on top.
Again, this is all publicly accessible because 99% of the use is not the intelligence services or military, it's instead random government departments, local government offices, hospitals, doctors surgeries, etc. This is designed to replace most computing purchasing in the UK public sector.
I have DoD clients currently doing app dev on new and replacement (for legacy) systems. This writers attempt at making the DoD contract look like a conspiracy is infuriating. The developer experience in the current data center ecosystem is toxic. The infrastructure and insecurity woes compound daily. I don’t care how much Bezos makes, he has a great product and DoD, nay the country, need this. GovCloud would be ice water for people in hell.
I’m mad about this because this affects individual contributors, on up to the country at large. Currently money is wasted and systems are built in ridiculous ways. Imagine each sub program in a branch of service rolling it’s own IAM. That’s just a glimpse at what’s going on. The undifferentiated heavy lifting is unfathomable.
Before you respond with virtue signaling about war know that most DoD software is for logistics. Done better it boils down to not wasting tax payer money.
> Before you respond with virtue signaling about war know that most DoD software is for logistics. Done better it boils down to not wasting tax payer money.
It is not "virtue signaling" to be actually anti-war. Most virtue signaling on this topic revolves around the motte-and-bailey of "support the troops". Furthermore, if one is anti-war, then making the military less expensive isn't actually a good thing.
(or you could have just left this last bit out and stayed less political)
The important thing to realize is that this is par for the course with huge government contracts. There is always rancorous mudslinging, submarine stories, etc.
The reality is probably that Amazon is using some shady tactics to try and close this deal, but so are all the other bidders (it's not like Microsoft, who is probably the second-most likely to win JEDI, doesn't have an extensive set of Pentagon connections too). This is just how the game is played and you've gotta roll with it. Eventually this will be awarded one way or the other and we'll all move on.
Another thing to look at:
Microsoft is almost definitely going to land DoD/IC-wide O365.
AWS taking IaaS compute might be the way of giving both Seattle juggernauts some love.
The cottage industry around this specific contract is going to be massive.
Whats really interesting though, is that, a huge swath of talent will be ineligeble to work on this as GovCloud/FedRamp requirements for US Citizen employees, physically located in the US to be able to access and work on any of these systems really does impose some limits on the pool.
And while, regardless of how you look at this, it is dysopian from the Corporate-yberPunk-Future perspective (small book seller is now the richest man in the world, and is responsible for the company which monolithically provides the largest government military its computing infra) - There arent really any other alternatives.
I'd say that the opportunity sitting right in front of the talent pool who couldnt be legally able to work on GovCloud implementations would be to ramp up their training and setting up consulting groups who can.
a huge swath of talent will be ineligeble to work on this as GovCloud/FedRamp requirements for US Citizen employees
That doesn't stop some DoD contractors from still subcontracting foreign nationals. Crazy what goes on.
Question: How do you vet a non-citizen for work on a DoD system? Let me give you a tiny example. Let's say you have a system for keeping track of jet parts. Super important stuff. What's to say a foreign agent couldn't make a subtle "mistake" that allows for misuse or misallocation of parts by not properly storing their flight history? Planes fail and fall out of the sky. This could totally happen (not saying it has although, again, the stories I hear JFC) and a competing state would be foolish not to try it.
>...a competing state would be foolish not to try it.
Yeah, there is no way that this contract is not the largest target possible...
I am honestly fascinated and really interested in the cyber-warfare that has, is and will be going on.
Just thinking about Stuxnet/Duqu is pretty amazing - the things that are going on that we don't currently know about and the things that will happen are very interesting.
What I would like to determine, as an emergent tell of the electronic warfare between states is the true disposition between "enemies" -- i.e. we know that Iran and Israel really are enemies and assassinating targets of opportunity (stuxnet ++ engineers previously killed), but it should be telling when publicly on state denounces another (e.g. US denounces NK), but then doesnt take any back-channel actions against the state - or the reverse...
Bezos was already on a Pentagon board [1], and already had contracts with the CIA [2]. Corporations run our government almost entirely, and are now getting into Military Industrial Complex. This is nothing new.
If you "want to do something BIG", that "BIG" thing is, by definition, going to have an impact. That is to say, it will exert some form of power. And you're probably going to have to do that with other people. Probably in the form of a corporation.
Why can't they build this themselves? If the DoD is such a large customer, has such precise needs and requirements, they should build their own datacenters with their own staff and manage it in-house.
> (snip) And you're probably going to have to do that with other people. Probably in the form of a corporation.
The government manages a nuclear arsenal and a space flight program. There's no valid reason they couldn't build and run their own server farm without giving it all away to Amazon/Microsoft/Google/Oracle. There's no good reason everything in the entire nation has to be privatized like this.
But maybe that would cost 2-3-4 times as much as buying it?
Why shouldn't they instead go with the best in the world?
Presumably, the whole reason why they are doing this, is because they DID build it themselves a decade or so ago, and it didn't result in the best outcome.
Government IT has quite a terrible reputation these days. The government should instead be pushing to have the best in the world.
While this does seem like an insiders rigged game, is that necessarily bad? Business is about relationships. It’s has always been this way and will always continue to be this way. People do business with people they know and like. While the govt is not technically a business as such, awarding a contract is still a business transaction. If the DOD know and trust people at amazon to do a good job, then why shouldn’t amazon get the contract? AWS is an industry leader (if not the leader? Somebody correct me) who obviously knows a great deal about doing a project of this magnitude. It seems to me they would probably do as good a job as any of the other candidates (which seems to be realistically azure or google cloud) so I don’t really see why this is a bad thing.
TLDR: amazon knows people at dod, they decide to do business together, who cares?
Given that Oracle only exists as a company due to the fact that they built one of the first/earliest RDMSs for the CIA makes me not shed many tears for them...
