
In Pentagon Contract Fight, Amazon Has Foes in High Places - hvo
https://www.nytimes.com/2019/08/02/us/politics/amazon-pentagon-contract-trump.html
======
jacksnipe
I’m not the biggest fan of AWS, but I feel like giving $10B to Oracle or IBM,
well, you might as well just burn the money.

~~~
tssva
The options aren't just give $10B to AWS, Oracle, IBM or Microsoft. The JEDI
should have been a multi-award IDIQ (indefinite delivery indefinite quantity)
contract. With a multi-award IDIQ the each task order issued under the IDIQ is
competed for among the multiple vendors the IDIQ has been awarded to. It is
actually fairly unusual that this type of IT services contract was not done as
a multi-award IDIQ.

Having to compete for each task order is thought to reduce costs to the
government. Of course having to compete each task brings its own additional
costs which don't necessarily make the cost saving proposition true. The big
benefit I see to this contract is that it helps prevent vendor lock-in. I
don't worry about the next 10 years and $10B, but what happens after 10 years
when the DoD is now locked in to proprietary AWS cloud features which make
migrating to a different vendor technically and fiscally near impossible.

~~~
dkhenry
You should not do infrastructure as IDIQ, this is an infrastructure contract.
The idea that all contracts issued by IT are for "IT Services" is part of the
reason digital systems at the DOD and other government agencies are such a
nightmare. HAving a single system to build off of is the best way to do it,
and not having that system be a customer solution for just the government is
way better then not.

Lock in is the only concern, but from my experience doing an IDIQ or multi-
vendor won't solve the lock in problem. The best thing to solve that is to
make sure your using bland normal things not customer government thigs.

~~~
killjoywashere
You know, I've been in some data centers and top 50 HPC clusters ... I don't
understand why the US government doesn't run it's own cloud systems, except
for the appropriations. There are enough computer scientists in the larger org
to field a major distributed systems engineering group.

~~~
acdha
Here’s the civilian pay scale – think of how it compares with AWS or Google:

[https://www.opm.gov/policy-data-oversight/pay-
leave/salaries...](https://www.opm.gov/policy-data-oversight/pay-
leave/salaries-wages/2019/general-schedule/)

Now, think of the political climate: there’ve been decades trying to “trim
waste” by steering money to the private sector, where efficiency is taken as a
matter of faith. First you’d need to hire the technical leadership to properly
plan something like that and then hire all of the people who’d build it, not
to mention procuring all of the hardware & data centers, etc. You’re looking
at years before you have a positive return on any of that – and all the while,
every big contractor is going to have their lobbyists talking about how it’s a
waste of money compared to going with them.

NASA tried this a decade ago with Nebula, which was part of the early
OpenStack history. I don’t know exactly what happened with that but the people
I know working on NASA projects are all using AWS.

------
MarkMMullin
Oh Oracle - first your sales people came every year and jacked up the
licenses, and we fled. Then you came up with plans to monetize Java, and we
fled. And then this. Just die already, OK - your old business model is dead
and your attempts to make a new one would even embarrass Elsevier

~~~
tootie
Larry Ellison is exactly the kind of guy Trump loves. He'll get one flattering
phone call and the contract is theirs. Then $10B will turn into 100 before
they hit the first release.

~~~
ethbro
For $10B, Oracle will gladly buy $8B of AWS services for you...

------
ghostcluster
Lots of drama to this story, between the apparent sabotage from Oracle to the
politics of the president not favoring AWS just for spite, to the Pentagon's
acronym for its cloud move: JEDI.

The article states that Microsoft does not yet have the security clearances
for JEDI yet, but is working on it, but nowhere in the piece does it mention
Google's cloud services. Are they in the running as well? Are there any other
giant enterprise cloud platform businesses that I'm unaware of? Dropbox uses
AWS.

The other interesting thing about this, circling back to Oracle, is the
changing of the technological guard here. One can imagine all sorts of
government databases running on IBM mainframes, using Oracle software. But
now, the shift to enterprise cloud has arrived. I guess this must be happening
across the entire industry.

~~~
cowsandmilk
> Dropbox uses AWS

They very loudly left AWS several years ago, not aware of them going back.

~~~
dehrmann
Wasn't it because they hit the scale where managing their own data store,
renting colo space, etc. was cheaper than AWS? I've heard first-hand from some
large players that self-hosting is cheaper. One is still-hosted, but one is on
the cloud as a strategic bet.

~~~
doctorpangloss
Scale? Cheaper? Self-hosting a single server in San Francisco is cheaper than
AWS too. AWS sells convenience and baked-in knowledge.

~~~
ethbro
Self-hosting is cheaper at both ends of scale.

Enterprise clouds are cheaper if you're in the middle and/or quickly changing
scale.

~~~
sam1r
IMO, I think you're undermining the multiple years of programming/sys-admin
experience it would require to self-host + scale on your own. It's much more
expensive when experience is minimized, as a function of time.

Unless you've done it before, of course.

~~~
jblow
Why have we made our systems so complicated that it takes years of experience
to run software on a computer?

~~~
ethbro
The consequence of abstraction is underlying complexity. And we've added a
_lot_ of abstraction.

------
danenania
I know we tend to think of cloud-agnostic architectures as over-engineering,
YAGNI, etc., but for a massive, massively expensive, mission critical
government cloud perhaps it’s a good idea?

Apart from fail-over/redundancy benefits, it would avoid lock-in and
drastically improve negotiating leverage that could force ongoing
competitiveness and accountability.

The engineering costs are significant but still a drop in the bucket compared
to the size of these mega-contracts.

~~~
solatic
A contract the size of JEDI can (and probably is/will) build in that kind of
independence as a requirement. Which isn't to say "Amazon must build and
maintain an OpenStack implementation for DoD", but rather, that Amazon must
turn over all APIs and source code used to build the implementation.
Obviously, source code and APIs isn't enough to keep everything running -
there still needs to be operational expertise within organizational memory -
but the DoD would be empowered to start to take over operations and plan a
migration to some other company maintaining the cloud infrastructure, if it
were really untenable to renew the contract in a decade.

Put it this way: there's an old aphorism about how owing the bank $1 million
is your personal problem, owning the bank $1 billion is your bank's problem,
and owing the bank $1 trillion is your country's problem. There would be a
parallel for cloud infrastructure: if you have a $1 million cloud bill then
you have a (vendor lock-in) problem; if you have a $500 million cloud bill
then your vendor has a problem (because you represent a huge chunk of business
that can evaporate if you move to on-prem); if you have a $1 billion cloud
bill then the wider market would have a problem (because of the reverberating
shock effects if your migration to cheaper infrastructure is unsuccessful).

It's the textbook definition of something which is too big to fail.

~~~
danenania
"the DoD would be empowered to start to take over operations and plan a
migration to some other company maintaining the cloud infrastructure"

That's still a massive project though that I presume would only be undertaken
in extremely dire circumstances, so Amazon only has to do the bare minimum to
make sure things don't reach that point.

I'm thinking more along the lines of switching off the primary cloud to a
"follower" cloud in a few clicks. I realize that's not at all easy, but I
think it should certainly be _possible_ with these kinds of budgets to build
in cloud-agnosticism from the ground up. It's hard but it's not _that_ hard--
it's just the facade pattern, which is quite commonly applied in other cases
like OSes/databases that are arguably just as complex.

~~~
ethbro
This is how you end up running on an extremely limited cloud subset, decades
behind current offerings.

Do you want seamless ability to transfer work to another cloud, or do you want
the latest managed services and features? Pick one.

~~~
danenania
I think "extremely limited" is hyperbole. The major clouds can all provide
every service that 95% of applications need. If you're building something with
unique needs that only one cloud can handle, then go ahead and use the
advanced feature. A little bit of lock-in here and there for edge cases seems
preferable to having your entire multi-billion dollar system deeply coupled to
a single company's cloud.

~~~
julianozen
Sort of. AWS Lamba (server functions), dynamodb (hosted nosql) and aurora
(hosted sql) are all pretty generic services, have significant auto-scaling
benefits to using them, and have competitors on competing platforms.

At the same time they are sufficiently differentiated that migration plans
would be non-trivial

Disclaimer: former aws employee

~~~
danenania
Right--I'm not saying it's trivial, just that it seems doable with many
millions in budget.

------
bmchnl
Half the shit the gov buys and mandates people use nobody doing work actually
wants to use. Picking Oracle or some two week old fly by night hub zone
contractor to handle this award would be typical. The unusual thing would be
for it to actually go to a provider with services that are useful and people
want to use. - speaking as someone who has done a significant amount of .gov
work.

------
bhouston
The winner probably should have been AWS and close behind was azure and then
Google.

IBM and Oracle just are not at the same level but I do suspect they have the
best sales teams for government deals.

~~~
hn_throwaway_99
Google took themselves out of the running a long time ago.

------
blackflame7000
The government already contracted General Dynamics to build their own cloud
called MilCloud. It had a lot of problems so naturally they are developing
MilCloud2.0

------
throwawayjava
I guess I don't understand the need for this contract in the first place.

Is it normal in the private sector for companies to spend $1B/yr on cloud
services? I feel like at that price point you're better off building out your
own infra for most stuff, and using the cloud only for the remaining $10M -
$100M fraction where you really do need dynamic scaling.

------
cronix
I guess owning the Washington Post (Bezos paper), and its overwhelmingly anti
Trump stance over the last 3 years, had its price. This is about Bezos IMHO,
nothing else.

~~~
akhilcacharya
I genuinely wonder sometimes if we are on a slide towards a banana republic,
and it makes me genuinely concerned that many people (40-46% of them) _just
wouldn 't care_ if we did.

~~~
mschuster91
They would happily live in a banana republic just to spite the "libruls". Some
people simply are not worthy of the privileges of living in a democracy, it
seems.

------
bediger4000
This headline seems... misleading. There's only one high places foe mentioned,
and that's Trump. Every other high level official mentioned seems to be doing
Trump's bidding on this.

Also, how is having someone high up intervening in a business process a big
violation of free market principles? I don't recall the article mentioning
this either. As I think about it, this just reeks of crony capitalism. Didn't
we just (2008 or so) decide that was a bad way to do an economy?

Isn't the NYT slant generally in favor of laissez faire free markets? Why the
switch here?

~~~
akhilcacharya
You're right, but unfortunately the idea of a President individually
attempting to punish individuals and businesses that criticize him is
considered normal now.

------
andy_ppp
What is it that the government is buying and how can the government ever trust
(for this sort of thing) a private company. And surely you can set up faster
(performance per buck) more secure data centres for 10 billion bucks?

I just don’t get it, but I guess someone will say government can’t do anything
and that’s enough.

------
resters
The worst thing about this in my opinion is that as much as I personally
dislike Trump, the WaPo has lowered the quality of its journalism tremendously
over the past decade and so unfortunately Trump’s critiques of the WaPo are
not lacking in substance.

Bezos should have stepped in to make certain that the WaPo improve the quality
of its journalism even if the easiest path to profits is just to provide
entertaining stories to the #resist partisans.

There is plenty to write serious investigative journalism about that will have
the consequence of weakening Trump politically (if that is the goal) so there
was no need to become a tabloid.

Arguably the WaPo’s reporting on Saudi Arabia led to the grudge match that
culminated in outing Bezos’ philandering, and the subsequent major decrease in
Bezos’ wealth.

KSA is led by an abhorrent group of royals, but the WaPo’s coverage of the
disappearance of khashoggi was tabloidesque and immature.

~~~
brown9-2
What does any of that have to do with Amazon?

~~~
resters
Bezos also owns Amazon. It shouldn’t really be relevant but unfortunately it
is.

~~~
Supermancho
> In Pentagon Contract Fight, Amazon Has Foes in High Places

> Bezos should have stepped in to make certain that the WaPo improve the
> quality of its journalism

The relevance is that this waffling over Oracle and AWS is born of an
interview in WaPo? It has nothing to do with Amazon. Especially when you're
commenting on the quality of WaPo journalism. The story exists without WaPo,
but not without Amazon.

~~~
resters
WaPo has been weaponized against Trump by Bezos, or Bezos failed to stop it.

For some reason doing solid investigative journalism is not a priority and
instead rallying partisans around talking points and feel good, in-group
stories takes precedence.

FWIW I personally loathe Trump and would love to see him removed from power
thanks to some solid investigative journalism.

~~~
kevinconaway
David Fahrenthold at the Post won the Pulitzer for his investigative
journalism in to Trump

[https://www.washingtonpost.com/lifestyle/style/washington-
po...](https://www.washingtonpost.com/lifestyle/style/washington-posts-david-
fahrenthold-wins-pulitzer-prize-for-dogged-reporting-of-trumps-
philanthropy/2017/04/10/dd535d2e-1dfb-11e7-be2a-3a1fb24d4671_story.html?utm_term=.4f902e41213f)

~~~
resters
Significant, but two small blips in a sea of front page stories about rude
remarks or other tabloid style drivel.

------
crb002
Amazon needs IBM mainframes in their data centers. They are still king of
highly consistent and available systems.

~~~
dijit
I don’t think that the irony should be lost on you that this generation of
service offerings do not guarantee reliability on purpose. The onus is on the
developer to make a reliable system out of faulty pieces.

What does more reliability mean when that was expressly not a target to begin
with?

