If you read their S-1, they list a dependence on Google cloud as one of their big risk factors. Yet they then go ahead and make this commitment instead of working towards eliminating it.
There's so many advantages to owning your stack and if Snap thinks that it's going to need 2 billion dollars to pay for cloud infra, they're at the scale where it makes sense to build your own infra. Just look at Facebook, they're able to create tailor made data-centers that fit precisely what they need. The success of Snap relies on huge scale on the consumer side, if they want to scale their infra to support that 5 years down the line then this sounds like a poor move since they will either need to play catch-up later on or prepare to pay serious dough to Google.
Paying for cloud services seems like a great idea when you are not able to predict your needs in the coming years, given a deal like this I don't think that's the case.
The key here is: Compared to published rates. They'll not be paying published rates, the same as there's no way Netflix is paying published rates at AWS.
They'll be paying extremely highly discounted rates, assuming they're negotiating team isn't staffed with a bunch of people who failed their business degrees.
Out of curiosity, at what point should someone look into negotiating lower rates with AWS (in particular)? And how would they go about that?
A lot of my consulting is on cutting hosting costs, and I've yet to have a client where we couldn't come up with substantially cheaper alternatives than AWS, but sometimes being able to show your account manager that you know how insanely high their margins actually are and that you have a credible alternative makes enough of a difference for them to end up sticking with AWS.
It also depends on ease of cutting the cost, I'd say. E.g. if 90% of your cost is bandwidth, and it's mostly serving up static assets, it's trivial to cut the cost dramatically by rolling your own mini-CDN outside of AWS (bandwidth prices at AWS are between 10x and 50x higher than the cheapest competitors depending on region if looking only at managed hosting or other cloud providers - more if you're large enough to look at peering options).
The exception is serving films. If you watch a film, you receive bytes served by hardware built to Netflix' specification, running in leased space at a mixture of colos and ISPs. That's a really big exception. The third of internet traffic is that exception.
That's…not true. Netflix has a huge CDN running their own custom-built servers that sits near customer endpoints.
I shudder to think of how much bandwidth they'd be paying to AWS if they didn't…
They have 1800+ employees as disclosed in their filing. So clearly people resources are not a problem for SNAP.
Do you believe its not possible to build 4 datacenter - 2 US, 1 EU and 1 APAC datacenter for 2 billion dollars? These are tangible assets that you can depreciate as well. The lifetime TCO of a center rack is $120K
Here is a DC build vs buy calculator. So can see independently those costs are about right:
Quite apart from the dollar costs and human capital required to build and maintain a DC, there's the lead time required to build the thing, and I'd speculate that that's potentially a significant factor in Snap's decision. Perhaps Snap are looking at e.g. how quickly Pokemon GO scaled their operation, and they're thinking for whatever reason that they might need to do something similar?
By the time you build your infra (5 years was mentioned), Google would have iterated on their offering and their own network and data centers for 5 years.
(Work on Google cloud)
Thinking about it more, there must be some other strategic reason behind their decision to go with Google.
Crunchbase says they have 100-250 employees.
Their LinkedIn profile says 50-200.
"1,859 as of December 31, 2016"
I hope you don't rely on Crunchbase or LinkedIn for anything serious.
It's a hedge. If they don't grow as rapidly as expected then they're betting that someone else will buy the excess reserved capacity from them for close to market rates. So in the case that they only use 1.5 billion dollars worth of hosting, they're betting that they'll be out, say, $25M rather than $500M. On the other hand they want to make sure that if they do grow rapidly then Google has the capacity to meet their needs.
Security wise it's no different than any other instance; specific machines aren't associated with the party who originally reserved the capacity in any way.
It doesn't mean they won't. Apple is reportedly spending a similar amount on Google cloud. They're also investing billions in building their own data centers. Those things go together.
Do you really think a company facing literally hundreds of millions of dollars in cloud service bills is not going to run the numbers on plugging in their own servers.
For ex. this is Twitter's risk item for running hosted services:
> Our business and operating results may be harmed by a disruption in our service, or by our failure to timely and effectively scale and adapt our existing technology and infrastructure
Continues bottom of page 26
On the other hand "someone who has an entire product devoted to that one area, and has many years of experience and is probably the largest provider on the planet for that service" also describes Enron at the time.
Google's no Enron, but Size isn't the metric one should use.
Sometimes it's just better to pay someone else to do a job.
I agree that too often "put it in the cloud" is abused, but there are areas where it really shines, and looking at their filings it looks to me like Snap has done their homework and that it's going to be a good fit for them.
They are signing on with one of the biggest, they are building out their own infrastructure over time, they are signing up with a secondary provider just in case, and they lay out their reasoning pretty well for why they want this to be handled by someone who has magnitudes more experience at it than they do (the big one being that they point out their demographic is VERY fickle, and any kind of performance issues, outages, or major problems could easily be the thing that switches someone to an alternate service forever. And they themselves don't quite have a grasp on expected growth so going somewhere that you can scale at a moments notice is a great benefit!)
When you have Adwords dollars pouring in it makes sense to go send out a few people to buy up fibre and land/power/water for datacenters on the down low....
"We have committed to spend $2 billion with Google Cloud over the next five years and have built our software and computer systems to use computing, storage capabilities, bandwidth, and other services provided by Google Cloud, some of which do not have an alternative in the market. We are currently negotiating an agreement with another cloud provider for redundant infrastructure support of our business operations. In the future, we may invest in building our own infrastructure to better serve our customers."
The rule of thumb I've heard is that you should start looking beyond public cloud once your cloud spend hits ~200K/month. Since at this level, engineering and ops investments you need to make to maintain your own infrastructure start making more sense. I think it's safe to say Snap is beyond that threshold right now.
Google Cloud is a good idea when you don't know how much infrastructure you need.
Of course, maybe they got some killer promotional deal with Google. Seems likely.
But unless you're still "in the garage" and a couple of DigitalOcean droplets are good enough, it's going to be much, much cheaper and usually much wiser to run your core infrastructure on your own colocated bare metal.
I've seen companies increase their server expenses by ~$1M/yr by moving everything to EC2, and they sit around congratulating themselves for it because now "they're in the cloud". There's no reason to do that!
Little humorous tangent: an AWS rep told someone I've worked with that Amazon really wanted to help them secure better pricing, because as new CFOs come from self-hosted companies and into AWS-dependent companies, the CFO's eyes bug out when they see the Amazon bills and EC2 becomes the first thing on the chopping block.
Script your stuff out in Ansible or something similar, run it on your own hardware, and use GCloud/EC2 as secondary data centers for failover/backup/support/emergency bursts/whatever. You can have the flexibility without paying through the nose.
Except then you have to run your own networking and when shit fails (as disks, links, and switches are want to do), it's now "your problem". Hybrid clouds and not being a tenant is nice, but not without time and monetary costs -- by the time you have geographically distinct failover, you've also spent a non-trivial amount of opportunity costs making phone calls, flying around, and writing lines of code and config for things customers don't even know exist.
Multi-AZ, multi-region complete failures are very, very rare. How often do you get a failure in your data center per year (that you notice)?
> You're going to be writing a lot of the same fallover code if you're running on someone else's hardware, so why rent?
The answer is in the question -- when rented things fall down and go boom™, your code runs and someone gets a text message with the receipt.
When a handful of the "wrong" disks decide to revert to air-blocking bricks or your upstream network provider has an outage, you're lucky if it's something you can fix by heading to the data center. I promise that AWS or Google is better at running a DC, and unless you're trying to enter the hosting business, I wouldn't advise spending the time and money to meet their uptime and features.
I've only managed data storage in the scale of many petabytes (and this was a handful of years ago) and honestly, I think it required at least 20 hours a week of babysitting by various staff. At Snap's scale and traffic patterns (viral content, lots of writes, so on), I imagine this is a very non-trivial spend on scaling, staffing, tech implementation.
At 2bb over 5 years, maybe Snap would benefit from rolling their own -- hiring 50 great hackers at a mildly conservative 250k/head (say 200k average + benefits + taxes + employee support costs (HR, payroll, recruiting, legal, etc)), eating a year or two of transition costs off their cloud hosting providers, then probably saving a bit of money even after hardware, bandwidth, facility, insurance costs. Hell, maybe they'd even open source some software and recruiting would get easier after conference talks of how they did it. Or maybe they get bought by Google or Facebook in a year. Snap's in the business of selling ads and getting more eyes on those ads. Whatever enables growth and doesn't serve as a distraction or speedbump is a "fine" decision.
First, if you don't notice some random/unexpected EC2 instance failures, you don't have a big EC2 deployment. Even though there is a lot of pomp and circumstance around the cloud, when it comes down to it, your instances are still on a physical server in a datacenter somewhere and they can, and sometimes do, fail. In that case, as in every other robust production deployment, your application (hopefully) performs an automatic and graceful failover to its standbys. The location of the standbys is usually an configuration value. Not seeing any unique value proposition here for "the cloud".
The point is that even when you're using EC2, you still have to set all of that up. Contrary to popular belief, EC2 is not a panacea that can magically make your software reliable and redundant. It's just a nice interface that makes it easy to rent servers from Amazon.
The only benefit you get from EC2 is that someone paid by Amazon has to go pull the box, but your company could hire such a guy in-house for _much_ less than it's paying Amazon.
The onus is still on the developers to figure out all of the application stuff that's necessary to accommodate failover and make sure that everything plays nice with each other, and getting that working right is by far the most time-consuming part of deploying a high-availability application.
So EC2 doesn't add any extra resilience; it's just outsourcing the job of pulling a server to an Amazon employee/contractor instead of YourEmployer employee/contractor. If your company is big enough (and at Amazon's prices, you don't have to be very big at all to be "big enough"), that doesn't make sense.
I know EC2 et al are popular because people like buzzwords, but that doesn't make it good business (or does it? Investors love cloud because it keep capex low, and because investors are buzzword-driven like everyone else; saying "cloud" will make them like you more and want to give you more money).
For companies that are still in the garage (literally in the garage), shelling out $20/mo for a couple of cheap VPSes from something like DigitalOcean is going to be just fine. But once you get bigger than that, there's no way to avoid paying attention to this stuff, even if paying Amazon tons of money creates a false psychological connection that makes you think they're doing the work for you.
>The answer is in the question -- when rented things fall down and go boom™, your code runs and someone gets a text message with the receipt.
Let me fix that for you: when things fall down and go boom, if your code is written and your deployment is configured to support it, your product continues to work, and someone, somewhere, has to get a broom and sweep up some ashes.
Whether or not cloud is a reasonable proposition is primarily a question of whether it makes more sense for that someone who sweeps up the ashes to be on the corporate payroll of YourEmployer or YourCloudProvider.
>I've only managed data storage in the scale of many petabytes (and this was a handful of years ago) and honestly, I think it required at least 20 hours a week of babysitting by various staff. At Snap's scale and traffic patterns (viral content, lots of writes, so on), I imagine this is a very non-trivial spend on scaling, staffing, tech implementation.
EC2 is not a silver bullet. It's just an interface to allow you to rent servers from Amazon. EC2 users still have to babysit stuff, just not the hardware (though they still have to monitor resource usage, clean up disk space, and be prepared for things to blink offline with 0 notice -- again, all the normal things; only difference is that your hardware jockey is accessed through EC2's web support interface instead of Slack/cell).
>At 2bb over 5 years, maybe Snap would benefit from rolling their own -- hiring 50 great hackers at a mildly conservative 250k/head (say 200k average + benefits + taxes + employee support costs (HR, payroll, recruiting, legal, etc))
Vastly overallocating here.
>Hell, maybe they'd even open source some software and recruiting would get easier after conference talks of how they did it.
Unnecessary, there's already tons of great open-source software to handle HA deployments (usually, this is the software underneath the commercial UI that makes everything work; it's surprising how much "revolutionary" commercial software is just glue code and a point-and-click wrapping around an OSS workhorse).
Of course, once you get unicorn-scale, everything has to go custom and/or highly modified because no out of the box solutions can handle the load, and that will be the case whether their hardware is hosted by Google or not. Again, "cloud" does very little to relieve workload for all non-hardware employees.
And the added benefit of being a trendy tech company is that after your company creates some extremely specialized solution, you can open-source it and watch with an uncomfortable mix of amusement and horror as 90%+ of other companies's tech departments contort themselves into pathetic, desperate architecture pretzels so that they can become cool by abandoning a stable, proven, mature stack for your company's experimental, sputtering, duct-taped abomination that requires a PhD to even get to compile.
This pattern has become so commonplace that reciting any specific example feels trite. You can probably name 12 off the top of your head. Hadoop in particular is a victim of many gross offenses of this type.
>Snap's in the business of selling ads and getting more eyes on those ads. Whatever enables growth and doesn't serve as a distraction or speedbump is a "fine" decision.
Sure, but they don't have to set massive gobs of money on fire for no reason along the way. But then, I guess they wouldn't be part of the Silicon Valley family if they didn't.
( Work at Google cloud)
The parent didn't claim they don't happen, just that (1) they were rare (a point you agree with, given the minimum usage needed to notice them) and (2) multi-AZ, multi-region failures nearly non-existent.
> The point is that even when you're using EC2, you still have to set all of that up.
It takes literally minutes to set up an ELB and Autoscaling group across five availability zones. How long does the non-cloud version of that take?
Because when something fails, you don't have to care about the "why" as long as you can replace it. I see about 4 instances needing a maintenance per month per 1000. That's reasonable enough to not demand someone be full-time focused on making sure that only the good lights blink on the hardware.
> The point is that even when you're using EC2, you still have to set all of that up. Contrary to popular belief, EC2 is not a panacea that can magically make your software reliable and redundant.
You're making a strawman by suggesting people think it's a panacea. The advantage is that a lot of the work, maintenance, and feature improvements for 'infrastructure as code' is handled for you. Cloud hosting means writing the software layer and being done, no managing the infrastructure services, facilities, hardware, business relationships involved with rack/stack.
> It's just a nice interface that makes it easy to rent servers from Amazon.
To be fair, it's a _very_ nice interface.
> I know EC2 et al are popular because people like buzzwords, but that doesn't make it good business (or does it? Investors love cloud because it keep capex low, and because investors are buzzword-driven like everyone else; saying "cloud" will make them like you more and want to give you more money).
If you think cloud hosting is popular because of op-ex or buzzwords, I think you're out of touch. EC2 and Google Cloud are popular because they let you focus on getting shit done, even when you have variadic workloads that are uptime dependent.
> For companies that are still in the garage (literally in the garage), shelling out $20/mo for a couple of cheap VPSes from something like DigitalOcean is going to be just fine. But once you get bigger than that, there's no way to avoid paying attention to this stuff, even if paying Amazon tons of money creates a false psychological connection that makes you think they're doing the work for you.
They _are_ doing a lot of work for you. You say $20 is the point that it makes more sense to self-host. I'll be charitable and round that up to $100, but even at that price, there is _no way_ you'll be able to get something as fault tolerant or low-cost as a cloud hosted solution. Do you really think that for $100 a month you can self-host geo-close servers with redundancy to the point that you don't have to think about it? Keep in mind that "two is one and one is none" when planning your hardware purchase.
> Vastly overallocating here.
No, that's conservative for a major US city (e.g. where Snap would be doing the hiring). Have you tried to pull a handful of really good system hackers out of thin air recently? Even if you can get them, they're not cheap, and you'd need a sizable team to pull off the highly-redundant world-wide install that Snap needs for its growth projections. It starts off expensive to hire good tech and gets more spendy the longer you're fishing.
And that's even ignoring the costs on productivity (for that and other employees) when an employee isn't happy or decides it's time to leave -- staffing also takes money and attention to maintain.
> And the added benefit of being a trendy tech company is that after your company creates some extremely specialized solution, you can open-source it and watch with an uncomfortable mix of amusement and horror as 90%+ of other companies's tech departments contort themselves into pathetic, desperate architecture pretzels so that they can become cool by abandoning a stable, proven, mature stack for your company's experimental, sputtering, duct-taped abomination that requires a PhD to even get to compile.
You seem like you're speaking from personal experience. Having a working infrastructure that isn't a barrier to growth isn't trendy or sexy, it's a base competency for any internet-reliant business model.
> Sure, but they don't have to set massive gobs of money on fire for no reason along the way. But then, I guess they wouldn't be part of the Silicon Valley family if they didn't.
This isn't setting "massive gobs of money on fire for no reason", this is going with a high-performance datacenter that someone else maintains. They clearly have something very big in mind and I doubt they made a multi-$bb commitment without asking themselves "are we lighting this money on fire?"
For sure. How much is this free marketing that Google cloud service is getting worth? I'm pretty sure whatever discounted deal Google gave Snap is more than made up by this free marketing blitz they're getting.
Also, there's a difference between something being public (like press releases) and actively generating buzz where lots of (relevant) people are actively talking about this.
I think if you were in a GCE sales meeting yesterday you'd have noticed a lot of people jumping up and down in joy. They've been playing second fiddle to AWS and in desperate catchup mode. Their next cold call got so much easier. Their next close got so much easier. Screw all that, their inbounds suddenly went through the roof. Lots of smaller startups etc. who would have never thought of Google cloud as an option are now seriously considering it. A lot of people who are already on AWS just signed up for GCE out of curiosity "just to see what the big deal is about". I don't think there's any way to overstate the impact of this news on Google Cloud's future.
If they ran on bare metal then they'd list that as one of their big risk factors too.
Just to be clear, how many billion dollar infrastructures have you built up? What about played a significant role in, witnessing the various trade-offs that have been made? Taken part of, in any shape or form?
Of course, if you have experience that's all well and good. But I would expect someone with experience to lead with it, not omit it, precisely because they know how important it is with all the trade-offs involved.
GCP isn't sticky. They can leave, or renegotiate, or even try to get Google to do some of their engineering work for them ("We'd really love if this API did this as well..."). Or some combination thereof to minimize risk vs max profits.
Similarly, services like AWS provide companies with that type of infrastructure (data or internet in this case) so they can focus on building out features.
But to your point, it gets expensive at some point and I am surprised that Snap is still outsourcing this instead of having their own data centers. I guess it's because the CEO is not strong Technology-wise so the company is focused on Product?
I imagine part of their plans are to move away from the dependency, but while it exists - this is exactly how you reduce your exposure to that risk.
Perhaps a discount large enough that prices match or are even better than bare metal servers.
Remember, Google builds its own custom hardware and probably gets much better performance per dollar than you would get from HP/dell/IBM/etc.
How about a HN reader who is taking part in a discussion? They offered their view and why they believe that. Nowhere did they they claim "authority." Unlike your snide commentary they are actually contributing to the conversation.
Good discussion would have ensued after a statement like "I wonder why they don't build out their own infrastructure because at this scale it is usually cheaper to...".
I could imagine that with this big a deal, they could get rate reductions that get the cost down to a comparable level to building out your own datacenters (at least 2) and hiring qualified ops people.
It might very well be a bad move, but you really can't say unless you know more.
>"This sounds like a really bad move on their part and they ..."
That's hardly a "factual" statement.
>"Good discussion would have ensued after a statement like "I wonder why they don't build"
Do we really need to preface our comments with personal opinion disclaimers? When someone says "it sounds like", it seems pretty clear that they are admitting they don't know all the details, so how can what follows that phrase imply authority? That's absurd.
Especially as a fellow iOS dev, there's a few UI things I'd like to reverse engineer. Seems like every year they introduce something useful or cool-looking and I think, "Wait, how did they pull that off?"
(observation based on the fluidity of their app)
For example: Elon Musk is a Twitter user. I too am a Twitter user. So's Kylie Jenner, Donald Trump, and random spambots. Using the same service does not mean they're equally qualified to speak with authority about the same things.
Ah right, I think I see the nature of our disagreement / misunderstanding. I totally agree with you on the general principle that quality of content should be allowed to stand on its own.
However, I believe that there are things that are context-specific things that the men and women in the arena will face. And these are things that those of us in the stands, however thoughtful and discerning, will never be able to appreciate them, because we simply do not know. (For a great read about this, check out Daniel Ellsberg's message to Henry Kissinger, on the reality of having access to top secret information: http://www.motherjones.com/kevin-drum/2010/02/daniel-ellsber...)
So for example. It seems obvious to me that the top comment is sensible and correct. Snap's CTO or whoever else made that decision is surely very familiar with the costs of being dependent on something like Google. So if they decide to do it anyway, I'm of the opinion that they're quite likely to have done it because of concerns that I am not able to appreciate, because I am not in their context.
Of course, there's a non-zero chance that Snap is making stupid decisions. But I think it's far likelier that they're making decisions that SEEM stupid to a 3rd party, but make perfect sense once you appreciate their context.
I could be wrong about this, of course.
You're right, we don't know everything that has gone into a decision, but that's part of the value of an exercise like this. Being able to debate about something, remain civil and deal with specific arguments is an incredibly valuable skill that only gets more valuable as you age.
Snap has competent engineering execs that have built a very strong company. I would have a hard time believing that they haven't put way more thought into this than OP.
It might be a bad move still but we don't have all the information necessary so it's strange to just assume we know better.
The days of needing massive ops teams to run on owned and colocated hardware are long gone.
It's an excellent argument as to why I will not be purchasing this stock.
- GCP will always have a margin that high
- Snapchat is paying the advertised rate
- They can find the requisite talent in a reasonable timeframe
Surely they don't.
>What if snap comes out with a statement saying Gcp sucks and they're moving to aws?
Then we question why Snap committed billions of dollars to infrastructure that "sucks", and wonder why they couldn't figure it out, but Google could.
You think people will take Snap more seriously than Google itself?
If Snap threatened to leave I'd wager that even the CEO of Google would get involved to keep their biggest cloud customer. Snap's revenue and brand name have enormous value to Google. There simply aren't that many $2 billion dollar cloud customers right now.
What about the cost of writing the software services that you could pay Google Cloud to use?
How much would it cost to write your own Compute Engine, or Datastore, or Pubsub, or Bigquery?
The parent made some good points about scale and infrastructure. I've worked with infrastructure on many projects and I can't come up with a coherent argument why any of the parent's points are wrong. Can you?
I agree that empty snark is usually unproductive, but in this case, I think there's a useful point being made. Still, always best to rephrase into non-snark. :)
I assume spending this much would give them a below market discount and they could recoup some of their losses if nec.
All this assumes it is a positioning tactic w/ a hedge. Maybe, they will just build a developer ecosystem a don't want to bifurcate their engineering output. If twitter fails, it would validate this choice as they can pay a slight premium and if their metrics increase build infrastructure in 2.5 years w/ more info to spec it
This is akin to dot-com era companies signing long leases on buildings or small business owners buying lots of inventory. Plus, 2 gigadollars could easily buy 4-10 soup-to-nuts datacenters that have tangible (although less) resale value.
Overexpansion is easy to do and super risky... these sort of moves increase expectations and scare off wise investors.
EDIT: for super scrappy startups playing with on-prem with lab-like reliability, https://unixsurplus.com/ is awesome.
Then everyone can honestly announce that Snap user data won't be handed over to Google - because it's already there. ;)
I don't regret hand-installing Kubernetes on AWS, if only because I know a lot more about how it works now. But I rather just spin it up with GKE.
We do invest a lot in making Google Container Engine a great experience, including integrating it with other parts of GCP (e.g., IAM ), but at the same time, the core is plain vanilla Kubernetes.
Dropbox built out their own environment (and did it migrating 500PB out of S3)  [1a]. As did Twitter . And Facebook . And GitLab  (too soon?) As well as Mixpanel . Even Twilio is multi-cloud (last time I checked it was split between AWS and Rackspace; this was several years ago during an interview, so maybe its changed). Sure, start in Google, or AWS, but at some point you will either need to use multiple compute/storage providers (redundancy) or go to your own gear (redundancy and cost).
Example: "In 2014, Moz CEO Sarah Bird said that it was spending “$6.2 million at Amazon Web Services, and a mere $2.8 million on [its] own data centers.” Simply put, the cloud killed its margins." 
simonebrunozzi: Forgive me, but when you're talking about hundreds of millions of dollars in spend, "easy" is relative. It is much easier when you're not relying on underlying primitives that are difficult to reproduce on your own at another provider (witness how terrible Open Stack is; no one wants to do that if they don't have to).
Am I minimizing the effort involved for this discussion? For sure. But the money involved...it solves most problems you would have migrating between providers.
> It seems to me that you have no serious experience in the real world.
You are entitled to your opinion. I have seen the pain, and it is relative. Its easier when someone says, "Here is the budget, just fix the problem", and your vendor's (AWS/Google) margins are 20-40% (these are real margins pulled from earnings reports); that's a lot of money you can put back in your own (or your shareholders') pockets.
If you we're spending $2 billion dollars, and I told you I could save you $400 million by spending $100 million, wouldn't you take that deal? Even at $200 million, its a bargain!
For less than what Snap is spending on Google cloud infrastructure, SpaceX built a rocket that can take a payload to orbit and return the first stage successfully (SpaceX has taken on ~$1.2 billion in funding over the last 14 years). Moving out of a cloud provider is comparatively hard?
EDIT: Maybe this is a roundabout way to kick back to Google in order to get preferential treatment on the Ad network. It sure isn't a logical decision.
EDIT 2: @ashayh: I'm not saying go back to good ol' bare metal. For $2 billion, you could build your own cloud provider out as an internal operation. The amount that's being spent on Google Cloud is egregious, and worse yet, common shares have no voting rights to push back against poor decisions like this.
EDIT 3: @hueving: HN throttles my posting; editing this comment is my only way to respond. Sorry about that!
 https://code.facebook.com/hardware/ | http://www.zdnet.com/pictures/facebooks-data-centers-worldwi...
Moving from one cloud to another, even with containers, is never easy at large scale.
(source: I have worked at AWS for 6 years, at VMware for 2, and I've seen hundreds of clients go through this exercise)
If moving operations around is insurmountably difficult, you built operations incorrectly. Put another, even broader, way with a few more implications: if you are totally reliant on one vendor for continuity of any part of your operations, you built operations incorrectly and are introducing unnecessary risk. If us-east-1 goes down and you cease generating revenue as a result, you have built operations incorrectly. That's really all there is to it. And yes, I realize this means 80%, maybe more, of the operations in in the world is built incorrectly. We just learned Snap's is. Maybe even yours! And that's fine as long as you're working on it. Good news: said exercise is a good chance to fix it!
Now that half the crowd is inhaling to bombastically retort that undoubtedly controversial, yet completely true, paragraph, allow me to quickly redirect:
What's "the real world," anyway? Most of HN forgets a Windows/.NET ecosystem exists, not to mention extra-valley gigs in, say, Nebraska. Would you say the lone sysadmin holding together a hospital in Des Moines is gaining "real world" experience and able to meet you in discussion? Seriously, I hate "the real world" and the people who fire it as a volley during an argument. Even your career is not indicative of "the real world." (Nor is mine.)
: Flagrantly so. I've spent the better part of an hour trying to concoct a scenario where that deal is even remotely in the win column for Snap. Still trying. You enshrined a business disincentive (nay, prohibition!) toward optimizing your opex into a five year contract and $400mm a year operating Snap was some kind of win? ... How on Earth? Even with a quarter billion DAUs...
If you're ever in Tampa or Chicago, let's grab a beer or dinner my treat. Would love to share war stories.
An additional point is that even the second tier PaaS/IaaS providers (like GE/Predix, for example) are trying to get out of the DC ownership business. There's no compelling reason for them to keep their own DCs when a) it's expensive to run & keep fresh, b) it's CapEx, and c) it costs them expensive heads to organize and manage everything.
That shows about as much of a lack of real world experience as you are bemoaning. It doesn't work that way in real life.
"Real world" and "real life" dismissals of said ideal are stupid, lame excuses for people who want to make a stupid, lame argument to cover up stupid, lame technical debt in operations by either (a) assailing the credentials of the speaker, as this entire thread has spent much time doing, and/or (b) providing a "well, everybody else isn't investing in this, neither should we" cop-out. And yes, you are doing it wrong. Rather than getting defensive, pulling the knives out, downvoting on sight, and trying to wipe away or justify doing it wrong, why don't you instead realize that it's motivation to do it better? We should all do things better, and every time a single AWS region goes down and takes out half the Internet I sigh because it's this exact, stupid, lame justification fest that results in that situation.
Work toward correct operations. You will never reach it. The cognitive dissonance of these two statements is totally acceptable. If you build a new service today and don't account for DR and security up front it just goes to show me that you haven't learned from the very public failures of those who have come before you, and that to me shows a lack of real world experience.
And yes, I'm aware of several shops that can literally flip a switch, and we're talking cage and ASN, not hobby-scale iOS backends. It's not like finding the holy grail and requires developers who "get" operations. It does exist. And it exists in real life! \u1F632!
When it comes to in house DCs, very few companies seem to have that top talent.
For example, most companies are simply buying regular Dell/HP servers, Cisco switches/routers, slapping them in cabinets and calling it a day. They simply do not know how to take advantage of high density platforms like open compute, or of SDN. They also throw millions at software vendors like Vmware/CA, instead of building their own provisioning or CPU/RAM/Disk aggregating solutions with open source or custom tools.
If they don't know how to do it, then the theoretical savings for certain companies, of bringing things in-house simply won't materialize. And then its better to throw money at AWS if your in-house "engineers" are also breaking things 10x more often.
In a company with a badly run DC, developers very quickly latch on to cloud benefits like not having to wait for 3 days for DNS request, and 2 weeks for a VM.
At this point, with datacenters having been in wide production for the last 20 years, there should be plenty of 7+ year veterans.
(though yes, I have known in-house service providers that couldn't set up a new VM to save their life)
I've seen innards of very large DCs for more than a decade. At one of my first jobs at a fortune 500, I was responsible for everything from rack and stack to the command prompt. The expected turnaround time for a single physical server was 4-6 weeks until the application could be installed on it! One of the reasons was that they did not have automated DHCP/PXE provisioning. I started the process to enable it.. going through all the political, security mess, it was 9 months until it was enabled. I was gone by then.
An extreme example for sure, but if AWS revenues are growing like they are, then surely such issues are everywhere to some extent.
Also them starting the project along with the knowledge they have internally scaling containers helps.
I'm going to go through all of the service offerings this weekend - from Docker Inc's Docker for Azure and Docker for AWS to the native container services on each.
FWIW, Most of this are advertising gimmicks though and Google has a pretty different internal infrastructure for orchestrating containers that has hardly much to do with K8s.
And it's not just the rather-large core team directly on GKE and k8s, nor the related products like Container Registry , Container Builder , and Container-Optimized OS . GKE and k8s benefit in other ways too: Google's internal kernel team helps debug customer issues when we trace them to the kernel, and people like Kees Cook are helping with the upstream Kernel Self-Protection Project  that make container technology more secure. In addition to that kernel work, Google also has rather-decent security teams and they work with us to improve security in other ways too.
Finally, re: toomuchtodo's question, "Why opt for Google if you're going to use containers in Kubernetes?" Because we hope you find that Container Engine is the best place to run Kubernetes -- and benefit from the other parts of Google Cloud Platform. If you ever find GKE is not that place, and you don't derive value from the rest of GCP, then exactly as toomuchtodo puts it: "You can even move to your own datacenter at some point (relatively) easily."
Am I totally off-base, if you're able to speak to this? (Maybe it exists and I've missed it, but I'd love to see a blow-by-blow of the differences and their rationale, too, because that'd be valuable insight on how Google learns.)
When I say k8s is like borg I mean: it has the same concepts of tasks, jobs, and allocs. The scheduling of those is handled by a k8s scheduler which resembles the borgmaster scheduler (a lot of hand waving here), and the containers themselves execute in an environment much like the borglet provides for containers.
Many of the valuable features provided by borgmaster and borglet are provided in k8s and you configure them through similar mechanisms.
Beyond that, how they are implemented specifically, there are a ton of differences but for an end user who is just using k8s, not setting up and managing k8s infrastructure, it's conceptually isomorphic.
I don't understand this statement. I have built systems that work with openstack and then scale up to public cloud. The primitive is the same as EC2, which is a virtual machine. What did you have difficulty with?
Due to the cache and read dependend nature of the database queries the latency impact is worth it.
Dropbox built out their own environment (and did it migrating 500PB out of S3)
Like most discussions in this thread, this statement is way too general. Dropbox moved their storage from S3. What about EC2 or other AWS services they were using? Did they abandon all of those too?
We have committed to spend $2 billion with Google Cloud over the next five years
In reality, it's probably more like $200M, $300M, $400M, $500M, $600M
> On January 30, 2017, we entered into the Google Cloud Platform License Agreement. Under the agreement, we were granted a license to access and use certain cloud services. The agreement has an initial term of five years and we are required to purchase at least $400.0 million of cloud services in each year of the agreement, though for each of the first four years, up to 15% of this amount may be moved to a subsequent year. If we fail to meet the minimum purchase commitment during any year, we are required to pay the difference.
Being frugal until you hit the $400MM doesn't matter anymore...
That's a lot for the first year. Not much scaling at all. Why would they structure it like this?
Otherwise they wouldnt have tried to buy snapchat.
It took Instagram Stories two quarters to catch up to Snapchat's total MAU.
If anyone wants to bet against them existing, I'll take the bet.
As soon as Snap goes public, you can short the stock. Go for it!
I'm not sure about Twitter, Groupon, or Uber.
(See how obnoxious this sounds?)
Perhaps I could have been clearer, but I didn't say that Alphabet would fail anytime soon anyway. I referred rather to G's habit of discontinuing popular services.
Normal expenses are things you pay, and then you declare how much you just paid.
This is not a normal expense, it's a long term contract. It states how much they'll pay IN ADVANCE over a long period of time. That can be used to make all sort of accounting magic , adjusted per year over multiple years however you like it.
That they didn't wait until after the IPO suggests Snap may see the partnership as a positive. Could also have been pushed from Google's end as it is a nice way of bragging about a big get but without having to formally announce it, and being associated with a hot IPO that will get a lot of coverage.
(they initially built their product on AppEngine)
The amount of hardware and services one would get for that bill is insane. Snapchat doesn't need that much computing power and storage.
$33M a month is the right magnitude. The more I think about it, the more I wonder if it's actually way too low, and they're getting really deep discounts.
Which means, 50 cents per Mbps/per month. Usually a 1Mbps means about 190GB of transfer over the course of a month, I think. So 1000 * 190GB = 0.190 PB per month.
412PB thus means ~ 2200 Gbit/s . Under my pricing, I get a whole lot less than $33M / month.
So, does your calculation involve storing that same 412PB?
Is my math off?
If I push 1 GB per month, https://www.google.com/search?q=1GB%2Fmonth+in+mbps&oq=1GB%2...
You can assume they are not paying list rate when they buy $400M of service.
If they push ~40gbps on average, that's 13PB - https://www.google.com/search?q=40gbps+*+30+days+in+PB&oq=40...
13PB is 13M GB. 13M GB @ .08/GB = $1.04M
$1.04M/40gbps = 1.04M/40000 = $26/Mbps.
I pay less than a 10th of that outside.
But given that Snap could be anywhere in the USA, they could locate their servers anywhere. I wasn't pushing Denver, or the colo I am in, as the solution.
It's just kind of shocking that people are willing to pay 20 times or more the going rate for bandwidth...
Assuming 2 Petabyte per user, mere 500,000 user will consume entire storage produced every year. May be they do, but 2 PB/user seems improbable. That's also $14k per person of data (assuming no discount from Google to this company).
https://med.stanford.edu/news/all-news/2016/08/stanford-medi... or https://www.broadinstitute.org/google
Terrance from Google Cloud Platform Support here. If you are having any issues interacting with our support team please drop me a line with some case numbers (email@example.com) and I will take a look and try and resolve them.
From the article...
"Google doesn’t break out revenues from its cloud infrastructure, choosing to lump it in with other non-advertising businesses like hardware and Google Play sales. But that segment totaled $3.4 billion in sales in the most recent quarter."
Assuming that all $3.4 billion is for cloud revenue last quarter, that is still less than 10% for last quarter ($400m/4 = $100m).
Isn't Snapchat supposed to delete data after 30s or whatever?
Why this insanely high cost? Can somebody shed some light.
2. Like Whatsapp, server has to store the message until the receiver is online again (until they open the app).
3. Whatever systems they use for advertising, tracking, profiling, and analytics is probably a significant chunk of their backend.
The server actually has to store it for a maximum of 30 days . Snaps get deleted after 30 days (even if they're unopened). They're deleted immediately once the Snapchat servers get confirmation they've been viewed.