My company has gone through 4 reps in 3 years. Every time we get a new GCP rep they just want to talk to us about "expanding our use of GCP offerings". The only thing they want to talk about is starting to use BigQuery - not my business at all.
I signed up for a Google Cloud Security summit, and afterwards a sales rep reached out me. It was obvious from the start they had no idea I was a gsuite or GCP customer. They then directed me to a NEW account manager (#4) even though I had been working with a different one. I had worked with the prior account rep going over our architecture to make sure everything was kosher (sustained use discounts etc). I even made them a schematic of our architecture on GCP at their request. Once I provided that to them I was met with radio silence.
It's really insulting, and to me obvious they don't care about my company at all. We're looking at other options.
I wouldn't be surprised if somehow this post leads to me getting an email from another rep "Wanting to start over and doing things right", which will inevitably devolve into the discussion of how can I use BigQuery.
Yep, drives me up the wall. We put a lot of work in making sure we're zero-downtime, pay for "Highly Available", and every couple of months I get the dreaded email. No explanation, nothing. Then we have to do the whole downtime dance.
If anyone from Cloud SQL is reading this, please, please stop doing this.
It’s insane that even with all of their HA and failover turned on they take the whole cluster down for as long as they like every few months!
To an engineer, things are things, because of how they architect and build them.
To an engineer who understands a customer, things are things and all the things people actually use them for. Big difference.
It also makes "Well, that customer is using it wrong" less of an exceptable engineering dodge.
I haven't worked inside AWS or GCP, but I've never seen product get everything they want, especially around maintenance/downtime. If "less downtime" is on the roadmap but engineering is constantly pushing back "that'll be really really hard and take a long time and they're just using it wrong anyway," I can't imagine it getting done as quickly as at a place where the engineering team was also focused on customer satisfaction.
It probably is hard and intensive. Engineering shouldn't lie and promise that it will be easier. Product has to take that engineering estimate and determine whether to work uptime or some sexy feature (and sexy features usually win because of perverse incentives).
Moreover, I have a hard time believing this for a couple reasons: first of all, I've scarcely met engineers who were opposed to improving product reliability, maintainability, etc. The portrait of Google engineers arguing that database services fundamentally shouldn't be HA (and customers are "using it wrong" for wanting HA DBs) is particularly incredulous. Secondly, I've never heard of an organization where engineering held political power over product decisions, but I have worked in several places where product dictated engineering solutions. Businesses trust product more readily than engineering because the things that engineering is always petitioning for are abstract and "costly" (deferring some immediate profit for reduced costs in the long run) while the things product wants are usually tangible and profitable.
If product has PM or financial incentives that reward without regard to technical debt...
If engineering has PM or financial incentives that reward without regard to user experience...
If you want people to cooperate, set up shared, team-based incentives!
Behavioral properties are end-to-end, like speed, security, customer success, uptime, etc. Everyone and everything on the hotpath for that has to be on board, as just one violator means nobody is achieving it. Operationally, part of achieving that means company norms, such as when collaborating, planning, executing, etc.
It's 100% fine not to provide any of these, and is even reasonable given it's hard to change one's DNA end-to-end... but better to be an explicit decision. Likewise, if starting a company... a lot easier to pick & ingrain these habits ahead of time.
Sure, you can have zone-redundant Azure SQL Databases, but not Azure App Service!
You can have zonal Azure App Service, but with a different network model than the default, so it's a breaking change for some apps.
It's as if nobody has actually sat down at Microsoft to build something similar to what their customers are building. It's all just tech demos, "get your blog hosted in 5 easy steps", and crap like that. Nobody at Microsoft has ever built anything of substance with the majority of their own platform.
It was the same story with Windows Presentation Foundation (WPF). It was hot garbage when it was first released, but Microsoft kept telling all their partners to prefer it over the legacy GDI+. People tried and failed to write GUIs in it. When after many years Microsoft finally tried to convert Visual Studio to WPF -- the first time they had used their own framework for one of their own apps -- magically the core issues were fixed and WPF become viable for real apps.
In general, Microsoft, Google and others are behaving really enterprise-y in that they are slow, you cannot reach anybody of importance to solve massive issues with their products and everything is actually quite expensive for what it does.
NO, RDS includes non-Aurora versions of many DBs, plus MySQL and Postgres-compatible Aurora servers, plus MySQL and Postgres compatible Aurora Serverless.
Which is fine. It can just be a little confusing as they drift and the caveats grow.
Current Aurora Postgres is compatible with pg 12.4; pg 10+ support has been around so long that several versions that support 10+ have already been EOL’d by Amazon. Even Serverless, which lags behind, is on 10.x.
That said, this is not some poor rep's problem. This is a problem of Google culture and one that will be very difficult to fix. They simple don't care much about their customers and never needed to.
I just watched a load test hit that issue, the developers cranked up the traffic until the response times were measured in minutes. At that point Azure SQL started throwing errors related to too many concurrent queries.
Compared to just locking up or timing out, it's probably better.
E.g.: the 4-CPU pool has only 210 Max concurrent workers per pool, but if you think about it, that's over 50 queries running at once per CPU core, which is nuts!
Ps: if you’re working on a production load system, and has decent traffic, lean towards premium/business critical tiers for true HA
“Downtime as part of Scheduled Maintenance will not be counted towards any Downtime Period.” - https://cloud.google.com/sql/sla
I’ve never seen an SLO document for CloudSQL though so the SLOs may be slightly higher internally?
AWS tells you when your maintenance window for an RDS instance will be and you can delay/reschedule as needed.
I wouldn't consider a maintenance window as "downtime". That 99.95 applies to all the time outside the window.
When we first launched on GCP, there was no question that it was the way to go (frankly, because of BigQuery). Working with AWS, when we launched, was going to cost us significantly more up-front, before we had even brought on our first customer.
Fast forward 5 years... AWS has closed the gap in every way that matters. I still, frankly, trust Google's Security more than Amazon's, but I don't encourage folks to use GCP the way that I used to.
Just the opposite. In 2021, no questions asked, it's either AWS for general compute, or something more targeted if your business doesn't need it.
Until you get to the point where your bill is larger than a dozen engineering salaries, you won't get any respect from these people.
> I still, frankly, trust Google's Security more than Amazon's
If you have the time, could you expand on this? While I'm not directly involved in security at AWS, I'd be down to forward your thoughts to people who do.
My comments are mostly backed up by my experience at startups and are not colored by my experience at Google (too different a beast).
GCP is great for teams that are also using GSuite because you can set permissions at the level of a Google Group and have them propagate to individual members. You can, of course, also create groups in AWS but they don't have the same semantics of Google Groups and don't cover the wide range of use cases that Google Groups does.
The AWS scopes -> policies -> roles -> resources chain of abstractions is less natural conceptually than GCP's GSuite accounts + service accounts with attached scopes per project.
Also the fact that each managed service (GCE, GKE, Cloud Builder) has its own service account that you can attach scopes to is really nice. GCP service accounts just feel more discoverable than AWS IAM roles - I think it's because the number of AWS pre-built roles is so overwhelming.
Just some thoughts off the top of my head.
I’m sure they exist, but over the last dozen or so years I’ve worked with public cloud offerings across 5 or 6 industries and domains, I haven’t found a use case that can’t be easily implemented in the simpler GCP model.
AWS support is really nice. That I miss.
Top of mind things I found weak/weird:
- Basically can’t do least privilege, only want a role to read messages from one pubsub queue? Nope not possible
- IAM policies seem bolted on, legacy roles seem much better fit in the ecosystem, but they suck obvs.
- different gcp resources have somewhere between very coarse grain to just acceptable iam operations. Might just be GCS that lets you do proper least privileges policies like you would do in AWS.
I was very surprised how bad it was, AWS IAM is some black magic shit that is deeply impressive and often taken for granted, even GCP can’t replicate.
But I agree, AWS IAM is some black magic and is very powerful.
The biggest problem I have with GCP is that something will say "you need the foo.bar.baz permission", and when I go to the IAM page to give that to myself... there is nothing in the search results for "foo", "bar", or "baz". Instead, I have to guess the "friendly name" for the permission.
Often the UI, and docs make it seem like everything is all over the place but AWS feels like lego with some pieces tucked away. That is where I think AWS can be improved upon with a better documentation UI and discoverability.
I do have to commend Google on Flutter + Firebase + Firebase Functions. I think if Amplify focused on serving Flutter users more it could pull me away from Google altogether.
Unfortunately, Google has done a fantastic job with making Flutter integrate with Firebase through Android Studio and there really is no product from AWS that matches its developer friendliness and low learning curve. This makes it very easy to switch.
I guess it is somewhat of a threat because the Firebase Cloud Functions also offer something of a counter to AWS Lambda as much as I love using it with API Gateway.
And telling them to "just learn it" isn't the customer focused mindset, it's the engineering one.
Custom roles created through Terraform helped a lot.
Yes its a bit of a pain having to add policies sometimes when you are first getting started but once it's up and running you can rest easy.
Learning it isn't that much more time consuming or difficult, its just a bit of effort that is all (we are talking a few hours at most).
The discussion is about GCP having much better functionality around it where projects and permissions are naturally integrated with gsuite organizations and users. This is objectively true. AWS has an archaic project system with a dozen different attempts at uniting it all but nothing comes close to GCP's smooth and easy manageability.
Integration with our org GSuite (this alone is a massive plus).
I trust virtually anyone's security over Google's. I've never had issues with AWS. I've consistently run into serious Google security failures. Google has airtight security for its own data, but not for its customers.
Examples range from Chromebook and Android security update policies (tons of expired machines on the public internet, in the case of Android, usually without people knowing), to pay-for-security on GSuite, to really difficult-to-audit Google Drive security (there's no convenient way to track and audit what was shared with whom or where data went), to just a ton of other things.
I've never seen Amazon be callous with my data. I've seen Google do things that even nineties "we don't need security" Microsoft wouldn't have imagined....
The only people I know who really trust Google security worked for or are close to people who worked at Google. There's a reality-distortion field based on how much Google invests in its own security that people fail to notice very basic failures, like millions of expired Android devices, or a lack of audit logs if someone physically accesses your machine to rifle through your gmail....
Nov 2019 - interviews
Dec 2019 - passed hiring committee
Dec to June 2020 - emailed recruiter every two weeks never heard back
June 2020 - cold email from recruiter “here is your offer letter to join Google! Let's talk team match.”
Needless to say I did not accept that job. This was for a software engineer role.
(edited to add line breaks)
Many weeks to generate offer and paper work and then 24 hrs to accept with a "take or leave it" ultimatum.
Just a very poor taste and I know some fine people who work at Google. Its unfortunate they don't seem to value warmth and humaneness when communicating externally
Noped out of that as quick as I could.
It crushed me because I found a team that I was unbelievably excited to work with.
Me and a friend of mine interviewed in Nov and passed HC in Dec 2019 as well (different recruiters between the two of us). Our experience was different from yours: Immediate timely followups from the recruiting coordinators, team-matched and hired in the same month. Hopefully our experience is more common...
Some European-based customer apparently had a requirement if we engaged with them, that our service be offered via an acceptable vendor such as GCP, for some reason AWS apparently wasn't, but it was such a nightmare to even prod about an architecture that would have feature-parity with AWS, it wasn't even worth it. Also as an fyi, I'm no AWS fanboy, I don't use it in any of my own projects to avoid vendor lock-in this company suffered from.
Since at least when we started spending about $100k/yr we've had a dedicated account rep we can contact at any time. They also get in touch to schedule a check-in every few months.
They've been genuinely helpful in several situations and have scheduled meetings with various teams around AWS (like, actual engineers) to get us answers to questions, support, and guidance. We've been put in touch with team leads and engineers working on beta features when we tried to use them and had issues to report.
Obviously this is all a sales tactic: if we have questions about X, putting us in touch with experts in X makes it more likely we'll successfully implement it and then pay them to use it. But it's the kind of sales we're getting value from, not just blindly pushing us to pay them more money.
We don't pay for any support package or anything.
Developer support $29/month, and business support is $100 (both go up if you spend more).
I paid for business support. You get
24x7 phone, email, and chat access to Cloud Support Engineers
Unlimited cases / unlimited contacts (IAM supported)
This is for $100/MONTH!! That is the deal of the century.
And they are ridiculously helpful.
I don't understand this - AWS must be losing money at least on support side, though they obviously get happy customers (myself included).
And even at $150 this would be great.
I had a client on gsuite with google 8 years ago - we COULD NOT get anyone to help with some weird admin state flow issue - it just was not possible to talk to a human being for ANY amount of money.
Working at a place with AWS enterprise support by contrast, for the second occasion, I would suggest that many of the places paying $15kpm don't cost that to support.
Google has treated me so badly so many times now on my personal account (as well as on business accounts, for that matter) in so many different ways that they've, conversely, lost MANY million dollars in business sales. It's hard to even count; a lot of people ask me for advise on decisions, and whenever someone even thinks about using Google Cloud in a business setting....
This is not a hole I see Google getting out of, except by eventually shutting down the Google cloud. Too many people have had too many bad experiences, and reputations take a long time to recover.
And the failures just keep on piling up.
Google is great for personal use, but I think they're diversifying in all the wrong directions. They're not structured for success there.
Like regional disks  or live migration of compute between hosts if problems are detected with the host.
When would you ever want to rely on this? Seems to me like you should have two hosts in the first place.
I'd guess they don't get many resource intensive support queries from the < $10k a month customers (and at that level you probably don't get the A team support)
The other poster indicated it is possible for stuff like this that you get bumped, even on the el cheapo plan, to someone actually on product team. While I'd hate to be on the product team having to answer these, it perhaps keeps folks aware of customer issues so they can at least improve docs for corner cases?
In general the AWS support has been great. In many cases, they've forwarded our requests to product teams who have even fixed bugs we've run into and contacted us directly.
Our other experience is with paid Azure support, which did little else than direct us to the (not related to the question) docs. They also had a really hard time understanding our technical questions about specific APIs. To their credit, they did eventually escalate to the PM of the service in question.
In general, the team responsible for the service really must be able to help out with support requests. In AWS this is definitely the case, in Azure as well but there's a bit of gatekeeping. Does developers and PMs in GCP participate in support?
Their model was fault-based, and a “bug” gets billed to the support group. So the game was always for MS to avoid assignment for non Sev-A cases, and our game was to find a product defect for anything.
Yeah... just like the other 5 dupes of the same suggestion with hundreds of upvotes that the product teams have dutifully ignored because it only matters to customers.
There are plenty of cloud agnostic platform out there that does VPS, load balancers. digital ocean, linode, etc.
If you want to be green there is also the Advania they use geothermal energy. And since it is in iceland better data protection policy than US companies.
I'm not affiliated with any one of them, it is just from year of cloud provider hopping.
Only thing I miss is firewalls for cloud (they have it for dedicated).
While everyone else is talking about game engines, design and programming techniques, Google's talks are mostly around their cloud offerings, and customer telemetry.
So why is Google this way? They already have a terrible reputation for their non-existent support. Never thought they'd treat actual companies this way though. It makes no sense...
Of course, now when they enter a business that needs direct customer interaction, it shouldn't be surprising that they aren't good at it. No supportive processes and probably no internal appreciation for what reps do. Thus the turnover. They will have to build out a sub-org that nurtures and develops this part of the business.
I personally find customer interactions problems to be epidemic with fast-scaling internet businesses. It does however create a space for others to come in and do it better. We can hope...
>> Hahah. They did that to me too. I doubt they even looked at it. It was a fun homework assignment though.
"Once I provided that to them I was met with radio silence."
>> Hahaha. Same here.
"which will inevitably devolve into the discussion of how can I use BigQuery"
>> This is a trap. Once you are on BQ there is no sane way off.
Experiences with support or lack of support concerning other Google product areas and divisions, especially those not designed for businesses really don't apply if you know how Google operates.
I don't use GCP in a personal capacity at this time and I work for a competitor (though am certainly not speaking on behalf of the competitor).
It is true there are individuals that shine, but across the board, Google sucks at support. It starts at the top with what kind of company they want to be, and goes from there.
They do not like humans.
You should have seen the extent to which the rest of the team and I tried to resolve customer issues.
Random App Engine app getting 500s calling an Open ID endpoint (the service wasn't strictly speaking managed by GCP). I absolutely could not reproduce this and had no other reports, but I knew it was happening for the customer. I examined everything that could be different... After 2 weeks I finally got it:
The customer app was running in a different data center compared to where I was testing. Calls to the API endpoint were timing out internally (pretty aggressive timeout interval). It turns out the Open ID service wasn't running in the same data center region. That should not have been the case but occurred because the internal BigTable service had to be upgraded in a particular region and hence all App Engine apps in that region were moved to another one - a fairly seamless process. The Open ID service unfortunately wasn't provisioned in that region (it's not a dependency of App Engine).
Once I identified the issue it was a matter of telling the on call SRE about it who then spun up Open ID in that region and thereby resolved the issues in a matter of 5 minutes.
Often a new support case meant discovering platform limitations - for example running into BigTable anti patterns because App Engine Datastore allocated IDs sequentially by default. That could cause poor read / write performance, in part due to automatic sharding behavior of BigTable. We later switched to snowflake IDs.
And then there were lots of Java folks used to Classpath scanning (default behavior of Spring Framework?) which for larger projects caused App Engine container startup to exceed the allowed time limit. That became a user education issue for which we then had to write an in depth article.
Good times :)
I absolutely hated dealing with billing inquiries though... so difficult to map someone's code to billable operations. Some SDK methods were of course utilizing multiple billable operations. Determining SLA violation refunds was much easier.
I'm not saying you didn't do great work, but I think the secret to success at AWS Support is normalizing this level of customer obsession.
We did whatever it took to resolve a support case in the quickest yet most satisfactory manner, even if it meant doing some crazy research, meanwhile we got of course overloaded with more support cases coming in.
I had 100% customer satisfaction from all surveys. That was definitely hard to achieve because happy people usually don't complete surveys. :)
These were the early days of many GCP services. I worked there when Compute Engine and Big Query GA'd. This was just before and possibly during the time the paid support tiers were launched (I don't recall). I left mid 2013. No idea what their support is like now.
And those technical account managers just bounced customer questions to us 3-5 support engineers (technical solutions engineer)
Anyways, your AWS comparison may apply to the GCP support culture now (I have no idea) but back then we were all extremely customer obsessed - by choice.
At the end of the day, I don't think the complaints you hear about GCP Support / Service has anything to do with the people working on the line. There's always better or not better examples of folks you encounter as a customer.
As a support consumer, I've learned that my feelings about the person on the other side are very much colored by the fact that I almost never call support when things are going well or I'm in a good place. I typically call support when all else fails and I'm probably ready to throw a product out the window. That's not their fault, and it shouldn't be their problem, either, to a certain extent. They're humans, doin' a job, just like me.
As a support provider, I've learned that the same is true for all of my customers. I need to know that this isn't who they are (usually) when they go home to their family, or out with their friends. They're under a lot of stress and, one way or another, we've let them down and have to deal with that.
I think the challenge with GCP support is all about the overall strategy as a business. My experiences with GCP support is that I have to work way too hard to get past the robots and the scripted responses in order to find someone who is willing to listen and engage with my question or problem. And then, to make matters worse, about 80% of my interactions with Google support involves them trying to introduce me to some consulting company or third party provider. That is pretty much never what I want. I don't want some third party billing integrator that is going to mess up my reporting with their "value added" service. I already pay for support, I don't want to hear a pitch about paying for more support when I feel like I'm not getting what I've already paid for.
I don't hold this against the support folks at Google.
I hold this against the leadership at Google.
This is their strategy to contain costs or to juice sales.
If it were working, I think their numbers would be very different in the market. So the proof is in the pudding.
Something isn't working at GCP.
I don't think it's the engineers or the teams. It's the leadership and the strategy.
A lot of support became outsourced and the original support team became more of a tier 3 backline support with the first 2 tiers being handled by vendors. This started for GCP just after I left in mid 2013.
The vendors responsible for support have contracts to meet certain customer satisfaction metrics and time to first response metrics (based on customer tier and ticket priority). The majority of inbound support cases would get auto assigned to the vendor team. Only the top customers would reach Google employees directly.
The vendor and their staff simply have no incentive to troubleshoot an unusual issue - they also cannot have access to the tools that may expedite such troubleshooting. It's all about pattern recognition to quickly resolve a ticket. As many vendors were overeager to escalate support cases to the Googlers it became necessary to impose more requirements on the vendor team to be able to escalate an issue - invalid escalations were being tracked and the vendor would be penalized for the total of these.
I built the internal tool via which GSuite and GCP outages or significant issues would be managed from a support perspective. This had the advantage that vendors didn't need to waste their time (poorly) parroting (with a delay) the status of an outage as related incidents could be (manually) grouped together and then all be updated at once by the current incident manager on the support team - freeing up lots of support folks to work on other support cases.
I definitely agree this vendor-based support model is a frustrating experience for everyone involved. I witnessed that myself after leaving Google. By the way, I'm very surprised GCP tries to upsell you on consulting services based on your support interactions.
Running a support organization for a very technical product with so many unknowns and variables at scale is hard. I wonder who does that well and how they accomplish it (from an operational perspective).
I have no idea how AWS manages support internally. I work at Azure - until last week as a Developer Advocate, now as a software engineer in Azure CTO incubations - I have no idea how our support organization is run or what their quality is. However, from what I hear Azure support is eager to help anyone and everyone. I remember being very confused I got a phone call to inquire whether I needed help within a day or two of signing up for an Azure free trial.
(no, I'm not and have never been a Google employee)
Back in the dotcom days companies would spend a fortune on Sun kit but I bet when averaged out over time a comparable company would be spending a LOT more on cloud billing.
I would like to learn more about this. I'd have thought costs should go down over time. Are we doing more or is the cost per unit (not sure what that means) is truly going up?
I think a number of factors add up over a 3-5 year timeline (obviously this will be more or less true for different organisations). There is the way the cost scales for a given instance in the cloud - in the old days, for example, doubling the memory or doubling the CPUs didn't double the cost of the kit, but it does for clouds VMs.
Another example is that the cloud bills you for everything, in the old days I could have a database server on a network and query it as much as I liked, the cost was fixed upfront for the lifetime of the hardware. Whereas it's very cheap to get started with a managed offering but e.g. BigQuery charges you for every query, Cloud Functions charge you per invocation, bandwidth is chargeable etc.
Speaking of hardware, in the old days you could look at your hardware and say, actually, it's fine, we don't need to upgrade/replace it this year after all, and it will just keep running. Whereas in the cloud the payment is continuous (perhaps offset by the fact that it's easier to "give back" excess capacity).
There will come a point at which the cost of DIY vs cloud will cross over, the question is whether you will reach that point, and if so, what you will do about it, since you may be well and truly locked in at that point.
You always can drive cost concessions from sales, especially for base workloads where you have time flexibility.
For a big company, cloud rarely saves money for many categories of expense. In a normal market, it is almost always faster time to market to rent, and always cheaper TCO to own.
This is a different market and one which is ultimately constrained by the availability of land. Notably, cloud prices do fall especially relative to the compute power. Specifically I remember when Fargate moved to firecracker and prices fell by like 40% or something similarly considerable.
Maybe managing your own internal cloud is indeed cheaper (especially if you don't account for support or maintenance!), but arguing that cloud prices don't decrease or making some housing analogy seems like poor reasoning.
It’s always better for you as a person to rent box truck to move. If you’re a company that needs a truck 3-5 times a month, there’s a probability that leasing may make more sense.
I’d say that businesses that suck at managing on-prem will not magically get competent in a public cloud.
It takes time to build a competency. If you have to figure out how to operate everything from the hardware and networking all the way up to application, then you might very well fail as a business. If you can pay Amazon to manage your networking, hardware, databases, managed services, etc while you work on your application-level competencies, you stand a much better chance of succeeding ("I don't know if I could build everything from the ground up, but I can probably fumble my way to a container image"). As you become very proficient, you can cut costs, and if you're super proficient you might get off a public cloud altogether or more likely you'll just use that as leverage to negotiate a lower cloud bill.
Interestingly, you don't hear about many businesses that are so competent that they move from the public cloud to their own private clouds. Some people will shout "Lock in!", but I don't buy that--I've migrated between public cloud providers before and it's work, but if you're competent enough to run your own private data centers then you're plenty competent enough to migrate to them.
GCE also isn't so much predatory as incompetent and apathetic. If a Google failure wipes out your startup, you're a statistic and it's okay.
The massive number of newer services are propriety, often buggy, and poorly documented.
I don't use any AWS services introduced past 2015 or so.
Not to say that isn't a useful article on its own, but it's hard to draw too many meaningful conclusions for the rest of us when the first line of the AWS bullet points is "reach out to dedicated YC email".
I'm in a similar position and those credits would be lifechanging.
The grants start out small (I think the first one was $3k) -- but then once you spend 75% of them you can apply for the next round, which for me they gave me $17k out of a possible $30k based on previous usage. After I spent around $15k for that over the next year I applied again and got $100k.
Just one note though that I already had an MVP that I was running on GCP at the time I applied (but I was within the $300 free credits that I started with so I didn't pay out of pocket).
Also good PR.
I like Firebase, but the moment the utility runs out we're heading back to AWS.
Just from a technical perspective I'd advise to avoid the firebase databases like the plague.
I've built a prototype on top of it and it's fairly rudimentary but you can make it work. My biggest concern is that there are no case studies I can find of people using it past the prototype stage and how the pricing/scalability works out at that point.
Love to hear from anyone who has gone beyond the prototype phase.
It's not completely wrong, but incomplete.
So if you want to work on building your business and not debug cloud provider issues, avoid the new upstart.
If you just want to get started with GCP, you just sign in with your Google Account for $300 in credit. When you run out you can just start paying with a credit card.
AWS: makes me apply to a program to get credits and if I want to price out anything it's almost a full-blown research effort where I have to dig through documents and cross reference tables between different services and I still probably miss something important. Free tier is opaque enough to frustrate even "use it at small scale and see," because you still have to do the research to see if they are pulling a "your first hit is free but try to scale and WHAM." Oh, but they're customer obsessed, so after much begging and pleading they'll refund half of WHAM, one time.
If I punch in the services I want to use, I can quickly see how much they will cost, and it's easy. I recommend clicking "Advanced" inside services on the calculator if you want to be sure you're not going to run into an edge case that costs lots of money unexpectedly.
If you use any of the megaclouds without first understanding the price structure, good luck.
Personally, I think a lot of companies would be better off using DigitalOcean or some other medium-size cloud. The pricing is much simpler, and pretty much all medium-size clouds charge $0.02/GB for egress bandwidth, either globally or in the US, depending on the provider.
In contrast, the megaclouds make shocking amounts of money off of their extremely pricey egress bandwidth, among other highly profitable aspects of their business.
Even still, AWS is a solid cloud platform, and they really do seem customer obsessed compared to what I've seen happen on GCP.
That said, I'll grant you that AWS is not the only megacloud leveraging opaque cost structures. They just leverage them more.
AWS support is genuinely a cut above, though.
I use GCP now. I find their tooling and UX to be about 100% better and will not use AWS in the future if I can avoid it.
Can you provide an example of a service that the AWS calculator doesn't compute the correct price for?
I honestly can't remember ever being surprised by what things on AWS cost, and I don't think I'm that shockingly good at detecting hidden costs.
When I shared this anecdote at Re:Invent, the sentiment at the table was "lol that's cute, here's my story with an order of magnitude higher price tag." There were 5 or 6 of us, and my story was the smallest surprise cost except for one other person who was even greener than I was.
> Can you provide an example of a service that the AWS calculator doesn't compute the correct price for?
Can you tell me how I should have used AWS calculator to prevent my surprise charge? You can't, because AWS calculator assumes you know exactly what you're asking for, and the problem with opaque pricing structures is that you sometimes don't.
Other clouds tend to be much more upfront about "this will cost X," "this is costing X," "you're out of free tier," etc.
Apples and oranges.
It's really nice to hear from high-prestige companies about their experience because it gives outsiders a sense of what prestige might fix. If you're a small startup looking for help using AWS, finding someone to introduce you might really help! Not so much with GCP.
All in all, it had taken our incubator at least 1.5 years to get enrolled in AWS Activate, and 4 days in GCP for Startups.
When we applied for GCP for Startups, we were approved in a week, and were contacted right away by an account manager and a support engineer who offered us help with anything we needed. When we applied for Activate, it took a month with lots of followups, and being shuffled between 4 support team members who were communicating with the Activate team (there was no way to contact them directly).
It definitely felt like our business was important for GCP, and for AWS, it wasn't. It was an easy choice.
Google has never had a good reputation for Customer service, they believe they can solve all Customer Service with bots and Automation, this will ALWAYS lead to a lower level of customer service.
Amazon started in retail sales where customer service is king, so it naturally has batter customer service philosophy than Google.
Google has shown zero signs it even desires to have a customer service philosophy that is remotely similar to Amazon, or even Microsoft Software which is somewhere between Amazon and Google on the customer service scale
My point is that obviously AWS thinks that on average being in YC is going to result in more revenue for them and therefore they prioritize support for YC companies. As a non-YC company I won't get the same treatment which makes any conclusions from this article less useful.
Recently, I needed to increase a CPU-limit quota from a small number (like 16 vCPUs to 64 vCPUs) - nothing crazy. In the past, the quota increase system was more or less automated and would only take a few minutes to process.
This time, however, GCP denied my quota increase and forced me to schedule a call with a sales rep in order to process the quota increase. It was the biggest waste of time and kind of goes against the entire point of instant cloud resizing.
It also feels like the velocity of new features, instance types, etc has slowed down dramatically in the past year. Also, while I'm ranting, Google Cloud SQL is probably the worst cloud service I've ever used (and it costs an arm and a leg for the pleasure!)
1) Increase the disk size to 100GB as this increases the IOPs
2) Switch to using private IP addresses. Huge speed increase
3) get rid of cloudsql-proxy. Another huge speed increase
^ Do not use cloudsql-proxy ever. GCP docs are wrong. DO NOT proxy all your db requests through a single VM.
Switching to private ip definitely had the largest impact by far on performance.
Interesting. I'm looking Cloud SQL right now and the advice seems to lean in the opposite direction: use public IPs for ease of connecting. Can you quantify the decrease in latency? All I can find is bits about reduced network hops.
I didn't have to worry about the 'ease of connecting' when using CloudSQL as all my GCP services are in the same VPC.
Also, private ip is one less security problem to worry about.
We abandoned cloudsqlproxy partially because the sidecar in Kubernetes caused random disconnects, but also because it just had bizarre network errors sometimes. (That was mostly when connecting from outside GCP though.)
MySQL for our DB.
- No way to upgrade major postgres version without full export and import into new cluster.
- Incredible delay between postgres versions. IIRC, it took nearly 2 years for them to add postgres 11 after it was released.
- HA is basically useless. Costs double, still has 4-5 minute window of downtime as it fails over, doesn't avoid maintenance window downtime (both primary/standby have same maintenance window) and you can't use it as a read replica. Honestly, feels like a borderline scam since I'd imagine a new instance could be spun up in the same amount of time a failover takes (but I haven't tested)
- With default settings, we experience overly aggressive OOM-killer related crashes on a ~monthly basis during periods of high utilization. On a 32GB instance, OOM killer seems to kick in around 27-28GB and it's incredibly annoying.
- Markup over raw instances is almost 100%, with no sustained use discount outside of a yearly commit.
It's just a lot of money to pay for a crashy, outdated version of Postgres.
To be fair, it looks like GCP supported Postgres 13 (Nov 5, 2020) before AWS did (Nov 27, 2020) and AWS currently marks Postgres 13 as a preview. Maybe GCP had a large initial engineer-cost to support multiple versions of Postgres and now the incremental cost to add new versions is small?
> It's just a lot of money to pay for a crashy, outdated version of Postgres.
Have you looked at other options? I'm evaluating GCP SQL and the comments in this thread are scary. Seems like Aiven might be a good way to go. I've also briefly looked at CrunchyData's Postgres Operator  for Kubernetes but it's a lot of complexity I don't really want.
Things that seem similar in AWS:
- For major version upgrades, you need to bring up a new instance from a snapshot and catch it up with replication.
- HA failover results in a few minutes of downtime. (They claim using their SQL proxy will reduce this.)
- Lag in providing the latest Postgres versions. GCP seems to be a bit ahead of AWS here.
Is there a managed Postgres offering that you prefer? Aiven looks nice, feature-wise.
Am using SQL proxy but doesn’t do much re: HA.
I don’t know, I’ll probably just run my own Postgres at some point. The only peace of mind that I get from Cloud SQL is the automatic backups.
I will guess that you are much higher scale than us.
AWS has a similar thing.
I am founder of another YC backed company. We based our startup on GCP infra. They have great tech (for the most part) but I regret it so deeply for two reasons:
Support or desire to help customers is non-existent. For any questions, they want us to upgrade to paid support and pay them at least 10% more every month for that (we ask like 1-2 questions a year). What? We are already paying you thousands of dollars every month! I get included support for all my software subscriptions - so this is my biggest beef. Also, when they do help, they keep passing you around from team to team and dont resolve issues as well I’d like. It is just not a company I can love as a customer.
Their status dashboard is a joke. They dont even report minor outages, when they do, they start after a huge delay and update very slowly. And worst of all - when it only affects a single zone or a single region, they remove it from historic reports so everything looks green/great.
I’ve experienced both these times multiple times.
I have to assume AWS is better.
Basic business hours support is $29/month or 3% of your service spend, whichever is greater. 24/7 is $100/mo or 10%, which also includes outage assistance.
I’ve also worked for places with enterprise support ($15,000/mo or 10%) but of you’re bringing in millions per month it’s definitely worth it.
The AWS personal health dashboard is also pretty reliable. The public status page is the source of many jokes.
For Media services, the supporter will almost always need to coordinate with an internal team, which there is no visibility over, and then it becomes a game of telephone to make the supporter relay the information in a way the internal team understands. I've had the same thing happen with peering/networking related questions.
For EC 2, VPC, DynamoDB kinda questions, they are indeed pretty good.
1.) Try to figure things out ourselves
2.) If we can't figure it out, subscribe to AWS Support.
3.) Get question answered and then turn off the support plan.
You'll have your support plan for the rest of the month and pay a prorated amount for the days during which you had support. It's quick and cheap.
I guess grass is not greener on the other side. Oligopolies for the loss.
This whole onboarding felt like some caricature of what I thought were exaggerated stories of how bad the support was.
Lol no. AWS NEVER updates its status page unless it is a massive incident that everyone notices.
As for support, if you pay for the most basic plan you will get a basic answer within 48 hours usually. Its decent. The more you pay, the better the support.
For most organizations role based production plan is good enough
Did you know that the grass is always greener on the other side?
Better would be to think about what the general initial usability of these services are. How easy is it to spin up the compute load? Create reasonable IAM policies? Debug problems?
My own experience (and bias) is that while AWS has vastly more features, GCP is much more usable. The latter feels like a coherent setup with projects and IAM. AWS always has a surprisingly amount of work around org accounts and IAM setups.
So maybe it's faster to get AWS credits, but it's much harder to make use of them.
AWS Organizations is atrocious compared to GCP projects. I truly do not understand how/why AWS is doubling down on the awful user experience of Organizations, cross-account access, creating multiple accounts, etc. It's truly the antithesis of customer obsession and yet they show no signs of moving away from that model. And not only is the user experience bad, it also just feels hacky. AWS Organizations and Control Tower feel like poorly applied band-aids rather than real solutions.
I actually prefer AWS to GCP in most respects, but the AWS Organizations shitshow is something I can't personally get over.
Effect of ToS violations
Google-wide disabled account
In some cases a Google-wide account (which covers access to a variety of Google products like Google Photos, Google Play, Google Drive, and GCP) will be disabled for violations of a Google ToS, egregious policy violations, or as required by law. Owners of disabled Google accounts will not be able to access their Google Cloud resources until the account is reinstated. If an account is disabled, a notification is sent to the secondary email address provided during the signup process, if available. If a phone number is available, the user is notified via text message. The notification includes a link for appeal and recovery, where applicable.
In order to regain access to their GCP resources, owners of disabled Google accounts will need to contact Google support and have their account re-enabled.
To minimize the effect of an account being disabled on Google Cloud resources, we recommend that you add more than one owner to all resources. As long as there is at least one active owner, GCP resources will not be suspended due to the one of the owners being disabled.
A few years back, our business credit card was somehow stolen and used to buy Google Adwords. We disputed the charge with our bank. A day or two later, at 4am local time, our GCP account was suspended for fraud (presumably because the same, stolen, card was attached to that account). All instances were stopped and our service was brought to a halt.
We couldn’t contact Google Cloud support because our account was suspended. We had to go through our network to get our account re-instated. Pretty awful way to start our morning, to say the least.
You can fairly easily swap billing accounts a project is using if you still have an organisation admin account that isn't suspended. I saw this risk coming at a previous company and tried to get them to treat billing accounts like any other important resource and go for redundancy, but the director could not see the value and our Google account manager denied the risk existed, luckily they have been more fortunate and this has not happened to them yet, although it did come close once with some issues with the payments.
We actually created a portal where you can report unknown charges to Google directly; https://payments.google.com/payments/unauthorizedtransaction...
Credit card companies tell you to work with a company before issuing a chargeback, as chargebacks are a last resort. The above form helps you keep you account active without being swept up in the chargeback process.
It's something we all want to keep improving. I'm hoping as SCA (strong customer authentication) rolls out across europe and maybe other countries pick it up, it should cut down on issues like this.
The only logical thing to do in the case of a stolen or copied card is to contact the bank directly, so that the card can be canceled and fraudulent charges reversed.
Hold on while I go mine myself a ton of crypto coins.
The idea that someone could use a stolen credit card associated with an account, buy something on some Google service, have a chargeback processed as fraud and that would trigger Google's suspension of services on GCP is absurd.
When even Google themselves is recommending gaming their system since they can't guarantee it won't screw you over, that should be a warning sign.
The support isn't perfect, nor is the product -- but I would say the level of customer service for GCP can't be compared to other Google products.
1) AWS was extremely expensive
2) Our GCP bill is about 1/3 of what AWS was
3) The Kubernetes offering is top notch
4) Google giving us credits and offering us consulting were the triggers that started us talking.
Always like to see why people make these big changes though, thanks!
1) Costs per customer are lower because you can fit more containers per VM due to kubernetes doing the scheduling for you. Customers also include developer environments.
2) The number of deploys is way up because there is a simple and established pattern that everyone follows.
3) The speed of creating new services has increased because of the established patterns with containers, kubernetes resources, and deploys. Thinks days vs weeks to get something running.
4) The number of Ops issues are lower because kubernetes handles so many things for you. For example, if a deploy is incorrect for some reason, the old service is sitting there running. No outage = no escalation = everyone sleeps at night.
Just don't use EKS. That is managed Kubernetes in the checkbox marketing sense only.
- Google has a lot of experience running containerized services because of Borg. Google Research says that they have been running these workloads since 2005.
- The above gives them insight into how to do this well. They would have the internal infrastructure, logging and monitoring already setup.
- They are the creator and still a major contributor to Kubernetes itself which means they can insert their influence on it.
Note: I am responsible for 5 clusters, 300+ nodes total and several thousand running services. Not huge by any means.
Amazon also has been running internally on containers for years before ECS. But workloads for massive FAANG companies designed by FAANG engineers turn out to be quite different than most AWS/GCP customers.
Just like EC2 wasn't Amazon selling its "spare capacity during off-peak" but always a purpose built service with completely isolated data centers and network fabric from day 1, the connection between Borg and Kubernetes is tenuous at best.
Honestly, I would encourage you to evaluate ECS and Fargate. Forget Kubernetes and direct instance management. Try a construct like https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws... and see how much simpler it is than all the K8S overhead.
Source: I'm at Amazon but I work on none of these services and opinions are my own. I played with K8S and am eternally confused how it's as popular as it is. I guess it gives you the illusion of not having lock in?
I don't see K8S being about lock-in at all. Using a cloud provider gives you some sort of lock-in and you should be utilizing that providers strengths.
Using Kubernetes is all about the patterns that it provides and how it removes a certain class of problems for you. Deployments, scaling, logging, etc are some of the patterns it provides and the consistency matters. How many of us have worked at companies where deploying two services have been completely different? One team runs the jenkins pipeline while another ftps the files over. Now multiply that by several services and several tasks (logging, scaling, etc).
The benefit is in the patterns.
2. Does ECS have something similar to pods, i.e. colocated containers which can communicate over localhost, share filesystems, etc? A quick search didn't turn anything up.
What were your pain points with ECS? Did you use ECS with or without fargate? What about EKS, since that sounds like a closer equivalent to GKE.
I've used ECS more than fargate. My biggest problem comes from reliability and having to manage things. We still run things on ECS and we get about 30x as many maintenance things we need to do, like retiring instances or instances just dying on us and we have to move things off of it manually.
I have a little bit of experience in Fargate from a previous job but not enough to make a real conclusion about it.
I've never used EKS but I do have friends that would rather roll their own version of Kubernetes on EC2 instance instead of using EKS.
This makes very little sense for Cloud Computing where each one your clients gives you large amounts of cash. Maybe they are too used to the first scenario.
edit: revenue GROWTH is declining