On: Google App Engine and Cloud Datastore Outages

[Legogris]

This is pretty bad - on a regional cluster:

$ kubectl get nodes The connection to the server XXX was refused - did you specify the right host or port?

BigTable is also not responding for some time now.

EDIT: This is us-east1. Responding again now.

[mstg → Legogris]

What region is this? GCP Console doesn't function properly (api errors) but kubectl and all apps work. (europe-north1)

[Legogris → mstg]

us-east1. Our cluster in europe-west1 seems unaffected.

[sethvargo → Legogris]

Correct, this outages is affecting a few (but not all) regions.

[solidasparagus → sethvargo]

Will Google please consider explaining to us why we continue to experience multi-region failures and what will be done so that we can build reliable systems on top of GCP?

I have been taught by AWS that we should expect occasional cross-AZ failures and almost no cross-region failures. This does not appear to be the case at GCP. I would like to have GCP as a cloud option - some of your tech is very impressive - but I have no idea how to design infrastructure on GCP so that I can be confident it won't fail due to a GCP problem that I cannot fix.