> We are experiencing an issue with Cloud PubSub beginning at 2019-05-20 21:35 US/Pacific. Current data indicates that approximately 1% of publish operations globally (15% publish operations in asia-northeast2) and approximately 20~25% of various types of admin operations (i.e. CreateTopic) are affected by this issue. For everyone who is affected, we apologize for the disruption. We will provide an update by Monday, 2019-05-20 22:45 US/Pacific with current details.
> Further investigation indicates that approximately 3% pull and 5% publish operations globally are seeing 5xx and 499 errors. Various admin operations like CreateTopic and DeleteTopic are also seeing >50% errors.
This is a pretty huge failure, would be interesting to see the PostMortem
Our application is calling GetTopic on a whole bunch of topics at startup to ensure that they were already created, so even though the publish error rate was only 1%, the app couldn't start because it kept failing the GetTopic calls :/
> Further investigation indicates that approximately 3% pull and 5% publish operations globally are seeing 5xx and 499 errors. Various admin operations like CreateTopic and DeleteTopic are also seeing >50% errors.
This is a pretty huge failure, would be interesting to see the PostMortem