Hacker News new | past | comments | ask | show | jobs | submit login
Tell HN: AWS outage in us-west-2 AND us-east-1
81 points by ceejayoz on Sept 18, 2023 | hide | past | favorite | 17 comments
Last ~40 mins or so we've been getting intermittent internal DNS failures in us-west-2 in our Lambda-hosted app. AWS status board now shows "Lambda invoke issue: We are investigating increased invoke error rates in the us-west-2 Region."

arn:aws:health:us-west-2::event/LAMBDA/AWS_LAMBDA_INVOKE_ISSUE/AWS_LAMBDA_INVOKE_ISSUE_ab21a726-de18-5a2f-b2ac-5cefd2794db9 on https://health.aws.amazon.com/health/home

edit: Status page now says "Operational issue - Multiple services (Oregon)" and "Operational issue - Multiple services (N. Virginia)"




Side note: Twitter used to be great for "is it just me?" on this sort of stuff, but their search no longer shows very good "latest" results for search queries. Quite the bummer, it was always comforting to see confirmation.



That’s where I went and I saw your tweet. So thank you!


I'm seeing issues with SSHing into new ec2 instances and getting those new instances to reach apt to fetch packages. I'm using packer. My last successful build was 4 hours ago. This is in us-west-2. AWS service health dashboard doesn't show anything for me.

EDIT: Health dashboard shows all services as green, not that the dashboard is down.


They seem to hide issues from folks they don't think are impacted.

Logged in (1 issue) versus logged out (0 issues) health dashboard: https://imgur.com/a/QRhZMf2


Just for those following along. Even after I login, I still have 0 issues. I'm not sure how AWS determines when to hide a outage from a user.


Ooop, just had "Operational issue - Multiple services (Oregon)" show up.

edit: Aaaaand "Operational issue - Multiple services (N. Virginia)" now.


yup, finally seeing it on my health page.


The issue I'm shown has "Service: Lambda" on it, so I wonder if it's only shown to folks with Lambda functions live. Someone tweeted at me that they're having issues on EKS, and that they use private DNS inside their VPC as I do.


Not sure. We have lambdas in this account, but not that have triggered in a several months.

I'm not sure if we use Private DNS. We have some custom resolvers and rules. I see now they are calling it network issues.


Thanks for the tip... that is mildly infuriating =/


The only power developers have over companies who lie on status boards is to hit them in their wallets and file a support request for every failure, so they have to triage that instead of just being honest and upfront.


We're experiencing the issuew. AWS just reported over the phone that Route53 DNS and “quite a few other services” experiencing outages.


Snowflake not working well either..


Yeah... very painful!


We started seeing evidence of high packet loss in us-east-1 at 17:38 GMT. All seems good now.


Last DNS failure (us-west-2) in my logs was 18:52 UTC; something's stabilizing.




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: