On: Gitlab.com is down

[mdaniel]

> Components: Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - GitLab SaaS Shared Runners, CI/CD - GitLab SaaS Private Runners, CI/CD - Windows Shared Runners (Beta), SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary

> Locations: Google Compute Engine, Digital Ocean, Zendesk, AWS

that's some _impressive_ blast radius and I actually struggle to think what would cause that much damage

[jtriangle → mdaniel]

DNS, obviously.

[mdaniel → jtriangle]

According to the title of the issue (2023-07-07: Blackbox probes for https://cdn.artifacts.gitlab-static.net/cdn-test are failing) you win the game:

  $ host cdn.artifacts.gitlab-static.net
  Host cdn.artifacts.gitlab-static.net not found: 3(NXDOMAIN)

They must have used that same healthcheck for every single load-balancer or something, though, for it to nuke AWS, DO, and GCP

[rewmie → mdaniel]

> that's some _impressive_ blast radius

Most of the entries look like batch job processing, either by gitlab or managing jobs in runners.

Perhaps authentication, which is mandatory when managing batch jobs including in third party nodes.

If auth goes down on any service, you'll see similar blast radius.

[trashburger → rewmie]

Well, consider the fact that "most of the entries" comprise 10% of GitLab and things like "Git Operations" comprise the remaining 90%. This is a total system failure.

[junon → mdaniel]

Might be a domain issue or a routing misconfiguration. But people are mentioning 5xx errors so it could also be FELB/NLB misconfiguration.