This relates into the problem first highlighted for Let's Encrypt's tls-sni-01 (and tls-sni-02 but that never entered production use) validation methods, which cascaded into the entire method 10 of the Ten Blessed Methods.
To recap that: Several popular CDNs or bulk virtual hosts let you type in any hostname and say you're going to host it with them. They figured that if the people who really own that name don't agree, they just wouldn't point their DNS records to the relevant servers and so it's fine. Let's Encrypt's tls-sni family of validations do a DNS lookup for the name to be validated, but then request SNI for a different invalid hostname as a signal. If a CDN or bulk host used for say, example.com allowed another customer on their service to say they wanted the invalid hostname as their "real" name, they could pass the tls-sni validation for example.com and get a certificate issued.
Naively we often assume that it's OK to let users pick names in a service without taking any step to validate that the person asking is entitled to the name. We may even kid ourselves that the custom name doesn't implicitly convey any legitimacy, that @MileyCyrus might be just some 14 year old from Swansea named Miley Cyrus, that mcdonalds.com could be a family web site of the McDonald clan, or that security@gmail.com could just be some bloke who works the doors at a local nightclub. But the reality is that users think names convey meaning even if you specifically tell them otherwise.
GitLab won't be the first to get bitten by this, and they won't be the last.
Isn't this the case with most hosting providers, such as Digital Ocean? You point your registrar's name servers at ns1, ns2,ns3.digitalocean.com then create requisite DNS records for a given droplet. Seems like the same lack of authentication.
The difference is that DigitalOcean gives each VM its own IP address. So even though your DNS may be pointed at DigitalOcean's servers, other DigitalOcean customers are not able to respond to incoming TLS connections for that IP.
I have a website hosted there, and manually renew a let's encrypt cert for custom domain pointing to it every few months. Unfortunately this involves removing and re-adding the custom domain.
I got a notification the cert was going to expire soon so I went through my usual renewal process, only to get redirected to the linked post from the new domain button immediately after deleting the custom domain. So the custom domain now 404s which is the last thing I would have wanted.
I'd strongly suggest a warning get attached to the delete domain custom domain button, or something similar, until this is fixed.
It involves using "certbot certonly -a manual" with a bunch of other flags on the command line (well in a script really), create the file it asks for and let it generate a new key.
Gitlab pages only lets you add cert/key details at the point you add the domain (afaik anyway), so you need to delete it and re-add it with the renewed key. It's tedious enough, but it's really only the last step that needs to be done manually.
At least when using the web interface, you need to delete the domain and then re-add it. I'm not sure if there is an alternate option via an API, never looked into it.
IMO, this is user error. GitLab adding validation on their side at app level is a nice to have feature for sure. Ultimately though, keep your DNS records up to date! If you stop using a service, stop pointing your records to it, simple. If you point any record to a service that matches based on CNAME, A or some other arbitrary value expect squatting/"hijacking" to occur if you delete your reservation of that name.
I am the security researcher that reported this issue to GitLab. There is more to the issue than is described in GitLab's security advisory and it was definitely a design flaw on GitLab's part. Hopefully, more details will be published soon.
There are so many services with domain ownership validation issues. It's honestly an extremely hairy problem.
Handling all the edge cases is extremely challenging to do so with good UX
When a customer validates ownership do they keep it forever? How long? Do you rescan for txt records? How often? Do you handle customers leaving gracefully? How do you handle new customers re-setting up their domain on a new account?
It's a big mess and it's a lot better to do what cloudfront does and give customers a unique name to CNAME to
As a customer of Gitlab who uses custom domains, I haven't got any email notification about this. Especially when the post says it requires user action. As much as I love gitlab, this negligence is unacceptable.
We apologize that you haven't received an email notification. We've sent email notifications to as many customers as possible, but obviously did not get everyone.
Rest assured that when domain verification rolls out, there will be an email notification you'll receive regarding a grace period to address your required user actions prior to re-verification.
If you have any further questions about this plan, feel free to contact us directly at security@gitlab.com
It seems that a notice would only need to be sent to users that used to have an active GitLab pages setup with a custom domain that does not any more. No point in alerting customers that are not affected yet in my mind. If you do have a currently active setup, not action is needed until a fix is issue, and then it would be necessary to alert people.
Unfortunately, I cannot disclose any further details until GitLab give me permission to do so. All that I can say is that GitLab has certain features for custom domains that GitHub does not have. I plan on publishing a technical write-up once everything has been resolved.
Is this related to the issues recently discovered with the TLS-01-SNI validation method for TLS certs?
Looking over how GitLab handles setting up custom domains[1], it's pretty clear they were affected by that. I thought it was pretty much decided that's more a problem with the Baseline Requirements than with individual service providers like GitLab though. Mozilla even went so far as to forbid CAs from using two of the Baseline Requirement validation methods as a result of that vulnerability[2]. Assuming the CAs comply this shouldn't be an issue anymore, right?
I examined a whole load of DNS Records (automatically) to check for this problem under DigitalOcean and Vultr a couple of months ago, but for "stray" name-servers rather than poorly assigned A records. There are more vulnerable domains than you might think.
You mean domains with NS records pointing to nameservers - but no existing records? Such that you can upload your own records, and essentially steal the domain?
I did something like that too, a while back. Particularly a problem for people who "retire" services without cleaning up, and who use DigitalOcean, CloudFlare, and similar services.
Yes, exactly. I found a few domains with "stray" nameservers... for instance, two nameservers for provider A, and another one (or two) for provider B -- where provider B was someone like DigitalOcean, with no associated records.
To recap that: Several popular CDNs or bulk virtual hosts let you type in any hostname and say you're going to host it with them. They figured that if the people who really own that name don't agree, they just wouldn't point their DNS records to the relevant servers and so it's fine. Let's Encrypt's tls-sni family of validations do a DNS lookup for the name to be validated, but then request SNI for a different invalid hostname as a signal. If a CDN or bulk host used for say, example.com allowed another customer on their service to say they wanted the invalid hostname as their "real" name, they could pass the tls-sni validation for example.com and get a certificate issued.
Naively we often assume that it's OK to let users pick names in a service without taking any step to validate that the person asking is entitled to the name. We may even kid ourselves that the custom name doesn't implicitly convey any legitimacy, that @MileyCyrus might be just some 14 year old from Swansea named Miley Cyrus, that mcdonalds.com could be a family web site of the McDonald clan, or that security@gmail.com could just be some bloke who works the doors at a local nightclub. But the reality is that users think names convey meaning even if you specifically tell them otherwise.
GitLab won't be the first to get bitten by this, and they won't be the last.