This screams for clarification and I'd love for someone more knowledgeable in the area to elaborate on it. Is this common practice for data-center providers? Do I now not only have to worry about my own infrastructure security but also worry that my IaaS provider hasn't installed some backdoor to my servers?
I cannot speak for the industry as a whole, but remote management systems like this are bound to be common; any large enough physical datacenter is going to need a more efficient way to access a misbehaving system than sending a tech physically running to the box to plug in a keyboard and mouse. It should be extremely uncommon to have these management interfaces open to the public though, and I'll bet that's what NordVPN is surprised by. Generally these systems should be private and isolated due to the power that an attacker can wield through them.
Cheap datacenters are favored by VPN providers for their unlimited bandwidth and lax abuse policies.
Many of them allow access to IPMI only over a VPN, but do not isolate each customer’s IPMI to a customer VLAN. I personally know at least three large budget datacenters which allow all customers access to each others’ “private” IPMI IP addresses.
The higher end ones are usually newer-ish, but there are lot of older "computer rooms" that offer acceptable-level benefits for a reasonable price. Lots of legacy customers and ISPs in these rooms, for the record. You get what you pay for but a lot of older DC spaces are just fine for most users; everyone thinks they need 99.999 but most don't.
There are also re-sellers and managed services companies who take out a footprint in data centers and then lease space in their cabs, sell bandwidth, IPs, etc. Using a series of resellers you can often get around restrictions as to what you're doing -- small fry MSPs don't ask a lot of questions -- but still get a data center footprint. These are sometimes one-man shows, and their quality and professionalism are often sub-par, which is how you end up with default iDRAC creds and the like.
Check out Data Center Maps or WebHostingTalk for some examples.
Source: data center ops manager for several different companies.
But, yes, remote management is pretty common in datacenters. The fact that NordVPN wasn't aware of them just shows incompetence.
This is the dumbest thing I've ever seen... unless your firewall is between your host versus every other host and there's no multi-tenancy, this will suck.
Never trust the network.
There were many IPMI/iDRAC/etc. exploits published in the past few years. Throw a dart at a list of them, and you'll probably find one that was unpatched in most systems as of March 2018.
In secure environments one pulls IPMI module from the server or only uses the modules that have their own dedicated NICs that have to be wired to their own management network.
There also quite a number of sysadmins that connect idrac's to the "regular" network, instead of the sysadmin VLAN ...
Now wait for the machine to get rebooted (or do it yourself using the BMC, e.g. 'racadm serveraction powercycle' for Dell/iDRAC machines).
Even SecureBoot won't help as you can just turn it off using the BMC.
See here for a bunch of examples for Dell machines using the BMC's HTTP API:
Power failure would require both of the power feeds in the DC failing simultaneously and would be easily verified by contacting the DC and asking if they had any power outages reported at the time. Of course there are cheapskates who don't go for redundant power supplies so it's possible but would be indicated in the IPMI logs
I've worked in production environments with thousands of machines and random reboots are a completely normal event for some workloads. Combination of hardware issues & kernel issues with hundreds of thousands of lines of code makes it inevitable. I would be surprised if NordVPN even noticed and their architecture wasn't designed to automatically start everything at boot.
You can't be perfect at scale - you just need to design your work loads to be redundant and fault tolerant.
""All servers we provide have the iLO or iDRAC remote access tool, and as a matter of fact this remote access tool has security problems from time to time, as almost all software in the world. We patched this tool as new firmware was released from HP or Dell.
"We have many clients, and some large VPN service providers among them, who take care of their security very strongly. They pay more attention to this than NordVPN, and ask us to put iLO or iDRAC remote-access tool inside private networks or shut down access to this tool until they need it. We bring [iLO or iDRAC] ports up when we get requests from clients, and shut them down when they are done using this tools. NordVPN seems it did not pay more attention to security by themselves, and somehow try to put this on our shoulders.""
Even a lot of VM/cloud systems have some kind of virtual management console (Linode has their LISH system that lets you SSH in to console and Vultr/Digital Ocean have similar web based consoles .. AWS surprisingly doesn't. You can get console output but can't send VMs any console input).
Not only should have NordVPN been aware of this hardware KVM, they should have secured it and had version checks on its firmware as an essential part of their security. I could see this oversight with other companies, but not with one whose primary business claims to be security.
Power on/off should be done via APIs that issue commands to a PDU, like Atlantic.net started doing in the early 200s.
And there's nearly zero reason to access "console" - configure your server to always but off PXE and fall through to disk if that intercept is not needed.
Why is this surprising? AWS seem to know what they're doing in general, and this is obviously the right policy in this particular area.
They talk about the AWS “Metadata Service” Attack Surface and how juicy a target it is. I was just providing support for that opinion.
This is like an author of a website vulnerable to CSRF (because it relies on IP for auth) blaming browsers for allowing cross site requests instead of require proper authentication. Except that Amazon is powerful enough to get away with pushing all the effort onto developers and admins.
I guess a machine-local service that takes ownership of the metadata service and implements additional restrictions (such as limiting access keys to privileged users) might be doable.
True data centers where you own the hardware shouldn't... they give you an ethernet cord and everything is on you.
Absolutely. We had similar situation with one of the DC vendors.
If you're lucky during network install, you'll never need out of bound access.
But rebooting is still a manual process.
However, if the attacker gains RCE many IPMI implementations theoretically allow for DMA, but this is a significantly more complex attack to mount in practice with no public PoCs available.
- Or, request a private network with no connectivity and a VPN device connected on that private network.
- Or, hardware disable LOM.
There's no reason you shouldn't request an ASA from your datacenter provider in this day and age.
Most (not all) servers have out-of-band management; of which IPMI is just one of many such solutions.
It's also worth noting that the hack could have been against in-band management if the Nord used an OS image provided by the DC hosts. However OOB feels more likely given their description (as vague as it was).
One presumes that because of NordVPN's business, they're colocating a server or two in many very many "POPs", presumably not all of them have tight controls on physical access. Its likely that there are none available in many areas where they seek to maintain a point of presence.
Anyone can do this, it’s not nearly as complicated to launch and secure as some would have us believe. (https://github.com/jenh/sevenminutevpn)
You do lose anonymity with personal VPN, but it all depends on your use case.
e.g. run this from the command line:
sshuttle -r example.com 0/0 -x example.com --dns
Whether it grants you any significant anonymity is debatable, but it works well for evading content filters and tunneling your traffic onto a more trustworthy network.
Shadowsocks is more resistant to censorship from adverse actors (such as the Great Firewall) than OpenVPN.
Outline's user experience is the best I've seen among self-hosted VPN solutions, as it includes apps for both the server and the client. The server app is suitable for use in organizations, and can manage VPN profiles for multiple individuals.
Plus I really enjoyed learning about the in’s and outs of setting it up. I poke around in the VM just for giggles.
I use Algo for the exact reason you mention ("this lan is sketchy") and have been pleased, but I always assumed even if my traffic was mingling, one (possibly secret) court order would out me since I paid with a CC tied to my real name.
And for those who have used various VPN solutions over the years but not Wireguard: it really is pretty magic. It Just Works, with fantastic performance.
Hopefully you don't have similar news to share about Mullvad...
Besides, the argumentation from that vpnscam website and its followers reminds you of the typical conspiracy retards that follow Trump.
Then, on the other hand, you have Mozilla and the EU (which has access to all European corporate records) vouching for Proton (since they partially fund Proton). We also operate in a highly transparent way, so all information debunking this is actually in public record, details here: https://protonvpn.com/blog/is-protonvpn-trustworthy/
Proton definitely has an office and subsidiary in Vilnius, it's not a secret because it's on Instagram: https://www.instagram.com/p/BxMz62oHb6K/ The office is inside a 30 storey building, so it is not surprising the address is shared with quite a few other companies. But that doesn't mean Proton on a whole is based in Vilnius.
The people spreading the false information are also falsely implying that Proton's subsidiary controls the Swiss parent company, which is never the case as it's always the other way around (parent controls the subsidiary). And its super easy to disprove because unlike most companies in the VPN space, the directors of Proton's Swiss parent company are in public record, and are all well known people who have been in the public eye for years (e.g. at TED: https://www.ted.com/talks/andy_yen_think_your_email_s_privat...)
Quoting from the blog post:
"We therefore set out to conduct a thorough evaluation of a long list of market-leading VPN services. Our team looked closely at a wide variety of factors, ranging from the design and implementation of each VPN service and its accompanying software, to the security of the vendor’s own network and internal systems. We examined each vendors’ privacy and data retention policies to ensure they logged as little user data as possible. And we considered numerous other factors, including local privacy laws, company track record, transparency, and quality of support."
It was quite intensive, with on site visits to our office in Geneva and discussions with Mozilla technical leadership.
Someone found certificates for those three VPN providers and posted them to 8chan with a message like "I don't recommend these VPN providers lol"
The good news is that they're only certificates, and they have now expired, but theoretically they could have been used for the past year without anyone noticing.
> We […] started creating a process to move all of our servers to RAM, which is to be completed next year.
What does "RAM" mean here?
> The key wasn't set to expire until October 2018, some seven months after the March 2018 breach
And here's a dump of their logs: https://share.dmca.gripe/hZYMaB8oF96FvArZ.txt
Out of my 110/12mbit connection Mullvad let's me use 108/11 of that, and even has wireguard support.
Something I find pretty neat about them from a technical standpoint is their account creation, user authentication, and payment processes. Sign-up literally takes less than a second, so even if you don't plan on using their service, I recommend you try creating an account.
This bug loudly announces itself on every pageload, it speaks of tremendous incompetence that they ever let this go into production.
The site used to set a cookie that looked like this:
Set-Cookie: SWIFT_client=a%3A1%3A%7Bs%3A15%3A%22templategroupid%22%3Bs%3A1%3A%221%22%3B%7D; expires=Wed, 28-Dec-2016 23:24:13 GMT; path=/; httponly
The SWIFT_client cookie gets passed directly into unserialize(), TLS has literally nothing to do with this.
FWIW rasengan is one of the PIA founders, he should know much better.
This response is so utterly silly I must wonder if this is all just an incredible display of incompetence instead of malice.
So, I spoke with our internal team and was able to find more details:
- We haven't used that machine since that exploit was made public.
- We were never exploited.
- There was no sign of intrusion of any kind.
- The specific machine was a backup helpdesk test server without any real user data.
Thanks again for bringing this up!
So what? You were exploited before kayako patched this bug, it was glaringly obvious to anyone who ever looked at the cookies set by your site.
>- We were never exploited.
This simply isn't true, either you're misinformed or lying.
>- The specific machine was a backup helpdesk test server without any real user data.
The specific machine (Which you took down really fast after I pointed it out! :P) I linked probably did not even exist in 2015, I was talking about your prod env.
I don't have a horse in this race, there's no incentive for me to lie about this. I know what you are saying isn't true.
There's a historical, almost accidental connection dating back to the infamous November 2015 DDoS against Proton, but zero connection today, and certainly not in the way it has been portrayed by people seeking to attack Proton. Android certs are permanent and can never be changed so that is why there is still one mistakenly issued Android cert out there today.
That fact that this had to be slowly pried out with changing explanations along the way?
When you say the claim that has been debunked - I expect the claim not to be confirmed.
Proton definitely has an office and subsidiary in Vilnius, it's not a secret because it's on our Instagram: https://www.instagram.com/p/BxMz62oHb6K/ The office is inside a 30 storey building, so it is not surprising the address is shared with quite a few other companies. That doesn't mean Proton as a whole is headquartered there, or that the subsidiary somehow controls the parent company in Switzerland, or that there is somehow data mining going on.
Those are the claims that have been clearly debunked. The fact that Proton has a subsidiary in Vilnius, or the fact that we outsourced our HR back in 2016, are not secrets, and is on our Instagram and the Reddit thread linked above. This is the truth, and this is not some wild EU-funded data mining conspiracy as some would have you believe.
If you would like, I can pursue this issue further given what seems to be confirmation here of a certificate violation.
I've got enough HN internet points that a few downvotes will be fine. Thanks!
I've heard this several times but nobody has ever been able to provide a source.
but was conveniently never denied by PIA that I could ever find. If it was a lie, it would be easy for them (PIA) to prove under libel laws in the discovery phase of a trial. I'd argue if it was a lie from ProtonVPN, it would have been in PIA's best interests to clear their name. After all, PIA and ProtonVPN are a few of the only providers who've proven in courts they don't have logs of users. We know they're legit because they said so in court under penalty of perjury. Also, the European Commission has investigated these exact claims, and would have privileged access to a lot of the business documents, and found the claims without merit.
Me? Just a happy protonvpn user who finds the oft repeated shilling for PIA dull. If you really want to hate protonvpn, use PIA, or use someone else. Better, don't trust any of them! Setup algo on a digital ocean droplet of your own: https://github.com/trailofbits/algo
However, this is meant for running over an untrusted network, not for maintaining internet anonymity. Use Tor for that.
Now that doesn't mean ProtonVPN is automatically compromised but I feel with stuff like no-log VPNs one should always err on the side of caution.
ProtonVPN is 100% owned by the company behind ProtonMail, which in turn is funded by the European Union, so this has been verified by the European Commission. Details here: https://bit.ly/35RDKzB
There is also an abundance of public record which demonstrates this is false. The bad faith of those spreading this information is also apparent from the hundreds of fake Twitter accounts used to spread the rumors.
If you are acting in good faith, then we ask that you also take a moment to verify your facts and discover the truth, much of which can be found here: https://protonvpn.com/blog/is-protonvpn-trustworthy/
There's a historical, almost accidental connection dating back to the infamous November 2015 DDoS against Proton, but zero connection today, and certainly not in the way it has been portrayed by people seeking to attack Proton.
Regardless of that, there is so much mud being slung I recommend anyone to just search for 'protonvpn nordvpn tesonet', read a few articles on the topic and form your own opinion. Like I said, you can decide if you want to err on the side of caution or if it's a risk you're willing to take.
In case anyone wants VPN recommendations, I have good experiences with TorGuard and Private Internet Access and can also recommend Mullvad. Other people (that I trust) say iVPN and Tunnelbear are also solid.
On one hand, there's anonymous websites, competing VPN companies, and hundreds of Twitter bots pushing a story that is demonstratively false (just check public records).
Then, on the other hand, you have Mozilla and the EU (which has access to all European corporate records) vouching for Proton, which also operates in a highly transparent way, examples here: https://protonvpn.com/blog/is-protonvpn-trustworthy/
Proton definitely has an office and subsidiary in Vilnius, it's not a secret because it's on Instagram: https://www.instagram.com/p/BxMz62oHb6K/ The office is inside a 30 storey building, so it is not surprising the address is shared with quite a few other companies. And that doesn't mean Proton on a whole is based in Vilnius.
I agree, the VPN industry is rife with shady business practices. But the story being pushed isn't 'demonstratively false'.
* TesoNet offers data mining services
* You did contract TesoNet employees
* Due to an error and unyielding policies by Google TesoNet holds your Android app signing keys in name
* There is a lot of intermingling between TesoNet and NordVPN and to a lesser extent TesoNet and ProtonVPN.
Like I already stated, it's very unlikely you are compromised. But unlike, say, a billing company that handles my energy or water provider (where I care much less if they have tenuous links to data mining) my standard is extremely high for a VPN. Internet traffic is supremely personal and for me to trust a company handling that there cannot even be the slightest sheen of misconduct.
For me to trust you you would have to completely cut out your Lithuanian subsidiary and any employees, board members, etc. that were or are related to TesoNet, as well as any reliance on their infrastructure. Obviously businesses don't operate with such 'scorched earth' policies and I don't expect you to gut your company based on a HN comment, but it is what it would take for me and many other privacy-conscious individuals to regain our trust.
Proton does not today, and has never, used contracted (outsourced) employees. As is common with startups, in the past we did not always do all our HR in house (it's all in house today), but employees were always working on Proton and for Proton.
There are no board members, directors, shareholders, or employees, related to Tesonet beyond the fact that a couple employees might have been employed there previously. This in itself is not strange, we also have some employees who previously worked at Google, the ultimate data mining company, but clearly decided they preferred to work for the other side. People can and do change jobs.
Proton has also always run our own infrastructure, and for ProtonVPN, this is publicly verifiable.
So, we don't have to "gut our company" to remove any "intermingling" because there was little to none to begin with, and certainly nothing today.
Indeed trust is super important, but it seems odd to trust anonymous internet accusers or those with a clearly vested interest in harming Proton, as opposed to reputable third parties like the EU or Mozilla who don't have a vested interest here and are independent.
Proton is still to this day, the only VPN company that has an address clearly published on our website, where you can show up, and find company management and board members, and that means something.
I realized another way that would work for you guys (but is out of your hands) is fighting a court case about this. You'd be legally compelled to tell the truth and very screwed if you deny but then it comes out there is logging or mining going on. It's not ironclad but it is how most VPNs end up being considered 'solid'.
Second, there has already been a court case. We were ordered by a Swiss court to hand over logs, and we stated truthfully (under penalty of perjury) that we did not have the logs requested. This case was previously disclosed here: https://protonvpn.com/blog/transparency-report/
I'm not terribly well-versed in the international (or Swiss) legal system but are portions of that request public record, or would it be possible to put portions of it online, verbatim?
It would really strengthen the case to your customers because whilst claiming you had a request when you didn't isn't illegal, falsifying court documents definitely is.
Trust serious organizations such as Mozilla and the EFF.
Mozilla trusts ProtonVPN enough to officially partner with them. That means a lot more than some random anonymous reviews.
Wait... that doesn’t sound ideal either.
Yes, unfortunately it's currently not possible to use Warp VPN on PC. Otherwise, quite good service.
https://airvpn.org/ is also worth mentioning.
Haven't tested it, just rembered datacenterlight from a HN thread about buying a mainframe.
The reality is, if someone else owns the infrastructure you're just pushing the risk to a different location.
And frankly, his alternatives are just absurd. Tor? Really? Has he ever tried to use Tor for usual daily browsing? Does he expect people to try to use Facebook, Instagram, Youtube over Tor? Really?
Nobody should be using a VPN provider, full-stop. It is structurally impossible for anyone to verify their claims, they have more incentive to lie than your ISP does, and they're cheap and easy to set up, so the industry is a cesspool.
You should assume that all of them are behaving badly.
Also, hasn't CloudFlare been audited to verify their no log claims? I know while Mullvad hasn't been audited to verify their no log claims, they have at least been audited to verify the security of their app.
"1. We don't write user-identifiable log data to disk;
2. We will never sell your browsing data or use it in any way to target you with advertising data;
3. Don’t need to provide any personal information — not your name, phone number, or email address — in order to use the 220.127.116.11 App with Warp; and
4. We will regularly hire outside auditors to ensure we're living up to these promises."
A lot of ISPs openly collect user data, so I don't know how much that factor matters. And while I can go get a VPN, I can't just get another ISP.
So instead of allowing their customers to do their own damage limitation, they left their customers in the dark and continued to expose them to a breach they weren't sure they had fully contained.
I wonder when that sort of thing will become a criminal offence.
Official response hides fact OpenVPN CA keys also leaked, so attacker could impersonate any other NordVPN server: https://gist.githubusercontent.com/Snawoot/85f77356e229d77aa...
RADIUS secret key also leaked, so propably it is possible to break into EAP session which infers session secret key for StrongSwan.
Could you elaborate on this? I am familiar with PKI so the first part makes sense, but I am not familiar with the intricacies of VPNs so I am not sure what this means.
Some EAP authentication methods (namely, EAP-MSCHAPv2 and EAP-TLS) export Master Session Key, which is exported to StrongSwan via MS-MPPE-Send-Key/MS-MPPE-Recv-Key EAP attributes.  So, MSK derived on RADIUS side and sent to StrongSwan. Eavesdropper with knowledge of RADIUS secret key capable to intercept and decrypt such EAP payload.
 - https://wiki.strongswan.org/projects/strongswan/wiki/EAPRadi...
 - https://tools.ietf.org/html/rfc5216#section-2.3
Assuming their IPsec was enabled with it (and OpenVPN should be enabled by default), them leaking their keys does not matter. The sessions can not be decrypted even if the master key is leaked.
TLS also has perfect forward secrecy by default.
Impersonation is an issue, but the article stated the CA keys have already been rotated and are out of date.
EDIT: I meant to reply to the post below me, but this is fine. Sorry about that!
They were not rotated back then in 2018, so we can only guess if MITM had place. Their line of defence appealing to keys which are NOW outdated is just ridiculous.
> Read up on "Perfect forward secrecy": https://en.wikipedia.org/wiki/Forward_secrecy
> The sessions can not be decrypted even if the master key is leaked.
It's not true. PFS provides cryptographic isolation between long-term keys and session key used to encrypt data. Obviously, if MSK compromised it is irrevelant, how it was inferred: with PFS or not.
MITM could have taken place anyway because the attacker was on the machines. They did not need the key.
> It's not true. PFS provides cryptographic isolation between long-term keys and session key used to encrypt data. Obviously, if MSK compromised it is irrevelant, how it was inferred: with PFS or not.
PFS implementations have the session key rotated automatically in software and dependent on the implementation multiple session keys are in use at any given time dependent on flow. The PFS session key would also be different for every VPN server in the NordVPN environment. The only possible way to compromise a session on another VPN node (that was not compromised itself) would have been to intercept it at the time of the session being created and MITM by injecting your own PFS session key.
That is why it is called "Forward secrecy": the session can not be decrypted in the future, only in the present.
Unless your assumption is that this is a state actor with the ability to MITM connections in the first place, or a rogue ISP BGP hijacking that would have been obviously seen on something like BGPStream [https://bgpstream.com/], it is safe to say that no other VPN node's traffic was compromised. Only traffic on this single host in this single location.
More reading: https://www.speaknetworks.com/what-is-ipsec-vpn-pfs-perfect-... (Step 5)
"Instead of making use of the DH Keys Calculated during Phase-1, PFS forces DH-Key calculation during Phase-2 Setup as well as Phase-2 periodic Rekey. The PFS ensures that the same key will not be generated and used again."
OpenVPN (using TLS) also uses PFS by default. There is a reason it is called "Perfect."
EDIT: PFS is commonly also implemented by Diffie-Hellmen (DHE) key exchange: https://en.wikipedia.org/wiki/Diffie–Hellman_key_exchange
EDIT 2: I am not defending them as well. I just believe extrapolating the technical details is fear mongering at best. Believe it is best to focus on the facts as it makes for a stronger argument.
There are two distinct severe security issues. First one is leaked CA key, which allows to certify any key as a valid key for NordVPN server certificate key. I think it is not necessary to argue about this: traffic decryption to any Nord OpenVPN server became simple as network MITM. Not likely a state-level BGP hijack, but local attack targeting channel of small group of users. Anyway, we do not have cryptographical guarantees since this point.
Second issue is leaked RADIUS key. Why it is a problem for encryption? Because EAP authentication and key derivation runs between VPN client and RADIUS, and VPN server receives derived keys as attributes of EAP message: https://wiki.strongswan.org/projects/strongswan/wiki/EAPRadi...
> The eap-radius plugin does not implement an EAP method directly, but it redirects the EAP conversation with a client to a RADIUS backend server. On the gateway, the EAP packets get extracted from the IKE messages and encapsulated into the RADIUS protocol, and vice versa. The gateway itself does not need special support for a specific EAP method, as it handles the EAP conversation between the client and the RADIUS backend more or less transparently.
> For EAP methods providing an MSK, the RADIUS server must include the key within the MPPE-Send/Receive Keys; Unfortunately, FreeRADIUS before 2.1.10 did not include these attributes when used with EAP-MSCHAPv2.
So, despite session encryption key is not bound directly to long-term secrets which server possesses, they can be extracted from communication between StrongSwan and RADIUS server.
PFS has nothing to do with all of it. In first case it is possible to issue VALID certificate for eavesdrop VPN server and redirect users traffic to it. In second case MSK probably can be extracted communication between VPN server and RADIUS server (I can't say if it will require MITM of RADIUS session or it is possible to decrypt EAP payload with passive sniffing and posession of RADIUS secret key).
1- Nord falsely blames its server provider.
2- Nord hides it from their users.
3- Nord claims all will be well with an “audit” (again, since they were already “audited”)
This is either criminal negligence, “security theater”, or both.
I don't see anything in the article about those claims being false. Where did you get that?
I believe that would be the section they're referencing.
They are casting blame on the provider. Providing remote access tools is not a fault. Failure by NordVPN to disable said access is the issue, yet they passed the blame on.
It's a bad look in any case.
That seems to be ExpressVPN, the main competitor of NordVPN.
If they have EU customers then article 33 of GDPR should see to that.
"In the case of a personal data breach, the controller shall without undue delay and, where feasible, not later than 72 hours after having become aware of it, notify the personal data breach to the supervisory authority competent in accordance with Article 55, unless the personal data breach is unlikely to result in a risk to the rights and freedoms of natural persons. Where the notification to the supervisory authority is not made within 72 hours, it shall be accompanied by reasons for the delay."
Unless the authorities in EU accept the explanation they are in trouble, but I think you shall report even if you think there has been a breach.
Does this always/necessarily lead to customers/the public being informed? But yeah, better than nothing.