
Gandi loses data, customers told to use their own backups - webrobots
https://status.gandi.net/timeline/events/2109
======
moralestapia
Whoops, so long with the "no bullshit" policy.

I stopped using them a while ago but for a different reason. I used to use
their website to check availability/whois for domains that I was interested in
buying. If it was available I didn't buy it at the time but until I finished
the website/app whatever I was going to put there, this took me a few months
obviously. It happened to me that when I was finally ready the domain had
already been sold to someone else. This repeated five times during six or so
years. Now, I know, "someone else could have thought the same thing" but I
find it very hard to believe that it happens so often. These domains were a
bit of niche words that were not hot topics at the time, some of them using
fairly uncommon TLDs (like .one). Another weird thing is that they were always
registered to someone living/or doing business at India, and it was a fairly
simple landing page with a "contact me" link. I'm a bit superstitious so I
don't think it was a coincidence.

Now, I don't think this is a GANDI problem per se, but my theory is that they
share this information (who is looking for which domains) with marketers or
something like that, or maybe it was a rogue employee trying to make some
money squatting domains. I would have expected this from BigDaddy or similar
sharks, but from a company whose motto is "no bullshit" I had much better
hope. Anyway, I decided to move (to namecheap if you're wondering) and
surprisingly the problem went away.

~~~
Operyl
They never had a real "no bullshit" policy. When I had my domains with them, I
had been asked to verify my identity 34 times in 12 months. 34 separate
fucking times. Because "ICANN says so" or some stupid shit (their words, not
mine). It stopped the moment I moved to Google Domains, where they asked once
and never again.

EDIT: And, to make things worse, each time I was threatened with the
"confiscation" of my domain, and the round trip on the tickets was so high
that each instance took 2-3 days to resolve. Frustrating as hell.

~~~
capableweb
Since you're giving a anecdote, let me do the same. I have about 30-40 domains
with Gandi, and have been using it for about ten years. I don't remember ever
verifying my identity, but guess I most have done it at least once. I have not
been asked to verify anything for at least the last five years of using it.

Disclaimer: I don't work there or have any relationship, except I'm a happy
customer

~~~
Operyl
It was a matter of them refusing to keep my identity on file, and the
threatening tone of each ticket. It grew tiresome quickly.

~~~
capableweb
Sounds more like a bug than anything. Why would they want to not make it
easier for you if they can?

Seems you missed my point though. Both of our anecdotes doesn't really say
anything, in terms of if Gandi is good or bad.

~~~
smnrchrds
His anecdote does say something though. It suggests that Gandi has a _" if we
have a bug, it's your problem not ours, sucks to be you"_ policy, which is
exactly what has happened with this data loss issue as well.

Actions speak louder than words. Google famously has a _" we don't have bugs,
you just don't know how to use it, talk to the hand"_ policy for example. It
is better to learn about the policies due to minor issues rather than major.
OP learned of it early on and moved away with little trouble. Others did not
learn until now and stayed, and now they are SOL.

~~~
Operyl
I’m a gal, but exactly. I really wanted to support a company at the time who
was supporting the community (they were a freenode sponsor), but I just hated
dealing with the stress and potential that my domains could just disappear
over night.

------
EnderMB
Oof, this Twitter thread looks particularly bad, especially the response from
the official Gandi account.

[https://twitter.com/andreaganduglia/status/12151991477012316...](https://twitter.com/andreaganduglia/status/1215199147701231616)

While I appreciate that there are real people behind these companies that are
probably having a really rough time right now, the criticism that Gandi are
getting as a company is justified - and if Gandi are truly a "no bullshit"
company they need to put something out to their customers asap.

~~~
danr4
Screenshotted in case (when) they delete it
[https://i.imgur.com/s3R1VVc.png](https://i.imgur.com/s3R1VVc.png)

Using memes after permanently losing customer data is extremely disrespectful.

~~~
petee
"Julie Pelloille @juliepelloille Replying to @gandi_net @andreaganduglia and 4
others

This post was disrespectful. It's not an excuse, but this is a stressful
situation and the thread was getting heated. Either way, I truly regret
posting it and it was my decision alone to do so. Please don't take this as
representative of the high standard Gandi sets"

"That said, for the sake of transparency, we won't be deleting the tweet --
Julie"

~~~
K0SM0S
I like that. Honest mistake. Simple, truthful apology. Transparency for the
record. Julie's one of the good guys.

Whatever the context / stakes (doesn't change anything in this case), this is
how people should behave in life (not just online).

------
jiggawatts
Just to play devil's advocate: This is in no way different to how Azure, AWS,
and GCP operate. They don't have backups either. They too rely on n-way
replication, a bit like a distributed RAID.

All cloud providers make it absolutely clear, in black & white, that
protection of your data is _your responsibility_ , not theirs.

What I find hilarious is that most cloud providers only provide built-in
backup functionality for a tiny subset of their services.

Ask Microsoft if you they have a "backup" button for Azure DNS Zones. Or Azure
load balancers. Or _anything else_ that isn't a VM disk, App Service, SQL
Database, or a Secrets Vault.

I mean, look at this insanity: [https://docs.microsoft.com/en-
us/azure/backup/backup-azure-f...](https://docs.microsoft.com/en-
us/azure/backup/backup-azure-files#limitations-for-azure-file-share-backup-
during-preview)

 _" Backup for Azure file shares is in Preview."_

After 10 years of operation, this trillion-dollar company has only a use-at-
your-own-risk beta for data protection!

Don't be too hasty to point fingers at Ghandi and laugh about how they're
unprofessional. Whatever you're using is essentially the same.

Ask yourself this: Could _your_ organisation recover if some malicious admin
simply deleted all Azure Resource Manager resources in one go using
PowerShell?

~~~
yjftsjthsd-h
> Ask yourself this: Could your organisation recover if some malicious admin
> simply deleted all Azure Resource Manager resources in one go using
> PowerShell?

We have streaming replicas for hot data AND regular snapshots shipped to
offsite cold storage, _because RAID is not a backup_. If we experienced an
equivalent event, we'd be fine.

~~~
jiggawatts
The equivalent scenario to recovering from a bulk erasure of all Azure RM
resources is this:

How long will it take you to recover if someone deleted your switch configs,
reset the SAN to factory defaults, wiped you firewall rules, deleted you
Active Directory accounts (or equivalent), and then ran a secure erase on
every every physical server just to raze everything to the ground and salt the
earth?

I mean in wall-clock time, how long would it take your team to _even figure
out_ what is going on? Where would you start?

Would you recover the switch first, or the server that you use to authenticate
to it using RADIUS or LDAP?

How will you securely connect to servers if your CRL and OCSP servers are
down?

How will you get access to your passwords if your file server where the key
blob is stored is saying "Insert boot disk"?

People think that disaster recovery is for "I deleted a folder".

Disaster recovery is for _disasters_.

Removing all Azure resources wipes _everything._ Your vNets... Poof! Your
public IPs... Poof! Your internet-facing DNS zone... Poof! Your authentication
credentials... Poof! Gone, gone, gone.

How do you plan to restore _dynamic IP addresses_ to their original values?

How do you plan to restore DNS Zones that get assigned to 1 of 10 randomly
selected server pools and hence have a 90% chance of requiring a change to the
NS server glue records on restore?

Do you even know which order things would have to be restored in to prevent
failures during a restore?

Could you _possibly_ work out what is missing if you log on to your cloud
portal and see the "Welcome to Azure, to get started click here" splash page?

Get it?

~~~
smnrchrds
> The equivalent scenario to recovering from a bulk erasure of all Azure RM
> resources is this

It just occurred to me how much easier it is to wipe everything in the cloud
age than the on-prem age. Doing all the things you said for on-prem takes some
serious effort. Some, like factory resets, may be impossible without
individual physical access. You would probably be discovered and stopped
before you can inflict much damage. In the cloud age however, it takes orders
of magnitude less time and effort to inflict the same damage.

It is kinda like how much easier it is to steal data now. Before the digital
age, stealing as much data as Equifax hack would have required moving
truckloads of paper without being discovered. It was simply impossible to pull
it off in reality. In the digital age, however, we have accepted massive data
leaks as not only possible, but unavoidable.

~~~
dragonwriter
> It just occurred to me how much easier it is to wipe everything in the cloud
> age than the on-prem age.

It's easier for physical facility damage to a single facility (whether hostile
action or natural disaster) to wipe everything out in an on-prem setup than in
the cloud, where multi-DC redundancy is a click away. But, sure, it's easier
to wipe out data without physically destroying equipment in the cloud.

------
webrobots
Dear customer,

This mail is a follow-up to the previous email we sent (on January 8th, 2020)
on this topic. As a reminder, yesterday, we experienced an incident on a
storage unit at our LU-BI1 datacenter, located in Luxembourg.

Despite the replication systems in place, and the combined efforts of our
technical teams throughout the night, we were unable to reover the data that
was lost on the impacted storage unit.

We sincerely apologize for the inconvenience that this situation has caused.
This type of incident is extremely rare in the web hosting industry.

In the event that you have a backup of your data, we suggest that you to use
it to recreate your server at a different datacenter.

To help you in this, we have provided you with a promo code that will give you
one free month for an instance, so that you can create a new Simple Hosting
instance in a different datacenter:

    
    
        XXX

~~~
abtinf
> This type of incident is extremely rare in the web hosting industry.

Why would they include that sentence? Are they trying to imply it is rare for
them because it is rare for the industry? Are they saying they are not as good
as the industry, so customers should move to other providers? Or are they
trying to show they apply the same inattention to their customer communication
as they apply to their data backup/recovery practices?

This kind of data loss should simply never happen. It’s one thing to say “it
will take us up to 30 days to restore your data because our fast recovery
options aren’t working and we have to bring up cold archives”, it’s entirely
another to say “your data is gone, tough”.

~~~
jermops
If you're not paying for backups... what archive?

~~~
icebraining
They say you can backup by using their snapshoting tool, but they lost those
snapshots too.

~~~
yjftsjthsd-h
The bright side is that now if anyone asks me why we would ever need the 3-2-1
backup protocol, I have a beautifully worked example.

------
matthewaveryusa
Looks like their backups only consisted of in-region backups on systems that
were homogeneous. Common pitfall. While technically a 3-node distributed
system may provide disaster recovery from one node failing, in practice, an
accidental rm -rf from an ansible script targeting all three machines, or a
bug in the software that's doing the replication, will leave you without a
backup plan.

If you're in such a situation, The easiest is to do filesystem level backups
with something like zfs and ship the backups to a third-party system that only
has write/append-only semantics (better yet, use a write-once-read-many (WORM)
disk to really guarantee it.).While there will still be _some_ data loss,
it'll let you recover since the last snapshot.

If you don't have zfs, a database backup that runs the db dump script and
scp/sftps it to a server running as a cronjob can also be an immediate remedy
while you get your shit together (and by that I mean buy yourself a product
with an immaculate reputation like aurora or cockroachdb to manage the db for
you)

Harder but better would be to tee the log of the changestream (all distributed
systems have such a log) to a third-party system. This is ideal because if
it's done synchronously it'll let you recover since the last committed
transaction.

And of course, test your backups, because backups are subject to code rot as
well.

~~~
namibj
What backup strategy are you implying for the case of cockroachdb? Streaming
the changefeed (including timestamps) to an external append-only system while
slowly and incrementally iterating through all tables using _as of system
time_ to reduce impact on active transactions and know how late this shard of
a "full backup" can be inserted into the "agumented" changefeed you'd generate
by interleaving these shards into the changefeed. For replay you'd use the
stream from the oldest shard up to the _select min(a) from (select
max(timestamp_resolved) as a from changefeeds group by table)_ newest
timestamp you know you have the transactions complete changesets for (the
resolved timestamp can be periodically emitted to confirm that no further
records in the same feed(/table) could have a transaction timestamp earlier
than it, inducing a partial ordering).

You could replay the (combined,sorted,agumented) changefeed in-order, or shard
it on the table's primary key to ensure per-key monotonicity when applying the
streams in parallel threads/transactions/nodes.

------
djmobley
Gandi have something of a cult following, but in my only experience with them
they literally lost my domain name during an inbound transfer.

Their response was awful and rude and completely unprofessional. I never got
my domain back.

Based on that experience, this incident doesn’t surprise me at all.

~~~
maximente
what can you recommend as an alternative?

~~~
djmobley
For domain registrations I use a mix of Namecheap, Cloudflare, GoDaddy and
Name.com, and haven’t had issues with any of them.

Gandi is the only domain registrar I’ve had an issue with.

~~~
NeedMoreTea
I've been burned by both Namecheap and GoDaddy, along with losing a few
domains in the infamous registerfly scam in the early or mid 00s. Namecheap
may have been simple cock up, rather than systemic pattern of intentionally
fucking over every customer. Avoid GoDaddy at all costs.

I consider GoDaddy to be one of the worst companies in existence, as bad as
anyone else you can think of, as free of corruption as current ICANN and as
fraudulent as registerfly. Clients _looking_ at available domains have found
them immediately registered and squatted at {hundreds}% markup. Their
incompetence lost me a few domains, and several freelance clients reported
similar -- all of whom were paying vastly over the odds for what they were
getting. GoDaddy make Gandi look an exemplar of ideal behaviour for behaving
as people are reporting in this HN post.

Their previous CEO had domain squatting and a complete lack of personal ethics
as sidelines. That's quite apart from their horrific upsells making a simply
renewal a 22 page nightmare of deeply dark patterned "no" clicks against
atrocious value "offers".

------
aosaigh
I don't have hosting with Gandi, but I do use them for domains and DNS. I'll
be considering migrating my domains from them after this.

Their response to this is exceptionally poor. To say essentially "this could
happen to any other web host" it nonsense. I've never had this happen with any
of the providers I've used for hosting and I'd be very angry if I had just
lost an entire VPS. The fact that they've lost all snapshots as well (which
are advertised as backups of the underlying volume) is unforgiveable.

~~~
eg312
I use Gandi for domains & DNS too. I've never had any problems so far but I
don't want any surprises... Where do you want to migrate? What is a better
alternative?

~~~
actuator
I like Cloudflare and find them to be a very good value proposition. They have
a domain registrar now as well, though I haven't tried that yet.
[https://www.cloudflare.com/products/registrar/](https://www.cloudflare.com/products/registrar/)

~~~
caymanjim
Cloudflare DNS is free, and they support DNSSEC (unlike Digital Ocean). The
web UI is good, and there's an API, and Terraform provider.

~~~
tptacek
Most providers (notably: AWS) don't support DNSSEC, because DNSSEC doesn't
matter.

------
M2Ys4U
TIL that gandi was bought by a private equity firm around a year ago.[0] This
may explain some things...

[0] [https://news.gandi.net/en/2019/02/futureofgandi-the-
adventur...](https://news.gandi.net/en/2019/02/futureofgandi-the-adventure-
continues/)

~~~
nachtigall
Where does it say it was bought? It talks about a new investor:

> we have found a new investor in Montefiore Investment, who have replaced our
> former shareholder!

Am I missing something?

------
SeanMacConMara
Interesting reaction. Is the highly negative reaction correlated with US
culture maybe ?

I've used them for many years and had several complex support interactions
with them.

Their customer service policy is very "API-like" in that you get exactly the
t&c you paid for and nothing more. Hand-holding and soothing noises are not
included in the t&c. They fuck up you get a refund, you fuck up they'll tell
you exactly that. Outside that they're very casual relaxed humans to
communicate with.

I find that far more trustworthy (in the mathematical sense) than a "slick"
twitter feed.

Politness does not imply trustworthiness.

------
anonred
Gandi is the absolute worst.

The last time I tried buying a domain through them, they took my money and
then demanded "identification" via government ID (citing some bullshit in
their ToS). I refused, so they closed my account and took the domain with
them.

Based on that, I'm not surprised at all by their CEO's response to this
incident[0]:

>If we led you to believe that you had nothing to do on your side when warned
multiple times to make your back ups, then we'll have to make it clearer, and
stop assuming that it's an industry wide knowledge.

[0]:
[https://twitter.com/StephanGandi/status/1215287619938062342?...](https://twitter.com/StephanGandi/status/1215287619938062342?s=20)

~~~
TheChaplain
I had exactly this problem too but with NameCheap. Told them to put their id
request and my money somewhere and left for Gandi.

After more than 8 years with Gandi, not had a single issue with them.

------
anarcat
I understand people might be upset because they lost data, but as a sysadmin,
my reaction is "ooh shit, poor guys, that must be a horrible week"...

And honestly, if you don't keep data of stuff you host on a server provider
like this, you kind of get what you deserve...

~~~
aosaigh
No you don't. While agree everyone should have their own backups, you should
expect your hosting company to properly replicate and backup their
datacenters.

~~~
anarcat
I don't, actually, expect them to do so. But even if I would, and Gandi, here,
_were_ doing backups and replications, no one is immune from errors and
catastrophes.

Pretending that the cloud is permanent in infallible is extremely dangerous. I
would seriously question the competence of any sysadmin relying on this as a
base principle.

Sure, they screwed up, but this stuff happens. We should actually be happy it
happens "only" on a "small-ish" provider like Gandi and not an entire AZ at
Amazon.

Can't wait for _that_ shoe to drop, I'll bring the popcorn, if there's
anything left of civilization then...

~~~
alpaca128
> Gandi, here, were doing backups and replications

As far as I understand correctly they only made snapshots on the same machine,
which is why there's trouble to begin with.

Considering they're currently "reminding" customers that backups are an
industry standard right after losing data due to missing backups I wouldn't
just shrug it off.

~~~
notyourday
That's probably because they bought into the sales pitches of the likes of
EMC. It's a nice pitch and in most of the cases it works exactly like EMC
promises. Snapshots work great, data is always recovered, etc, etc, etc.

The fun, of course, starts that one time when it does not work and you realize
that no one looked at the corner case that bit you.

------
jchw
Gandi is absolutely not the company I expected this to come from.

With that having been said, everyone please stop assuming your data is safe.
It’s never safe, but it’s extremely not safe single homed somewhere. Make
backups. Anything that’s saved locally on one machine only? Consider it gone
until it’s backed up.

Cloud providers may be able to give you better assurances, but if you really
care about data give it at least 2 independent homes. I’ve lost data more than
I care to admit. BuyVM lost one of my VPSes years ago. Who’s fault was it
really?

When you are ready to stop kidding yourself about your data, check out some
backup solutions. I particularly like Borg Backup:

[https://github.com/borgbackup/borg](https://github.com/borgbackup/borg)

And if you do not have network attached storage anywhere there are services
that provide it as a service.

(Note: I think needless to say it’s also a good idea to back your NAS up to
other places too, although I haven’t gotten into this practice yet. Synology
supposedly has a lot of features around this.)

------
teh_klev
The key question is, did Gandi offer and explicit backup service for your data
on their plans? I just had a look and I don't see this being offered.

As a former hosting engineer, at the risk of pissing on everyone's outrage
parade, but unless an _explicit_ guarantee of a backup is included in your
plan's contract, or you can pay for backups as a bolt-on, then if you've lost
data it's your fault for not planning for this scenario.

And I mean proper backups where you get, for example, twenty eight days of
hourly backups and you can pick a specific version of file to recover in that
28 period. And where those backups are stored on different hardware or off-
site. We offered this as a bolt-on (in-site and off-site). Tt was 20 quid a
year for in-site, the off-site was a bit more. But a great many customers
chose not to pay for this add-on, even despite the great big red bold warning
text explaining that unless they paid for this add-on we made no guarantees
about the permanence of their data in the event of a storage problem. Guess
what....

Now that's not saying we didn't take snapshots of the hosting environment, but
they were for internal use and to allow _us_ to recover quickly in the event
of something unexpected going wrong, but now and again stuff breaks.

Sure, it's unfortunate some lump of storage hardware has failed and whatever
mirrors they may have had have been taken out as well. They possibly could
have done better but shit happens sometime.

You shouldn't rely on an "implied backup" from your service provider, if you
want that then you're going to be paying a shedload more for hosting your
Wordpress and Woocommerce site. It's up to you to make sure absolutely sure
your data is safe if it's critical to the day-to-day running of your business.

Edit: ok, so this is tucked away in their docs (thanks to itake below):

[https://docs.gandi.net/en/simple_hosting/common_operations/s...](https://docs.gandi.net/en/simple_hosting/common_operations/snapshots.html)

But it does say:

 _> Snapshots do not make a backup of your databases. If you would like to
perform a backup of your databases, we recommend you perform an export, or
launch a dump script via crontab._

The bottom line...is it guaranteed in your contract? Always check. And as per
my follow up comment, those plan prices are are just too cheap for that
facility to be taken seriously for business continuity. They're a convenience
to quickly recover a version of a file, not a serious backup.

~~~
itake
> Easily recover backups of previous versions of your website's files, thanks
> to our automatic Snapshots system. It's free!

[https://docs.gandi.net/en/simple_hosting/common_operations/s...](https://docs.gandi.net/en/simple_hosting/common_operations/snapshots.html)

They are supposed to be providing backups.

~~~
martius
I believe nobody should count on backups provided by the product that stores
your data.

There are different kinds of backups here:

* the ones that are part of the offer, where the provider gives you a convenient way to recover from your mistakes, this is a feature they provide when their services are operational (in this case, the snapshots feature).

* the ones they put in place to mitigate incidents and maintain their SLOs. If you accidentally delete a file, you don't have access to them, they are useless to you. These backups are a mean to reach their service level objectives. Nobody can offer you 100% guarantee that they won't lose your data in an SLO. If someones promises you this, just... don't believe it.

(edit: formatting, typo, mention snapshots in case 1)

------
corford
I'm a long term user of Gandi for my domains but have wanted to get off them
for some time now.

Can anyone recommend a domain registrar "equivalent" of a Fastmail or
Letsencrypt or DNSMadeEasy i.e. truly no bullshit, geek friendly and polished
at the same time ?

I'm not too bothered about price. I just want a well run outfit that has a
wide selection of TLDs and ccTLDs (and ideally isn't a mega corp like google
but is big enough that I don't have to worry about them disappearing
overnight).

~~~
dangravell
NearlyFreeSpeech?

~~~
corford
Close but sadly they don't seem to offer ccTLDs :(

------
yardie
Azure Shared Responsibilities [0]

AWS Shared Responsibilities [1]

Flipping a switch that says "Backup" does not mean you are handing your
responsibility to them. At most, they will fail to meet their SLA, write you a
check for according to the TOS and be done with it. At best, you'll be able to
bitch about it on Twitter, possibly threaten a lawsuit (you read the ToS?) and
still be in the same position because you did not share the responsibility of
securing your data.

[0] [https://docs.microsoft.com/en-
us/azure/security/fundamentals...](https://docs.microsoft.com/en-
us/azure/security/fundamentals/shared-responsibility)

[1] [https://aws.amazon.com/compliance/shared-responsibility-
mode...](https://aws.amazon.com/compliance/shared-responsibility-model/)

------
Eikon
> We sincerely apologize for the inconvenience that this situation has caused.
> This type of incident is extremely rare in the web hosting industry.

Why are they speaking of the "industry" as a whole when they are to blame?

It's even crazier they are not even explaining the source of the data loss and
why the "replication systems" didn't help.

IHMO they are trying to sweep this event under the carpet. They should instead
explain why they should be trusted in the future and _why_ this would not
occur again.

~~~
anaphor
Replication is not a backup as was already mentioned. A great example of this
is when the KDE project almost lost all of their Git repos because they were
mirroring a corrupted copy of the data.
[https://www.phoronix.com/scan.php?page=news_item&px=MTMzNTc](https://www.phoronix.com/scan.php?page=news_item&px=MTMzNTc)

~~~
Eikon
A backup _is_ a replication of the live dataset, although, usually out of sync
to be useful when the main dataset goes bad.

~~~
y4mi
You might want to read the Wikipedia definition, because you're technically
mistaken.

[https://en.m.wikipedia.org/wiki/Backup](https://en.m.wikipedia.org/wiki/Backup)

~~~
ben509
That's a long article; please quote the part you're referring to so we're all
looking at the same text.

> a backup, or data backup is a copy of computer data taken and stored
> elsewhere so that it may be used to restore the original after a data loss
> event

Since a "replica" is a copy, that seems technically correct.

~~~
y4mi
all fruits are apples because apples are fruit, right?

~~~
ben509
Your definition claims "a backup is a copy."

The original claim was "a backup is a replication of the live dataset,
although, usually out of sync to be useful when the main dataset goes bad."

The only thing that makes a replica special is that it's in sync. Once you add
the caveat that it's out of sync, it's just a copy.

------
l0b0
The "no bullshit" motto is mentioned a few times here. _A motto is just
another marketing device_ ­– a way for a company to pretend to have any sort
of principles beyond making as much money as fast as possible.

Why would anyone believe a motto is anything other than a marketing device? It
is only believable if people follow it _contrary to pragmatism._ Any company
is eventually going to have a fair share of people who believe being pragmatic
is more important than their motto. And in Western culture at least it's
usually considered rude to bring up the "big guns" and have a fundamental
values discussion when everybody just wants the meetings to end and to start
making more money.

~~~
scandox
In fact a motto is usually chosen to cover a weak spot. So "No Bullshit" reads
to me as "We're Kinda Cowboys". "We Care" \- "People Know We Don't care".

Fujitsu – “The possibilities are infinite” ... "The Ways in Which we can Screw
this Up Are Infinite"

Intel – “Leap Ahead” and “Sponsors of Tomorrow”. "We've got to protect our
entrenched position".

LG – “Life's Good”. "Life is Actually Objectively Bad".

Google - "Don't be Evil". "How We Actually Make Money is Evil But Our Mission
is Good".

------
totaldude87
shit happens...but the way their philosophy is fine tuned makes me wonder..

Above all, "no bullshit" is our golden rule—to treat our users how we want to
be treated. It's a promise to respect your rights and to level with you about
our shortcomings.

[https://www.gandi.net/en-US/no-bullshit](https://www.gandi.net/en-US/no-
bullshit)

ex:
[https://twitter.com/andreaganduglia/status/12151991477012316...](https://twitter.com/andreaganduglia/status/1215199147701231616)
(thanks op)

We will listen to you, and be honest in our replies, even if it means you
won’t always like what we say.

~~~
ChikkaChiChi
> We will listen to you, and be honest in our replies, even if it means you
> won’t always like what we say.

They are actively treating their customers like shit, and that tone starts at
the top. No bullshit does not give creative license to be assholes to people
that are panicked because of something you directly caused.

------
YesThatTom2
That's why I keep all my DNS configuration in DNSControl and push the results
to Gandi (and NameDotCom, and Route53, and GoogleDNS, and AzureDNS)

[https://github.com/StackExchange/dnscontrol](https://github.com/StackExchange/dnscontrol)

(Terraform users have a similar benefit)

~~~
gingerlime
Sounds really cool. Do you also have NS records for all of those? or just in
case you want to switch-over? (can you actually hold a domain on multiple
registrars?)

------
benguild
I hate to say it but Gandi seems like they’re in a quality freefall. I had a
domain there a year or two back because they were one of the only registrars
that supported that particular extension... and man, so many problems just
with simple tasks like updating the WHOIS info and credit card for renewal.

This is basic stuff.

------
zulgan
I had a co-worker who was super chill during outages; especially at night, we
were 10-15 people on the call fixing issues related to his work almost
monthly.

those outages costed millions of euros, and he never picked up his phone at
night, once I asked him why he never picks up, he told me:

"I used to be a general surgeon, when someone calls me people die. Relax,
nobody is dying during our outages."

now I think I am taking myself(and my work) too seriously.

~~~
kbr2000
It's likely the only way to stay sane in a corporate environment. The problem
is, you don't need much people practicing that, before it drags everyone down
to the same niveau. You can choose to try to continue your quest into doing
work seriously (this will likely drive you insane over the years), or to join
in that kind of negligence (goodbye spine), or to quit. In the end it got us
where we are now, a world filled with fake companies selling their fake little
products as they were qualitative, and making a game of disrespecting their
own customers. Pure facade, been there...

I've done it before, but I'll recommend to you Scott Adams' book, The Dilbert
Principle for some light reading about forces like that at work.

------
terom
From the incident timeline:

> we have a problem to import zfs pool on the unit storage

I really want to know what went wrong to a) break ZFS b) prevent recovery from
backup.

~~~
terom
Re myself, it looks like they're using FreeBSD-based ZFS filers with iSCSI/NFS
exports using a user-spce NFS server:

* [https://news.gandi.net/en/2019/09/exporters-detect-micro-inc...](https://news.gandi.net/en/2019/09/exporters-detect-micro-incidents-and-improve-storage-performance/)

> Gandi’s storage infrastructure consists of two environments: one for IaaS
> and one for PaaS. Both are based on FreeBSD-based storage units (filers),
> that stock each volume (disk) as though it were a ZFS volume.

* [https://news.gandi.net/en/2019/03/tracking-a-storage-issue-l...](https://news.gandi.net/en/2019/03/tracking-a-storage-issue-led-to-software-change/)

* [https://www.bsdcan.org/2016/schedule/attachments/351_FreeBSD...](https://www.bsdcan.org/2016/schedule/attachments/351_FreeBSD%20based%20high%20density%20filers-bsdcan2016.pdf)

No mention of what they're doing for backups / "replication systems",
unsurprisingly/unfortunately. I'm anxious to know what the failure mode for
`zfs send | zfs receive` replication is here?

~~~
tomatocracy
Sounds very much like they weren't doing zfs send | zfs receive to anything
sufficiently physically separated. For example, if you send and receive in the
same pool, it's replication but still leaves you vulnerable to issues where
the pool can't be imported due to corruption in the wrong places (it can
happen) or significant hardware failure (eg a PSU fault that takes out too
many of the drives in the pool).

------
iseeyou
last update says they were able to restore a version

Updated on Thursday, 9:58 PM +0200: we're not sure we will be able to provide
the data but we were able to recover a version of the filesystem from right
before the crash

I have been using them for DNS and some minor hosting for a long time and I
will stay with them. I think it's important to avoid the
monoculture/centralisation which is otherwise happening.

Sure Gandi has their flaws, they are humans.

I expect they do will a proper post-mortem on what went wrong and how they
managed to fix it. Seems they were using ZFS and relied on it a bit too much.
Or if they indeed managed to restore the last snapshot, then their only error
might have been the classic one of underestimating how long
restoring/investigating several terabytes take even on modern HW.

------
privateSFacct
There have been a number of threads suggesting places like Gandi over AWS
because they are so much cheaper. I've always been skeptical about building
key apps on these types of places but folks INSIST it's the right choice.

3TB at Gandi costs $6 + you get compute with it. 3TB of bandwidth at AWS might
be $270.

Has anyone tried this instead of using cloudfront etc? Get 100 $6 hosts and
pump out content for your ipv6 connecting clients etc?

------
newscracker
This seems like data has been lost from servers hosting sites/services.

Since Gandi is mostly known for domain registrations and DNS, I'm curious if
you (as an individual who hosts websites/online services somewhere on the web)
backup your site's DNS records periodically (or whenever they're changed).
What if your authoritative name server lost data and all the caches of those
records across geographies expire while you're asleep/away? If you do back
these up regularly, how do you do it in an automated way on a *nix system? I
found this article [1] when I searched about this, but it's not a simple shell
script. The scripts that I did find on some of the Stackexchange sites seemed
to have specific subdomain names hardcoded.

[1]: [http://www.programblings.com/2012/07/23/do-you-back-up-
your-...](http://www.programblings.com/2012/07/23/do-you-back-up-your-dns-
records/)

~~~
dylz
Gandi has an API and lets you download entire zonefiles if they host your DNS.

------
aspectmin
I helped co-found a large Dropbox-like white label product. We used AWS and
especially s3 for storage.

After many many years of experience with systems, I made sure we had as many
possible ways to recover user data as we could. The initial solution was a
large Postgres database for all the metadata/indices and s3 for the actual
storage.

Despite much pushback we built in little things like an individual meta file
on the file system for each file we stored. That way, if we lost the Postgres
dB for any reason, we could create a script to rebuild the dB and restore
access avoiding massive counts of orphaned files. A simple and probably stupid
solution but...

Well guess what - the DB got corrupted and after some ado, we restored all
access and none of our customers lost anything.

No it’s not full backups but...

------
Neil44
The problem with cheap hosting is they want to use backups as an upsell, but
you should still have backups to cover the companies ass even if the customer
doesn’t get to use them. 123 Reg lost a load of customers VPS’s a year or two
ago also thanks to a faulty script.

~~~
PHGamer
Not excusing them but..

with modern container hosting you really should be able to make your own. even
with cheap VPS hosting. There is no reason to live in a world where a server
goes down you lose anything anymore.

------
Hoasi
As a customer, I only have good things to say about Gandi.net but have to
admit this is subpar customer communication right there.

Lost 3 sites built with WordPress. Will rebuild as static sites repo separate
from host, no more database, lesson learned.

------
nojvek
In the world of cloud, this should be pretty trivial. Upload your daily dumps
/ asset metadata to S3/GCS/ABS.

Set a retention policy I.e even if someone ran some delete command it wouldn’t
delete. Someone with retention lock permissions is the only one that can
remove the locks And delete.

There is cold storage and other things even cheaper. But cloud object prices
are pretty cheap per GB it’s ridiculous.

I think they make most of the margins on bandwidth.

Losing customer data. All customer data is pretty ridiculous. I can understand
downtime. I can understand losing a day of changes. But everything? That’s
just unacceptable business.

------
deanmoriarty
I have a custom domain with Gandi and take advantage of their mail forwarding
option to forward the emails sent to the custom domain (my “no lock-in” email
address) to my personal Gmail account.

Considering how critical email is for me, seems like I won’t be trusting their
MX servers to process all my inbound mail anymore and will soon be looking for
another solution that works well with Gmail (don’t want to pay for GSuite),
and possibly also transfer my domain to another registrar.

That support tweet is such bad taste.

~~~
fchu
Depending on your email volumes, mailgun could be a viable alternative

~~~
deanmoriarty
Thanks, I'll absolutely look into this soon.

------
dkarras
This is like living in an alternate universe, I've been heavily involved in
all things programming and webdev for years, following trends and whatnot and
it is literally the first time I'm hearing of this particular company. What is
(was?) so special about them that they attracted the HN crowd can someone
briefly explain? Why would I buy domain from them when something like
namecheap, even google domains exists? Why would I even host something there?

~~~
porker
> Why would I buy domain from them when something like namecheap, even google
> domains exists? Why would I even host something there?

If you're in Europe, they're cheap for many European countries' domains.

Back 10-15 years ago they were special because it felt like a hacker kind of
company. They gave free WHOIS privacy, what seemed like good DNS control/UI at
the time. But it was the WHOIS privacy that got me onto them.

I still use them because they're around half the price for .co.uk than many
registrars - and many others I've used have become more rubbish than Gandi
has.

All my DNS is hosted elsewhere now, and I never understood why Gandi
introduced hosting et al. I've never used it and never would, it seemed a
terrible diversification for a good domain registrar.

~~~
thrwaway69
Can you explain how you not using their hosting makes it a terrible
diversification?

I think majority of people buy domains for hosting websites so it makes sense
they would want to setup one using one click WordPress or something similar.

~~~
porker
It wasn't their skill-set. They had problems with their hosting from the
start; reliability problems IIRC, not data loss.

------
nachtigall
News post about it:

[https://news.gandi.net/en/2020/01/major-incident-on-our-
host...](https://news.gandi.net/en/2020/01/major-incident-on-our-hosting-
infrastructure-in-luxembourg/)

A site of mine is also hosted as their PAAS at Luxembourg, but was luckily not
effected. Probably my site was on another storage unit ("on one of our ZFS
storage units").

PS I also always thought that the snapshots were backups.

------
kuon
I have about a hundred of domains registered at gandi, I used to like the
formed management interface, but I really hate the new one.

Is there a registrar you would recommend as an alternative, I don't need DNS,
nameservers and glue records and I'm ok.

The main selling points are stability, transparency and simplicity. I don't
care if it's not the cheapest.

------
whorleater
Gandi is never really impressive, but they're one of the few registrars where
I can get .af domains without a hassle.

------
blackethylene
All my domains are registered through Gandi. What good registrars would you
suggest? I'd like to move them out.

------
Macha
I have some domains here, mostly secondary domains to not have all my eggs in
my namecheap basket (e.g. if anything happens to namecheap or my namecheap
account).

Will likely transfer those to elsewhere after this. Probably Name.com, I
guess.

------
smileybarry
(1h22m before this comment)

> Updated on Thursday, 9:58 PM +0200:

> we're not sure we will be able to provide the data but we were able to
> recover a version of the filesystem from right before the crash

Maybe it's not _all_ gone.

------
Johnny555
_The assessment is taking a long time because there are several TB of data on
the filer_

Is that a lot of data? That sounds like a very small filer that could have
easily been backed up.

------
kup0
Regardless of _any_ of the technical aspects of this disaster, the attitude of
the company and its customer service means I will be staying far away from
them.

------
RantyDave
For not the first time I'm left thinking that the "big filer" model is not
such a great idea :(

------
cenourinhapt
Always make your own backups. Shit is going to happen any time soon.

------
jigglypuffs
'Gandi' means bad in Hindi. Some coincidence huh!

------
babycake
What kind of data was affected? Was Email messages affected?

------
pinewurst
Does anyone know what Gandi is using as a “filer”?

~~~
corford
ZFS by the looks of the status updates:

>we have a problem to import zfs pool on the unit storage. Our engineers are
still working on it.

------
buboard
There s an upside to knowing your data is not backed up somewhere, and that
when you delete them they 're really lost. They should offer that as privacy-
conscious hosting.

------
markorigho
Hey Guys sorry for being a philistine about this but does that mean we have
lost our domain name and how can we migrate it to another hosting platform?
Cheers

~~~
mratsim
If you only had a domain you're likely not affected (metadata). If you also
have a website hosted there (data), you may be.

------
markorigho
Just a quick question, how do I transfer my site adress to another hosting
company, is that too is lost? Sorry for being a philistine about this...
Cheers

