
Verelox Wiped by Ex-Admin - jonmarkgo
http://www.verelox.com/
======
tgtweak
This exact thing happened to realitychecknetwork hosting about 6 years ago
(now rebranded to serverstack and digitalocean).

There was 250+ dedicated servers, 2-3 weeks of restoring week-old backups
(thankfully they had these weekly intervals kept offline). Mass exodus of
clients.

"Ex-employee" used root keys and a boot zerofill drop and rebooted every
server resulting in severe data loss. Their online backup systems were also
using these keys and we're not spared.

They said they would have to shut down the company as a result, but ended up
securing capital and eventually launching what would become digitalocean.

They said it was highly probable that it was an ex employee and that the FBI
was investigating buy nothing was released about it.

Good cautionary tale for segregation of credentials and proper user key
management.

~~~
mintplant
Looks like DO has managed to keep this detail out of the record for the most
part, save some forum posts that pop up if you search "realitychecknetwork
digitalocean". It would explain the odd gap in time in their Wikipedia page
history:

> In _2003_ , Ben and Moisey Uretsky who had founded ServerStack, a managed
> hosting business, wanted to create a new product which would combine the web
> hosting and virtual servers. The Uretskys, having surveyed the cloud hosting
> market felt that most hosting companies were targeting enterprise client
> leaving the entrepreneurial software developers market underserved. In
> _2011_ the Uretskys founded DigitalOcean, a company which would provide
> server provisioning and cloud hosting for software developers.

~~~
tgtweak
Yeah Ben and Moisey are both incredibly competent sysadmins and shrewd
businessmen. They deserve the success they've had but they were very close to
desperation after this incident. I can't blame them for omitting it from the
records, the reality is likely that digital ocean and serverstack are both
much more secure as a result.

------
treyfitty
This may be an unpopular opinion, but I want to preface this by saying:
"before passing judgement, context is always necessary."

Mario Savio was a Free Speach Activist and organized a protest to protect the
Freedom of Speech at Berkeley around the 60s. In his speech to protestors, he
says "there's a time when the operation of the machine becomes so odious...
that you can't take part... and you've got to indicate to the people in charge
that unless you're free, the machine will be prevented from running at all!"
Applied to free speech, this notion of disrupting the functioning of an
organization was lauded, because freedom of speech is just that important.

But let's shift to employment. Without employment, it's very hard to survive.
And here's a situation where the people in charge has the upper hand in every
arena- hiring, pay, work Place behavior... etc. How do we know that the ex-
admin wasn't blackmailed by the CEO to come back to work for free to fix
something, or future references will be negative? Why are we so quick to side
with the employer in this matter when we know nothing of the situation at all?
Why do we start calling the employee a felon? He hasn't even been charged yet.

My point is, context is important. Fine, corporations have the power to ruin
your life as a deterrent to keep you from acting against their interests, and
that's just the way society is. And fine, We're not all rational at every
instance of life. The calculus of establishing status quo equilibrium of those
two conditions/constraints is hard, but without context to the situation, who
are we to decide who's right or wrong? Would you label Mario Savio wrong for
protesting and urging protestors to prevent the operation of the college from
functioning in the name of preserving Free speech wrong? No, because you've
learned the context.

~~~
idealogue
I agree with you. The majority of people in this thread have completely
internalized that property rights are more valuable than human rights. Only
one comment I've seen has acknowledged that all we have is a single webpage
with a single accusation, and yet many people leap straight to the conclusion
not only that the individual in question exists and is guilty, but that there
can be _no justification_ for a person destroying a business. Businesses
destroy people all the time. I for one would like to see the tables turned
more often.

~~~
xupybd
We have responded to the article based on the information presented.

There is nothing to imply human rights have been violated. If such information
is presented im sure people will respond in kind.

~~~
DashRattlesnake
> We have responded to the article based on the information presented.

> There is nothing to imply human rights have been violated. If such
> information is presented im sure people will respond in kind.

What are you talking about? There is no "article," only the statement of one
involved party. It's right to be skeptical and theorize about what else might
have gone on that was omitted from that statement out of self-interest.

------
yardie
So in addition to the criminal side of things I guess the ex-admin wants to
work in manual labor or fast food. There is no way in hell he'd have the
references or pass the background.

BTW, we had a netadmin interview a few months ago. Guy was really smart, aced
the technical and group interview. We were really looking forward to hiring
him, and only needed to pass a background and reference check. HR told us in
no uncertain terms to run the other way. They didn't share what was in his
check but it wasn't good.

~~~
ugh123
>They didn't share what was in his check but it wasn't good.

Always ask to see the background check details if you're the hiring manager
(you should have the rights to see that despite what HR might tell you). Could
be just a personal issue an HR employee might have with a former colleague. Or
discrimination-based (happens).

~~~
Gibbon1
Reminds me one of my old bosses hired someone. And then HR fired them because
they had a pot conviction. So then my boss hired them as an outside
contractor. Every six months HR would discover he was on the 'no hire list'
and terminate his contract and my boss would cut him a new contract.

~~~
pkaye
HR doesn't need to approve of contractors?

~~~
valuearb
In a good company, HR has as little authority as possible.

------
preinheimer
Wayback Machine link if you want to know who they were:

[https://web.archive.org/web/20170603212121/https://verelox.c...](https://web.archive.org/web/20170603212121/https://verelox.com/)

~~~
michaelbuckbee
Wow, they were a VPS hosting provider. Lots of unhappy customers (and this
likely qualifies as a data breach for all of them as well).

~~~
__jal
Yeah, that could become legally awful. Which is appropriate; it appears $admin
ruined a lot of people's days.

I suspect that person will soon have a fair amount of time to decide on what
new career to pursue to pay down the fines when they get out...

~~~
redm
I suspect they assume it was an ex-admin probably because of timing. There may
not be any proof to back up that claim and thus there may be no repercussions.

Until they release specifics if ever, it's hard to know what to make of the
status message.

------
pmarreck
This is so stupid. If you have a problem with your employer, either you quit
or they fire, you move on, full stop. If you're in a relationship and someone
isn't happy enough with you and breaks up with you, the dignified response is
NOT to key their car. I see employment relationships mostly the same way.
Either it works (for both) or doesn't (for either or both ends).

And having switched jobs quite a few times, the next one is always better for
you, regardless.

~~~
HipHopHacker
> I see employment relationships mostly the same way

Apparently they did not.

At the end of the day, the people working at the company are the ones who are
doing the work, and who have control of the means of production. The ex-
admin's bosses probably thought they were the important ones, and that this
worker was a replacable cog, but they found out the hard way that this was not
the case.

I worked at a Fortune 100 investment bank where this happened. Everyone knew
layoffs were coming. One week after layoffs came, a digital "bomb" went off
wrecking many servers. So security went through, trying to find evidence
(nothing incriminating from what I heard, although they had a strong suspect)
and also looking for more bombs. They missed out on finding and defusing one,
because another one went off a month later.

The view from the pinnacle, people counting the dividends on the checks that
they inherited is that they're the job creators, and everyone else is
dispensable. This company just found out that is not the case.

~~~
hoodoof
Integrity is choosing to behave in the right way, even though you have the
option of behaving otherwise.

Therefore, to behave with integrity, you must have formulated your own set of
values about what is "the right way" to behave.

Every minute of every day, we all have the option to behave with or without
integrity in a whole range of ways.

You earn respect by demonstrating behaviours over time where you have taken
the interests of others into consideration, generally people consider someone
who behaves like this to have "integrity", especially when they continue to
behave that way when no-one is looking.

Saying things like "The ex-admin's bosses probably thought they were the
important ones" indicates a childish set of values where there is a power
struggle between employers and employees ........ of course the "bosses" are
the important ones, they act for the business which is an independent legal
entity, upon which many people depend for their lives to work effectively. If,
as an employee, you feel poorly treated or otherwise dissatisfied, then the
right thing to do is leave in a polite and respectful manner, even if you feel
you were not treated in that way. Depending on the circumstances, if you were
actually treated really badly, then the right thing to do is pursue your
complaint through the appropriate legal channels.

Someone important in my life once said to me "the only thing you have is your
reputation". Take that reputation, defend it, enhance it, nurture it and earn
the respect to grow it. Don't throw it in the garbage by smashing other people
(or their business) in a childish tantrum. I admit this is hard to do - I
regret many things I have done in my life, but I try to lead a life consistent
with my own sets of values that I think are meaningful and I get rid of people
from my life who I think don't have integrity, or whose values are different
from mine in critically important ways.

~~~
throwaway5728
>You earn respect by demonstrating behaviours over time where you have taken
the interests of others into consideration

The relationship in most companies is entirely asymmetrical. If times are
tough, employees are expected to work unpaid overtime, to sacrifice on pay and
perks, to accept layoffs. If times are good, shareholders and executives see
all the profit. Employees are expected to show absolute loyalty, but are shown
not one shred of loyalty in return. Productivity is soaring across the
economy, but wages have been stagnant since the 1970s.

Most employers will never truly respect their employees, ever, under any
circumstances. Employees aren't people, they're a "human resource", a cog in
the corporate machine as interchangeable as any hardware. More so, in fact - a
piece of machinery would be hired on a fixed-term lease, but most employees
can be dismissed at will.

I don't endorse vandalism, but I think that it's utterly naive to expect that
you can earn the respect or loyalty of corporate America. It doesn't matter
how honest you are or how hard you work, you'll still be discarded like an
oily rag if you're surplus to requirements. You'll still be lowballed on every
pay rise while executives and shareholders make record earnings.

~~~
DashRattlesnake
> Most employers will never truly respect their employees, ever, under any
> circumstances. Employees aren't people, they're a "human resource", a cog in
> the corporate machine as interchangeable as any hardware. More so, in fact -
> a piece of machinery would be hired on a fixed-term lease, but most
> employees can be dismissed at will.

> I don't endorse vandalism, but I think that it's utterly naive to expect
> that you can earn the respect or loyalty of corporate America. It doesn't
> matter how honest you are or how hard you work, you'll still be discarded
> like an oily rag if you're surplus to requirements. You'll still be
> lowballed on every pay rise while executives and shareholders make record
> earnings.

You are absolutely correct about corporate America, the executives, and the
shareholders. The hard thing is that you _can_ earn the respect and loyalty of
the other cogs that you work with, which can be difficult to disentangle from
that underlying truth.

------
wilhil
Other than treating staff well, how would you go about stopping something like
this?

As my own company is growing, we fully trust all employees, (limiting only
what is essential), but, a dev ops guy if he was so inclined could technically
do something like this... It always scares me.

~~~
benjohnson
We swiped an idea from launching Nuclear Missiles: Typically you need two keys
from two people to launch.

For really important accounts - we have three people who each know two thirds
of the password. It requires two people to then log in and do damage.

For example if the root password was CatDogFish then

Person1: CatDog_

Person2: _DogFish

Person3: Cat_Fish

Two people can then log in and watch what the other person is doing.

~~~
rcthompson
Maybe someone should implement a PAM module or something that requires
authenticating as two different users with sudo privileges, so you can
implement this for as many users as you like and let them each have a single
password, instead of half of a password for each possible pair of users.

~~~
seanp2k2
Use something with 2FA like e.g.
[https://duo.com/docs/duounix](https://duo.com/docs/duounix) and give person A
the password and person B the 2nd factor (phone with this Duo profile
configured or whatever).

~~~
rcthompson
That doesn't eliminate the need to set up an account & password for every
possible pair of users, rather than just every user.

------
yangtheman
Proper exit procedure should have disabled all access from this ex-admin...,
unless s/he had some sort of cron job or launched some process that would
execute commands at certain time? I am very curious to know how it was done.

~~~
redm
That's all true, but in reality, we often count on the better nature of
people, goodwill, for a proper break on both sides.

If someone is planning a malicious exit, it can be very hard to stop them
depending on how "integrated" they are.

~~~
weaksauce
IT admin people have the keys to the building and pretty much all data at the
end of the day. Trust is everything and reputation is extremely important.
This will not go well for the ex-admin one way or another either by lawsuit or
blacklisting.

~~~
idealogue
And yet, they are often abused, underpaid, and treated as replaceable. The
only surprise should be that this doesn't happen far more often.

~~~
weaksauce
At the core, IT people are usually seen as cost centers and not revenue
generators. Not that I disagree with the business owners a lot of the time
because IT is usually not the thing that makes a lot of companies money.

------
stevenh
If I ran a hosting company and all of my servers were compromised by ring -3
malware exploiting the Intel AMT vulnerability, the first thing I'd do is
privately inform Intel that I intend to go public with the story and sue for
damages, after which Intel would perhaps offer a very generous bribe for my
silence and a week-long window to replace all of the server processors for
free, on the one condition that I bury the truth by fabricating a story about
an imaginary ex-employee who improbably was both smart enough to gain an
administrative position in a large company while also being stupid enough to
risk decades in prison for petty revenge over workplace drama.

~~~
midnitewarrior
What kind of hosting company is going to have any kind of reputation after
admitting their security allowed an ex-admin to ransack everything?

Rephrase the question -- what idiot customer is going to do business with such
a place that allows such a lapse in security to happen?

Intel would basically have to buy the company.

~~~
hoodoof
>>admitting their security allowed an ex-admin to ransack everything

What, exactly can be done to secure a company against a malicious systems
admin? These are the guys typically with not only the keys to everything but
also the knowledge of how it all works.

You say that the company cannot be trusted for "allowing" this to happen.

I know quite alot about this stuff, and for MOST companies, they simply have
to trust that the people with the keys to the castle with behave responsibly.

There are ways to design infrastructure such that it is protected from its
builders and keepers, but this is very very hard and complex and expensive.

Presumably you work for a company that has taken steps to ensure this will
never happen, what are they?

~~~
kelnos
You should be trivially able to secure your company against an _ex_ admin,
though -- why was this ex-admin's credentials not revoked immediately after
their employment ended?

~~~
sqldba
If they leave the company, their access might be left running for a while in
case they're asked to come back on and fix something during that post handover
period.

Or maybe they had backups running under their account and when their account
was disabled everything failed so they re-enabled it while they sorted out the
mess...

Or they had multiple accounts as part of "security" and HR only knows to
disable one and didn't find the other one in time.

There's a whole bunch of reasons why shit like this goes wrong. Every time.
You'd cry.

~~~
kelnos
I would consider none of those to be _acceptable_ though, and would point to a
huge process failure around security at the company.

------
bobbob1921
Lots of comments are interpreting "ex-admin" as someone who was fired and
_then after_ went and did this. Just want to float the possibility that "ex-
admin" could also mean someone was employed there, _then did this_ and is now
no longer employed as a result of doing this.

(Btw, IMO there is no excuse or justification for any admin or exadmin to ever
do this. Among many other issues is the fact he deleted the data/work of
individuals who had nothing to do with whatever "problem" he has with Verelox
)

~~~
sqldba
What if their information was all secretly being leaked to intelligence
agencies and there was no way to ever let them know except to burn everything?

There's probably excuses and justifications. I personally wouldn't do it and
they're probably wrong for doing it but I don't want to jump to conclusions
and moral absolutes so easily.

------
tw04
It's always interesting watching startups learn the lessons that thousands of
enterprise learned along the way. "Why would you ever want offline tapes
sitting in iron mountain, how inefficient".

Nothing is foolproof, but anytime you've got constant network access to every
last copy of your data, you're begging to lose it. It's the reason why people
who think one copy (redundantly dispersed or not) in AWS S3 is sufficient
scares me to death. Is it unlikely Amazon would get hacked and have the entire
thing blown up? Sure... but if we go to war with China I wouldn't want to bet
my company on it.

~~~
nulagrithom
If we go to war with China I don't think you or anyone else will care much
about your company.

I know you meant it as an example, but this sort of extreme attitude towards
security is just another footgun.

~~~
kstrauser
> If we go to war with China I don't think you or anyone else will care much
> about your company.

Why? We went to war with Europe and Asia a few times and businesses kept
chugging along here in the states.

~~~
sabujp
that was before people had lots of nukes

------
pavement
I would expect to see some kind of police report, and prosecution of an
individual charged with a crime, no?

~~~
Cshelton
Yes, it is criminal and the ex-admin will go to jail. The cases for this are
pretty straightforward... not sure why someone would throw their life away by
doing it...

~~~
slantyyz
>> Yes, it is criminal and the ex-admin will go to jail.

What if the admin was a remote worker in a country that doesn't have an
extradition treaty with the Netherlands (Verelox hq country)?

~~~
wand3r
Presumably they would have a warrant out in Netherlands. If issued in the EU
or US I wonder if it would effectively ban them from most Western countries
similar to Assange.

If this was a "cyber crime" they could possibly be picked up to stand trial in
any cooperating country which would narrow their choices of travel.

However, in that situation they would probably be fine except potentially
limited ability to work for Companies who do background checks

------
jldugger
Been waiting for a company to announce shutdown after this was posted:
[https://news.ycombinator.com/item?id=14476421](https://news.ycombinator.com/item?id=14476421)

Possibly related?

~~~
Dreami
Well it's kind of the opposite, isn't it? ;)

~~~
jldugger
If you take Verelox's word for it. Or if you want to be more conspiratorial,
you could see your firing on the wall, tweak the new hire docs to lead to
trashing prod, and just wait for the inevitable.

Or these two events are unrelated. Or the whole deleted prod on day 1 story is
made up.

------
zokier
Wiped by ex-admin, or by the ineptitude of current admins that can't maintain
proper exit procedure?

~~~
ascendantlogic
Ok, I'll bite. If you are an admin generally you have God powers over
everything within your umbrella. If things are going south with you and your
employer and you're a giant asshole you can most certainly use your God powers
to quietly open firewall ports, install software in dusty "corners" of
networks, etc. We can all armchair quarterback this but in the end humans make
mistakes and beyond that the presence of actual malicious intent by someone
with keys to the whole kingdom makes the detection and/or mitigation
difficulty factor go up significantly. It's real easy to make righteous
proclamations about other people's misfortune when it's not you.

~~~
zokier
I'll admit, my tone might have been tad overly poisonous (too much internet
for me..).

I'd agree that defending against malicious admins is really difficult. We have
really little context to go by here, but I think there is important
distinction to be made if the malicious actions (planting backdoors or
whatnot) were done while the malicious actor was still employed or after their
employment was terminated. Proper exit procedures protect against the latter,
but generally are not that effective against the former.

> but in the end humans make mistakes

And it is useful for us outsiders to highlight the real mistakes so that we
can learn from them, because that is really the biggest value of stories like
this for the majority of people who are not directly impacted.

------
jacquesm
This is why you have the backups stored under a different account than the
primaries and you make sure that nobody has access to both accounts.

------
cannonpr
A lot of 'managed' hosting providers are pretty bad with security, there still
is a major provider that just gives root credentials to all servers to all
techs not just admins, doesn't audit who accesses which credentials, and
doesn't rotate credentials, doesn't rate limit dumping credentials... That's
before we go into more interesting issues with their security. Frankly I am
surprised this sort of thing doesn't happen more often ? In some ways it both
restores some of my faith in people while reducing some of it at the same time
in a different vector.

------
keithpeter
[https://www.lowendtalk.com/discussion/116329/what-s-up-
with-...](https://www.lowendtalk.com/discussion/116329/what-s-up-with-verelox-
being-down)

Some posts from Verelox staff towards bottom third of this forum page search
for user name Verelox

------
CM30
This is why a hosting company needs to both segregate credentials to only what
an employee needs for their job, as well as to revoke them the minute they
leave the company.

Otherwise while the vast majority of your staff will be decent people and not
cause problems like this, it just takes one angry ex staff member with a
grudge to cause problems.

They also need to revise their backup system too. There should rarely if ever
be a risk that any data is 'unrecoverable', yet their update says some data
will just be impossible to get back.

As for the employee involved... well I hope they like the inevitable lawsuit
their selfish, stupid actions will bring them. I don't care what you think of
a company you worked for, there's no excuse to destroy their business through
actions like this. Also, good luck getting any jobs in the industry after too.
Because with this on your track record, no one will touch you with a ten foot
bargepole.

So yeah, what a disaster all round.

------
ceejayoz
Yooooowch. They appear to be VPS and dedicated host.

~~~
rexicus
There goes the 99.95% uptime guarantee.

~~~
oneplane
In theory that still gives you a lot of downtime in a 365-day span :p Then
again, once you are down for more than 4 hours (as they are), that uptime
starts ticking away rather fast.

~~~
wujonesj2
I believe a 99.95% uptime guarantee indicates that you will only have 4.38h of
downtime _per year_. So, yeah, they've blown their allowance for the year and
now need to execute perfectly.

------
gaius
Who even knows it was an ex-admin? Could be the current one fat-fingered it
and is trying to shift the blame! We just don't know.

~~~
philfrasty
Exactly. Mind-boggling how easy it is to present people a logical reason.

------
quicksilver03
The thread title should be changed to "Verelox allegedly wiped by ex-admin":
we only know one side of the story.

------
svakacast
Anyone of us know what is the best way to get refunded? My Company lost
20.000€ for this joke.

~~~
bm1362
Lawsuit, most likely.

------
antfarm
I wonder whether the ex-admin was already an ex-admin at the time he wiped the
servers.

------
kashif
Use Vault from Hashicorp where possible.

~~~
oneplane
That would most likely have made 0 difference.

------
Sir_Substance
Dick move from the ex-admin, but I'm curious to know what would compel an ex-
employee to take such a brazenly criminal and traceable yet damaging action.

I'd like to know more, I think...

~~~
jacquesm
There is no excuse for it, no matter what the company did this affects the
customers of the company just as hard as it affects the company. Likely some
of those will go out of business, people will lose their jobs, go bankrupt.

~~~
idealogue
That's the smallest thing I've ever seen "too big to fail" applied to.

~~~
jacquesm
The company could very well still fail, and no matter how large they are their
customers should never rely on their provider not failing to the point where
they themselves will also fail if something drastic were to happen to their
provider.

But that does not mean that a single individual can put themselves in the
judge, jury and executioner role all at once without any kind of oversight,
that's at best a misguided case of vigilantism and at worst an act that is
disproportionate against innocent bystanders and possibly a far larger crime
than whatever happened before.

