Hacker News new | comments | show | ask | jobs | submit login
Verelox Wiped by Ex-Admin (verelox.com)
241 points by jonmarkgo 11 months ago | hide | past | web | favorite | 380 comments

This exact thing happened to realitychecknetwork hosting about 6 years ago (now rebranded to serverstack and digitalocean).

There was 250+ dedicated servers, 2-3 weeks of restoring week-old backups (thankfully they had these weekly intervals kept offline). Mass exodus of clients.

"Ex-employee" used root keys and a boot zerofill drop and rebooted every server resulting in severe data loss. Their online backup systems were also using these keys and we're not spared.

They said they would have to shut down the company as a result, but ended up securing capital and eventually launching what would become digitalocean.

They said it was highly probable that it was an ex employee and that the FBI was investigating buy nothing was released about it.

Good cautionary tale for segregation of credentials and proper user key management.

Looks like DO has managed to keep this detail out of the record for the most part, save some forum posts that pop up if you search "realitychecknetwork digitalocean". It would explain the odd gap in time in their Wikipedia page history:

> In 2003, Ben and Moisey Uretsky who had founded ServerStack, a managed hosting business, wanted to create a new product which would combine the web hosting and virtual servers. The Uretskys, having surveyed the cloud hosting market felt that most hosting companies were targeting enterprise client leaving the entrepreneurial software developers market underserved. In 2011 the Uretskys founded DigitalOcean, a company which would provide server provisioning and cloud hosting for software developers.

Yeah Ben and Moisey are both incredibly competent sysadmins and shrewd businessmen. They deserve the success they've had but they were very close to desperation after this incident. I can't blame them for omitting it from the records, the reality is likely that digital ocean and serverstack are both much more secure as a result.

Problem is at some point some person or more likely group of people has to be entrusted with the Master Keys. Using different keys for production machines and that sort of thing is all well and good but they're kept somewhere, meaning if an admin decides to screw you, there isn't a lot you can do.

Seems like the better option is keep your admins happy as much as possible.

By that logic, all employees should be kept as happy as possible because they might commit arson.

All employees should be kept as happy as possible anyway, shouldn't they? Why would anyone want unhappy employees?

The point is, it's not a trade-off between employees' job satisfaction and 'committing a crime'. These are not two sides of the same or even related issues. You don't keep employees happy because they might burn the place down if you don't and and someone's departure from a job is not a reason for them to commit a crime. Equating these things is silly.

> By that logic, all employees should be kept as happy as possible because they might commit arson.

If that's the only logic that successfully gets through to the boss, it's good logic.

Well in general, you catch more flies with honey than with vinegar. I can't help but notice that most of these companies that run into these sorts of issues also had some pretty bad stories on Glassdoor.

Not necessarily causality, but I'd say there's at least a correlation, and a good enough reason to make office life as bearable as possible.

I need to watch Office Space again...

This may be an unpopular opinion, but I want to preface this by saying: "before passing judgement, context is always necessary."

Mario Savio was a Free Speach Activist and organized a protest to protect the Freedom of Speech at Berkeley around the 60s. In his speech to protestors, he says "there's a time when the operation of the machine becomes so odious... that you can't take part... and you've got to indicate to the people in charge that unless you're free, the machine will be prevented from running at all!" Applied to free speech, this notion of disrupting the functioning of an organization was lauded, because freedom of speech is just that important.

But let's shift to employment. Without employment, it's very hard to survive. And here's a situation where the people in charge has the upper hand in every arena- hiring, pay, work Place behavior... etc. How do we know that the ex-admin wasn't blackmailed by the CEO to come back to work for free to fix something, or future references will be negative? Why are we so quick to side with the employer in this matter when we know nothing of the situation at all? Why do we start calling the employee a felon? He hasn't even been charged yet.

My point is, context is important. Fine, corporations have the power to ruin your life as a deterrent to keep you from acting against their interests, and that's just the way society is. And fine, We're not all rational at every instance of life. The calculus of establishing status quo equilibrium of those two conditions/constraints is hard, but without context to the situation, who are we to decide who's right or wrong? Would you label Mario Savio wrong for protesting and urging protestors to prevent the operation of the college from functioning in the name of preserving Free speech wrong? No, because you've learned the context.

> without context to the situation, who are we to decide who's right or wrong?

You can comfortably make a determination about what is right and wrong. We don't know the facts, but if the claims are true, wiping out not just that company's property, but that of their customers, is a crime.

Now, sometimes a crime is justified, but I don't think it is a rush to judgement to work from a starting point that criminal behaviour is bad until proven otherwise.

I disagree. Once again, we can find a scapegoat at the lowest possible level and "stone" them or we can do a proper post mortem and try to learn from it.

Why did this happen? How can we prevent it from happening in the future? These are the questions we need to stress.

In particular, why does an ex-employee still have access to production? I say when something like this happens and heads must roll, they must roll at the top. Fire the CEO. Fire the board. Leave the sysadmin alone.

This is a civil matter. My tax dollars should not pay for a criminal lawsuit. Screw that.

Oh and by the way if you're reading this: please help repeal cfaa.

I think someone at the org needs to be responsible for not ensuring former employees' access is not revoked, but that doesn't mean an employee who takes advantage of that lapse isn't responsible for his/her actions.

Put another way, someone at Verelox screwed up and left the door unlocked, but that doesn't mean that the person who walked in broke stuff is in the clear.

Look, someone, without authorization, accessed a former employer's network and maliciously destroyed data. That's a crime. Sure, the CFAA is overly broad and is abused, but this is not one of those cases: this is a textbook example of something that should be prosecuted under the CFAA.

I disagree. The point of focus should be verelox and the criminal negligence on their part. Verelox is the criminal, not the victim. The clients may be the victims.

We should not equivocate on cfaa. It is good for nothing. Full and unconditional repeal should be our only demand.

> The point of focus should be verelox and the criminal negligence on their part. Verelox is the criminal

Nope. The affected customers probably have grounds for a civil suit, but no sane prosecutor would think bringing criminal proceedings against Verelox to make sense at all.

The ex-employee who perpetrated this is at fault as well and must bear responsibility for his destructive actions. Having the ability to do something (even due to a lapse in security) does not make that action moral or legal.

We have no reason to suppose that the ex employees actions were justified. None whatsoever. He committed both a civil wrong and a crime and the default assumption ought to be that he ought to face both civil and criminal penalties. You tax dollars pay for punishing people who vandalize a car, I have no idea why they ought not to pay for punishing someone who vandalizes a server.

We may remain open to additional information without presuming that uncivil and illegal vandalism was justified indeed without inventing a narrative from whole cloth as you have done. The logical conclusion is that drawing from your own life experience you identify so strongly with the narrative of the wronged sysadmin that you desire to fit a narrative to sparse facts that has no basis in fact.

We are merely commenting on a story on hacker news. We aren't members of the jury and don't face the same burden or power. I'm down with repealing the cfaa because its badly written, I'm down with figuring out who dropped the ball as far as giving the sysadmin access post firing, but as to the sysadmin himself, burn the witch!

> I say when something like this happens and heads must roll, they must roll at the top. Fire the CEO. Fire the board. Leave the sysadmin alone.

IIRC, the Soviet Union had a policy rather like this, referred to as something like the Vertical Stroke, where anytime there was a screw-up at a low level, they would fire the screw-up-ee's manager, and manager's manager, and so on, up to a very high level. The practical result was a drastic decrease in innovation and risk-taking. CEOs and others at that level usually aren't close enough to the guys actually doing direct work to supervise them all closely enough to ensure they don't make mistakes. All they can do is create a culture where there's a book of rules, and you don't deviate from the rules ever, for any reason, no matter what. So that's what they do, and that's the resulting culture and economy that you get.

Maybe we shouldn't rush to judge either sysadmins or CEOs, but instead figure out who, if anyone, actually did something malicious, and let everyone else take the lessons they've already learned from what happened.

The Soviets wouldn't actually fire them. They just lost their jobs due to missed time in Siberia or from having bullets in their brains.

Ah, well, that's ok then!

If corporations want us to treat them like people, I'm all for treating them like people. This means they are able to defend themselves in a criminal lawsuit and the board is willing to go to prison or the chair for the scrims of the organization.

They can't have it both ways.

> I disagree. Once again, we can find a scapegoat at the lowest possible level and "stone" them or we can do a proper post mortem and try to learn from it.

If he did it (intentionally wiping the servers), then he is a criminal and a bad person.

If he is a scapegoat, then he didn't do it, so that's a totally different situation.

I don't see how you can disagree with either of these statements.

And discussion of how he still had access is an unrelated matter.

> why does an ex-employee still have access to production?

Maybe he became an ex-employee after (and as a result of) wiping production.

Thanks for pointing it out. That is certainly a possibility given the ambiguity in the announcement. In fact, it's not even clear at this time whether wiping production was intentional or not.

I've heard stories where employees wiped production by accident and promptly became ex-employees, followed by their ex-employer trying to put all blame on them. We don't have enough information at this time to determine whether this is what happened at Verelox.

>This is a civil matter. My tax dollars should not pay for a criminal lawsuit. Screw that.

So if an aggrieved ex employee does enough damage that the company has insufficient resources to sue, or has enough resources themselves to make that difficult, it's all good?

Individuals and companies should not be vulnerable to attacks like this based on their resources. It's in all our interests to ensure this sort if activity is dealt with severely because we are all vulnerable. Mutual defence in the form of criminal prosecution of offenders is the way they go IMHO.

Absolutely. Let's deal with it severely. Let's put the CEO and the board behind bars. I don't see how verelox is a victim. It is like saying a police officer who leaves his keys in the patrol car is a victim when someone takes of with the patrol car. The patrol car isn't his! Similarly, the data isn't verelox's!

Let's be practical. What is our end goal? Is it vengeance or is it prevention? If it is the latter, our actions must not be centered on retribution but rather on logic and reasoning. We should ask what can we do to prevent this from happening again? Throwing someone in prison is not the answer IMO.

If you are a nefarious administrator, you don't think you could install something on your machines that would let you back in after they revoked your access?

If some one hacked your bank account and stole 20k you'd be singing a different tune.

If someone hacked my bank account through no fault of my own, I'd be taking it up with my bank.

And in a lot of cases the bank would say Fuck you as with the increasing number of frauds committed in the UK where crims impersonate a house buyers solicitor and steal large sums of monney


If the bank does say fuck you, I'd file a complaint. If they still say fuck you, I'll reach out to the financial ombudsman service. And if they still say fuck you after that, I'll take them to small claims court.

What do you expect me to do? Investigate the matter myself? Buy myself an unlicensed firearm and look for the hacker to get it back at gunpoint?

In the multiple cases mentioned above the bank says not our problem and only reacted when it was on the front page of the daily telegraph

That's not exactly hacking, I've had a similar problem (paid rent by bank transfer and the "landlord" just vanished). The bank said "you transferred the money yourself, there's nothing we can do, it's like paying cash". Which is ridiculous, but technically it's not the bank's fault :/

I take it you're one of those folks who doesn't care for the whole "innocent until proven guilty" concept.

We don't even have a name yet so we are talking about whomever did it. The unnamed guilty party is guilty by definition.

yes, the whole thing is a fiction

You'd be wrong. You may have noticed I qualified my statements with "if the claims are true".

That the actions described in the claims are criminal is not in doubt. Whether those actions actually took place is unproven, as is whether this unnamed ex-administrator performed those actions.

So, I'm not ready to say that this person is a criminal, or that a crime was committed, but I am ready to say that the actions described in the claim are criminal and wrong.

The point of "innocent until proven guilty" is that, in these situations, we don't argue from an assumption that the claims are truthfull.

Actually, we talk about what is and isn't a crime all the time. That's how the laws are made in the first place. ;-) We also say that people are alleged to have committed a crime. When you charge someone with a crime, you don't necessarily have definitive proof that they committed a crime. That's potentially established in a court of law.

Finally, all of that is applicable to the courts & government determining criminal acts. The courts don't determine what is moral or ethical. Lots of things are wrong that aren't criminal. Saying something particular behaviour wrong really doesn't have anything to do "innocent until proven guilty".

Saying that, "if someone did what the story said they did, it's wrong" is really not saying anyone is guilty of anything.

Trying to think of how to summarize this better. We can have right and wrong without having any guilty parties. Saying that murder isn't a crime isn't the same as saying that anyone was murdered or that a particular person committed the crime.

Who defines criminal behaviour? And why is criminal behaviour always placed on the little guy when companies can always get the leeway of claiming ignorance?

Verelox is certainly not a major player in this space - aren't they little guys?

How many little shops rely on Verelox - aren't they little guys?

How many end customers depend on services that depend on Verelox - you know someone like you - aren't they the little guys?

"Who defines criminal behaviour?"

The police/prosecuting agency in the country where the company is based I imagine. That appears to be Holland [1]

[1] https://www.ripe.net/membership/indices/data/us.verelox.html

The fact that it was an ex-administrator does suggest foul play as opposed to a mistake compounded with poor backup processes. We shall have to see how the story pans out.

Netherlands and no, the police doesn't define what is criminal - the PARLIAMENT, ie the law-creating body, does.

I should have been clearer: The police/prosecuting authorities will decide if the behaviour in this case was criminal and if there is enough evidence to bring charges under the law - the law(s) in question having been passed by Dutch Parliament. Thanks

> Who defines criminal behaviour?

Sigh. I can't believe we're going there on this one. Just so we're clear, you think that maybe the definition of criminal behaviour shouldn't include wiping out systems without the owner's consent?

> And why is criminal behaviour always placed on the little guy when companies can always get the leeway of claiming ignorance?

I don't know what you are talking about. Criminal behaviour is placed on the criminal... and ignorance of the law is not a defense.

> Who defines criminal behavior?

We do.

The hell you do.

actually.. no. we do. We do it by proxy via our elected officials but our elected officials are supposed to follow the wishes of their constituents. They're the ones who create the laws that define criminal behavior. The courts use and refine those laws... but ultimately, yes the definition of criminal behavior is the result of the will of the populace.

Employment is a voluntary transaction, entered into by both parties. One should always focus upon increasing their own skill set such that an employer may be able to make your life temporarily difficult, but your skills and value to an employer (or future employer) are such that they can't "ruin your life".

Employment is a relation of subordination, inherently unbalanced. People also happen to need money to live, making work not a voluntary transaction, unless you consider death a viable option. That is not considering other factors, such as having to feed other members of your family. It is made even more unbalanced by the amount of influence your employer can have in your life by firing you, as opposed to the influence you can have on him by quitting. Employment is coercion.

I wish HNers would get their heads out of their tech arses and face the reality. Most people do _not_ have a choice where they work. Most people cannot afford to quit. Most people don't have jobs where they can increase their skillsets. Most people do not have time to read HN while they enjoy their 10 AM pause in their comfy sofa while working from home. You are, for the most part, amongst the most privileged people in the world. Yet you continue to spew the "employment is voluntary" propanganda because you never had to face actual hardships. Worse, you push people of lower classes even further in the ground, when what you should be doing is elevating them to make a better and fairer society.

People don't need money to live - they need food and shelter that is typically provided by someone else's work. Find a way to pay them directly and you don't need to work for an asshole boss anymore. The problem is, a lot of people are in a situation where they don't have that much to offer.

Employment isn't voluntartly: there is strong economic coercion to be employed and high degrees of friction are introduced to make that coercion more effective.

Urbanization fundamentally has been engineered to create "wage slaves", who are basically modern sharecroppers (though the system has some extra steps).

> Urbanization fundamentally has been engineered to create "wage slaves"

Suppose someone was born out in the forest, and there were no cities, no societies, no technology. Would you call that person a sustenance slave? Would you say that their foraging was involuntary?

>> Urbanization fundamentally has been engineered to create "wage slaves"

> Suppose someone was born out in the forest, and there were no cities, no societies, no technology. Would you call that person a sustenance slave? Would you say that their foraging was involuntary?

No, they are not a slave because they have no master except nature.

A wage slave has at least one master: his boss and the system that forces him to work for one.

Foraging (or farming, hunting, etc) is involuntary. Nature has compulsive power via biological needs and imperatives.

The difference lies in the fundamental difference between natural law and synthetic human systems. Entropy isn't your master, it's a fact of your own existence.

By contrast, the present economic system is a synthetic construct made by other humans to exploit you, designed with largely that purpose, co-opting your biological needs for coercive power, and doesn't necessarily need to happen.

That's what makes one slavery and the other not.

The difference is so obvious, I actually am confused why you think conflating physics with political norms was appropriate.

If the employment market is equivalent to slavery, it not obviously so. Saying it was designed for our exploitation implies a level of conspiracy and central control that I find dubious.

They don't have a boss, they can't be fired, they don't generate wealth that someone else keeps. On the other side there are huge material benefits to coordinating with many people, even if that coordination is done via a system that exploits you. Not a good comparison.

Very insightful question. This gives me an alternative context in which to frame the "wage slave" concept. Thank you.

Also for context: Just because an alternative is worse or "just as bad" doesn't mean that the status quo is okay. The United States is fucked up in many ways, regardless of the fact that Somalia has had it much worse.

depends if their mom and pop sent them to work the fields. There's many such communities throughout the world. Some even keep real slaves. (particularily in Saudi Arabia)

Most of society (and the wage slave concept) is about human farming.

would you say you are a farmer or livestock?

the distinction is that there is a willful effort to exploit and to enslave on the part of someone (be it a person, corporation or market

> the distinction is that there is a willful effort to exploit and to enslave on the part of someone (be it a person, corporation or market

That's right, and it's a shame it's been downvoted.

Being employed isn't voluntary because there are finite resources available to humanity and no one wants to waste those resources supporting a lazy person who doesn't contribute anything.

Being employed at a particular organization is, in fact, voluntary. You are not obligated to stay at one employer forever.

> Being employed isn't voluntary because there are finite resources available to humanity and no one wants to waste those resources supporting a lazy person who doesn't contribute anything.

Don't say "no one".

Productivity keeps rising. It becomes possible to support non-workers with a smaller and smaller fraction of that. And thorough safety nets help non-lazy people too. They even help people create their own businesses!

This is much like my response to the old employer gambit, "you say you find your job very fulfilling so why do you want a pay raise?"

I do my job because I love it. I do my job for you because you pay me more than the next guy.

What a nice false dichotomy, we are free as in free to chose our salve master, and there are only 2 possible states of humanity, to be a slave or to be lazy / a useless eater .. lets not talk about the slave masters and the hollywood-production american dream(tm) illusion that _every slave_ can make it to the top (which, for everyone who actually checks is just what it is, propaganda) .. I want to applaud the institution that managed to wipe..ehm teach you such a illuminating world view, very, very .. convenient

Calling a situation in which an employer trades you money for your time and effort "slave labor" is a disgusting metaphor and an affront to actual slaves. You should be ashamed.

I enter into employment agreements knowing full well that it's a business transaction. The business and I have agreed to an amount of money (and other benefits) in exchange for my labor.

It's not about the American dream. It's not about "slave labor". It's about you give me money for services I provide. When either of us decides that deal isn't advantageous for us anymore, then we end that agreement.

Stop being edgy.

Do you make median income or less? (For a full-time worker, that's about 41,000/yr.)

I think most of HN is in the "baronet" class (say, 2-3x of median income) or higher, and doesn't necessarily have a good perspective on what most people (ie, those at or below median income) deal with.

Per capita income has flatlined for a decade (since 2008), after 60 years of steady growth. But that largely doesn't impact programmers, entrepreneurs, etc. Instead, it impacts the people on the bottom.

Just because the present system works for you doesn't mean it works for everyone.

It was always good to be king.

Vast majority of people are in jobs they didn't want getting paid very little. Many are working several jobs. This is all by design of corrupt politicians and self-serving elites in business who have forced higher productivity, cut jobs, and shifted almost all rewards to least-productive people on top. Usually barely-skilled, biased managers are deciding who gets hired, how they're treated, and why they're fired. This is often related to politics more than job performance.

So, overall, it looks more like a form of economic slavery than some cooperative, voluntary, win-win system. A good hint is that most of the wealth has moved into few hands who usually didn't work their way up into it from the bottom like the people producing it. The people producing it didn't want that arrangement either. They have no control, though, since the elites outspend all of them combined on politicians passing these laws, media shaping perspectives, and public education limiting their intellectual growth while simultaneously reinforcing submission to authority (esp arbitrary requirements) daily for 12+ years. Most of America is born into a new form of slavery that few ever escape.

> [..] since the elites outspend all of them combined on politicians [..]

"[the illusion of]"Democracy", "Organized religions", "history of our society", esp. the last 300 years or so", 3 of the most profitable lies in todays day and age. Its wrong to say that this "happens to be happening because A, B .. Z", this is "works-as-designed". A world viewed from the top has no nations, only powerful blood-lines, it has not morals only self interest, it has no humanity only contempt of the "human animals" they have inherited and extended control over. It's them, a new species, and us, the soon-to-be transformed, stripped of everything that could challenge them, rest of the world.

This is insightful. One test we can apply is to ask ourselves, "If I needed to stop working and quit my job permanently because <put compelling reason here, e.g. illness, old age>, could I do so without enduring a financial disaster leading to <something horrible, e.g. homelessness, relying on the kindness of strangers, suicide>?"

I suspect that, for many, if not most of us, the loss of the ability to earn income would be an unspeakably grave event. This puts to the lie that we have any real choice about employment, assuming we are one of the vast majority of people who is not financially independent.

> Calling a situation in which an employer trades you money for your time and effort "slave labor" is a disgusting metaphor and an affront to actual slaves

Incorrect. Throughout history there has been many societies where people would legally sell themselves into slavery (e.g. to pay a debt, under threat of being imprisoned) and even receive a meager salary.

Yet it's still slavery; you can look up the definition in a dictionary.

If you wouldn't [work for a "slave-master"] you'd end up homeless on the street, no option to enter university, change your qualification, travel and get some experience abroad, start a business, nothing, absolutely 0 social mobility. Todays slavery does not need chains, it only needs ideology "on the west"(of which direct application you are a great example of), and an unprecedented murderous machine abroad(which you have no idea about, not the slightest)

Real slaves would find your pedantic definition of "wage slaves" offensive.

No one has the perfect job, everyone thinks they should make more, have more benefits, more freedom. But everyone has choices. You don't have to participate in the rat race, you can stop letting material possessions drive your lifestyle decisions.

Everyone has choices, it's true.

The choices that are available, however, have an extreme variance from person to person. At the extremely shitty end, Viktor Frankl, while a captive of a Nazi death camp, worked out that when all other choices are taken from a person, the only remaining one is to die with dignity.

But it's a choice, right?

At the other end of the spectrum, choices may include where to vacation this winter or if the maid should be fired.

The bottom line is that the quality and quantity of choices vary so much, depending on luck and pluck, that it's a bit glib to say "everyone has choices", including, I guess, the choice between working a crappy, abusive, mean job or two to make ends meet, or starve.

"It could be worse" fallacy.

I don't think everyone has the luxury to philosophize about giving up material possessions and exiting the rat race, either. A lot of people are living from paycheck to paycheck (or worse), have dependents, and don't have the prospects to find a new job. Those people can't afford to say "no" to their boss' unreasonable demands.

Depends on the slave, but I call bullshit on "don't have to participate". If I don't participate, in 3-4 months I'd be homeless and with nothing to eat. What gives?

I didn't say you didn't have to work.

I said you didn't have to participate in the rat race.

For most of human history, choosing not to work for 3 months meant starvation and death. Hunter-gatherers had to work.

My brother is 50 years old and has delivered pizzas for about the last 20 years. He only works enough to afford to rent a room, own a car, and pay for an annual ski pass, and skiing/volleyball gear. He spends his free time on the mountain and the lake.

He obviously doesn't own many material possessions and his retirement isn't going to be pretty. But that choice was his, he had opportunities to work corporate and make far more money, own more stuff, and maybe raise a family, but he chose to have more freedom and the lifestyle he preferred.

> Why are we so quick to side with the employer in this matter when we know nothing of the situation at all?

It's called collateral damage. There is no context, outside of fantasy, where the admin could be in the right to do this.

Eh, I'm inclined to agree with you that the ex admin isn't "right" but the essence of my statement is that he may have been carrying out what was his optimal strategy.

If you treat people bad, they'll treat you bad. If you nuke Russia, Russia will nuke you. It's the Nash Equillibrium where each party is faced with a game and certain situations call for your best move. But your best move should account for what I'll do, and that should be factored into your initial move.

Mutually assured destruction is actually a powerful deterrent. Do we know what the ex-admin's situation was prior to all this? Only then should we pass judgement.

Morally, I'd say the ex-employee doesn't have a leg to stand on. If the employee had been wronged and their actions only negatively impacted the employer, then I'd be sympathetic, but all of Verelox's customers have been damaged, some of them perhaps irreparably, by this. Getting back at your employer with such huge amounts of collateral damage is never acceptable.

I find it hugely improbably that ex-admin's situation entailed being wronged by all of the customers whom he impacted by this action.

You are basically justifying the 'killing of innocents' here..

If we assume that the firm did awful things to the employee, we can't also consider those who employed the firm to do those awful things "innocent".

What if the customers' data was preserved, archived, and sent to them as links? And just the VPS was trashed? Now, we're talking a brief disruption instead of total loss which the customers indirectly signed up for by going with a cheap, small player instead of one qualified to provide higher availability and security.

The collateral damage would low to non-existent for the users depending on how critical the service was. The direct damage would be to the employer. If the employer was the bad guy (hypothetically), would you still state there's no context or any situation whatsoever where the employee should cause them damage?


If their employer was a James Bond level super-villain about to shortly kill millions you could construct this scenario.

Otherwise there is no scenario where inflicting what you term a "brief disruption" (but what they'd likely describe as an awful, painful and expensive) to thousands of innocent customers is justified. Customer costs could easily be millions, if 1,000 customers each spend over ten hours of worker time dealing with that disruption.

If the employer did something illegal, report them to the authorities. If they wronged you, other employees, or customers, take them to court and publicize their malfeasance.

If you can't do any of those things because what the company did wasn't wrong or illegal, and instead wipe their servers, you are a sad little loser who can't handle the fact that they just didn't like you, and for good reason.

Man, your comment is so far off base and patronizing, that I think you're the sad loser here.

The ex-admin in question probably understood the impending customer harm. But the customers are the veins of a company- without them, they're not a company at all. The ex-admin just played dirty- something that he thought was the only way to win a lopsided battle against a force much greater than he.

In David V Goliath, Goliath was taken down... with a slingshot. A weapon. And he's celebrated as someone over coming a stacked challenge. The customers were the ex-admins slingshot. The difference is that the law doesn't allow this behavior

"Otherwise there is no scenario where inflicting what you term a "brief disruption" (but what they'd likely describe as an awful, painful and expensive) to thousands of innocent customers is justified. Customer costs could easily be millions, if 1,000 customers each spend over ten hours of worker time dealing with that disruption."

That same logic would make it unjustified for them to be using such a small, unproven service to begin with. I mean, the first thing I do for mission-critical stuff is to see if the hardware, software, or service has supported long periods of uptime with easy maintenance and security patches. Also, has anything really bad like preventable breaches happened? And how are the servers configured by default?

Further, if you're worried about downtime so much, you have two providers in a setup with replication plus failover. The only people responsible for service going down and data destroyed completely in a world where basic HA is cheap are the customers. They should assume some shit could happen. They should mitigate it if it matters. Those that didn't took the risk willingly. Backups in particular are also really cheap these days.

"If the employer did something illegal, report them to the authorities. "

People have reported all kinds of big companies to the authorities for breaking the law. Goldman Sachs nearly destroyed the financial system. They got criminal immunity + $1 trillion from the government whose Treasury was run by their ex-CEO who profited off that activity. They and most of the rest like them still in business mostly without anyone doing time on the top. What's your next move for punishment if law doesn't care or is receiving bribes (esp Congress)?

"If you can't do any of those things because what the company did wasn't wrong or illegal"

Wrong and illegal are two different things. Slavery was legal but wrong. Locking child workers in buildings that might catch fire to force overtime was horribly wrong but legal. Civil forfeiture... taking an innocent person's money or property w/out charges... is wrong but legal. All kinds of abuse of employees, esp regarding promotions or references, is harmful to all but people on top and legal.

You're clear that no illegal action should be taken in response to a wrong. I'm guessing you oppose the underground railroads that freed slaves since they were illegal. You would have griped about it at best while all the harm continued to those people if working within the legal framework.

I don't know man. I can't see any situation where these actions are justifiable. Revenge generally helps no one.

For what it's worth, revenge doesn't require a net benefit to the situation. The word "revenge" is kind of similar to "retaliation" where an actor is purely seeking retribution for a past event without regard.

To us, it's not justifiable. Hell, the ex-admin may think so too. But that doesn't mean we should automatically side with the employer and subsequently crucify the ex-admin. We don't have any information yet.

How do we even know the story is at all truthful? Maybe someone who still works there did something stupid they couldn't recover from and is now blaming it on "an ex-admin" to make it sound nice.

Is your first assumption always that the victim made up a story to cover up their own misbehavior?

Is your first assumption that people only post their first assumptions? WTF dude. There are plenty of people in this thread already ready to tar and feather a completely unknown person. I'm just trying to point out that we don't actually know anything about what is going on.

Free speech: telling your boss off, quitting, organizing a walk out, refusal to work, picketing, posting your story online to warn other potential employees and customers just who their doing business with

Not free speech: punching your boss in the face, burning the building down, vandalizing the office, deleting all the data on the servers and hurting a bunch of people that had nothing to do with your conflict with management

I'm all about workers right and have walked out on many a job because management were raging assholes. I have convictions and I stand by them but conviction and doing the right thing in life often come with sacrifices. I'm currently underemployed because of my convictions but I can sleep at night knowing I'm doing the right thing. The way for us to take back power in the tech industry is to organize, unionize and refuse to work for abusive and exploitative employers. We're the ones with the skills. They need us not vise versa.

Mental gymnastics.


Wow. That just shows just how reliant I've become of autocorrect. It didn't autocorrect, nor did it present a red underline, and I had a feeling "Speach" looked wrong. But I doubted myself and figured autocorrect would have my back.

I agree with you. The majority of people in this thread have completely internalized that property rights are more valuable than human rights. Only one comment I've seen has acknowledged that all we have is a single webpage with a single accusation, and yet many people leap straight to the conclusion not only that the individual in question exists and is guilty, but that there can be no justification for a person destroying a business. Businesses destroy people all the time. I for one would like to see the tables turned more often.

" I for one would like to see the tables turned more often."

I don't know you.

But what I have learned that people who do not care about other peoples property, care very much about their own. Destroy other people business? Go for it! Destroy my car which I could kill someone with? No way, it's mine, I've worked hard for it. Vandalize houses of rich people? Go for it! Steal my iPhone? Hey, where is the police when you need them?

In Berlin people cheer the burning of other peoples BMWs - yes this is a thing. The same people go to court when the police scratches their table tennis table during a raid.

We have responded to the article based on the information presented.

There is nothing to imply human rights have been violated. If such information is presented im sure people will respond in kind.

> We have responded to the article based on the information presented.

> There is nothing to imply human rights have been violated. If such information is presented im sure people will respond in kind.

What are you talking about? There is no "article," only the statement of one involved party. It's right to be skeptical and theorize about what else might have gone on that was omitted from that statement out of self-interest.

>I for one would like to see the tables turned more often.

The most prominent example of this in recent memory is Peter Thiel sponsoring Hulk Hogan's lawsuit against Gawker. Do you approve of what he did?

HN is full of baronets arguing in favor of the modern social order -- because while they're not real aristocrats, they're not peasants.

Corporations function as de facto titles of nobility in the US, so crossing one is crossing social rank.

> HN is full of baronets arguing in favor of the modern social order -- because while they're not real aristocrats, they're not peasants.

I think that's very on point. While HN is a technology site, it's also one for "founders" * who aspire join the elite of that social order, which explains the exaggerated empathy for the interests of shareholders and companies.

* (and those with fantasies of being one)

you have to be "someone" to become "someone", even if you suddenly earned 30% of shares in facebook, it would not make you "part of the elite", there is only so high you can get without [the right] family background (see billy gates who's mother - the less interesting part of the picture - was co-chair at / IBM CEOs personal friend .. so much for the hollywood story of the "deal of the century" ft. young billy and DOS .. aaand FTCvIBM but the cast/story gets even more interesting later on

"Businesses destroy people all the time".

Lol. Citation please.

I'd like to see the justification for forcing thousands of customers into expensive disaster recovery because the company fired a sysadmin, likely for good reason.

So in addition to the criminal side of things I guess the ex-admin wants to work in manual labor or fast food. There is no way in hell he'd have the references or pass the background.

BTW, we had a netadmin interview a few months ago. Guy was really smart, aced the technical and group interview. We were really looking forward to hiring him, and only needed to pass a background and reference check. HR told us in no uncertain terms to run the other way. They didn't share what was in his check but it wasn't good.

>They didn't share what was in his check but it wasn't good.

Always ask to see the background check details if you're the hiring manager (you should have the rights to see that despite what HR might tell you). Could be just a personal issue an HR employee might have with a former colleague. Or discrimination-based (happens).

Reminds me one of my old bosses hired someone. And then HR fired them because they had a pot conviction. So then my boss hired them as an outside contractor. Every six months HR would discover he was on the 'no hire list' and terminate his contract and my boss would cut him a new contract.

Reminds me when I worked a big telcom (BT) on of the guys in our section was an ex teenage Phreak and had been done for hacking PRESTEL.

One day one of the feared internal security team who had caught him years ago was in the building and met him and freaked out - our centre mangers politely told him to f off when he demanded that we fire the guy.

Was not the manger involved in the hearing in most big companies its the mangers who make the firing decision with input from HR.

HR doesn't need to approve of contractors?

In a good company, HR has as little authority as possible.

Yeah let me guess this dude was white. Unbelievable, the world must be so different in that skin.

My boss was French.

Socially Adept Frenchman: 1 Human Resources: 0

I'm not sure what you're referring to. If you mean a formal background check, which summarizes your criminal history or your credit history, then candidates are entitled to copies of the specific report the company claims to rely on in taking an adverse action, but nothing else.

If you're referring to private reference checks, of the type that would surface "personal issues people might have with formal colleagues", you're not entitled to anything whatsoever.

If you're a hiring manager in an organization where HR handles background checks, you personally as the hiring manager are entitled to nothing. I would venture further that it's inappropriate for HR to provide criminal/credit background check information to hiring managers.

>you [as] the hiring manager are entitled to nothing

You mean in any kind of practical or legal sense or in your personal opinion?

I see this happen all the time and am not aware of any thing barring this in the U.S.

Think about the two extremes. If a startup of a few people run a background check there is no HR dept so the CEO/hiring manager is seeing it.

On the other hand if the hiring manager is the CEO at an F500 company trying to hire a handpicked rockstar exec and a background check pops there no way that person will be rejected only on the word of HR without understanding what the issue is and the context around it.

Somewhere in the middle of those extremes companies may have policies to address the situation, but I don't see how any blanket statement can be made here.

I asked the author what they meant, and then laid out what the facts are. It's simple:

* Candidates are entitled to copies of background/credit reports used to make adverse decisions against them, under the FCRA.

* Candidates are only entitled to the specific document, collected under the FCRA, that was used to make the adverse decision.

* Reference checks --- calls to previous employers and coworkers --- are not governed by the FCRA or, as far as I know, any other law. Candidates are absolutely not entitled to any information about reference checks, but a candidate that "flunks" a reference check might be told so when they're refused a job. Or they might not.

* No law entitles hiring managers to any documentation whatsoever.

Only the last point I made was one of opinion, but I think it's a widely held and pretty common sense opinion:

It is terribly inappropriate for HR to share background check documentation with hiring managers. Companies that employ background checks should have simple, static rules, like "no previously undisclosed felonies" or "no violations relevant to the job", that are evaluated by HR.

It's already a grave violation of candidate privacy to do these kinds of checks in the first place (which is why they have to collect special permission from candidates to do them at all!). It's negligent to then pass that information around the org chart to help others in the company read tea leaves from them.

My opinion here is informed by experience working with companies that do background checks, but I wouldn't be surprised to learn that there are fucked up companies where HR passes copies of credit and background reports to other employees.

> It's already a grave violation of candidate privacy to do these kinds of checks in the first place

True, but in general, it is likely that whatever a background check turns up on someone, such that they're unemployable, is probably available to anyone with a web browser and some time on their hands. The people in the org who want to satisfy their curiosity or sense of self importance as to why someone didn't meet the company's standards can probably find a way to do it.

The reason one really shouldn't what to see this info at all if they can help it (and should have a trustworthy HR office handle this stuff) is that pretty much everything done in the hiring process can be used later on in court. The safest possible position for a hiring manager if the question comes up is "HR indicated the candidate did not pass our background check and so we moved on." And only an idiot wants to answer questions later on about with whom they shared damaging information about a candidate.

Well, the thing is sharing background issues is not limited to rogue or obscure companies. For example if you search the archives of WSJ it comes up reliably at some well known places.

To be clear I don't think a lot of this stuff is right. It drives me nuts to see someone ace technical interviews and then not be hired for some insignificant drug related issue. How do companies not realize this probably hurts them more than the candidate? A lot of tech companies don't do drug tests at all, I'm guessing because they've figured this part out.

Its not the law here HR is working for you as the hiring manger so dam right I want to see the docs - and BTW I have more experience in IR/HR issues than many junior HR.

HR is there to serve the organization, not the other way around.

If you are trying to hire a talented employee, and HR says no way, you better demand they justify it. It's your job to advocate for people on your team, even when they haven't come on board yet. Otherwise no one will.

It's far to easy for HR to auto-reject because of hard-wired criteria, they have no incentive to challenge their own rules for a good candidate.

I know a kid who once was a chip-runner at a casino. The way the job worked was he took cash/chips from poker players and went to the cage to get them chips/cash. He got paid minimum wage plus tips, and was on the hook if he got shorted. But the plus was he'd get $1 tip for each transaction, sometimes more.

But this kid optimized the whole job. He hustled like crazy, minimized cage trips by carrying 7 or 8 racks at a time, and even took optimal routes through the poker room. Watching him work was inspiring, the kid was so driven.

He was making a ton of money, far more than the other chip-runners, and someone ended up complaining. So the room decided chip-runners should pool all tips together. He quit in a rage the next day and chip-running service in the room became awful, the remaining chip-runners now saunter at their own pace, carry a single rack at a time and minimize their risk by slowly counting everything three times.

A couple years later the kid got busted for selling weed out of his house. He ended up serving a few years in prison. He's going to now fail any HR background check/screen. I'd still hire him in a fucking second, and expect him to rapidly work his way up from any entry level role.

One of the reasons I doubt any sane HR department would hand a hiring manager the results of something like a criminal background check is precisely to protect the organization.

Didn't know I could do that.

I've had my own background checks in the past and no one has shared them with me. Speaking to HR, unless there is a negative item on your record they aren't required to.

In EU even something like recruiter scribbling notes over your printed CV during job interview automatically grants you right to look into them as it is considered personal data processing.

If the background check was done on you, I believe they have to provide a copy of all the data they received if you request it.

At least at the federal level, no they don't. Unless they decided not to hire you (or fired you afterwards). Then they only have to provide "a notice that includes a copy of the consumer report you relied on to make your decision" - https://www.eeoc.gov/eeoc/publications/background_checks_emp...

I've requested it, just for my own amusement. They told me no. Unless they didn't hire me based on something from that background check they didn't have to give it to you.

Depends on the state.

> Didn't know I could do that.


Why would you assume you couldn't?

Discrimination is exactly why you want to leave that decision in the hands of HR. In the event of a lawsuit they'll have evidence to show that they followed the process your lawyers signed off upon and the decision to say "no hire" was based solely on permissible factors. At large companies, this will go as far as "here's a database dump of all the applicants with X on their criminal record... as you can see, we didn't hire any of them".

As soon as individual managers want to make those decisions for themselves, it becomes much harder to refute claims of discrimination.

That's a terrible policy. You need to optimize for getting the best employees, not minimizing discrimination complaints. How often is a felon going to sue because the company once made an exception for a specific felon because of specific outside factors, and how can that cost you any suit?

"Your honor, my client, the recently released child rapist with an iffy work record, is suing because BigCorp says they normally don't hire felons, but a few years ago BigCorp made an exception for a felon who sold marijuana, got his college degree in prison, then after he was released worked at non-profits helping the poor while getting an advanced degree in computer technology and writing several highly regarded academic papers on improving user interfaces for the disabled. That's not fair!"

The specific facts matter, and HR should be fired if they have blanket polices that ignore them. HR needs to serve the business, not the other way round.

I had a manager who was consistently underpaying me, significantly. When I quit in disgust, he threatened me that he'd make sure I never got another job. He had been milking my success to get lots of bonuses and raises for himself, even though I did absolutely all of the work and he just "managed" (which meant, ignored me completely). So he wasn't happy that I was quitting, and though he could bully me. Luckily for me, I didn't have to use him as a reference, as 2 other PMs agreed to vouch for me.

So, just because a guy has a bad reference it does not mean that he's necessarily bad.

"So, just because a guy has a bad reference it does not mean that he's necessarily bad."

This is so true. Years ago I was brought into a company specifically to improve their software quality process, but without being aware one of the company owners (of which there was two) was against be being employed for that purpose. I uncovered a lot of incompetence and outright corruption with some employees. The 'good' owner went on stress leave, and then pressure was brought to bear on me, resulting in me quitting on the spot one Monday morning; not my proudest moment, but I couldn't take the pressure any more, and my ally was no-where to be seen. Fast forward a decade, and I interviewed with someone who used to work at the same company; it turns out that my leaving had been framed as 'fired for incompetence', and word had been put around part of the local industry that I was hard to work with, unreliable, bad at my job etc. I laugh about it now, but at the time it really bothered me about the possible damage done to my career and reputation.

First, I'm not a lawyer and this is not legal advice, just my own limited knowledge and experience.

I don't know what country you're in so I won't speculate on what's legal or not in your area, but here in the U.S. it's illegal for a prior employer to provide false information to a prospective new employer during a background/reference check. If your former boss tried to blacklist you like that, he'd put his company at risk for a civil suit. And, while it's not illegal to truthfully say that a current or former employee is a bad employee, doing so rides the thin line of opening the company up to a libel lawsuit.

Generally speaking, a company might "dish the dirt" on a former employee if there are criminal charges to back up the claims. Even then, legal and HR will likely frown upon it. Usually, when the new company calls the old company for a reference, the old company will say something like "Yes, $employee worked here from $startDate to $endDate" and refuse to divulge any other information in order to avoid any semblance of libel.

Once again, this is my limited experience in SMB and government settings, I have no experience in mid-to-large businesses, Fortune 500 companies, and Silicon Valley startups. We've all heard stories about managers at such companies going around HR and discussing potential hires at the bar or on the golf course.

I am actually in the US. On paper, suing the ex-employer sounds nice; but who's going to put bread on the table while the lawsuit works its way through the courts? Who'll pay the lawyer? And how can an individual compete against a company with billions in revenue??

> I guess the ex-admin wants to work in manual labor or fast food

You'd be surprised. I did something somewhat similar and was convicted of a federal felony. No fast food place or retailer would touch me with that record. But ironically I've found plenty of IT work with smaller companies.

Ironically, we are much laxer on those with more responsibility. When was the last time a middle manager or senior executive had to pass a piss test to be employed? It's routine in blue collar work.

That's a vertical thing. I've had two financial services employers require a drug test.

Do they fire you if you're not testing positive for cocaine?

Not in Wall St for sure. Or they make sure the "right people" know it's coming ahead of time so they can buy the right stuff to cheat the tests. Plenty of drug use by managers in a lot of organizations I've been involved in. None were ever fired for it. A few were fired for the damaging behavior that resulted from it.

You missed a word in jon-wood's comment (emphasis added):

> Do they fire you if you're not testing positive for cocaine?

Glossed over it. Good catch. Interesting enough, my post still is pretty accurate reply if you drop the "not" at the beginning and look at it from who gets ahead in their special clubs.

> HR told us in no uncertain terms to run the other way. They didn't share what was in his check but it wasn't good.

I always have a hard time with those contexts. HR sometimes has the wrong idea of what is unacceptable and what is appropriate. There really isn't a good reason for them not to tell you. The only thing that would make sense is if they shouldn't know it.

He had the references to get hired at this job, right? So just leave the job you screwed off the resume.

A similar thing happened to a client. Sysadmin logged into GCP and Azure immediately after termination and just deleted everything. He was in the UK, we were in the US. Wasn't worth it to try to get someone to prosecute, and I'm sure we're not listed as references.

Did get people motivated for a multifactor delete bucket for extra backups.

We hired a guy once who, a few weeks after starting, had to take three weeks off to serve jail time for his 3rd DUI. Good times.

I had a retail customer once (I sold pc stuff to SMBs in college) who made a big purchase via credit card. The cashier got a weird code, and got transferred to a person who asked for a supervisor (me), and then asked yes/no questions about the physical description of the guy. Then they gave me some code to clear the transaction.

I was checking out the Sunday paper a few days later (this was the 90s), and it turned out that the guy was a fugitive who basically killed his wife and fed her to the fishes.

Pretty freaky stuff.

A little bit off-topic, but stories like this lead me to wonder why mandated ignition locks are not more prevalent.

I once saw a pitch for Google VC by a company selling mandated breathalyzer ignition locks - and insurance. The founders were a criminal lawyer and an insurance salesman (brothers). Their angle was to get state legislation passed, and win per-state government monopolies on the interlocks - and then cross-market auto insurance to drivers who had to buy the interlocks, because most carriers won't insure drivers after a DUI.

It was the creepiest, most cynical thing I've ever seen.

Those guys are gonna be RICH.

That's interesting. You can read it a few ways, one sounds like a company tailored to the problems of being a post-DUI offender. If the traditional systems fail you and you need a vehicle for transportation this might be a solution. The other is a company that has a captive market and squeezes them like the rest of the justice economy.

It's not really two ways of reading an action. It's two separate actions. One is providing insurance to people with interlocks. The other is using a monopoly to promote that insurance.

In oligopoly form, it's exactly what the car insurance companies did paying off legislators to require their product, setting the standards for how the insurance is supposed to be done, selling that now-mandated product at nice profits, and then municipalities and states making nice profits off the tickets and fines (esp off poor people).

And then we add a mandatory breathalyzer check to the remote VPN login. Because if people believe they can still drive, they might just believe they can still answer the pager.

YOLO is not really a phrase you should use on-call. :)

Every time I've seen them discussed, I see brought up that they're not especially reliable.

Denying transport to someone who's got a history of doing anti-social things behind the wheel seems reasonable. But making them required for all vehicles (if I'm understanding your intention correctly) seems likely to have undesirable results.

FWIW Toyota had a drunkenness-sensing steering wheel at least in concept for ~10 years ago: http://www.autoblog.com/2007/01/03/toyotas-sweat-sensing-ste...

Breath tests are notoriously unreliable/inaccurate, to such a degree that in some jurisdictions the law is now phrased entirely in terms of a machine reading since it's been established there's no reliable relationship between the machine reading and blood alcohol level.

Seriously, read up on the history of breath-test machines sometime. Everything from difficult chemistry to literal "oops we misplaced a decimal point in the software".

Not for evidential breath testers used here by police in NZ. The blood tests always back up the breath test, and have done so for years.

They're very expensive and they're not especially reliable.

Like, stupid expensive. Kids, don't ever put yourself in that situation. Call an uber/lyft/whatever.

Yeah, it's a seller's market. If you get a DUI and have to drive to get to work, you have to pay it. And the public is (understandably) largely unsympathetic to extraneous costs for drunk drivers.

There were a few guys at the bar I used to hang out at that had several DUIs, had the ignition lock, yet somehow seemed to get on the road after closing the bar.

"We hired a guy once who, a few weeks after starting, had to take three weeks off to serve jail time for his 3rd DUI'

You flunked your own background check.

Is this deep background check again a US thing? In Germany it's even unlawful to ask potential employees if they are pregnant. It's essentially discrimination. Maybe some HR will check your social media accounts and search on Google for you but generally police records (at least my experience in IT) or even health records are not requested (or against law!) unless it's a high responsive job like e.g. becoming a police officer or maybe military.

Asking potential employees if they are pregnant, married, plan to become pregnant, have children already, etc... is illegal under US law as well. Common German practices like asking for a photograph on an application or CV or asking about a prospective employee's religion are also usually illegal in the US, as it's pretty hard to think of non-discriminatory reasons an employer would need that information.

Asking previous employers about their experience with an employee, however is not illegal. It's usually not illegal for them to say something negative if it's true, though some businesses are conservative about what they will say out of fear of being sued for slander. Accessing public court records or news stories about criminal cases and using that information for employment purposes is usually not illegal. Asking prospective employees if they've been convicted of serious crimes is usually not illegal.

Who asks for a prospective employee's religion in Germany? Apart from church affiliated positions this is illegal here as well.

I think I may have confused it with Ireland or Northern Ireland where there's a legal mandate to keep track of it for anti-discrimination reporting.

It does seem to be conventional to include one's photograph, age, sex, marital status, children or lack thereof and place of birth on a CV in Germany. It's not clear to me if it's actually problematic to exclude those, but it's my impression that not following conventions tends to not go over especially well in Germany.

I even remember learning to include my parent's jobs on a CV but thankfully that is gone for good. The things you listed are still common with variations depending on the industry. More often than not you can exclude them without negative effects but sometimes those are still seen as important information.

And many responsible HR people don't look at peoples sm accounts for the same reason they dnt ask for a photograph

What are you talking about? At all (german) companies I ever worked at is was common to request a "Polizeiliches Führungszeugnis" (essentially police records or the lack of such) first thing during the trial period. This is absolutely common, maybe you just forgot about that?

Not sure at which companies you worked at but outside certain fields it is not common and actually illegal as the Führungszeugnis may contain entries irrelevant to the job. Exceptions include working with children/youth, public officials, security guards, insurance, and some positions in the field of logistics.

Common?? Maybe it is more and more common but it never happened to me in my professional career.

I've encountered many companies that are scared of lawsuits from bad references so responses, mandated by HR, are often limited to employment dates, and other very generic information. The mentality is, that person isn't our problem anymore, let someone else deal with them.

On this case the person's problem is law enforcement. Basically they just need to pass off details to the FBI / local police and have the ex-admin arrested and put on trial. What he did was a crime

If a company walks away from this and doesn't take legal action; they should themselves get sued by their customers.

Destroying company property is a crime; wether it's defacing a website or ... it's not your property; you are just hired to maintain it (in one shape or another)

They are based in The Hague Netherlands, so they don't want call the FBI for sure.

Why didn't you ask to see details from HR? There's no transparency in their decision making - how do you know he was turned away for a good reason?

HR's view point is that the candidate has a right to privacy and their personal problems are none of the hiring manager's business.

Any HR manager with that viewpoint should be fired on the spot. They work for the hiring manager. The hiring manager is the only advocate the candidate has, certainly not HR which will only cover their own ass.

Background checks don't stop things like this. The problem is you have to hire trustworthy people in these positions.

Regardless of what crime they may (or may not) have made; you are looking for the person and what they do. Not what they did.

This is a common problem; we look at the past a bias of the future. Life only works out that way if the person is too unwilling to change; and that again is something you should look for in the hiring process.

Lastly; hiring ex-hackers isn't a bad thing. Caring about a background check when hiring an ethical hacker or someone who turned their life around; only shames them and pushes them back where they came.

So be careful or you only end up criminalizing being a criminal.

Sorry. If someone intentionally destroys company property I would want nothing to do with them, ever. This isn't a simple b&e where the perp rarely knows the victim. He was given the tools to do his job and intentionally used them to undermine his company and colleagues.

Also, he is not a hacker, just an asshole. Having the admin login and password doesn't make you elite. The only weakness he exploited was himself.

Being a criminal or commiting a crime has nothing to do with destroying company property. That's the problem with asking for people's background.

What they did; and more importantly have they recovered and not commited another crime of similar circumstances.

P.S. I knew a kid when I was younger who helped an ISP start up in Colorado Springs in the late 90's... he got fired and he hacked them and spent 6months in jail; eventually they re-hired him after he got out of jail only for a repeat offense.

There is a point; he did it twice. The first time they could pass it off as they didn't know; the second time they hired him they were legally liable.

So this gentleman should spend some time in jail for what he did.

The problem is you have to hire trustworthy people in these positions

Yeah, that. Also, secure backups and compartmentalized systems and data access.

Secure backups where the admin doesn't have access to. As someone who runs servers for a living I don't have a problem with this.

Automated tools may need to delete and list; but heck off site secure backups are an amazing thing.

Wayback Machine link if you want to know who they were:


Wow, they were a VPS hosting provider. Lots of unhappy customers (and this likely qualifies as a data breach for all of them as well).

Yeah, that could become legally awful. Which is appropriate; it appears $admin ruined a lot of people's days.

I suspect that person will soon have a fair amount of time to decide on what new career to pursue to pay down the fines when they get out...

I suspect they assume it was an ex-admin probably because of timing. There may not be any proof to back up that claim and thus there may be no repercussions.

Until they release specifics if ever, it's hard to know what to make of the status message.

They should be able to use https://github.com/hartator/wayback-machine-downloader and get at least a static version of the website back online.

If a VPS provider didn't manage to have even an offline backup of their own website they should dissolve the company.

If it was an ex-admin he might have deleted all the backups also. But they should be able to tell this publicly, otherwise it's similar to recent case where the newbie deleted the prod DB without any backup.

"If it was an ex-admin he might have deleted all the backups also."

One of the very neat things about rsync.net is that your account is on a ZFS platform and you have snapshots enabled by default and the snapshots are totally immutable.

Which means that if you back up your VPS (or your VPS company) to an rsync.net account and someone owns you and owns your rsync.net credentials, the worst they can do is delete the very latest backup ... the snapshots cannot be altered.

Just saying.

So you're telling me if I have a git server (for example) on rsync.net and a newbie developer accidentally commits sensitive credentials there's no way to remove them from the git database? That's very troubling.

I think at that point you should just rotate the credentials rather than trying to erase history.

You misunderstand on several levels here ...

First, rsync.net is "cloud storage for offsite backup" - you can't run a git server[1] (or anything else) there. It's not a VPS or a web host. It's a remote unix (ZFS) filesystem that you can access over SSH.

The other point you are missing is that the ZFS snapshots I refer to are immutable as far as you are concerned. Of course we can remove them[2], and could do so at your (vetted, verified) request. Further, we don't have unlimited disk space so the snapshots rotate out (expire) over time. Every day the 7th one is removed to make room for the new "yesterday" snapshot, and so on.

The point is, an attacker can gain full access to your backups with all of the control you have ever had over them and they can't destroy/delete the snapshots. That would have helped the victim in this story immensely.

[1] You can, however, put git repos there and interact with them, using git, over ssh.

[2] Although it requires root and is an involved, manual process - which is good.

Great explanation. Aside from using standard tools, that's the best benefit I've ever seen for your business. I rank it as best since my specialty in INFOSEC is high-strength attackers and subversion (eg malicious admins). There's rarely 3rd-party services designed to technologically counter the latter. Your solution is straight-forward, too.

One concern I have is right here:

"your (vetted, verified) request."

What's that mean specifically? There's potential for attacks there. For instance, a request from several email addresses might come from computers the admin controls. Same with some 2FA's. One would have to be careful here. I got a voice idea that just passed through my head that could leverage their smart (or dumb) phones. Also maybe dedicated tokens, apps on their phone or home computer, or something that come from your company. I'm curious what you're already doing, though.

"What's that mean specifically?"

It means that the owner and founder of rsync.net stares at your request and decides how he feels about it. Then he weighs the financial security of his family and the reputation of the business firm that has become (over these last 16 years) his life's work ... and decides if one of the engineers should call you on the phone and vet your request just a little bit more ...

Aligned interests and "skin in the game" ... those are powerful things.

The intuition of a qualified, incentivized reviewer that optionally will call a person whose voice or knowledge are known a prior. Stronger than some and weaker than others.

From my understanding of the new EU General Data Protection Rules (GDPR), it will even make rsync.net unusable for most EU companies starting 2018.

They have a no-snapshots plan, which I am using.

Or, say, one gets sued over information that you're not supposed to have and a court orders you to delete all copies. How do you comply with that?

The parent poster talked about an offline backup. That is the big idea of offline (and even better, offsite) backups, that you cannot delete or destroy them from a computer, just by destroying the physical media.

An ex admin would probably have access to the physical backups too, this could have been done before he left the company.

If he was still a full employee when he wiped the servers, then theoretically yes. It rather sounded to me that he wiped the machines after he left. And off-site backups are often handled by storage companies, making it more difficult to access them.

Isn't this the time where you put stuff on tape and store it somewhere outside of physical reach? Is tape even used anymore?

Yep, I'm not involved at all but I believe my workplace does its offsite backups on tape with a service like Iron Mountain http://www.ironmountain.com/Services/Data-Management/Tape-Va...

A patient disgruntled admin would have altered the backups to write garbage to the tapes for however long the rotation cycle is, then kicked off the production melee.

Damn, they are fucked. How could people trust them any more???

This is so stupid. If you have a problem with your employer, either you quit or they fire, you move on, full stop. If you're in a relationship and someone isn't happy enough with you and breaks up with you, the dignified response is NOT to key their car. I see employment relationships mostly the same way. Either it works (for both) or doesn't (for either or both ends).

And having switched jobs quite a few times, the next one is always better for you, regardless.

"I see employment relationships mostly the same way."

It's not an equal relationship. One side usually has significantly more power than the other.

What is "relative power" in the context of, say, an employer with excellent business acumen, a manager with excellent interpersonal acumen and an employee with excellent technical acumen?

The relative power is determined by who has the least to lose by walking away from a potential deal.

People assume that that is all about skills but it isn't necessarily. An employee with a serious debt problem and a mortgage payment due soon will be more likely to accept a bad deal than an equally skilled employee who doesn't.

LOL. You've obviously never had to try to hire good employees.

Oh, I have. The most eye opening part was realizing all of the host of variables which were factored into wage negotiations which had nothing to do with how good the employee was, including:

* The irrational biases of other companies (even if we didn't care about you being a university grad, our commensurately lower wage offering reflected the fact that other companies did).

* If the employee telegraphed an air of needing the job (e.g. people with a debts), they got offered commensurately less.

It's an ugly process, truth be told.

> I see employment relationships mostly the same way

Apparently they did not.

At the end of the day, the people working at the company are the ones who are doing the work, and who have control of the means of production. The ex-admin's bosses probably thought they were the important ones, and that this worker was a replacable cog, but they found out the hard way that this was not the case.

I worked at a Fortune 100 investment bank where this happened. Everyone knew layoffs were coming. One week after layoffs came, a digital "bomb" went off wrecking many servers. So security went through, trying to find evidence (nothing incriminating from what I heard, although they had a strong suspect) and also looking for more bombs. They missed out on finding and defusing one, because another one went off a month later.

The view from the pinnacle, people counting the dividends on the checks that they inherited is that they're the job creators, and everyone else is dispensable. This company just found out that is not the case.

Integrity is choosing to behave in the right way, even though you have the option of behaving otherwise.

Therefore, to behave with integrity, you must have formulated your own set of values about what is "the right way" to behave.

Every minute of every day, we all have the option to behave with or without integrity in a whole range of ways.

You earn respect by demonstrating behaviours over time where you have taken the interests of others into consideration, generally people consider someone who behaves like this to have "integrity", especially when they continue to behave that way when no-one is looking.

Saying things like "The ex-admin's bosses probably thought they were the important ones" indicates a childish set of values where there is a power struggle between employers and employees ........ of course the "bosses" are the important ones, they act for the business which is an independent legal entity, upon which many people depend for their lives to work effectively. If, as an employee, you feel poorly treated or otherwise dissatisfied, then the right thing to do is leave in a polite and respectful manner, even if you feel you were not treated in that way. Depending on the circumstances, if you were actually treated really badly, then the right thing to do is pursue your complaint through the appropriate legal channels.

Someone important in my life once said to me "the only thing you have is your reputation". Take that reputation, defend it, enhance it, nurture it and earn the respect to grow it. Don't throw it in the garbage by smashing other people (or their business) in a childish tantrum. I admit this is hard to do - I regret many things I have done in my life, but I try to lead a life consistent with my own sets of values that I think are meaningful and I get rid of people from my life who I think don't have integrity, or whose values are different from mine in critically important ways.

>You earn respect by demonstrating behaviours over time where you have taken the interests of others into consideration

The relationship in most companies is entirely asymmetrical. If times are tough, employees are expected to work unpaid overtime, to sacrifice on pay and perks, to accept layoffs. If times are good, shareholders and executives see all the profit. Employees are expected to show absolute loyalty, but are shown not one shred of loyalty in return. Productivity is soaring across the economy, but wages have been stagnant since the 1970s.

Most employers will never truly respect their employees, ever, under any circumstances. Employees aren't people, they're a "human resource", a cog in the corporate machine as interchangeable as any hardware. More so, in fact - a piece of machinery would be hired on a fixed-term lease, but most employees can be dismissed at will.

I don't endorse vandalism, but I think that it's utterly naive to expect that you can earn the respect or loyalty of corporate America. It doesn't matter how honest you are or how hard you work, you'll still be discarded like an oily rag if you're surplus to requirements. You'll still be lowballed on every pay rise while executives and shareholders make record earnings.

> Most employers will never truly respect their employees, ever, under any circumstances. Employees aren't people, they're a "human resource", a cog in the corporate machine as interchangeable as any hardware. More so, in fact - a piece of machinery would be hired on a fixed-term lease, but most employees can be dismissed at will.

> I don't endorse vandalism, but I think that it's utterly naive to expect that you can earn the respect or loyalty of corporate America. It doesn't matter how honest you are or how hard you work, you'll still be discarded like an oily rag if you're surplus to requirements. You'll still be lowballed on every pay rise while executives and shareholders make record earnings.

You are absolutely correct about corporate America, the executives, and the shareholders. The hard thing is that you can earn the respect and loyalty of the other cogs that you work with, which can be difficult to disentangle from that underlying truth.

You've never had to hire or manage anyone.

This is a highly normative opinion. Your version of integrity requires you to cow to a system that has been built for the purpose of your oppression. Your beliefs in following legal channels, turning the other cheek to abuse, and failing to defend your fellow man against dehumanization are the result of indoctrination by a system designed by the wealthy to keep themselves wealthy at the expense of the many.

I get rid of people from my life who I think don't have integrity, or whose values are different from mine in critically important ways.

On this, however, we agree entirely.

I'm not religious, but one of my ideologies is to do unto other as you would have them do unto you.

I think if everyone treated others the way that they would like to be treated, then the world would be a better place.

"I'm not religious, but one of my ideologies is to do unto other as you would have them do unto you."

That belief was created and pushed by religious, political, and business elites who themselves did not give equally to other people or treat them fairly. They always schemed out more for themselves. Your rule is best modified to do onto others as they would actually do onto you to the best of your knowledge. Otherwise, your rule will result in more evil happening overtime as the good people will work within the schemes created by the bad people. That's already happening.

Illustrated nicely in Hawk-Dove game:



Even includes retaliators which are relevant here.

The platinum rule, of course, is don't do unto others as you wouldn't have them do unto you.

Maybe that is too cynical, but it works well as a rule of thumb.

This is a very submissive approach to handling things. I'm not condoning the opposite- pure aggression, but it seems that you're holding what others think of you as something worth preserving, which is fine. But when you attach words like "integrity" and "reputation," it has this "holier than thou" feel to it.

The large majority of the population will succumb to "what's normal" vs. "what's for the best" to avoid being casted unfavorably. Your regimented approach to seek out the carved out channels of recourse, dictated by authority, suggests you won't combat for change.

We live in a society, yes, and we have norms to abide to. But when we propagate the notion that everyone should put their heads down for the sake of reputation, then the world will never progress.

> But when you attach words like "integrity" and "reputation," it has this "holier than thou" feel to it… The large majority of the population will succumb to "what's normal" vs. "what's for the best" to avoid being casted unfavorably…

Especially when considering the asymmetries involved in interactions between a corporation and an individual, i would even suggest that it is a form a bias people can have which could be considered to be a conformity excuse[0].

[0] http://www.overcomingbias.com/2017/06/conformity-excuses.htm...

Yes and I know a guy who went to prison for a revenge-deletion after leaving the company. He's a really good guy and he made a mistake in anger. While I disagree with the exact outcome in that case, I also think it's wrong to regard it as heroic (as implied by your post).

> The ex-admin's bosses probably thought they were the important ones, and that this worker was a replacable cog, but they found out the hard way that this was not the case.

I don't think this really indicates that at all. Maliciously inflicting damage on the company when you're fired is very different from being irreplaceable. It makes it risky to replace you, but that's not the same thing.

Right. If they were truly irreplaceable, they wouldn't have been fired in the first place since there would be no one to replace them.

A person who can destroy servers is no more indispensable than an employee who can destroy a physical office.

Yeah, a better case would be firing the admin then the site goes down and nobody can fix it.

Why be in a technology business if you can't deal with stuff like that?

Because money. God help my company if I get hit by a bus tomorrow. Until that happens, or they piss me off enough that I go down the road, they are happy to make money off my expertise.

"He who can destroy a thing, controls a thing."

I don't think it's realistic to expect this company to now treat their employees less like replaceable cogs than they were previously (to the extent they were at all). What is far more likely is that this company will now just have more red tape in place to ensure that this doesn't happen again.

> The ex-admin's bosses probably thought they were the important ones, and that this worker was a replacable cog, but they found out the hard way that this was not the case.

Sabotaging servers doesn't mean you're unreplacable any more than a terrorist attack means Western culture is depraved.

I'd believe you if the servers simply started falling apart without this person around, but that wasn't the case.

> The view from the pinnacle, people counting the dividends on the checks that they inherited is that they're the job creators, and everyone else is dispensable. This company just found out that is not the case.

The solution to this, should it become a regular occurrence, is to make the folks with the keys to the kingdom replaceable.

It's doable, companies just don't do it because most people don't want to destroy their high paying and relatively comfortable careers committing felonies and getting sent to prison because they had to spend a few weeks or months looking for a new job.

So if the company doesn't have enough revenues to afford all the employees, it's a bad guy for laying some off so it can stay in business and have jobs for the rest of the employees?

Other than treating staff well, how would you go about stopping something like this?

As my own company is growing, we fully trust all employees, (limiting only what is essential), but, a dev ops guy if he was so inclined could technically do something like this... It always scares me.

It has parallels everywhere. How can you stop people poisoning food in a shop; walking into a school and stabbing a bunch of young kids; irradiating coins; deliberately driving cars into people; spreading sexual diseases; closing organisations with bomb threats etc. There's plenty of things you could do with practically zero chance of getting caught (and not everyone is worried about that) and for a low cost. You can't run a society that has a 100% success rate of preventing these sorts of things; you have to trust that they won't do it.

to your point: chasing terrorism is like trying to solve for all these (and many more) nefarious ways in which society isn't looking for wrongdoing. It's basically impossible to safeguard against every type of threat. Manage to lock down one sufficiently so that it's no longer attractive to accomplish ${evil_stuff}? Move on to the next one. We now see people getting run down with large trucks, since air travel is much more restricted. Everyone still suffers from restricted air travel, but will the TSA start inspecting every car driving over a bridge or an on-ramp now to prevent these new attack vectors? I sure hope not.

We swiped an idea from launching Nuclear Missiles: Typically you need two keys from two people to launch.

For really important accounts - we have three people who each know two thirds of the password. It requires two people to then log in and do damage.

For example if the root password was CatDogFish then

Person1: CatDog_

Person2: _DogFish

Person3: Cat_Fish

Two people can then log in and watch what the other person is doing.

This is like an informal version of Shamir's Secret Sharing


Maybe someone should implement a PAM module or something that requires authenticating as two different users with sudo privileges, so you can implement this for as many users as you like and let them each have a single password, instead of half of a password for each possible pair of users.

Use something with 2FA like e.g. https://duo.com/docs/duounix and give person A the password and person B the 2nd factor (phone with this Duo profile configured or whatever).

That doesn't eliminate the need to set up an account & password for every possible pair of users, rather than just every user.

If would also have to do session sharing so both people can watch each other and see what commands they run.

What if one of the people that has part of the key gets mad and leaves without sharing it?

Then you still have two users with a complete password set, from which you can either reconstruct the third permutation or generate a new set.

That only sort of helps. Do both people also watch each other while they run commands?

Because if not, once you are admin, you can install programs that let you become admin again at will.

Yes! We swiped the idea from the military. In certain dangerous places, they require two people to be there or nobody. One person is not allowed.


Well whoever set that up knows the whole password right?

Nope. It could be system generated and disseminated over secure channels.

Password rotation becomes necessary, and a little bit riskier, because now you have to deal with accidental lock-outs in a sane, coordinated manner.

and who implements the system that generates and disseminates over secure channel?

You are stuck trusting somebody, no matter what. Its turtles all the way down.

Eh, 3 pieces of a randomly generated string... hashed in memory, and only the result of the hash function is stored. Pretty basic concept.

Soon, dear throwaway, you'll be telling me we should live in fear of the locksmiths, for all their key blanks and such fiendish metal files to abrade them with. What if they should file down a butter knife into the shape of my precious bicycle chain's key?

I didn't say you should be afraid, simply that trying to build a system that doesn't require you to trust people is a fools errand.

So one person runs a keylogger and captures the full credentials?

But it's not a bad system.

Since the title says "Ex-Admin", revoking credentials when someone leaves the company is a must. If there's centralized auth (AD), this is pretty straightforward. If not, then at the very least it makes sense to use Puppet, Ansible, etc. to lock that person's account on all production servers.

Beyond that, be sure to keep regular backups (and test them), and audit all user actions. (feed the logs into something like Splunk, running on a separate machine)

I bet the ex-admin was also responsible for keeping backups and auditing/logging all access, in addition to managing pretty much everything else.

There really should be more systems that require 2 keys to do things like delete servers, load balancers, etc. I'm not aware of any. It's crazy to think that if an admins AWS/GCP/Azure account is compromised (or the cloud provider is compromised) a few commands/clicks to an API server could delete everything.

You prevent this by really trusting people (Trust, but verify). Don't screw them over. Treat them better than fairly. Give your employees raises before you give yourself anything.

And do backups. And then backups of those backups.

Yeah, the key is inspiring loyalty and creating a nurturing environment. I loved my first two employers and would go in to either, today, if something I worked on was broken and they couldn't fix it.

That's probably because when my grandmother died, my boss at QueBIT said "Ok, go home, call me when you can work again - however long that takes." There was never a discussion of PTO/HR policy, just human treatment.

Also remember to test restoring your backups or they don't count.

As crappy a job as it is to think about, start planning now. Have a checklist of everything everyone has access to. Have a procedure in place for what happens upon termination.

I worked retail to pay for college. Could always tell when a manager was getting the boot; they'd order new cylinders for all the doors. You kind of have to have that plan in place in IT too.

There should be a backup device or system offsite somewhere that automatically grabs incrementals of everything (and yells at you if it can't for some reason), essentially nobody should have any kind of remote access to it, and the fewest possible people should have physical access to it and gaining physical access should be such a hassle that you don't do it unless it's critical.

That's how I do it, anyway.

We can grant that this can be logistically difficult at certain scales, but it doesn't fall into the "engineering-impossible" bucket until you reach Facebook's size.

What you're describing here would require the company to be able to access their clients servers and data.

I don't know of any server provider (bare metal or cloud) that forces users to allow the company full access to their data (outside of managed providers were you voluntarily give this up , as you're paying them to fully manage your server)

If you are in AWS, you can set the employee's account permissions to not be able to delete servers or drives. In AWS you can also us CodeCommit and set permissions so that the Repos can't be deleted either.

If the CI/CD is done right, then no DevOps staff has any access to any servers and no one can delete anything except a scripts and AWS configurations.

The whole problem with limiting permissions is that you have to do all the work of deleting files, servers and drives.

If this was truly a former employee (wasn't made 'former' after the incident occurred) then all of their accounts should have been blocked from having access. Unless the guy built a back-door or something. IT and Network team may have dropped the ball on that one.

Usual DR stuff covers this and everything else, no one person should have the keys to the kingdom or in this case access to the offsite backups. Offsite because fire, earthquake, zombie apocalypse, covfefe,... google 'disaster recovery'. If you are a small company take the backups home with you and lock them in the safe in the den. Trust nobody including yourself.

So THAT'S what it means!

Proper exit procedure should have disabled all access from this ex-admin..., unless s/he had some sort of cron job or launched some process that would execute commands at certain time? I am very curious to know how it was done.

That's all true, but in reality, we often count on the better nature of people, goodwill, for a proper break on both sides.

If someone is planning a malicious exit, it can be very hard to stop them depending on how "integrated" they are.

IT admin people have the keys to the building and pretty much all data at the end of the day. Trust is everything and reputation is extremely important. This will not go well for the ex-admin one way or another either by lawsuit or blacklisting.

Well, assuming he doesn't change his name and fake his employment history. Or just deny it. Or threaten to sue for libel if anyone claims it was him that did it. If it was me I'd claim they screwed up (restored a backup onto the backups, something like that, happens all the time) then blamed me. Let's be honest, they're more screwed than he is.

And yet, they are often abused, underpaid, and treated as replaceable. The only surprise should be that this doesn't happen far more often.

At the core, IT people are usually seen as cost centers and not revenue generators. Not that I disagree with the business owners a lot of the time because IT is usually not the thing that makes a lot of companies money.

Often people about to get fired knew long before that axe was coming. Making sure everything is properly backed up and secure is a better option and what you should be doing anyway.

So true. A former employer continues to share google docs with me -- not just updates to docs I had been using but new docs as well. I don't read them because they're no longer any of my business, but I haven't been able to stop it from happening.

If they're being added to a folder, you can remove your own permissions from the folder. At least if you have write access.

If they're sharing them individually with you... Then clearly they're not paying attention

> should have disabled all access from this ex-admin

You can't. Not from an admin.

Same as how if you are rooted the only advice is to reinstall. It's simply impossible to reliably undo everything from inside the machine.

If you are a company, reimage the machine, then reinstall everything, and copy the code fresh from known good source control (and hope someone was watching source control that the admin did not check something in).

Of course you can. The moment his account is deactivated, he should not be able to access any machine in the system. Unless of course, he installed proactively backdoors, which is a criminal offense, at least here in Germany. And with a proper setup, he should not get random remote access.

Unless you have exceptionally good controls it's very hard to be sure there is not an SSH key sitting on some machine that would allow access and possible nefarious activity by a dishonest ex-administrator

This is one of many great reasons to rotate them regularly in an automated way. e.g. https://derpops.bike/2014/06/07/ssh-key-rotation-with-ansibl... or update it in your master image / wherever it comes from if doing immutable system images for deployments.

edit: also, use a bastion host which has the keys on it and don't allow them to be removed / used from laptops directly.

Let your Puppet/Ansible clear out all non-managed keys. If it's not in version control, you don't know who did what when. That's a nightmare as soon as you are more than two admins.

Also, the CA mode of OpenSSH is great. More people should use it. It's like PKI but sane.

Dead man's switch? Embed a process somewhere to trip when an account is removed from AD/some other trigger.

Bad employees know they are bad employees. Sometimes they plan ahead for the dismissal.

A "proper exit procedure" designed and implemented by who exactly? And what if they go rouge?

This problem is not as simple as you are pretending it is

Once you turn rouge you'll never be rogue again.


If I ran a hosting company and all of my servers were compromised by ring -3 malware exploiting the Intel AMT vulnerability, the first thing I'd do is privately inform Intel that I intend to go public with the story and sue for damages, after which Intel would perhaps offer a very generous bribe for my silence and a week-long window to replace all of the server processors for free, on the one condition that I bury the truth by fabricating a story about an imaginary ex-employee who improbably was both smart enough to gain an administrative position in a large company while also being stupid enough to risk decades in prison for petty revenge over workplace drama.

What kind of hosting company is going to have any kind of reputation after admitting their security allowed an ex-admin to ransack everything?

Rephrase the question -- what idiot customer is going to do business with such a place that allows such a lapse in security to happen?

Intel would basically have to buy the company.

> What kind of hosting company is going to have any kind of reputation after admitting their security allowed an ex-admin to ransack everything?

The kind of people that:

  - Use Gmail, iCloud, etc. post Snowden
  - Buys SSL certificates from Comodo, etc.
  - [put other companies here]
So, pretty much everybody, me included. Your idea that mis-management can damage a company's reputation permanently, has been proved wrong. The complexity of moving an infrastructure in and out of a service provider is apparently bigger, os as you say, we're all stupid - we manage to survive somehow though.

>>admitting their security allowed an ex-admin to ransack everything

What, exactly can be done to secure a company against a malicious systems admin? These are the guys typically with not only the keys to everything but also the knowledge of how it all works.

You say that the company cannot be trusted for "allowing" this to happen.

I know quite alot about this stuff, and for MOST companies, they simply have to trust that the people with the keys to the castle with behave responsibly.

There are ways to design infrastructure such that it is protected from its builders and keepers, but this is very very hard and complex and expensive.

Presumably you work for a company that has taken steps to ensure this will never happen, what are they?

You should be trivially able to secure your company against an ex admin, though -- why was this ex-admin's credentials not revoked immediately after their employment ended?

You're assuming he hasn't built in back doors.

At my one and only sysadmin job, the network was secured from likes of this guy by the senior engineer, a Vietnam vet with trust issues and whose talents weren't only technical.

Nothing says "web of trust" like knowing your boss could show up at your door with a shotgun demanding answers.

> why was this ex-admin's credentials not revoked immediately after their employment ended?

Maybe they were but he'd set up time bombs.

Maybe the admin was fed up, knew he wanted to burn things down, wiped everything remotely and then never turned up for work and he became an ex employee in the aftermath.

It's decidedly not trivial to secure your company against a malicious admin who has control of much of your infrastructure.

There's nothing trivial about that unless the organization is tiny with the most boring and controlled IT ever. There's lots of little cracks for admins to slip in. That's not to mention physical devices with remote, wireless access planted in the building somewhere. Or even in keyboards of important people.

At best, these companies will be keeping out the riff raff. Fortunately, that will stop majority of attacks since most admins aren't geniuses or spending personal money on attack toolkits with 0-days.

If they leave the company, their access might be left running for a while in case they're asked to come back on and fix something during that post handover period.

Or maybe they had backups running under their account and when their account was disabled everything failed so they re-enabled it while they sorted out the mess...

Or they had multiple accounts as part of "security" and HR only knows to disable one and didn't find the other one in time.

There's a whole bunch of reasons why shit like this goes wrong. Every time. You'd cry.

I would consider none of those to be acceptable though, and would point to a huge process failure around security at the company.

Who's saying they weren't? I'm sure that administrator would have been able to plant subtle backdoors, or find and not fix vulnerabilities and misconfigurations, during their time there. In which case it wouldn't matter that their credentials were revoked.

Honest question: Are Ansible + peer reviews enough to secure sysadmin work? Of course they can log on the machine to open a backdoor, but they are supposed to be regularly destroyed and rebuilt regularly, so we're theoretically safe, aren't we?

Unless they put the backdoor in the image, then it's on all machines. I don't think there's an option here

VW has record sales.

Either way, their reputation would be trashed. Intel knew about potential issues in their products for many, many years. I'd shed no tears if lots of people do this to them now.

Weird. That's exactly what happened to the data centers at Area 51 too.

Also similar to when Gillette paid a very generous bribe to bury Occam's and Hanlon's razors. /s

I'm sad both that I can't upvote this more than once and that I can't think of any way I'd ever get to steal this line.

No, these were stolen by aliens but CIA covered it up by pretending to be covering up AMT malware ;)

Seriously though, would Verelox still be running unpatched AMT many weeks after the disclosure of this authentication bug? Or does GP think there are more bugs which Intel hopes to sweep under the rug forever by individually covering each incident? They would spend quite a money on these bribes while AMT bugs can simply be fixed with BIOS updates.

This Intel conspiracy doesn't make sense. It's aliens, folks, I know it.

What's the opposite of Occam's razor?

How Intel would most likely respond to you: "HahaHAHAHahAHAHAHahahAHAHahAHA, fuck off"

Lots of comments are interpreting "ex-admin" as someone who was fired and then after went and did this. Just want to float the possibility that "ex-admin" could also mean someone was employed there, then did this and is now no longer employed as a result of doing this.

(Btw, IMO there is no excuse or justification for any admin or exadmin to ever do this. Among many other issues is the fact he deleted the data/work of individuals who had nothing to do with whatever "problem" he has with Verelox )

What if their information was all secretly being leaked to intelligence agencies and there was no way to ever let them know except to burn everything?

There's probably excuses and justifications. I personally wouldn't do it and they're probably wrong for doing it but I don't want to jump to conclusions and moral absolutes so easily.

It's always interesting watching startups learn the lessons that thousands of enterprise learned along the way. "Why would you ever want offline tapes sitting in iron mountain, how inefficient".

Nothing is foolproof, but anytime you've got constant network access to every last copy of your data, you're begging to lose it. It's the reason why people who think one copy (redundantly dispersed or not) in AWS S3 is sufficient scares me to death. Is it unlikely Amazon would get hacked and have the entire thing blown up? Sure... but if we go to war with China I wouldn't want to bet my company on it.

Heck, it doesn't even have to be Iron Mountain if your data size isn't too big! Just rotating tapes to a safety deposit box at your local bank every week will do a damned good job.

If we go to war with China I don't think you or anyone else will care much about your company.

I know you meant it as an example, but this sort of extreme attitude towards security is just another footgun.

> If we go to war with China I don't think you or anyone else will care much about your company.

Probably not in the immediate aftermath, but someone might decades later, if the company actually does something valuable.

> If we go to war with China I don't think you or anyone else will care much about your company.

Why? We went to war with Europe and Asia a few times and businesses kept chugging along here in the states.

that was before people had lots of nukes

You're assuming war will involve armed conflict from the start. If we go to war with China that will be the end of the war.

I would expect to see some kind of police report, and prosecution of an individual charged with a crime, no?

Yes, it is criminal and the ex-admin will go to jail. The cases for this are pretty straightforward... not sure why someone would throw their life away by doing it...

Well, whats actually not clear from their message is if the person who is currently an "ex administrator" was an ex-admin at the time that they deleted everything or if they were fired because of a (bad) mistake and are now an ex-admin.

Being fired for an honest error made in good faith is a misclassification of such a problem. If a single current, acting employee causes an outage by way of human error, then process and planning are to blame for the destruction.

If there isn't a police report, and charges aren't pressed against a malicious individual, then the company may be at fault.

Certainly the employee could willingly and pro-actively step down, out of personal guilt and feelings of shame, but then, one need not qualify with "ex" as simply "employee" will suffice. A criminal incident is worthy of such a clarification, indicating that the incident is a deliberate attack, but human error is not.

>> Yes, it is criminal and the ex-admin will go to jail.

What if the admin was a remote worker in a country that doesn't have an extradition treaty with the Netherlands (Verelox hq country)?

Presumably they would have a warrant out in Netherlands. If issued in the EU or US I wonder if it would effectively ban them from most Western countries similar to Assange.

If this was a "cyber crime" they could possibly be picked up to stand trial in any cooperating country which would narrow their choices of travel.

However, in that situation they would probably be fine except potentially limited ability to work for Companies who do background checks

  > 4 low-latency locations in ISO/IEC 27001 
  > certified Tier 3+ data centers in United 
  > States, Netherlands, France and Canada.
Entirely possible.

Hrm. I wonder how this would work if the ex-admin is working remotely from a country that wouldn't extradite for this.

A good argument against "offshoring".

Although that's perhaps not the right word. There is contiguous land between e.g. The Netherlands and Russia, so there's no "shore" involved as there would be between e.g. the USA and Russia. I do think the word's meaning is now more "another country", rather than denoting crossing physical land/water boundaries.

It's impossible to happen from offshoring because there are contracts in place to prevent it! (cheeky grin)

Assuming that the police care about cyber crime and can find enough evidence.

>Yes, it is criminal and the ex-admin will go to jail.

Maybe in North Korea, or the US. Relatively unlikely in civilized countries.

I wonder if some of the downvoters would like to expand on why this guy wouldn't be given a suspended sentence.

I've been following and partaking in various European cybercrime trials for years and unless this guy is a repeat offender or acted in a particularly methodical manner it seems utterly ridiculous to make blanket statements like "this guy will go to jail". He most likely will not.

I downvoted because it compared USA to North Korea and suggested they both are "uncivilized".

Do you believe that's an unfair comparison? According to various statistics both states have very similar incarceration rates.

Compare US incarceration rate of 693 people per 100,000 to 69 people per 100,000 in the Netherlands.

Seems like nothing short of a felony. I can't conceive why someone with half a brain would do this.

Been waiting for a company to announce shutdown after this was posted: https://news.ycombinator.com/item?id=14476421

Possibly related?

Anecdotally, the timing doesn't match. Based on this https://www.lowendtalk.com/discussion/116329/what-s-up-with-... it looks like they went down on early on the 8th give or take a bit and the other was posted around then 2nd ...

Edit: Make that the 7th based on: https://www.facebook.com/Verelox/posts/1886196381643427?comm...

Well it's kind of the opposite, isn't it? ;)

If you take Verelox's word for it. Or if you want to be more conspiratorial, you could see your firing on the wall, tweak the new hire docs to lead to trashing prod, and just wait for the inevitable.

Or these two events are unrelated. Or the whole deleted prod on day 1 story is made up.

In both cases, critical data with no backups was deleted by an ex-employee. In the "first day" story, it just happened to be both his first day and last day. I guess the difference is that one was a developer and the other an admin. No telling from this story if the person who deleted the data was an employee when they deleted the data.

Wiped by ex-admin, or by the ineptitude of current admins that can't maintain proper exit procedure?

Ok, I'll bite. If you are an admin generally you have God powers over everything within your umbrella. If things are going south with you and your employer and you're a giant asshole you can most certainly use your God powers to quietly open firewall ports, install software in dusty "corners" of networks, etc. We can all armchair quarterback this but in the end humans make mistakes and beyond that the presence of actual malicious intent by someone with keys to the whole kingdom makes the detection and/or mitigation difficulty factor go up significantly. It's real easy to make righteous proclamations about other people's misfortune when it's not you.

I'll admit, my tone might have been tad overly poisonous (too much internet for me..).

I'd agree that defending against malicious admins is really difficult. We have really little context to go by here, but I think there is important distinction to be made if the malicious actions (planting backdoors or whatnot) were done while the malicious actor was still employed or after their employment was terminated. Proper exit procedures protect against the latter, but generally are not that effective against the former.

> but in the end humans make mistakes

And it is useful for us outsiders to highlight the real mistakes so that we can learn from them, because that is really the biggest value of stories like this for the majority of people who are not directly impacted.

Exactly. Happened to my shop on my watch. Fired a contract admin... about a week later the network went to hell. Unfortunately I couldn't nail the bastard, but at least we found and closed the back door.

The current admins ineptitude did not wipe anything. Yes, with better procedures this could possibly have been prevented, but the current admins are not the malicious party.

Well, for one thing, if you can't ensure that a given privileged person can be locked out of your systems quickly, you have a problem to fix. This should be a 1-minute operation.

Note that I don't know the details and am making assumptions that may be wrong about the case in question, but in general, if you can't deny access quickly to any given account, you really want to fix that. Not just because of rogue ex-employees - what happens when $important_person's account is compromised?

> This should be a 1-minute operation.

No. It's easy to revoke access to a user. An admin is different - an admin can install whatever he wants to give him backdoor access. Or a timebomb.

Yes, but backdoors/malware are a different question. I was talking about authorized access - LDAP, ssh keys, etc.

Detecting unauthorized software from a rogue privileged user is a different problem with very different mitigations. It is a great topic that I'm personally interested in, given that I'm implementing controls for that, but I wasn't discussing that.

My most favorite of all razors is Hanlon's: "do not attribute to malice what can be explained by stupidity"

Not sure how that's applicable here? Seems like an ex-admin deleting things is pretty clearly malicious...

He's saying the current admins are stupid for not investigating every possible route an ex admin could take for tanking the business.

Don't necesarrily attribute the error to the ex-admin, because the current admins were stupid enough to let it happen. Yes, the ex-admin is wrong and shouldn't have done it, but it is a crime of opportunity.

If I leave $20 on the sidewalk the thief is wrong for stealing it, but it is partially my fault for being stupid enough to leave $20 on the ground.

I'm not sure I can agree with the reasoning here. An ex-admin, who had the conviction to wipe an entire company's database, probably wouldn't have been stopped by exit procedures.

This can easily happen to anyone -

    5PM Friday - $admin and $ceo have a fight
    6PM Friday - $admin decides he's had enough with $company and $ceo, and wipes everything
    7PM Friday - $admin is fired
Where are your exit procedures now? Are the current admins stupid for not having the foresight that $admin and $ceo's fight would have resulted in the worst?

This is way more like throwing away your old roommate's belongings because they didn't change the locks when you moved out. Anyone would take a $20 bill on the ground with no way of knowing the previous owner. Not everyone would maliciously destroy someone else's property just because they had access. Yes, current admins could and should have done more to prevent this. However, this goes way beyond opportunity and firmly into malicious intent.

Not following/having a proper exit procedure for employees with a lot of access is being negligent. Wiping servers of your ex-employer out of spite is criminal (and being pretty bad at that whole being a human thing).

Was he an ex-admin at the time he did this? It's not made clear here.

Who says he used his credentials and didn't leave and use a backdoor?

This is why you have the backups stored under a different account than the primaries and you make sure that nobody has access to both accounts.

A lot of 'managed' hosting providers are pretty bad with security, there still is a major provider that just gives root credentials to all servers to all techs not just admins, doesn't audit who accesses which credentials, and doesn't rotate credentials, doesn't rate limit dumping credentials... That's before we go into more interesting issues with their security. Frankly I am surprised this sort of thing doesn't happen more often ? In some ways it both restores some of my faith in people while reducing some of it at the same time in a different vector.


Some posts from Verelox staff towards bottom third of this forum page search for user name Verelox

This is why a hosting company needs to both segregate credentials to only what an employee needs for their job, as well as to revoke them the minute they leave the company.

Otherwise while the vast majority of your staff will be decent people and not cause problems like this, it just takes one angry ex staff member with a grudge to cause problems.

They also need to revise their backup system too. There should rarely if ever be a risk that any data is 'unrecoverable', yet their update says some data will just be impossible to get back.

As for the employee involved... well I hope they like the inevitable lawsuit their selfish, stupid actions will bring them. I don't care what you think of a company you worked for, there's no excuse to destroy their business through actions like this. Also, good luck getting any jobs in the industry after too. Because with this on your track record, no one will touch you with a ten foot bargepole.

So yeah, what a disaster all round.

Yooooowch. They appear to be VPS and dedicated host.

There goes the 99.95% uptime guarantee.

In theory that still gives you a lot of downtime in a 365-day span :p Then again, once you are down for more than 4 hours (as they are), that uptime starts ticking away rather fast.

I believe a 99.95% uptime guarantee indicates that you will only have 4.38h of downtime _per year_. So, yeah, they've blown their allowance for the year and now need to execute perfectly.

Who even knows it was an ex-admin? Could be the current one fat-fingered it and is trying to shift the blame! We just don't know.

Exactly. Mind-boggling how easy it is to present people a logical reason.

The thread title should be changed to "Verelox allegedly wiped by ex-admin": we only know one side of the story.

Anyone of us know what is the best way to get refunded? My Company lost 20.000€ for this joke.

Lawsuit, most likely.

I wonder whether the ex-admin was already an ex-admin at the time he wiped the servers.

Use Vault from Hashicorp where possible.

That would most likely have made 0 difference.

It's a process problem, not a product problem.

Dick move from the ex-admin, but I'm curious to know what would compel an ex-employee to take such a brazenly criminal and traceable yet damaging action.

I'd like to know more, I think...

There is no excuse for it, no matter what the company did this affects the customers of the company just as hard as it affects the company. Likely some of those will go out of business, people will lose their jobs, go bankrupt.

That's the smallest thing I've ever seen "too big to fail" applied to.

The company could very well still fail, and no matter how large they are their customers should never rely on their provider not failing to the point where they themselves will also fail if something drastic were to happen to their provider.

But that does not mean that a single individual can put themselves in the judge, jury and executioner role all at once without any kind of oversight, that's at best a misguided case of vigilantism and at worst an act that is disproportionate against innocent bystanders and possibly a far larger crime than whatever happened before.

So do whistle blowers. That line of reasoning doesn't set the bar for what's right and wrong.

Are you sure no matter what the company did? What if the CEO threatened the ex-admins family? Or if the ex-admin found child porn on the CEOs computer?

There's a fine line between right and wrong in most situations. The most egregious acts of disobedience can be seen as defiance or foolish. It's not for you to decide- especially when there isn't any context to this whole situation.

> So do whistle blowers.

This is not a whistleblowing case, it is a case of wanton destruction by a former employee.

> That line of reasoning doesn't set the bar for what's right and wrong.

Dragging in all kinds of stuff that has no bearing on the case doesn't set the bar either.

> Are you sure no matter what the company did?


> What if the CEO threatened the ex-admins family? > Or if the ex-admin found child porn on the CEOs computer?

In that case you go to the police and file a report with them. Hurting the company, the employees and customers when your target is the CEO is ineffective and illegal besides.

> There's a fine line between right and wrong in most situations.

No, it's crystal clear that this was wrong in any way you would like to look at it.

> The most egregious acts of disobedience can be seen as defiance or foolish. It's not for you to decide- especially when there isn't any context to this whole situation.

This is a criminal act, pure and simple. If the CEO did anything illegal this guy/girl is an idiot for doing something illegal himself.

I think this one would qualify as a criminal move, not just a dick move

Well, I’m guessing it starts with rationalization of why it is not criminal, and how you’re so smart that it won’t be traceable. Wrong on both counts, usually, but that’s my guess.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact