Hacker News new | past | comments | ask | show | jobs | submit login
SolarWinds CEO blames intern for password leak (cnn.com)
305 points by tnolet on Feb 27, 2021 | hide | past | favorite | 214 comments



> "They violated our password policies and they posted that password on an internal, on their own private Github account," Thompson said.

So let's analyze the various scenarios under which the "intern" might have been responsible, and why each one is bullshit.

1) The intern exposed a company password on their own account.

Counter: What kind of "password policies" allowed such a weak password in the first place?

2) The intern came up with the weak password themselves, thus violating "password policies" not just for secrecy but strength/security. This password was then used for several critical, production applications.

Counter: Why was an intern in charge of deciding a password used for anything critical?

3) The intern came up with the weak password and exposed it, but it was only used for the intern's own corporate accounts (e.g. their Windows workstation).

Counter: Why did an intern have a level of access such that their account being breached could lead to this level of compromise/exfiltration?

Conclusion: There is no conceivable scenario under which this makes sense.


Years ago, not long after I started at Amazon, there was a huge Netflix outage[0]. It surfaced - or at least was widely speculated - that the cause was a pretty green employee running a DROP TABLE command against a prod database instead of a dev environment.

One morning when I came in and sat down at my desk, all of the old-timers were having coffee and discussing the fiasco. I was very happy to hear all of them talk about how mistakes happen, and the last person to be blamed for such an outage is the poor guy or gal that hit the ENTER button. Rather, blame falls (to various degrees) on: the engineers in their orbit who should be backing them up; the managers helping to onboard them; the chain of command; the entire system that is in place to prevent inappropriate access.

One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way.

[0] https://netflixtechblog.com/a-closer-look-at-the-christmas-e...


> One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way.

owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers.

Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents.

Something we tell employees during our onboarding in a technical comes down to this.

- reason about a problem by yourself first - think about impact before you do a change, if in doubt, ask and doublecheck. - admit mistakes when you realise them, explain your reasoning and why you did the action. - learn from your mistakes, but accept that being error-free is simply not possible.


> owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers.

If you are in the right company. I made a big mistake at one point. We had all of the people responsible for one of the payment methods away on holiday (this one payment method was managed by another team in another country, as it wasn't using our normal payment gateway and provider that my team was maintaining).

Huge panic, someone needs to fix this, you can't use X any more for completing your order. I'm in the right team that's handling payments and fulfilment, only one at work at that point so I'm told to fix it. I do, I fix it, send it over to testing, get the green light, fix is deployed, everyone is happy.

2 hours later, we figure out the payments are working, but the orders are not being finalised and are still in unpaid. We realised that that single payment method done by this other team in another country was not using our standard payment processing workflow and it has a different way of actually getting the confirmation from the payment provider. This was quite a big company, we had around tens of thousands of Euros blocked in those 2 hours. I own up to it, I admin I made a mistake, go deeper (we did not have anything about this documented) fix it again, we unblock everything, all good. Until 1 month later I got fired (there were layoffs because of larger financial issues, but I was on the list because of the incident), it was the only time in my career this happened to me.

In the same time, someone else in my team made a mistake, hid it from management, even though we knew about it internally, fixed it and bragged about fixing the issue (no mention of him being the one who caused it) and got somehow (didn't even know we had such a thing) employee of the month and big praise in the next department meeting from management.

My lesson from this? Screw these companies and the people running them. I was asked to help, I did it, I made a mistake, fully aware of that, but then I'm the only one thrown under the bus for it.


Man that sucks. You deserve better for jumping in and helping. Hope you're in a much better company now.


Leaving that company was one of the best things that happened to me. It also thought me that when a company, especially a big one, blames a low level employee for a big mistake or some visible incident, there is something very wrong there. There is usually so much politics, so many layers of management, that blaming one person that does some actual work is the easiest way of hiding your actual problems.


in my experience this is detrimental for a company in the long term. It results in people who are in charge of taking leadership not actually leading the company and it shows everyone in the company taking risks is instant failure in their eyes.

To give you a counterexample.

The same company my prior example came from, also had some other "silly mistakes" made by an intern. He had to do inventory of a couple of old servers and remove hard disks from these servers. The servers where to be sold.

Sadly, no one told him we had additional servers in the back of the storage room which he forgot to check because they where not on the same pallet as the batch he was told to check.

Result, an couple of servers got sold with disks still in them. Luckely the company we sold to was friendly enough to give us a headsup about it and it resulted in no further issues, but still. Our company director personally took this as a reason to spearhead a plan about improving operational security and change processes (Aka, remove the hard disks when the machines are put out of service instead of half a decade later when their sold).

The intern felt pretty bummed and thought he was responsible for the mistake, but in my opinion he done the job that was asked of him, he just got incomplete instructions. This was also explicitly communicated with him by his direct supervisor.

In my experience, not throwing people under the bus to hide organizational or process failure, but simply admitting the processes could be better and striving for improvement does absolute wonders for morale and team building.

Being perfect is impossible, organizations should keep people to impossible standards, especially to hide incompetence.


Great for you. The company that I currently work hired a new manager, he is amazing on how he shows that we are a team and a "person" mistake, is actually, the process which need better thinking/layout.


"Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky"

Third option: they don't realise when they make a mistake, either because they are not smart enough or too full of themselves.


Like the engineers who never have to refactor, because their code is good enough in the first go.

The only way to guaranteed never have mistakes in your code, is to not have any code at all.


To be fair sometimes people hyper focus on refactoring rather than fixing business problems, which isnt really helpful either, have to find a decent middle ground.


You are absolutely right, I forgot about the other extreme.


> Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents.

Bingo. I’ve made mistakes that have taken down systems or caused them to silently fail. The worst mistake I’ve made took down basically my entire company for about 20 minutes. This turned out not to be critical, because our site was still operating, and it was just external data feeds that weren’t getting updated, but I freaked out about it for a minute. After that minute, I went and got help, and we fixed it. Had I not, I probably could have fixed it myself, but it would have taken much longer and cost much more than it did.

If you’re in an environment that doesn’t recognize that, you aren’t in a place that actually values and understands engineering work.


I've also found that being forthright about your mistakes gives credence to your _disavowal_ of responsibility. When it's not your fault and everybody's looking for a root cause, if you're known as someone who is quick to admit fault, then when you deny it everybody accepts it.


I was on the other side of that Netflix outage, as the main contact point between Netflix and AWS at the time. Amazon did give us a detailed rundown of what happened, but was very specific to not name names, nor did we ask. We all agreed it was an excellent learning opportunity for both AWS and Netflix.

That outage is what drove us to rearchitect all of Netflix to be multi-region.


The bottom line is, the PR damage done by SolarWinds' CEO to his company stands out the most, and he can and should fix it.


The CEO blaming an intern is the big thing that looks bad, but it's also very telling of their culture and how they got in this situation in the first place.


The CEO saying this, they are inadvertently signaling that they want a certain class of customer, a customer who agrees with and would make similar statements.

Any customer that can see through the BS will immediately turn around on their heels. And SolarWinds will be happy that they just lost that problem customer.

SolarWinds is looking for an Equifax not a Netflix.


What’s left unsaid is that not just the CEO, but every manager down the line has made it clear that it’s okay to toss the blame down the line.

Every individual contributor in that company has just learned that they need to cover their ass for any action that could possibly go wrong.

The cultural outcome is that accountabilities will be spread across managers so that blame can’t be assigned to an individual.


I am deeply ambivalent about my tenure at Amazon overall, but I unconditionally applaud the level of psychological safety in their dev culture: there is a shared understanding that any system failure caused by a single person is necessarily a failure of the process that governs changes to that system.

If there were any singular behavior I wish every tech company would adopt, it is that level of perspective on what makes a system reliable.


My experience there was overall very negative. My comment above, though, was actually a very bright spot, and I want to call out the positives that exist.

At the time, I had several years in the industry under my belt but was new to big tech companies, and especially the tech that Amazon had built. I lacked confidence in myself but I was super conscious of actively seeking out the help of others. And it failed spectacularly - one SDE3 in my team with whom I was sort of paired would quite literally back away slowly when I was showing him an issue I needed his advice on.

Big companies like that have such variation in teams, and I think I just got rather unlucky in my experience.


At well managed big tech companies, stepping on a landmine like that is not only not detrimental to your career, but it can actually help you (unless you didn’t do that by mistake, but by being reckless and on purpose skipping all the protection layers). You can own your accident and, depending on the complexity, create whole teams to properly fix it, so no one else can cause an outrage like that.


My colleague typically asks this question in interviews, and I've occasionally borrowed it from him: tell me about the last problem you caused and how it got resolved. It's one of those questions designed to just get them talking and there aren't too many wrong answers, except maybe, you know, not being able to think of anything.


Worst scenario: The intern is malevolent. Counter: Credentials should be given progressively as trust is built (and as lessons are learnt), and an intern can’t have access to production.

I have an employee who I thought I could give more responsibilities, but he keeps not locking his computer when he walks away. He has very limited access to everything and it would impede his career if he didn’t also have the same attitude about other issues. (My question is, how do I make him diligent — It’s real potential wasted).


At a company I used to work for, if someone left their computer unlocked, we'd send the donut emoji into the #everyone channel on slack from their computer, and they'd have to buy donuts for the office


Oh! Story time!

We had a similar thing at one gamedev place that I worked at where and email would go out to the team if you left your computer unlocked(I forget the exact phrase but it was fairly silly).

We had shared offices and one of the programmers had the office right next to the kitchen. One day we all heard the senior programmer shout "WHAT THE FUCK!" and all ran over to see what had happened.

It turns out one of our engineers had walked into the kitchen and left his computer unlocked. The senior developer seeing this had opened up outlook, started a new message and began typing in the subject. What he didn't know is the developer had hand-rolled a keylogger with a match pattern for the message that everyone would send and dispatched Windows+L via key injection to the main window loop.

The trap was sprung and the machine locked right in front of him as he typed the last letter unable to send the email.

There was all sorts of other shenanigans at that place(like a fake "April 2nd" firing, they got the person who did that back with an annoy-a-tron over a 6 month period) but that was one of the more memorable ones.


This too but with croissants. It even became a verb. To croissant someone or be croissant-ed!

Edit: Also singing over the top praises of employees using the victim's account. "MonkeyButton is truly the best coworker I have ever had "

All this fun has gone away now with Covid and remote working.


You're french right ?


For us, it’s teapots. Inspired by the info set who noted his cube mate couldn’t learn to lock his screen and started sending, from his open laptop, “I’m a little teapot...” first to just him and then the wider org when he failed to adjust his behavior.


I strongly recommend just sitting him or her down and explaining why this is a problem (you've articulated it well here). Either they get more diligent (great) they can't get more diligent (not great) or they refuse to be more diligent (really not great).

I think that there's sometimes a tendency to overthink management topics - there's no need to overthink this one.


Why is locking your computer at work a security concern? It's certainly another layer in the onion but a rather weak one and certainly not one worth losing someone over. If it's so big an issue, get him a smart watch that works with their OS of choice and enforce a bluetooth device locking policy.

Physical access is considered game over, no?


It's actually a legitimate concern, and it goes far beyond coworkers embarrassing you in slack.

For example, posted yesterday: https://tailscale.com/blog/rotate-ssh-keys/ (https://news.ycombinator.com/item?id=26249380)

> One company had an employee they fired for bad behaviour. A few months later all their production servers got wiped out. ... A month later, it happened again! Everything gone.

> What happened was that the rogue employee, months before they were fired, had waited until everyone went to lunch, walked around from one PC to the next, and collected the ssh private keys from any of them that weren’t locked. ... Then, a while after getting fired, he used a co-worker’s ssh key to login and destroy everything.

Of course, screen locking would not prevent a determined, disgruntled (or desperate) employee from pulling off something like this, but it would stop the casual collection of secrets while everyone is at lunch. Add full disk encryption to PCs and you have a reasonable chance at defending.

Oh, and if you're doing anything with credit cards, PCI DSS requires you to enforce a workstation locking policy. Failing to do so opens you up to massive liability in the event of a breach.


I agree, I’m sure it helps but seems like if a malicious coworker wants access they can trivially steal your password with a keylogger or just watching you type it in. May prevent spontaneous acts I guess but feels like if those are really a risk you’ve hired the wrong people.


Right. There are some intense security environments where they also deal in airgaps and the like, but this is insane behavior at a regular job.


This isn't a solution you can implement yourself since it sounds like you're in a position of power over the employee and it would be harassment. But having a culture where leaving computers unlocked and unattended is an open invitation to getting embarrassing YouTube videos opened on full screen by your peers works wonders for getting people to lock their computers.

I wish I had an answer for the second part; it's hard to see someone with talent be the one to get laid off after months of asking them to be more diligent in their work.


How on earth is it harrassment to tell a subordinate that he needs to improve his security habits?


The harassment would be the doing embarrassing things with the unlocked computer. A manager or team lead shouldn't be picking on their employees like that. It's something more OK for a peer to do. Ironically, it's the interns who I've seen get into it the most and make it a game. They really got a kick out catching the full timers.


You were replying to laurent92, who didn't say anything like that (picking on employees or embarrassing them); that was this other comment, which was replying to laurent92:

https://news.ycombinator.com/item?id=26286183


> I wish I had an answer for the second part; it's hard to see someone with talent be the one to get laid off after months of asking them to be more diligent in their work.

Oof, this hits close to home. I don't have an answer either.


Well I've heard of one office that had an established rule that any unlocked laptop could be used to promise the rest of the team free cake on behalf of the person that forgot to lock their laptop.

If nothing else at least it promotes awareness (and cake!).


2-factor authentication allows you to bias yourself towards suspecting malevolence.

It's reasonably believable that they leaked their password because. Interns.

And the second factor?


Have you explained this to him?


Further, have you really listened to his explanation of why he isn't bothered about locking his computer?

It might be that in your organisation this is a cargo-cult security practice that he's not bothered by because he knows it's not an effective practice.

Or it could be that he knows he doesn't have enough permissions to do any damage, so he doesn't bother locking his computer. Trusting him with a little responsibility might change that.


It's too bad you are on here disparaging a current employee.


Agreed. All this statement does is reinforce the speculation that there was a failure of process but adds to that that there is a culture of blame rather than a culture of improvement.

If a mistake happens, don't blame the individual, blame the process then find a way to fix that process. If a company has a blame culture people spend more time covering their own arse instead of building safer processes.


This is such an extreme example of a culture of blame, testifying to Congress that an intern did it 3 years ago?


> > So let's analyze the various scenarios under which the "intern" might have been responsible, and why each one is bullshit.

"you can delegate authority, but you cannot delegate responsibility." [1]

This CEO is a very poor leader.

1. https://www.theleadermaker.com/you-cant-delegate-responsibil...


If Solar Winds CEO, CTO, and everyon below them had taken the responsibility of security even slightly seriously, they would have made the system secure against such leaks. Some really simple steps:

1) access restrictions such that even malicious interns, and certainly careless interns can do little damage *when* the inevitable leak happens

2) Actively scan everyone's online presence, and let them know that this is a requirement of employment.

3) Require 2FA

4) Much better training so it is reduced

5) Internally firewalled and airgapped systems

I could go on... but none of these were done

The fact that they blame the intern shows that they are insanely unqualified for any job related to any sort of security. These CxOs are active hazards in the industry.


> 1) access restrictions such that even malicious interns, and certainly careless interns can do little damage when the inevitable leak happens

this should be something that is implemented in any organisation beyond a very small scale, mainly because even if not malicious, people should not be able to make critical mistakes in systems they have no know how off.

The intern leaked the password, but how was he able to know this was critical information? Not to mention he should never have been put in that position in the first place.


They said in the article they don't know if this password was even related to the breach (meaning all of these points are not really describing reality, especially #3). It's not very clear what the password was for from the article. Sounds like the CEO doesn't have a very good understanding of what Github is though unless this intern was POCing github enterprise or something though.


How could this password even be “leaked”, it’s so trivial that it would be on a list of first 10 passwords you would guess.


Also what is the password rotation policy if this password was valid for years and intern had access to it? Reasonably I would expect all of the shared passwords to be rotated every time some person leaves.


It's the wrong question; it's incompetent to even use passwords. SAML all the things, and protect your IdP with yubikeys.


> Counter: Why did an intern have a level of access such that their account being breached could lead to this level of compromise/exfiltration?

I've worker with interns whose major redeeming quality was that their internship was fixed length and would be over soon. I've also worker with interns who demonstrated ability and responsibility sufficient to get the same access that I had (and, clearly, an offer letter). That it was an intern doesn't mean the level of access was inappropriate; of course, if it were my intern, I would take blame for them leaving their password on github.


Is it not usual for an intern to get access to the repository and production databases like a new employee would?


It shouldn't be usual. Temps don't get unsupervised access to sensitive systems. Too much risk of abuse and misuse from short-term less mature employees.


Are you a college student? Or an independent contractor? I don’t know how anyone who has worked in the real world can have these misconceptions.

1) Every password policy allows for dumb passwords in certain places. Because password policies are only enforced on systems that integrate with the password policy enforcement mechanism. Which never covers everything. Even with a password policy, it’s easy to make dumb passwords.

2) It doesn’t say that the intern chose this policy or that it belonged to something critical. There has been no link established between that password and the breach. A random researcher said they found the password a year ago and reported it. It could have been used, but there’s no reason to believe it’s relevant.

3) Nothing suggests that they did.

Conclusion: It’s better to not be an armchair quarterback after a breach, especially when it’s still under active investigation by actual professionals with access to actual data, and they aren’t even making the claims that folks here are making.


In many companies any account password is critical, because any account allows access to the company network and internal security is weak.


If the entire security of the company rests on every intern being fully competent, that's the higher ups fault.


Absolutely right, and further, we in tech know better as we could walk into so many big name shops and not be surprised to find huge, simple, obvious security holes. We know it’s standard operating procedure too often, so this blaming an intern rings hollow.


The same is true of passwords now since the invention of rainbow tables. The best way, aside from voluntary disclosure, to compromise any password is brute force. The best way to eliminate brute force is to require a long minimum character count and to hash passwords using a 512bit hash algorithm.

Imagine the size of a rainbow table of SHA512 or SHA3-512 hashes for a 60 character password. A 60 character password could be as simple as:

* I enjoy driving my tiny white car with a standard transmission.

* My big cat, Ace, really sleeps a lot during the day while I work!

* Growing up my favorite song was Time by Pink Floyd about regret :(


For most passwords, you should be using a password manager, which means long, high entropy passwords (which will not be memorable as a result).

For the few that you need to manually enter, something long is good, but it should ideally use characters or different classes, and ideally not be comprised solely of dictionary words (which your first example is), otherwise the search space is greatly reduced.

Also, manually entering a 60 character password is not going to be fun :) I think the longest "manual enter" password I have is 25 characters, and it's a PITA to enter in a password field!


> high entropy passwords

You only need that to impose a greater character width against brute force attacks. That is the only value in high entropy.

The actual reason people think they need this is because it was written into a NIST publication a very long time ago and it just became common practice. As a proof of concept what is the published standard that imposes that practice? I bet you think you need this but cannot find the written standard guidance suggesting it.

The guy who originally wrote that standard later came out and said it was a mistake. Bad advise that he wishes he could take back, but it’s too late everybody thinks they need it and they don’t know why or where that guidance even comes from.

> and it's a PITA to enter in a password field!

Only on a touch screen.

My first example also contains uppercase and punctuation. Think of it like IPv6. When the key space is large enough you don’t need a bunch of bullshit and gimmicks to ensure uniqueness.


> You only need that to impose a greater character width against brute force attacks. That is the only value in high entropy.

That's exactly the reason I meant - to make brute forcing take much longer.

> Only on a touch screen

Yep, certainly worse on a touch screen, but I hate entering long passwords even on a keyboard - it's OK if it's a field that let's you see what you type, but if it's just asterisks, then I find it hard not seeing that visual feedback.


2-factor authentication would've mitigated any one of the above failure points.


4) What was the user name? A password should be no good without a user name.


The point of being a CEO is that you are ultimately responsible for what decisions are made. Even if decisions are delegated. Because guess what, you as CEO are responsible for delegating correctly.

Failure to realize this is shameful.


Each time leader blames his team for a failure, it’s as clear of a signal as you can ever get, to runaway, both as an employee and customer.


This ought to be a motivational poster.

Throwing an intern under the bus is silly and disgusting.


Exactly. It may not be direct responsibility - but it's still your fault that password policies weren't made obvious on the onboarding process for IT hires for example. It's your fault there isn't a culture of technical supervision, or regular auditing, etc etc.


It’s a good thing. Now we know this isn’t an isolated incident and that the company needs to be fully shunned.


The CNN headline is actually "Former" CEO, because he did in fact resign.


Couldn't agree more. The reason why CEOs are so well paid is to come up with processes to prevent such things from happening in the first place.

The management team should be the first to be blamed when such incidents arise.


I have never been a CEO so take this as a question rather than a statement. Someone in that position can lay our rules, regulations, governance standards etc for people under you, but how realistic is it that a CEO has any visibility on how they are implemented or enforced?


Yeahh, red flag not to work for that company whilst under his leadership


There's a reason "shit rolls downhill" is a common idiom.


There's also a reason the Ottomans needed to make an idiom to explain organizational failure: "The fish stinks from the head down."


“The buck stops here” is also a well known saying. President Truman had a sign on his desk with the saying.


Alas, thats a relic of a bygone era.

More recently, our politicians simply “don’t recall”, or worse directly lie to us.

For C-level folks, its simpler. Take no responsibility ever, unless forced to by the courts. Even then, taking actual responsibility is so rare that I have no examples.


Sadly, now the people with the most power also want the least responsibility (and the most money).


Yes, this describes nicely what sociopaths seek.

That this is common in most organizations shows that most larger organizations end up being selection systems for filtering sociopathic to the top.

It is not because their are deliberately designed this way, but because this is what sociopaths seek, and the organization fails to actively filter against it.



Great article - I'd seen another on the S/C/L model, possibly by the same guy, and this one went into more depth - it's a really interesting insight - thx!


My pleasure. It was posted a long time ago on HN and also a few months ago. Actually I didn't notice the recent repost. See https://hn.algolia.com/?q=Gervais


>It is not because their are deliberately designed this way, but because this is what sociopaths seek, and the organization fails to actively filter against it.

The sociopaths are the ones designing the organization and creating the legal and bureaucratic frameworks ostensibly meant to filter them out. That the end result nurtures and rewards them and allows them to use their subordinates as a bullet sponge seems entirely deliberate.


+1


The story line may be being twisted by CNN but the idea of a CEO blaming his staff is fundamentally wrong. The message should be “I created a culture which meant an intern could put the whole company at risk. This was my mistake”


Exactly, blaming the intern raises even more questions about internal security practices.

When will IT companies learn not to use telnet, ftp, and easy to guess passwords on their intranet.


This is a silly standard that nobody is held to in the real world. Nearly every company can have a breach due to an intern or another employee leaking credentials. There’s nothing the CEO can do about that.

It is often the case after these breaches that HNers start assigning blame in ways that indicate their lack of practical experience in the security world. After 20 years working in security, from startups to fortune 500s, including the most well funded security teams in the world, EVERYONE has stupid password problems in some corners of their company. The password leak was absolutely not the CEOs mistake.


With all due respect to your experience in the industry, if knowing that an intern’s password is solarwinds123 is enough for an attacker to sign and release an unauthorized binary and publish it to hundreds of customers, then that’s an operational problem that the CEO is ultimately responsible for.


Well, nothing like that even happened, so why are we talking about it? solarwinds123 was a password for a public ftp server. It did not let anyone sign anything and it didn’t push updates out. It isn’t even related to the solarwinds breach.


That’s the reason they’re having the hearing, the point is that the CEO is bringing this up and blaming an intern for the massive breach when it’s pretty clearly either a red herring or a complete lapse in security.


Ted Cruz claimed it was his children's fault for pressuring him into their trip, e.g. there is no lacking in incompetence and integrity in many systems, organizations, companies, communities, families. It starts with the individual which requires someone caring about how they look, about accountability, chain of command/responsibility.

How do we solve this though? Who is the CEO responsible to - who do they care, if anyone, about how they appear to? How do we shift this behaviour for the examples of CEOs and politicians who maybe haven't been outed yet by such a security breach but are vulnerable to such lacking of a rigid structure of command/responsibility?

Edit to add: I'm beginning to think HN is full of really lazy people.


> Edit to add: I'm beginning to think HN is full of really lazy people.

Because of downvotes? That's probably just because you brought partisan politics into the discussion.


> Because of downvotes? That's probably just because you brought partisan politics into the discussion.

Wut? Mentioning a politician’s name isn’t partisan politics. He said nothing about the politics of Cruz, he compared the Senator shifting blame onto his kids with this CEO blaming an intern. Don’t be like that.

(Edited to correct typo)


> Mentioning a politician’s name isn’t partisan politics

Of course it is, why would you think otherwise? By calling out a particular politician by name, especially a controversial one, OP introduced a political slant to the discussion. Could just as easily have said "this is a problem we commonly see with politicians, or other people in leadership positions" without calling a particular one out.

> Don’t be like that.

Like what, exactly? OP made a comment, and when it started to go grey he maligned the HN commentariat as a whole as being full of lazy people. All I did was help him understand what aspect of his comment was the likely trigger for the downvotes. HN by and large does not like political discussions, it's even right there in the moderation rules.


Thank you for pointing this out - it's unfortunate that people can be immediately triggered to react after simply seeing a name that prevents/blocks them from continuing on to understand or critically think to understand what's actually said; it's definitely a sign of the times and state of our health and our thinking ability as a society.


This is uncharitable. First, you assume people were offended (and you used the loaded term 'triggered' to describe it). Then you go on to assume this means that society must be degrading, because so many people are losing their ability to think (the implication being that you are not part of that group).

A much simpler explanation is that people who read and comment on HN do not want to see every conversation degenerate into yet another political fistfight, since it seems to have infected every other part of our lives. One of the core values of this community is that we mostly manage to avoid that.


My perspective is valid, you've done nothing to prove it's not - other than putting your own "much simpler explanation;" where you're also making assumptions. Hope you enjoyed the thought exercise writing out your reply.

It's interesting how I can reference, with no personal opinion attached to it, a factual, recent situation that's comparable to the thread - about a well-known person who happens to be a politician - happens to be a politician, and as you said, because you're exhausted of hearing about them, you'd prefer they be censored/avoided in conversation. It's interesting to say the least.

P.S. Triggered is a valid word, I used it in the correct context - and you were in fact triggered by using it and are trying to gate keep its use.


What you are doing here is a pretty good real world demonstration of Russell conjugations. Also very non-productive, so I don't think there is any reason to try continuing the discussion. Have a great day!


Re: Russel conjugations - there was no "spin" to anything I said, I stated facts that were neutral - yes, that neutrality which is certainly a better light than you're appearing to believe Ted Cruz deserves (which has nothing to do with my own perception of him - it's your own belief or reaction you're applying to the facts). There is a difference - and if you read the examples at Google for searching "what is russell conjugation" - https://tomdehnel.com/what-is-russell-conjugation/ - then none of them fit what I said because I was just listing fact, so my points still stand and I wasn't using any fallacies or bad faith arguments. I won't analysis the conversation further because it'll become inflammatory. I'd love for you to show exactly where I am applying bias to the facts I listed.


Your comment was down voted because it mentioned a political figure. In today's hyper-polarized society, any mention of politics in a discussion about something apolitical preemptively draws silence or disregard. No one wants to be dragged into an adjacent unproductive argument.

Looking at your other posts, it also seems that you easily show a pessimistic attitude, similar to how old people will complain about younger generations. No one responds positively to such generalizations.


What you seem to be claiming as generalizations I can only assume you speak to the holistic view that I often speak from.

Mentioning a political figure doesn't necessitate baiting one into a conversation - however yes, it's clear that people are tired, exhausted - and no longer actually read to comprehend, stunting critical thinking throughout what they're reading - and then making an assumption as to what they're reading, and then doing a lazy downvote for a dopamine hit to feel fulfilled - as if they were diligent, reaffirming their assumptions.

And I disagree that "no one responds positively to such generalizations" - it doesn't make them untrue, and usually I also reference the solution, policy proposals, to solve the problems.


> I'm beginning to think HN is full of really lazy people.

> Ted Cruz claimed it was his children's fault for pressuring him into their trip...

I'll bite. (Stick with me; I'll get to the point after some analysis that may be objectionable or seem unrelated.)

He's got responsibility to two parties: the citizens of his state and his children. Both are important in different ways, and the needs of each conflict in certain circumstances.

Did he want to get somewhere warm? Probably.

Did his children want their father with them? Probably.

Was he unable to advise while in Cancun? Doubtful.

Was he unable to legislate while in Cancun. He was not.

So, despite the inappropriate optics of his trip (and reflection of poor character for a leader of the state), it appears as though he made a judgement that favored his family and his self, presumably because he judged no apparent harm to the state by his actions.

I bring all this up because it's relevant to the parallel you bring: he blames his children for the choice. The problem is, neither you nor anyone else I'm aware of have demonstrated harm by his choice. So his "blaming" his children appears to reflect his choice between two conflicting priorities.

The intern, on the other hand, caused demonstrable harm and the CEO blames his own decisions on said intern.

Cruz did not blame lack of preparedness on his children. He blamed bad optics on his choice to acquiesce to his children. Far as I can tell, you're bringing a straw man to a knife fight.

I don't think your assertion is incorrect, that incompetence and lack of integrity are responsible for the CEO's choice, but I don't think you demonstrated that properly with your parallel. I find Ted Cruz to be pretty unpalatable, but I believe the hype around his trip is exaggerated, and your use of it to affirm your assertion is misplaced.

However, in an effort to make this more interesting, let's say the parallel is appropriate. I would, for the sake of argument, challenge that the fault, instead, lies in the hands of the people of Texas. If "the buck stops here", then let us remember that there is an entity above that of Ted Cruz, and that is the people of Texas.

If we truly believe that mistakes are not the fault of the individual who made the mistake, but rather the person at the top of the chain, then it stands to reason that politicians are not at fault, but rather the citizens that voted that politician in.

Since voters have ultimate responsibility to decide the way in which their community is governed—just as a CEO makes choices that bubble down—then it seems to me the people are ultimately responsible.

And in this context, your parallel seems even more misplaced, because the people of Texas (and the rest of the country) are blaming the "little guy" when those in charge of actually deciding who's hired and who's fired are the responsible parties.

Personally, I find the ultimate lack of integrity in "the people", because everyone blames "government" when "government" goes wrong, but fail to acknowledge that they are, in fact, said government.

I charge you to challenge my assertion and justify your parallel.


> they are, in fact, said government

Great! where do we sign up to vote on legislation?


I see this, though I've exhausted my mental focus for today - I have severe chronic pain to contend with every day. I promise you I will respond likely by tomorrow, perhaps only will get a draft done and then Monday will reply. First though, thank you for taking a bite and spending energy engaging - I appreciate that alone.


Thank you, and I hope you feel at least a little better tomorrow.

I'd like to clarify one thing that I think was nebulous in my post: while I think "the people" are the responsible party, I meant to suggest that choosing who is the responsible party at all is close to arbitrary.

That is to say, arguments over who's held responsible more reflects who we want to be responsible than who may actually be responsible, which is in fact shared among several people. (The CEO, the intern, the person who gave the intern the password, the person who configured the system to require a password that could be leaked, etc.)

We choose the CEO (or Ted Cruz, or whomever) because of our own biases, in spite of the fact that responsibility is shared, and in spite of the fact that who's "in charge" is nebulous at best.

We do the same when blaming the 12 US billionaires (or however many there are) for income inequality in the country, or blame all white people for racial injustice, and so on.

That doesn't mean there aren't responsible parties, or that people and groups should be free from criticism. The truth of the matter is that—no matter how justified it is to assign blame—it's exceedingly rare for anyone to take responsibility for their own failings.

This shouldn't necessarily contort to whom and how we assign blame. But, rather, it should inform us of the breadth of the problem, our own contributions to it, and help us become better citizens, instead of shift the blame as the CEO or Cruz did.

If we find that unpalatable, but don't acknowledge our own complicity, then all we've achieved is a pitchfork-laden witch hunt, and not actual direction towards resolution.


Cruz had a choice: 1) Flee the state and do nothing 2) Stay and do ANYTHING to help

He chose 1.


Cruz using his children as an excuse shows either of these:

1) He's a pushover and his children, that we can assume he loves, he doesn't have control or authority over them for whatever reason; and in fact they'd have control over him.

2) He's using his children as an excuse to avoid taking accountability, and the problem down the whole line is a lack of accountability.

I agree the necessity of being in-person is debatable, however regardless, the lack of common sense to understand how it would be easily perceived is quite astonishing.

You're blaming the intern here - in fact it was the policies and protocol that allowed that whole shenanigans to unfold how it did: it shouldn't have been the interns responsibility to begin with - protocol should have prevented it, protocol and process that in a chain of command/accountability should reach the CEO if the protocol is adequate.

Of course the hype about his trip is exaggerated, that's to be expected - perhaps we've all been attempted to be gaslighted by Trump for so long that this doesn't seem like a big deal, voted in representatives not being actively engaged in military fashion to protect and help "his" people. We're so used to unqualified, inadequate, incompetent politicians being voted into power - that this seems okay - how many people died so far, how much damage was caused leading to economic damage and reducing productivity of those harmed - especially if they have to fight insurance companies for damage; it speaks to Eric Weinstein's commentary that it's only during wartime that the true leaders appear out of necessity.

Indeed, the people of Texas have been manipulated, arguably controlled subtly for many decades, directed by the duopoly - crafted heavily by industrial complexes - to put relatively bad people into the spotlight, for the mainstream media complex, controlled by a handful of conglomerates, to then amplify the two main narratives of either side of the duopoly. So is it really the people's fault? Yes and no, of course with the right policy proposals all of this can be countered strongly, and the change will cascade quickly, exponentially - policy proposals such as Andrew Yang's core policies; it's all the yin-yang dance of the universe, people need to suffer to learn - to pay attention, and hopefully we're at a part of a cycle where healing and creation is rapidly approaching, hopefully this isn't just a foreshock - where Trump's wave is still building to a tsunami - and we're yet awaiting the major earthquake before everything destabilizes and perhaps we're going to lead into a civil war - and perhaps even global war will start in parallel, if democracies can't be stabilized enough to create multi-lateral trade agreements to economically funnel resources away from the known bad actors who aren't behaving or falling in line to peace and freedom.

It's complex, there are systems that most don't know or understand or consider - and these systems self-perpetuate like a virus to keep counter attacks at bay; the autonomous immune system of America and the world is failing, inadequate. Luckily the purpose and value of distributing power, control - separating Federal government from the States - and further diversifying risk via having many States with their own varying rules - allows neighbouring States to shine light/the truth onto the States that have been darkened, influenced, lead too far astray by bad actors (domestic and foreign).

It's a lack of integrity in the systems - these flows that lead to multi-generational trauma, karma - action, consequences - passed down from one generation to the next. It's why my focus for all my projects is health: there are still ripples passed through stress-trauma of parents of war onto their children, and onto their children, and so on. We are generating new active trauma while not doing enough to support new people born or migrating to our more/relatively stable countries - and not enough to heal the existing population.

Also, I'm not sure I understand the parallel question. I do however think it's an elected person's responsibility to counter any claims - arguably by showing he was doing everything in his power he could help the people, and that there was nothing more he could have been doing + with proof to back it up; let alone traveling during a pandemic, where most people are prevented from traveling - that alone is terrible role modelling vs. AOC doing a social media effort to raise $5 million for the people of Texas. The issue is really the contrast is sickening - from the weak inadequacy of doing what's ideal and right vs. not - we need to purge the system ideally through non-violent means, America needs to throw up - clear all of the shit - the whole "clear the swamp" idea, the mantra that Trump misappropriated to rally and align with mainstream media narratives of demonizing the unknown - when in fact it's the devil himself who's managing and rallying the swamp; allowing those not blinded by or manipulated, indoctrinated into their control and lack of critical thinking and reasoning, and skills for integrity.


There is a stark contrast in the former SolarWinds CEO response versus Berkshire Hathaway CEO Warren Buffet's attitude. I noticed an interesting statement in Buffet's 2020 earnings report:

"The final component in our GAAP figure – that ugly $11 billion write-down – is almost entirely the quantification of a mistake I made in 2016. That year, Berkshire purchased Precision Castparts (“PCC”), and I paid too much for the company. No one misled me in any way – I was simply too optimistic about PCC’s normalized profit potential. Last year, my miscalculation was laid bare by adverse developments throughout the aerospace industry, PCC’s most important source of customers." . . . "I was wrong, however, in judging the average amount of future earnings and, consequently, wrong in my calculation of the proper price to pay for the business. PCC is far from my first error of that sort. But it’s a big one."

WOW. I rarely see that level of accountability at senior organizational levels.


SolarWinds: “My intern had prepared a report on PCC’s profit potential, which I used to decide the purchasing price. It turns out the intern had not followed our financial analysis framework correctly. This mistake therefore lies squarely on the interns shoulders, please talk to them if you have any complaints. My company and I should not be blamed.”


There is a man and Man.


Good companies: “this was a failure of process, no individual is responsible”.

SolarWinds: “the intern revealed our ‘solarwinds123’ password.”


Yes, an individual was responsible: the CEO. Ultimately that job comes with the responsibility for everything happening inside the company and as a CEO you ensure that you have the proper culture and control mechanisms in place that that sort of thing does not happen, and that if it should happen that it gets detected.


By your standards, zero companies have a proper “culture” then. That’s just not how things work anywhere. You might think or even assume they should, but they don’t.


The fun thing about being in charge is that everything is your fault. Blaming the intern, the person least able to defend themselves, is ridiculous. If an intern can destroy your business, you're doing it wrong.


What’s more damaging is that a simple password is the only thing keeping your business from destruction, not to mention exposing all Americans with a security risk.


And its at the expense of a young professional just starting their career...


The fact that the SolarWinds CEO dared to blame the whole security breach on a single intern is a damning testament to the rotten security culture at SolarWinds. Would you trust a company where a single intern can compromise security of all of its clients? The fact that the CEO does not understand how damning his admission is, makes the whole situation hopeless.


So, who is responsible for giving an intern - likely not even strictly under an employment contact with the company - access at that level. And who, in turn is responsible for failing in their oversight duty of this situation? And who in turn is responsible for a the department that this fell under where such oversight duty failures are possible? And so on. Before long, you're back at the CEO, but that time the question will stick.

As a CEO you've got nobody to blame after your first 90 days of employment.


I don't think I have ever been more than one week in to a job before I was given some form of admin level rights.

In small companies I can understand that. But I've had it also happen in government jobs, in big companies and startups that collect a lot of sensitive personal data.

In one case I was with a startup that had names, addresses, DOBs, phone numbers and debit/card details in their DB. When they hired me they didn't ask for references or ID. I was given full admin rights to their Azure account on day one.

If I were in charge I would at the very least want to validate a new employees ID before giving them any form of access to IT systems, and only elevate access privileges once they had an established track record within the company.


As an intern? I would find it somewhat worrisome if an intern at some random company would be given access to mission critical systems and data within a week. That would definitely qualify as an oversight failure, and 'small' is a pretty relative affair, if the company has four people then I could see your point - maybe - but if it had 30 or more than it really isn't ok.


We always use interns to do security critical work -- the less experience, the better. We also have exacting security standards, and a Critical Password Policy, since there are no non-critical passwords.

Critical Password Policy

Rule 1: All critical passwords must contain the company name, lower case, with no special characters or spaces.

Rule 2: All critical passwords must have between 3 and 1024 randomly selected sequential integers appended, each of no less than 1 and no greater than 3.

When you are creating your critical passwords, it is critical that you follow the rules in the exact sequence as stated above, and that you do not introduce any external sources of randomness or entropy.

Failing to follow this Critical Password Policy may result in your dismissal and later blame before a Congressional committee.


That’s all well and good but you should really make sure to disable any kind of TLS on these connections so the security team can monitor for breaches. Any ops team worth its salt will make sure to secure all endpoints by physical device address and remove single points of access like gateways to widen the attack surface and ensure attackers can’t focus on one location. Store any critical data in a basic S3 bucket and remove any firewalls on production databases so that anyone on the team can fix a problem or restore a backup if needed. And you should really disable most logging, it’s a performance bottleneck and can tell attackers what’s happening in the system.

I’ll type these notes up in a PowerPoint and we can deliver a presentation to key stakeholders.


Here's more context about what the password was for.

https://twitter.com/vinodsparrow/status/1338431183588188160

This tweet has a screenshot of an email from Vinoth Kumar (named in the CNN article) to SolarWinds saying:

    Hi Team,

    I have found a public Github repo which is leaking ftp credential belongs to SolarWinds.

    Repo URL: https://github.com/

     Downloads Url: http://downloads.solarwinds.com
     FTP Url: ftp://solarwinds.upload.akamai.com
     Username:
     Password:
     POC: http://downloads.solarwinds.com/test.txt

    I was able to upload a test POC.
    Via this any hacker could upload malicious exe and update it with release SolarWinds product.
(The tweet blanks out some things including part of the github URL, the username, and the password.)

My thoughts:

(1) I assume this means when it comes to technical measures to prevent a weak password, SolarWinds would have to rely on Akamai.

(2) The researcher was able to upload to the root directory of downloads.solarwinds.com. As an educated guess, this may have been a shared account and many people knew this password. When many people share an account, they tend to choose passwords that are easy to convey to someone else. If so, the intern probably didn't create the password and was only responsible for leaking it.


Also note from that thread: SolarWinds gave him $0 for finding and reporting this. That in itself is completely irresponsible and a reflection of the companies systemic issues, which can’t be blamed on one intern.


I think they know this is a pretty desperate story. Some of the other coverage [1] suggest Solarwinds had so many different overlapping security failures they may never be able to attribute to a single cause.

Still there's some interesting things that could help a (much smaller, less critical) software vendor decide where to focus their security efforts. Perhaps near the top of the list should be: 1. Who in your organisation has access to your build and distribution toolchain, and how secure are their credentials? 2. How good is your record keeping? Are all your builds traceable back to a specific revision in your source control, and are you keeping build logs somewhere they can't be tampered with?

[1] https://www.bloomberg.com/opinion/articles/2021-02-26/deepen...


This is great and very publicly (unintentionally) shows exactly why SolarWinds got hacked. A CEO scapegoating an intern is what you would expect from a company with very deep systematic problems which led to such an embarrassing failure.

If your least experienced employee can accidentally topple the entire company, it is the entire company at fault. There are cases where individuals can undermine a whole organization and be at fault, but that requires sophistication and corruption which go above and beyond solid safeguards.


Waitaminute...The password was “solarwinds123,” and the CEO is blaming an intern?


Yeah because everyone knows you go all the way to 9 like you are slapping the piano keyboard in movie Big.


Oh those interns... Few years ago, police launched an investigation into a European dating company. They were prohibited by court order to destroy any evidence.

Turns out in an unfortunate and unforeseen turn of events, an intern wiped their production Hadoop cluster just a week later with the backups having some issues.

He was fired pretty quickly, but I heard he hasn‘t been too bad off since...

Don‘t hire any interns, they can do quite a lot of damage!


Ultimately, the CEO should take responsibility for everything. Finger pointing is not a leadership skill.


Somebody appointed the CEO, no?


Ah, yes. To blame someone, we must first invent the universe.


Sure, SolarWinds is a corporation, it will have a Board of Directors representing the interests of its shareholders, it has at times been a private company (so its shareholders were some handful of private equity investment companies) and public (so they will have included funds and the great unwashed) but in either case the Directors are responsible.

So in terms of specific people that's Bill Bock, the chairman of that board at the time.

They did indeed replace their CEO. I would assume after agreeing to pay off the old one as this is the usual practice and, unlike some no-name intern, a handsomely compensated CEO can afford expensive lawyers if you try to kick them out uncompensated for incompetence based merely on the evidence that their inadequate oversight cost you billions of dollars.


If your companies existence relies on interns not making mistakes. Security breach is the smallest of your issues.. Company board is probably like, fingers crossed, hope the other interns are good...

The last time an intern made such a disproportionate impact was in 1996.


Did these guys really think people would buy it when their excuse is a literal cliché? What’s next? Did a dog eat their homework?


Wow, way to remove your product from consideration ever again.

Security breaches happen - fix up the issues and show people your reform. Blaming the intern just makes you look like an ass whos company I never want to do business with again.


According to the article, SolarWinds doesn't seem to think that the password itself is a problem, only that it was leaked. And they "took it down", that sounds as if they deleted the leaked document. It doesn't sound as if they changed the password.

Amazing.


Seriously. Blaming the intern is a bad look, but here it reinforces the idea that they don’t even understand what was wrong. The fact that it leaked is only relevant is that it shows 1) how bad their password is and 2) how deficient their process is for dealing with (let’s face it, inevitable) leaks.


This just means that the CEO doesn’t understand security risks and must be fired.

Any company that continues to rely on Solarwinds after this, if the CEO is not fired, is accepting that their security is only as good as a SolarWinds intern.


He already resigned - the actual CNN headline is "Former" CEO.


"A loss of X dollars is always the responsibility of an executive whose financial responsibility exceeds X dollars." - Gerald Weinberg's 'First Principle of Financial Management' and 'Second Rule of Failure Prevention' [1]

[1] 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993


As an intern, you usually work for free or for subpar money, and expect to be taught the basics of the job and some good fundamentals. The fact this intern lacked the security knowledge any mom and pop company employee should have suggests the company simply wanted free labour from them.


While this of course is pure BS, with the way they handled all this so far my trust in them has been lost quite a while ago. And I guess I am not the only one. My question though is what alternatives are people moving to now? Any experiences?


> Confronted by Rep. Rashida Tlaib, former SolarWinds CEO Kevin Thompson said the password issue was "a mistake that an intern made."

Imagine a bank where the CEO says "the problem with all the money going missing was that an intern dropped the keys to the security vault and we had told the guards to never question anyone who had the keys, just let them take whatever they want without question". Seems to me that you can't pin anything on the intern. The problem was the extreme lack of security practices at the company, which ultimately falls on the CEO, who's trying to blame his own incompetence on a intern.


I prefer to remember only one long passphrase and enter it in only one place – my password manager. And then I trust my password manager to generate, store and submit the right password to the right portal and only upon my consent. This is vastly better than I remembering and entering the passwords myself. But it is still not perfect. I would rather all websites and apps universally moved to a standardized machine friendly authentication API and then I have a authentication user-agent (my password manager) do the actual authentication.


Wouldn't that be introducing a globalized single point of failure?


The password managers can have many different strategies for keeping their seed secret – it could be local hardware backed or cloud based but locked to a few of the devices you own etc.


This exists - webauthn!


Intern, yea sure, the cheapest excuse ever


So, if you want to compromise the security of 100,000’s of IT departments, get an internship at SolarWinds?


This is an interesting way of looking at it. Undoubtedly there must be nation state actors using such attack vectors


A chain is only as strong as its weakest link.

This means that the CEO, CTO, and all management down the chain failed to create any kind of robustly secure system.

Their entire system was never more than a single password leak away from complete failure for the security of the entire USG.

With literally thousands of opportunities for such leaks per day, it is inevitable.

So, a system must be designed such that when the inevitable occurs, the consequences are minimal. The most basic of fail-safe designs.

These people, from the CEO on down, utterly failed to do this.

Yet they made millions from false claims to have succeeded in creating a secure system, when they created a highway for global espionage. They literally could not have provided our adversaries a better avenue for espionage if they tried. The free world would literally be better off if all of that company and it's management had never existed.

Meanwhile, as a small manufacturer who has some govt-related work, the blizzard of new security certification requirements for Controlled Militarily Critical Technical Data and the like coming down the pike is like nothing I've ever seen before.

Sure, some of it is due to increased threats, but much of it is definitely because of a*$holes like those CxOs who so utterly and deliberately failed in their most basic duties, and should never work in the industry again.


> Current and former top executives at SolarWinds are blaming a company intern for a critical lapse in password security that apparently went undiagnosed for years.

The SolarWinds execs have provided us with keen insight for the root cause of the SolarWinds attack, but the insight they conveyed is probably not the insight they intended to convey.


A few years back, I caught a security consultant from Thales pushing his credentials on GitHub as he was working on what seemed to be a personal project during office hours and trying to get throught the corporate proxy... Considering Thales creates the most sensitive system, I completely lost trust in companies who claim to make secure products


Personally, I am more comfortable socializing benefits and privatizing (my own) mistakes. Which might not be the best way of phrasing it, but if someone else made a mistake, whether it's widely known or just something I've been privy to, I'm happier saying that "there was a mistake" rather than pointing the finger. However if I made the mistake, I'm content to say it was something I personally did wrong, it either case I will follow up with the steps we can take to prevent it happening again.

Conversely, if I delivered something of noteworthy benefit for the project I will talk about it in terms of how 'we' achieved the success, and if it was someone else I will call them out by name and congratulate them on the solution.

This ethos is probably career limiting as a dev, but it does help me sleep soundly.


And the same culture runs through Microsoft-

Rep. Katie Porter: "You and your company were supposed to be preventing the Russians from reading DoD emails!"

Microsoft President Brad Smith: "There is no indication, to my knowledge, that the DoD was attacked"

That's accountability! dilligence! Microsoft at its best.


What's next, the janitor was responsible for the companies poor quarterly results?


Related ongoing thread:

SolarWinds CEO blames intern for password leak - https://news.ycombinator.com/item?id=26284782


The password is invalid on its face Eg - Maximum number of days a password may be used.

Minimum number of days allowed between password changes.

Number of days warning given before a password expires.

Says that you have to use any password within 0+x days - otherwise it would have expired - the older the posting the more unlikely it would be valid. Would you risk detection with a password without mix case and consecutive numbers. Most would avoid traps or suspect a trap


Password expiries are not really considered good practise anymore.


@dang any chance this could be updated to the correct headline? So many comments in this thread calling for the firing of a CEO who already resigned.


New CEO should be fired too. They are failing at crisis management 101.


Dig him up and fire him again.


This CEO needs to go, he apparently has zero experience in crisis management. He has signaled to all of his employees that yes he will throw them under the bus. He has signaled to his customers that there is not enough oversight on operations. As such he has now lost all credibility.

It's a shame really, to blow a totally manageable crisis where solar winds could emerge stronger as a result.


Blaming the intern instead of root causing it as a systemic failure just leads me to wonder if they will actually take the correct steps to prevent this from happening in the future.

Fwiw this is a good archive of mostly well done post mortems: https://github.com/danluu/post-mortems


The only thing the CEO of SolarWinds is telling us by saying this is that he's a terrible leader. We don't even know that the hackers used the exposed password at this point. But even if they did, this is an obvious process failure and by blaming an individual intern, Sudhakar Ramakrishna is loudly demonstrating his own hopeless incompetence.


Pretty terrible when you think of all the govt money this company took to provide these services. Likely charged at massively inflated rates for the benefit of their "expertise". Guarantee you that intern was probably billed out at 10-20x what it cost SolarWinds to employ them. Doesn't even pass a level 1 smell test.


The same intern worked for Volkswagen as a developer and is solely responsible for the whole miles per gallon debacle.


What a good look... one of the people at the highest tier of a company blaming someone at the lowest tier.

Are we all this blind?


No, we're all mostly just distracted by so many problems and failures throughout our society. Someone commented the other day mentioning how it doesn't take a genius to see the West is degenerating. To me it's obvious it's the industrial complexes involved in regulatory capture - fuelled by bad actors, greed, targeting the richest nation in history - leading to bad policy for people, individuals, for too many decades now; including policy allowing the duopoly to exist, thrive, and the tightening and expanding conglomerates of a handful of mainstream media companies also then perpetuating the two main narratives of the duopoly, along with whatever other for-profit interests decide to influence/manipulate us through shallow, cheap advertising. Thank Goodness for the internet, technology, to allow anyone with a voice a better chance at gaining a following and a communication channel - so we can bypass the control of those systems to educate people, and share different narratives - truths.


If you have a system setup where a single intern can destroy your company, you are doing something wrong.

Let's even say the intern was malicious and trying to do harm... that is still your fault. One person in the lowest position in the company can break everything? Again, you are setup incorrectly.


This is the equivalent of stock crashes being blamed on what the media has dubbed “fat fingers” of a trader


Remember the old nortel network equipment default passwords like admin/admin that telecoms didnt change?

Good times.


This environment sounds like a case of no accountability and negligence. If a intern can make this type of decision without review and accountability, what other decisions are being made under the same environment of no review and accountability within the company?


That is a symptom of an organization that improperly allows interns to access important information.


The buck stops with intern. That’s why the intern gets paid like 20 million dollars a year.


It shouldn't be possible for an intern to leak a password, or if it is, the blast radius should be limited. In other words, it's the job of senior people to make this sort of thing hard or at least not super damaging when it occurs.


Would you blame the intern if they ruined the production DB after being given write access?

I wouldn't. I'd blame senior technical leadership for putting processes into place that allow failures like that. Most especially, the CEO.


That really doesn't make them look any better. If a single intern's password mishap can breach a security company's systems on this level, they've lost the fight long before this incident.


They really should have tried harder to find a more convincing scapegoat.


The article says the "former CEO" said that, not the current CEO.


CEO’s are typically quick to claim large bonuses “because of the huge responsibility I have” and even quicker to not take responsibility (and keep their bonuses) when things go wrong.


Maybe it was the intern's fault to set a weak password but it is the CEOs fault for setting up an organization where things like these can slip through security review or monitoring.


Lol SolarWinds is never going to be able to find an intern again.

Who would want to work for a company that throws them under the bus for the largest hack in recent history


This is modern American corporate culture after all. Lie, cheat, steal, win at all costs and if you lose blame someone at the bottom of the ladder.


It’s extremely weak leadership to blame the intern... makes me think it’s probably the CEO’s fault with such a lack of character.


As a 20 person startup we have a KMS with only two authorized users. Seems like SolarWinds tried pretty hard to be hacked.


This is so outrageous a claim that it warrants the technology community responding to it. Where is Bruce Schneier?


Cool so its totally not his fault then. If only there was someone in charge that could handle this situation.


Someone didn't see Spaceballs.

"12345? That's amazing I've got the same combination on my luggage!"


Let us step back one. What policy in place which allows such a password in the first place?


This is so bad. A proper ceo, cto, executive, manager, leader does not point any fingers.


This is the weakest leadership reaction possible in a giant fuckup scenario like this


It was either an intern or a third-party contractor, and they went with intern.


Next week: SolarWinds board blames CEO for password leak


lol if an intern can leak critical passwords you prolly got a ton more problems than just the intern leaking critical passwords


We still doing the blame the intern thing eh?


Somebody hasn’t read Extreme Ownership.


Always the honourable thing to do!


ohhh really bad move blaming the person instead of the process, CEO.


tl;dr: If this CEO's security systems were so weak that a single password leak by an intern can compromise his entire company, the entire US Govt and many others, and much of the Fortune500, then his security systems suck - all the way up to insane values of suckage.

The fact that he is blaming the intern shows that he knows his systems suck and is attempting to divert.

The entire management chain from CEO down to the intern's manager is at best too incompetent to work in the industry, and more accurately, a major threat to national security.

Anyone who trusts these people to do so much as run out for a pizza is a fool.


poor intern is getting thrown under the bus


what a shitshow


I worked at two big, well known software companies which had the same default password - company123, albeit for low security stuff, where you wanted wide access but a password was required for some reason.


I find it interesting that some companies don't secure their test domains, but do rely on merging from test to prod.


Well, a lot of companies are doing it in reverse , by not properly securing test sites, and bringing live sensitive data from prod to test . I don't know which is worse


Should have used an encrypted complex password and a password manager.

It is the fault of both of you. ¯\_(シ)_/¯


In no way is this an interns fault. If your entire infrastructure relies on the secure password of ...

checks notes

... a single intern! then you're doing it wrong.


> "secure password"

Whatever that means.

This would never have happened in the first place had they used an encrypted complex password and a simple password manager.

The whole company takes the hit with blunders like this. It's everyone's fault responsible for the infrastructure allowing this to happen in the first place. Those who pass the blame on others very quickly are equally to blame which means the CEO is just as to blame as the 'intern'.

Clearly the whole company doesn't train their interns.


The password was coded into a file that was checked in to git.

The git repository just happened to be public.

It's entirely reasonable to think that the person in question possibly didn't even stop to think that Solarwinds123 was an actual secret that needed to be kept, as it is the equivalent of common passwords that are published publicly in manufacturer documentation.


I think the point is at no point is it acceptable to be in a position to be able to do something deeply damaging to a company with something as simple as a intern leaking a password. The intern should never been put in a position where this was even possible.

I’d say in all the ways that matters this was basically everybody BUT the interns fault.


You can't really blame someone who is explicitly there to basically do what they're told and learn as much as they can for a short period of time.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: