Real Story Here. A colleague and I developed a training course on the Foundations of DevOps. We started the course off with a discussion about risk in Software Delivery and IT in general. Attendees would often surface the "Key-Person Risk" or the "Hit by a Bus Risk" to which we would use as a teachable moment for pair programming, cross pollination, and measuring and improving the time it takes for new joins to become productive with the systems and or code base. We would illustrate that one of the ways we mitigated the Key Person Risk with the course was to always deliver it in pairs.
My colleague was tragically struck by a car and killed about three months ago. The course suffered an immense blow to the knowledge capital it had - but due to the fact that we had planned to mitigate against key-person risk by pairing - the course was able to survive and retain a fair bit of the legacy my colleague left behind.
I just share this to say that the getting "hit by a bus" scenario is something that, much like this post, we throw around in jest and treat in a laissez-faire fashion. The fact is that it is a tragic, unimaginable, event that does in fact occur from time to time and something I will take seriously for the rest of my life.
I've heard "wins the lottery" as a less morbid metaphor for the same concept: one of your employees stops working in a low-probability event. Of course the higher-probability event is leaving their job (but somehow it's less polite to just say that); you usually get two weeks to hurriedly start documenting things, and with the current state of employee/employer relations you might easily get less. Or they might have several days of vacation saved up.
As far as I can tell, "hits by a bus" is actually the euphemism. The highest probability way to lose an employee is just the employee quitting to work somewhere else. But you can't say that because it implies that your current workplace is suboptimal. Whereas "hits by a bus" is out of the control of your employer, so safe to say...
It's hit by a bus because an employee leaving, or dieing of a prolonged illness leaves you time to plan for a departure or possibly ways to retrieve information from them after they are gone. Immediate unexpected death is a worst case, and "hit by a bus" neatly sidesteps a lot of questions such as whether there was someone with them, whether the other party in the accident was hurt, etc which can derail the point.
Perhaps its "hit by a bus" because assuming that an employee will give you a solid 2 weeks of documentation on leaving and will be able & willing to answer questions after they leave is very much not a safe assumption.
I don't know if this is the norm outside of SV. I typically stay four years at a position and almost every interview I've had, people will comment about how short of a tenure that is.
It's definitely not unusual for developers to stay in roles for a decade or more. Especially very good ones.
I see all the comments about needing huge pay bumps every 2-3 years and then I also see things about how you can’t get a job in sv over 40. I wonder if it’s related
I'm not so sure the 30% pay increase with every job hop is necessarily the norm either. Maybe at first when you're moving up from entry-level to senior. But in some markets, you hit the regional salary cap after four of those and you're stuck with 2-4% annual raises -- even with in-demand skill sets.
If I posted the job offers I've received for having experience with Spark/Hadoop, building / productionizing machine learning models (since 2011 even), an AWS certificate, and 15 years worth of other worthwhile skills, people here wouldn't believe me because they are so low. Definitely well below the startng salary of someone at Google. Yet I've had three, all of which were basically identical.
Hmm. Been at the same company over five years because there were drastic changes/new things every six months. Luckily they were able to do a few promotions and stuff... :)
Hmm, data compiled from "Public Profiles". I wonder where they got "Public Profiles", listing people's profession and length of tenure. Could it be places like LinkedIn or Dice, and could it be that such places overrepresent engineers inclined to switch jobs?
Personally, I'd be disinclined to interview a candidate with >= 3 jobs on the resumé, none of which lasted longer than 2 years. Taking into account the time spent for them to get up to speed, and the time their successors would have to spend taking over their code, it seems unlikely that their tenure would make much of a positive impact.
Furthermore, a key driver of experience in software engineers is maintaining their own code over multiple product cycles. Job hoppers not only don't acquire that experience, but may actively fall into a habit of bailing as soon as they get sick of their own code.
- Salary compression especially early on in your career is real. Most companies don’t do market adjustments after you are hired and only adjust salaries by a predetermined very low amount. The only way to get market salaries is by frequent job changes.
- If you have a job and people are willing to hire a job hopper, why not leave? At some point it will be a negative but when that time comes, just stay at a job until you get a better one. Also, if you do change jobs make sure it’s for the right reasons. For me, it’s technology, environment, and money in that order.
Same for me. I've been at SAP for nearly 7 years now, and I know only a handful colleagues who left the company. Some people moved to different teams though, so that may count as well, but retention is still way way way higher than 1.5 or 2 years.
In the "wins the lottery" and "left for a new job" cases there is at least the possibility of knowledge transfer. I was brought in on a situation where the primary IT lead suffered a stroke, rendering him unable to communicate for weeks. We had to reverse engineer everything, crack passwords, rebuild servers, etc. This is, unfortunately, my go to example of the need to mitigate risk of a central point of failure.
the problem with 'wins the lottery' as a metaphor to me is basically that one is calling the winner a jerk..
if I suddenly could retire without a care in the world, I probably wouldn't mind dropping in for a day or two every now and again for knowledge transfer, even if I didn't like the company overall, to help my teammates..
I am the same way, but we are not the norm. In fact, most people don't even want to provide the occasional consulting once they leave the team (but stay at the company), let alone the company. There's also the problem of where to draw the line. Is a quick question ok? What about a bunch of questions? What about asking for a training session? What if the question requires diving back in to the source code to answer? Everybody has a different opinion and tolerance threshold.
I’m happy to provide consulting if you’re happy to actually pay me for my time. Otherwise I’ve got hobbies. I’ve had one job total where the company asked me questions and they stopped the instant I mentioned cash after 7 hours of chatting. I asked because it was actually taking measurable time to answer things. It’s up to them if they want to trade 60 hours of someone relearning a thing instead of an hour of me contracting. Not sour grapes actual numbers from a thing an employee mentioned at a party they did (gdpr) that I had 3 simple solutions to in minutes.
I will never do consulting again for a former employee unless I get paid upfront. A previous employer "adjusted" an incredibly reasonable bill. Then two months later their nginx server goes down and he calls me in a panic. I required them to deliver the check before I even took a look at it. Out of spite I charged him a near egregious amount because of that. On the plus side the office manager had a grin on her face, she forced the owner to pay the full amount of the previous bill. She thought it was incredibly underhanded.
Yep. In every company I've been in here in the UK, I have pretty much never heard of an ex employee returning to explain their work, fix a bug, do training, etc. And well, the more people leave a company, the less likely they are to come back to help at all, cause of a 'toxic' atmosphere or sub par working conditions.
I think pride plays its part here. People want to prove that they could indeed cope without the person. I think there is also the point where people think, "we are going to have to learn this stuff one day, it may as well be now".
In some key-person areas long notice periods (3months, for instance) are the norm. However this is never going to be the most productive 3 months of anyone's life! Serving 3 months notice becomes very dull by the end...
I don’t like when people use “wins the lottery” because it leads to distracting tangents. A person that wins a lottery could do anything. How much did they win? Maybe they’ll still keep working? Maybe they’ll go off to start their own company and take a bunch of key employees with them? Maybe they’ll be broke in a year and come back to their job? Maybe they don’t leave right away? These are just distractions you don’t need when illustrating a point.
By contrast, a person being struck by a bus and killed has a definite finality to it. There is no going back. There is no hope of extracting knowledge from the once living individual. There is not even time to prepare. Their existence and their output has ceased and everyone alive must now live in this new reality.
I think it's pretty clear what's meant by it, and "hit by a bus" has very similar problems if taken too seriously. Were they hurt? Are they out for a few days, just shaken, decide to change their approach to life etc.
> How much did they win? Maybe they’ll still keep working? Maybe they’ll go off to start their own company and take a bunch of key employees with them? Maybe they’ll be broke in a year and come back to their job? Maybe they don’t leave right away?
Well exactly. You don't know, and now you've given yourself four scenarios to think through:
1. Employee remains.
2. Employee leaves forever, maybe with others.
3. Employee leaves for a while but would come back.
4. Employee leaves, but with some time to plan.
1 should not be a difficult thing to plan for, and 2-4 all ask the same major question of "what if they leave". There's perhaps a slight benefit of not having to say "well assuming you are dead..." - I know I've had some time this year where I was trying not to focus on a death.
The key point is the risk. If your answer to "what if X wins the lottery" is "well as long as they don't want to leave..." then you're already setting yourself up to answer the question it's intended to propose - what if they are no longer working for the company?
I agree, and the "don't want to talk about morbid stuff" shows an overall immaturity in thinking about risk and controls. People die, people get sick, people leave for other jobs or roles, people steal. The institution has to be able to survive.
That's not to say folks should be callous about it. But failing to think about things like this is also unfair and stressful to employees who are not dead.
Hm. I guess if you read it as a euphemism for getting a better job offer elsewhere, winning a medium-sized lottery prize is more accurate: you can, in theory, contact a former employee, buy them drinks in exchange for getting their advice on a thorny problem, counteroffer them either now or in a year, etc. (And with a larger prize it's not impossible they'll use the winnings to start their own company or maybe even split the winnings with a coworker they're friends with.
Winning a lottery prize large enough to never talk to anyone again, and getting hit by a vehicle, are both very low-probability events. Employees (especially good employees, and perhaps in groups) leaving is a much more pressing risk.
The great thing about hypotheticals is you can just say "wins the lottery, leaves the job, and never comes back" and it's implied in the scenario that you shouldn't read too much into it.
Some people wouldn't dare speak to their former employers, let alone help them, if they won the lottery. Death or not, winning the lottery can be just as effective in getting rid of someone.
That's really tough. Sorry to hear about that. It's all abstract until it actually happens and then suddenly all those lessons bear fruit. You of course hope you'll never need them, but just in case you (or those that take over from you) do you will be very happy (insofar as the situation allows) that you were prepared.
It is an interesting phenomenon that I hear several times per year 'but what are the chances of this actually happening?'. The chances in any particular case are pretty good. But with 43 companies we looked at in 2018 and 100+ over the last years there are multiple such instances that I can refer to so even if the chances are not super high they probably are a lot higher than you think they are!
Actuarially, just go with 1 per 1000 employees for bigger companies. Go 1 in 3000 as a lower boundary, but as high as 1/500 or less for companies with older or lower skilled employees.
I do agree with other commenters that your procedures should keep you safe in case of regular churn (say 1 in 3 to 1 in 10) or even non-regular churn (say due to a toxic team-lead and a whole team quitting).
Ideally procedures should be in place that treats all these events quite equally and favor robust practices.
Upper management tend to travel quite a lot and tends to be under high levels of stress. So turnover there, due to buses, disease, burn-out or other mishap is substantially higher than in the population at large. And of course they can simply stop showing up one day for reasons all their own as well. In all of those cases it might as well have been a bus.
We had a colleague pass away abruptly a month-ish ago. So much disappeared with him. And I kept finding myself typing out, "oh ____ would know that" before reality caught up with me.
It started to get really frustrating and upsetting. I'm not exactly a "pair programming" zealot but I definitely find value in overlap of responsibilities.
The methodology seems flawed in that it does not take into account that, if the buses were self-driving, what operating system was used as this would affect collision lethality.
On a much smaller scale, I realized that I am probably the only person in my group (of stage/ audiovisual technical folks) who understands everything fully: I'm stopping next year, so I'm documenting how to do everything I do.
You can never have too much documentation - which isn't exactly what this dilemma is about - but still it raises an interesting bit of planning which should always be considered.
Speaking of getting hit by a bus, a friend and I are considering a consulting service that could help mitigate bus factor risk - would love any feedback on this idea.
We've both noticed that for nontechnical owners of small businesses that have a heavy software component, that owner is completely at the mercy of their lead developer. If that person leaves or "gets hit by a bus" their company might just die, especially since they don't really know what a Github is or how to log into anything. My friend is such a nontechnical person and hired someone to build his SaaS app. He has felt this risk himself but we wonder if he's just particularly perceptive.
The service would be, for a retainer fee per month a developer sits in on your planning calls, looks over all code committed to your repo, and over time documents how everything works and stores that documentation in a system outside of the company (maybe an external wiki or something). Then if your lead developer leaves, we'd help you find a new one and then train them on the codebase using the documentation created.
Over on the Let's Encrypt Community Forum we get quite a few inquiries from people who say "I'm the site owner and my developer [disappeared|quit|was only hired for a one-time setup task] and my certificate expired; how do I renew it?".
I assume this is a really common pattern for web sites in general. :-(
I don't imagine that this segment would be interested in paying for "continuity insurance", but they might grudgingly hire someone for "tech rescue". I guess that's a very different kind of service; I'm not sure if there's anything in between the two.
> I don't imagine that this segment would be interested in paying for "continuity insurance", but they might grudgingly hire someone for "tech rescue". I guess that's a very different kind of service; I'm not sure if there's anything in between the two.
Very sadly I find myself agreeing with your assessment in general. However I was once seeking almost exactly this service, and I have to say I struggled to find what I wanted.
You should start with convincing people that it is a non-negligible risk and then sell an apt mitigation. Nobody expects to die suddenly, and with a technical co-founder you don't expect them to leave the company to die, either. Depending on the actual risk (which I'm guessing is low), it seems much more efficient to have an insurance that will pay for costs incurred due to their untimely death/departure: pay for a replacement dev (for a reasonable amount of time so they can get to know the product), perhaps hiring costs, perhaps missed income for the company... Like with contents' insurance, I expect there would be a configurable amount that is paid out for all of these things, where lower amounts mean a lower premium.
Having a developer continuously kept up to date might only be affordable for larger companies, which might as well have someone doing that internally. I also don't think a single person can memorize enough about multiple startups to be worth it: either they hear about each customer's tech stack at least once every month (that already seems quite little, but is already relatively expensive so far as insurance-type services go), or they won't be worth more than written documentation.
If there were 200 Linus Torvaldses then after one gets hit by a bus there would be 199 left. Even after the experiment there were at least 108.9 remaining. That should be plenty to keep Linux going. So what's the problem?
Unfortunately, they all exist in parallel. So unless we figure out how to cryogenically freeze humans for hundreds of years, we'll run out of Torvaldii (which I think should be the plural for Torvalds) very soon.
I think this use of the genitive is a form of patronymic, like if someone named Torvald had a son named Einar, maybe that person could be called "Einar Torvalds" (that is, Torvald's Einar).
I'm not sure which declension it should belong to as a name (and I'll ask a friend to speculate), but the possibilities there seem to be Torvaldsor, Torvaldser, and Torvaldsar.
There are no Latin plurals in -i when the nominative form doesn't end in -us or -um. If you make a Latin plural of Torvalds, it would probably be third declension and something like Torvaldes (compare urbs 'city', urbes 'cities'), unless the consonants change in the oblique cases.
I want to ask a genuine question to folks here about disaster recovery and management. It sounds pretty horrible, but at what point do you just throw up your hands and give up? How many people in your company would need to be hit by a bus? Or maybe if aws went down completely for a week? I guess I'm wondering: how far into the rabbit hole must we go?
I guess the changes of a coworker passing away are more than aws going down for a week. I guess I'm just overwhelmed by how many scenarios we need to plan for.
That's not a traditional form of re-distribution. In legal terms, I guess it's a gray area.
The article is now on the blockchain. It is not published (re-distributed) anywhere in particular. It's also encoded and not in the actual form of an article.
Involving a technically convoluted mechanism doesn't change the fundamental legal question here. It's just made it harder for you to undo your potential violation.
A temporary web mirror has at least a narrow time scope and a justification. Choosing to embed a copy of content you don't own into a permanently-public record is a greater offense.
> Choosing to embed a copy of content you don't own into a permanently-public record is a greater offense.
Anybody making copies of that record, and anybody distributing them, would be (and probably already is) breaking the law.
But then, the one embedding it only broke the law once, by making and distributing one copy of the protected work, same as reposting it on a temporary web mirror. Worse, with a web mirror you make and distribute copies for every requests...
But for better understanding, I would love to hear more about "embed a copy of content you don't own into a permanently-public record is a greater offense". What is this based on? Are there any known precedents?
What’s crazy is that I voluntarily try to set aside time to document and cross train but I get push back. I don’t think anyone has ever voluntarily left the small company I work for and they don’t think about what would happen if I leave.
I try to cross train for a few reasons - I would never want to leave the company in a lurch if I find a better opportunity, I don’t want to be stuck on an implementation forever because I’m the only one that knows how to do it, and after awhile, I forget how the systems I designed works together.
There's a lot of comments considering the general case of losing a key person on a project, but I think it'd be interesting to speculate on what happens to Linux when Linus leaves the project! Will it go embrace, extend, exterminate? Will there be a bunch of forks? Will Redhat/IBM/Canonical types keep it up?
I got hit by a bus two years ago, granted it was more of a rude push by the side of the bus that I was riding for 6 hours from Athens to its last stop Preveza and the bus driver was just about to park his bus, but getting hit by a bus is not necessary lethal. I simply got back on my feet and dusted my pants.
I'm curious how startups or venture capitalists handle risk around DevOps, especially certificates, encryption keys, access to dns/domain names etc.
Legal processes take time, you typically need this info quickly in production. I'm not a VC, but if I was, I would want a backup plan in place before handing over an XX Million $ check. A smaller startup, or a seed level/pre-launch team may only have 1 or 2 founders with "admin" level access to this info.
I always thought a "Swiss bank account" type service for DevOps would be useful. It could handle this situation in a modern way (with a secure, modern API etc) and would execute a certain disaster recovery plan reliably.
Obviously centralizing that kind of data/ would be a security nightmare in itself. I'm curious how a larger VC with many companies deals with this consistently.
I know there are services available in different areas that specialize in handling certain data (corporate branding/identity, domain names etc) but not as a whole.
For example, there is a copy of the most core encryption keys, certificates/passwords etc, stored with a 3rd party, who is specialized in dealing with this kind of info in a modern way.
Similar to how a Swiss bank is familiar with storing valuables/currency or executing a will.
This type of service would need to be modern in the sense the info can be updated over an API etc.
Anyone can create a DR plan and handle these issues, but then it isn't consistent, especially from a partner (VC etc) perspective.
It also names Miguel de Icaza as a similarly important character, it was in 2000 because gnome, event considering Mono as an important technology nowadays, it's not as significative or critical as Gnome was back then.
I did read the text. It's humorous. But the question itself is no less important. And considering what happened with Torvalds' position this year, one might think it is a follow up to that question. So I think adding '(2000)' would definitely be helpful, if not just accurate.
English is pretty casual about borrowing other languages' plural forms, and Latin second declension nominatives -- alumnus/alumni, millenium/millenia -- are probably the most familiar non-English singular/plural pairs to English speakers. To an English speaker, Prii seems like a reasonable plural for Prius (though if it's a Latin word, it looks like a neuter comparative, in which case the plural is "priora"). Similarly, "Lini" looks like the plural of "Linus". It's an interesting philosophical question: given that English speakers are familiar with this Latin pattern, can apply it spontaneously to new words, and perhaps are even unaware that it comes from Latin, has this become a backwater grammatical rule of English? We're willing to say originally French words like apartment and address are English. How about Lini as the plural of Linus?
Afaik it's only with Latin that English-speakers insist on universal adoption of the foreign plural. For other words the irregularity is a rare exception―while for Latin it's pretty much regularity instead.
For some reason, people don't seem so keen on using French plural forms for French words: e.g. "animaux" and "journaux." Same with German ones, Italian, Yiddish or Arabic. Probably because borrowed words are mangled to the rules of the adopting language, like it's done everywhere.
IMO not only using Latin plurals in English doesn't make sense and is rather arbitrary, but it's kept around by people insisting on correcting others contrary to those others' intuition. It's time to stop.
My colleague was tragically struck by a car and killed about three months ago. The course suffered an immense blow to the knowledge capital it had - but due to the fact that we had planned to mitigate against key-person risk by pairing - the course was able to survive and retain a fair bit of the legacy my colleague left behind.
I just share this to say that the getting "hit by a bus" scenario is something that, much like this post, we throw around in jest and treat in a laissez-faire fashion. The fact is that it is a tragic, unimaginable, event that does in fact occur from time to time and something I will take seriously for the rest of my life.