I've seen memory drains during down-turns in the economy. The company will offer the oldest, most-experienced, highest paid employees a buy-out or they'll offer them five years extra pension/retirement if they leave now. Mgt figures that consultants can be hired if they ever need X, Y or Z done again at a fraction of the cost of the employees.
Many senior employees accept the offers, and then later mgt realizes that 95% of their Fortran or Cobol (or name your old programming language/product) experience has vanished. Thank god the software is mostly in maintenance mode, but client problems do still arise and what once took 15 mins to fix, now requires a week and phone calls to retirees begging them to help fix a client problem. They'll pay them loads of money too just for a few minutes of work.
Consultants turn out to lack the domain specific knowledge required to fix the problem.
Bottom line is that sometimes, mgt is so focused on hitting short-term financial goals, that they lose site of the big, long-term picture. Business needs less 20 something MBAs who only see this quarterly report and more managers who understand that business is a decades long proposition and that senior employees are key to maintaining an edge and passing along knowledge.
It's easy to blame the popular meme of "short sighted management", but they aren't necessarily as short sighted as you think.
Often management gets rid of the old guard because the old guard actively prevents the business from improving it's processes. I know someone (maybe multiple people) who works at a large financial institution. A particular department there is running on ancient systems using ancient processes, most of which can and should be properly outsourced to $BIGVENDOR. The fed agrees, and is constantly hitting them with MRIA's (Matters Requiring Immediate Attention).
But right now there are many managers of 50-100 employees who will become redundant when 49 of their employees are replaced by a few servers in NJ. These people tend to passive aggressively slow down the replacement, hoping to delay it until they retire. There are other people who have stable jobs, based not so much on competence as the fact that they hold lots of institutional memory. They don't want to help build a system that will make their knowledge obsolete.
When management fires these people, it doesn't do so because it doesn't understand their value. It does so because they are actively hindering the company from moving forward.
Sometimes this process fails as you describe. But when properly managed, this process results in a streamlined department half the size it was before, processes properly documented, everything automated, where people are fighting to improve efficiency from 99.7% to 99.9%.
I upvoted you because you explained your downvote. Anonymous voting is what I like least about Hacker News. I prefer that people vote publicly. On my own site, I'm about to start offering actual cash (very small amounts, mere pennies) to people who explain their votes.
I would be interested in learning how that experiment goes. Social hacking fascinates me and at one time I spent quite a lot of time performing "private" experiments/tests with regards to influencing forums/email lists. I would be especially interested in seeing a write up done beforehand as to your logic/reasons for doing it and what you hope to achieve and then a write up afterwards to see what the actual results were. The information is a lot less valuable/interesting to me if there is no beforehand piece of it, put in writing before actual implementation. However, it probably should not be published until you get results, just written down. Publishing it could contaminate the data as it could influence the outcome and make it impossible to tell if the money or the article was driving change.
That is a great idea. And I understand the need to keep the prediction secret till later. I just wrote this up and sent it to you as an email. We will see how things work out over the long term.
+1 to documenting the hypothesis+ prediction and the results.. I would love to see that too.. yep.. am ok with not publishing the hypothesis first too..
This depends a lot on what time frame a business is planning for. It used to be industrial businesses were planning for 30-75 year timelines; now it seems rare to find an MBA who cares about 10 years hence.
If what you care about is the health of the business in a year or two, the chances that that institutional knowledge will be important is almost nill. It is only if we had a way to prioritize lasting value that this becomes a management concern.
It's not always that. Sometimes (especially in smaller companies) the technical staff just leaves. Sometimes they get better offers. Sometimes they go off to start their own company.
I know some of what the OP is describing. I'm dealing with a system that's got three layers, each written in a different programming language. It was written by a pair of very brilliant programmers, who were unfortunately rather bad at documenting. They left the company when an after-hours project they were working on turned into a viable business opportunity.
When I deal with this system, I can piece together the what, but the why often escapes me, only to bite me later when I try something and it fails horribly. Luckily, I'm only dealing with software. I suspect that it'd be much more costly to deal with failures in OP's case.
This is sad, scary, and as far as I've seen an inherent part of any long running engineering project. As software developers, we have a hard time understanding the rationale of some of the code we wrote last week, let alone someone else's code from a year ago (I don't want to imagine any of my company's code running 30 years from now!).
In a recent series of design meetings I was running for a large project, I started to tackle some of these problems by emailing out the final decisions we had made at the end of each meeting. The following week we'd still have problems in understanding/remembering the chain of arguments that led to the final decisions, and we'd lose some time re-tracing our steps. So I then adjusted to not only summarizing the decisions at the end of each meeting, but noting step by step the arguments/debates that led to it. Then at the beginning of each meeting I'd review them to the group. This worked, but it was an awful lot of note taking.
I almost wonder that now that more and more of these discussions are done through work email, IM, version controlled documentation (and code), the job of future archaeologists will be much more manageable (though still hard and fraught with risk). Maybe more and more tools will develop to piece together a project's evolution, rationale, and implicit assumptions and decisions.
The problem with going through people's old email and such is the signal/noise is extremely low, and often of a timely nature, and may not represent the final outcome, especially if work has been done on a system after the fact.
What we need is testable documentation. I do IT work most of the time, and document using a mashup of Markdown and text snippets in a specific format for network documentation, all kept in version control.
I then have a motley collection of scripts and tools that run against my text files and do simple testing like pinging all the hosts mentioned, checking DNS entries, etc. Frequently I'll write up docs before I start work, then by the end of work, all the tests will past. It's TDD applied to network design.
I almost wonder that now that more and more of these discussions are done through work email, IM, version controlled documentation (and code), the job of future archaeologists will be much more manageable (though still hard and fraught with risk). Maybe more and more tools will develop to piece together a project's evolution, rationale, and implicit assumptions and decisions.
It's funny, Wave was actually a really fabulous tool for exactly this.
We aim for self-documenting code. If there is a decision that absolutely can not be reflected in the code itself, a comment goes with the code as close to where the decision is implemented as possible.
We have a 10-year-old code base that is doing well and continues to support rapid, robust and maintainable development. The learning curve is higher than if the documentation were external to the code base and accurate diagrams existed, but unlike those artifacts the code will always be right about what the code is doing.
In theory some quality systems (ISO 900x etc) should help with institutional memory.
Bob has a job; he performs a series of tasks to do that job; he writes down what he does for each task, including what information (also product) comes in (and who from) and what information (also product) goes out (and who to), also including all the documentation he needs to complete for each task.
Someone else edits that puts it in the company QA manual. In future new employees just read that documentation, and maybe do a bit of shadowing of existing employees.
Unfortunately, that's not what happens. What happens is that Bob has a job, but Charles (Bob's boss) writes a bunch of stuff that he thinks Bob should be doing. And he doesn't write sentences like "Set the machine for 190 degree Celsius. Place widgets in machine when it's at temperature. Leave widgets in machine for four hours." but he writes abstract ambiguous business speak that people can't use to actually do the job. Then Edwin (Charles's boss) adds another bunch of stuff. Then that gets edited into a god-awful mish-mash. The people who did the software (with the "help" of Edwin) add some stuff about information. Bob gets to work to an impenetrable document that bears little relation to his actual work flow.
You seem to miss the part in the original post where he states that things are running just fine. Everybody knows Bob has to set the machine to 190 degrees.
What they lack is the knowledge why it's 190 degrees. That's a bit harder to just pin on management. Anybody who's worked on a project for a long time has seen that drain of knowledge.
Alice, who wrote subsystem XYZ, leaves for greener pastures. Bianca and Christy worked on it with her, but they were juniors when they first started. Some practices, they just adopted without thinking about the why.
Now XYZ needs to be ported to a new environment, call it dotWEB. The question why things were done that way inevitably arises. Unfortunately, Alice works for a competitor - talking to her would violate all sorts of security rules.
That is the problem he's talking about. And like the original author, I'm not sure there's an easy to fix this. Not even if everybody including the bosses is supremely competent.
Charles acts aghast when Bob discusses with him the lack of actual direction and information in the document, they have an argument where Charles pronounces that if Bob were an expert in his field he should just know what to do anyway.
Later, Bob presents his work to Charles, who pronounces it "great!". So they move up to present it to Edwin for final sign-off, where Edwin pronounces it "crap! not what it's supposed to be at all!". Then takes the work, sells it anyway, pockets the bonus, trickles a couple percent down to Charles and Bob gets fired for incompetence three days before Christmas.
My first job out of college was at an ISO9000 certified place. It was weird. If I wanted any documentation, I had to go to a specific desk and fill out a form, checking all the documents I wanted. She had to go make copies, mark me as having them "checked out", then, after a few days, I finally got the documents that I requested. Of course, often they turned out not to be the documents I actually needed, so the process started over again.
I'm sure ISO9000 is an improvement over the worst-case, but dang if it doesn't introduce an amazing amount of bureaucracy into just getting things done. Not to mention adding bureaucracy to documenting things once you've done them.
In my experience, the best thing for institutional memory and process documentation in general is an internal wiki. Syntax can me learned in minutes, the barrier to documenting is extremely low, reorganizations are easy, and old versions are archived.
ISO9000 doesn't require what you describe, though. It requires that you have documented processes for quality management of processes, for recording exceptions, and for communicating exceptions to involved parties (including customers).
The process you describes sounds to me like a measure put in place because there is a requirement that documentation be kept up to date. I did a contract once to streamline updates to the ISO 9001 quality handbook for a large company, and part of the hassle was that while they mostly relied on an electronic copy on their intranet, they also needed printed versions and they needed to be kept in sync.
They wanted to minimize the use of the printed version exactly because they had to keep track of the copies and ensure every one of them were updated (or destroyed) when there were changes, so that everyone were working from the current versions. When I was there, they had ca. 300 ring binders of printed versions, and every time a page was added, removed or changed, someone had to track down all of those 300 ring binders to update them...
All of this hassle was down to the paper versions - the intranet version was a simple set of documents with some basic hyperlinking, and the update criteria were satisfied by simply prohibiting making copies of it.
You could likely satisfy ISO 9001 documentation requirements with an internal wiki - none of the bureaucracy it introduces prevents it AFAIK. What it does prevent is taking a lax attitude to updating that wiki.
I've been in the same job only five years and I have newer people coming to my desk asking "what do they mean by that?" because these emails get sent out with instructions that don't really make sense with current processes. I can infer what folks above me are trying to tell me to do, but the phrase used is a case of institutional memory and the process described/referenced is obsolete. No one has bothered to update the phrasing to fit the current situation and newer employees then don't get a detailed explanation of what it means (what the goal or purpose is) so it is slowly turning into gibberish. Over a longer scale, I can well imagine this can get much uglier. And I have an entry level job, not some high paid job with super specialized training and all that.
This is like a family of 1st, 2nd and 3rd generation immigrants. The 1st generation (the managers sending down gibberish instructions) use language that most people don't understand and refer to people, places and customs from the old country that have no meaning to anyone here.
The 2nd generation (you) are responsible for keeping communication going in both directions, and making substitutions and translations in both directions, sometimes changing words and orders into something that makes sense even though it wasn't at all what was asked for.
That's an excellent analogy but I don't agree that it should be on me to clarify instead of on the better paid, more powerful "first generation" folks to stay current and make their instructions legible and coherent and stay in touch with the processes of their people, especially since those new processes get handed down from on high. I assure you, we folks at the bottom aren't making them up for shits and giggles.
Suffice it to say that if I could find a means to monetize my websites, I would be gone. And it's a good company to work for. It really is. I occasionally note that if this is an excellent workplace environment, god help the world if you put me in a bad one cuz I will go postal and show up with a machine gun one day. I have little patience for such things. I was a homemaker, full-time parent and homeschooling mom for years. I am used to an environment where I am in charge and I have the power to effectively implement my "unreasonable" (high) standards and expectations. "A reasonable man conforms to his environment. An unreasonable man expects his environment to conform to him. Therefore all hope lies with the unreasonable man." The Fortune 500 company I work for seems disinclined to bend to my will (what with me having an entry level job and zero authority). If I can cross the chasm between here and monetization, I will be happily on my own at some point.
Oh, not at all. It just reminded me of the many immigrant families I've known, especially how the middle generation acts as a bridge between old and new.
My mom is a first generation immigrant. I often let her assumptions and misunderstandings of my sons simply stand. My sons tend to get a lot more of those explanations out of me, though sometimes only months after the fact (long, machiavellian story there). A similar pattern is emerging at work. I do explain stuff at work when asked. But I guess my policy at work mirrors my policy at home: I explain it to people I trust and who ask nicely.
Side note: The scrolling on this page is horribly broken on Android. I can't even read the whole article because I can't scroll down.
After loading the article on m laptop I must say that it is a fascinating article and will change how I think of documentation. For example, soon my company will be making a change that will render the old wiki url's incorrect, the problem is that many older documents will still refer to the old url's. I think we should have some sort of list of the old wiki url's (there is an even older set of url's as well) to help future archeologists in piecing together the information.
I glanced at your scroll.css and saw a bunch of rules that attempt to fiddle with the overflow/position attributes. This sort of thing confuses all browsers. Why not just use a normal scrollbar? It's fine to change the way it looks it if you want using the -webkit-scrollbar stuff you've done, but hacking out the normal web page scroll bar confuses everything. There are, for example, optimization opportunities in web rendering engines that only hit when the page is "normal" (which probably explains the poor performance other people are encountering).
(I noticed even on desktop Chrome the scrolling isn't smooth, unlike other pages. Probably for the same reason.)
I'm sure they saved money in the short run by going with the cheapest option each time they tried to manage their documentation the last time around.
On the other hand -- at the time, they didn't know which parts would be valuable, and which parts would be wasted effort, so it probably was the right choice.
However, if a major incident were to happen around that plant, could the company tell the emergency responders what they would need to know?
Many senior employees accept the offers, and then later mgt realizes that 95% of their Fortran or Cobol (or name your old programming language/product) experience has vanished. Thank god the software is mostly in maintenance mode, but client problems do still arise and what once took 15 mins to fix, now requires a week and phone calls to retirees begging them to help fix a client problem. They'll pay them loads of money too just for a few minutes of work.
Consultants turn out to lack the domain specific knowledge required to fix the problem.
Bottom line is that sometimes, mgt is so focused on hitting short-term financial goals, that they lose site of the big, long-term picture. Business needs less 20 something MBAs who only see this quarterly report and more managers who understand that business is a decades long proposition and that senior employees are key to maintaining an edge and passing along knowledge.