This is one of my favorite excerpts. I once worked in a lab where we would have frequent catastrophic failures because there was never any disaster planning or contingency management plan. I personally triaged 3 such incidents alone or with people who happened to be there when the problem arose and attempted to disseminate some suggestions for how to prevent similar problems in the future. No one was interested. People were primarily interested in tearing my head off because I hadn't handled the problem the way they would have done it (of course, they were out drinking beers or sleeping while I was dealing with the issue at 12 AM or on a weekend).
After the third time I said fuck it, the next time there is an issue I am going to insure my own projects are safe and then I'm going home and turning my phone off. Let someone else deal with it. That is the not the culture you want to be promoting.
This was an internal transfer. There were definitely warning signs in that interview. I was desperate because they were outsourcing my job in an obscure tech (Filenet) and we were expecting a kid.
The hiring manager said something to the effect of, "I was surprised anyone internal even applied to this job".
'Warning flag' doesn't do this justice. I have no idea what to call it, but desperation required I ignore it.
Because they keep getting shoulder tapped to put out fires. Because they’re the only one who knows the system. Because there is no documentation…
If any random person can break it, it's already broken.
If any employee can break it, it's probably broken (there are very small scales where even this doesn't apply. Ever worked for a company with less than ten people? There's probably something any employee can break).
If any employee that's an engineer, sysadmin or developer can break it, well now you're at least reducing the problem to a more specific set of people.
If only the people on a specific team responsible for a system can affect the system, now you've reached fairly good point, where there's separation of concerns and you're mitigating the potential problems.
If only a single person can break the system, you've gone to far. That effectively means only a single person can fix or work on the system too, and congratulations, you've engineered yourself into a bus-factor of one. Turn right around and go back to making sure a team can work on this.
Finally, realize that sometimes the thing only one team can break is an underlying dependency for many other things, and they may inadvertently affect those. You can't really engineer yourself out of that problem without making every team and service a silo that going from top to bottom. Got a shared VM infrastuture, whether in house or in the cloud? The people that administer that can cause you problems. Don't ever believe they can't. Your office IT personnel? Yep, they can cause you problems too.
Some problems you fix by making it so they can't happen. Other problems you fix by making it hard to happen and putting provisions in place that mitigate the problems if they do.
For example code review and LGTM ensures that a single individual can't just break the system by pushing bad code.
Often there are other control planes that don't have the same requirement, but I think the idea that there must always be one person who can break the system isn't clearly true.
Assuming code reviews are 100% effective at catching issues, particularly issues that cross multiple projects. Yes.
LGTM ensures that, for example, a single individual can't push a code change that drops the database. On the other hand, that same individual might be able to turn of the database in the AWS console.
Personally, I've seen LGTM let slip complex bugs in accounting code (admittedly, not great code) that went on to irreversibly corrupt hundreds of millions of accounting records.
Yes, it will catch "DROP DATABASE", but when it's still letting through major bugs that similarly require a full restore from backup... It seems functionally equivalent?
> There are lots of places where we require that no single person can break the system at least in a certain way.
I don't think code reviews are a solution. I mean, they're one of the better solutions I can think of, but they're not actually a solution.
There's always someone with rights to push code manually, or tell the system to push an older version of code which won't work anymore. Someone needs to install and administer the system that pushes the code, and even if they don't have direct access to push the code to where it eventually goes, someone's access credentials (or someone that controls the system that has access credentials) has access somewhere along the way.
But who controls that the code system is up and available even allow checkins? Can one person break that? What about who controls the power state of the systems the code gets checked in on? Is that also ensured not to be a single person? What about office access? What about the power main at your building? Is it really impossible for one person to cause problems there?
It might sound like I'm changing the goal posts, but that's sort of my point, these are all dependencies on each other. It's impossible to actually make it so one person can't cause any problems, because you can't eliminate all dependencies, and you can even accurately know what they all are. What you can do is focus on the likely ones, put whatever in place you can that's sane, but focus all the crazy effort you would have to do to track down the diminishing returns of trying to make failure impossible and start spending that time and effort on making recovery quick and easy.
Unfortunately, some work that goes into attempting to make sure any one person can't cause a problem might actually make that harder. Requiring someone to sign off on a commit to go live is great at 2 PM Tuesday, but not so great when it's required to fix something at 2 AM Sunday. This is the tightrope that needs to be walked, and also while even if you don't necessarily know about it, there probably is someone that has access to break something all by themselves, because they're who is called in to makes sure it can be fixed when the shit hits the fan and all those roadblocks to prevent problems need to be bypassed so the current problem can actually be fixed.
Any system that doesn't have some people like that at various levels persists in that state only until they have a problem and in the incident assessment someone needs to answer why a 5 minute fix took hours and the answer includes a lot of "we needed umpteen different people and only a fraction were available immediately".
Even at Google (which I see you work at from your profile), my guess is that people in the SRE stack can cause a very bad day for most apps. My guess is that even if the party line is that no one person can screw anything up, you probably don't have to ask to many SREs there before someone notes that it's more of an aspiration than a reality.
Sorry if that's a bit rambly. I know you weren't specifically countering what I was saying. I've just had a lot of years of sysadmin experience where it's pretty easy to see the gaps on a lot of these solutions where the face presented looks pretty secure.
I say this as someone who’s worked at large tech companies that are “internet scale”.
I think what is needed is a culture of -ownership-. That's basically people saying "I'm responsible". Not one where everyone tries to avoid responsibility, and not one where peopel point fingers.
You could argue that "It should be assigned" - yeah, it should. But assigning it implies either "here is the team that is responsible for it", i.e., this is the team responsible and they need to be told to fix their shit (which very much sounds like blame), OR it implies "here is the team that I am entrusting to fix it DESPITE their obviously not being responsible for it", which is just as bad, since it implies that the team that 'is' responsible for it is incompetent.
The only healthy option is that the 'responsible' team stands up to say "hey, that's ours; we'll fix it", and the only way they'll do that is if you have a culture of safety and ownership.
Also, one thing to make clear - ownership = responsible = blame. They're all words for the same thing, just different implications. You can't have someone 'own' something without making them responsible, and apt to be blamed if you don't ensure the culture is one that does not attach blame. That's really what I was getting at; of course you shouldn't blame. But, you can't also avoid ownership. But ownership implies you know WHO to blame, and so blame comes very easily. And it's very easy to mistake pointing out responsibility/ownership for something as blame; I have had multiple managers tell me "it's not us vs them" when I've raised up the fact that I'm unable to deliver to deadlines because I have been unable to get anything from product.
You don't say "this incident was your teams fault". You say "your team is responsible for ensuring that this incident can't happen again".
In an open discussion of the root cause, many times the issue is across multiple services / organizations within a company. You’d assign tasks appropriately across teams as needed. The key is to find and create actionables to address the root cause, not to punish / blame individuals.
Expertise and proximity are facets of responsibility (well, technically they are facets of knowledge, but ideally knowledge, empowerment, and responsibility are aligned, else things ALSO won't get done). Resourcing is a red herring; I've seen things get assigned to teams based on "they have the capacity", without it being an area whose domain they're familiar with (i.e., they don't work in that area, and ergo are not responsible for the outcome) - those things rarely get done, and never get done well.
At the end of they day you are going to have someone stand up and say: yep we should have planned for this, and we will correct this in x, y, z, ways.
It's business owner who's responsible, because ultimately he's getting all the expenses when critical event happens, client leaves, client sues the company, and so on. Other people are not really responsible, they just pretend to be.
That is, "you're responsible for this" - if they do it, and it succeeds, what happens? If they don't do it, and it fails, what happens? If the answer is "nothing" in either of those cases, they're not actually responsible. If the result is too detached, they're also not actually responsible (i.e., if I decide not to do one of the ten tasks assigned to me, and I don't hear about it until review time, if at all, then I was never responsible).
Responsibility is innately tied with knowledge and empowerment, but without going on at length, and to just give an example - if I'm the one woken up by the pagerduty alarm when something breaks, I am responsible for that something, because its success or failure directly affects me. If, however, there is a separate ops team that has to deal with it, and I can slumber peacefully, responsibility has been diluted; you won't get as good a result.
I can code with a few drinks in me; in fact the activity of coding seems to reduce the amount I drink (I'm a functioning alcoholic).
It's decades since I drank during working hours. In my early career in the City, it was the custom to drink at lunchtime. Those days are passed.
Not sure about producfion code though, the values are different
So maybe write drunk, edit sober does work
(Even without getting drunk, I've always found it useful to consider a hard decision once analytically and once intuitively, and if I don't agree with myself, think about it some more.)
I agree but think it is more than just _documentation_: effectively communicating ideas through text was one of the most underrated skills in software engineering. I say "was" as I think there is much more focus on it now with remote work and video call fatigue becoming the norm for many.
goal-oriented how-to guides
understanding-oriented explanations or discussions
information-oriented reference material
Two important lessons I learned:
1. Formatting and direct communication is very useful. It can make the difference between someone stopping and noticing critical information, or skipping it because they're lazy readers.
2. You probably don't know the correct way to convey information, and the audience probably doesn't know how to tell you how to convey it either. You need to listen for when the docs fail: when somebody says they read the docs but still don't know something or don't do something right. That means your doc has a "bug" in how it gets through to the reader. Experiment, change things around, add/remove information, etc until the bug is gone.
And it turns out those kinds of docs are pretty useful to my colleagues in piecing it together also.
* Rust Library Documentation: Most libraries have complete and up-to-date reference documentation, but are lacking even basic introductions (tutorials/guides) on how to use the library. This is totally just my personal experience so maybe I've been looking at the wrong crates, but with most of the crates I spend several minutes looking trough all the modules in order to find that all the juicy functions are hidden in the Connection struct, or something similar.
* Linux Kernel Documentation: The Linux kernel has excellent in-depth explanations on several high-level concepts, but on the other hand a little more systematic reference documentation on the supporting library code would help a lot.
* While I can't think of a good example right now, a lot of projects have a few basic getting-started tutorials but don't explain advanced concepts or high-level design at all, leaving you to wade through the sources yourself in order to understand how to actually use them.
That said, I find high-level documentation for larger systems to be very valuable. I also find Python's docs to be lacking compared to Java's; I'm often left wondering about the definition of what type is returned, exactly what parameters should be, and which exceptions are raised. Java's docs are very explicit about all these things.
I'm a pretty firm believer that documentation has to live next to the code. Otherwise it's nearly guaranteed to be out of date and/or incomplete
Thoroughly commented code tends to end up a couple changes out of date. A separate file in the same directory ends up a couple major refactors out of date. A separate file in a separate system ends up a couple company-wide reorgs out of date.
It works very well, and doesn't really add any overhead to my work.
I write about how I do documentation here: https://littlegreenviper.com/miscellany/leaving-a-legacy/
It's a long read, because it's a big topic.
Depending on the project, various other documents may be required, e.g. installation guide, user guide, operations manual, architecture diagrams, networking diagrams, module/component diagrams, information flow diagrams, high-level design, low-level design, docs at various "views" (such as "business view", "information view", "technology view"), design decision tracker, ontology...
I'm actually in the middle of using Postman to generate a more "modern" inline docset for a new engineer that is coming into a project that uses that server.
But I also like to keep docs to a bare minimum, as they become what I term "concrete galoshes": https://littlegreenviper.com/miscellany/concrete-galoshes/
We need to be very careful, as the docs can prevent agility, in a big way.
I find that if I can keep the docs pinned to the code, itself, as much as possible, it helps to keep some flexibility.
Totally agree about not having too much documentation - sometimes outdated documentations is worse than no documentation at all.
People managing work, from what I've seen, still prefer to babble over their scribbled 5 basic points than taking the time to do their job and create relevant textual information. Then you listen, you take notes, and then you go and produce whatever documentation of the objective is required to at least understand if it's going to work. Of course you still will have gaps in your understanding, so then more calls, and repeat. In the end, unnecessary/missing features, a whole bunch of time wasted in crap, deadlines missed, burnt time, all of which could have been avoided if someone just had taken the time to do their supposed job - this is not to say it wouldn't have to be discussed, or that there's no need for back and forth and calls, etc, it's just instead of starting halfway you start from -50% or something.
Even in outsourcing platforms there's been a shift contrary to that. At least two years ago and before that, video calls or calls weren't really usual unless you were in some months long collaboration - now even there everyone expects video calls on the interview... It doesn't matter if it's a $100 one time job or whatever.
And management is usually/always ok with this short-term optimization.
Code says what is done. Docs says why
Like dude, tell us why we should read the code in the first place.
Trying to write ops documentation on an unfinished project can be this way.
So I argue that the issue is writing skills, of which technical documentation is a subset speciality of writing skills. I will add, similar to math problems or programming, writing wants you to do it over and over so it can get better.
Agreed, wholeheartedly. Will hitchhike on your comment to recommend two things: Brett Victor's pdf stash  and, specifically, the Walter Ong essay "The Writer's Audience is Always a Fiction".
Long story short, we form our audiences by subjecting them to our writing. In writing software documentation, we are implicitly informing the next generation's thought by the simple power dynamic that underlies all technical documentation: "you must understand this in order to do your job properly".
It is no wonder that "form", "inform", and "information" are such closely related words.
We dictate the level of rigor and intelligibility we expect out of our technical documentation, when we write technical documents. It almost sounds like a tautology when put this way, but "bad docs" are exclusively the result of a professional culture that puts up with the existence of bad docs. I've been there; too tired and overworked to care about writing something properly, or wanting to avoid writing a doc enough that I setup some autodoc thing and called it a day. We literally don't get paid for writing documentation.
But good documentation is what made us into good developers (if we are good developers). We should get paid for doing that...
Absolutely. So many hour long meetings could be shortened by better communication skills (just more targeted), or even a small email chain.
Communicating in short form confidently is a skill. Many people, including myself (something I've been working on) struggle with saying something that was a complete idea in a meeting and not stopping because they feel like they need to say more. Short and sweet is the way to go pretty much whenever you can, for technical work that is.
I find 'small email chains' don't help at all. Too many people just don't read past the first sentence or paragraph. And email is slow.
I usually 'escalate' quickly. Support didn't understand a ticket comment I made on how this isn't a bug or why there really is a workaround for it?
I send them an IM trying to coax it out of them/get them to understand for a few minutes. Doesn't work? Quick call and do screen sharing. Problem usually solved after a few minutes. Problem is they might stay longer than that took just to 'catch up' ;)
This is really not that different from when we were all at the office just that the last part would be walking over to their office and looking at the computer together. In some situations that makes it even easier nowadays because I don't need to take an elevator down 20 stories and back up again after.
Having asynchronous means of communication is great. But as soon as the back and forth is more than maybe twice on each side, there's probably a miscommunication happening somehow that will be much easier to get solved with a really short feedback loop. Some people you have to call right away coz they just type soooooo slowly ;)
Example 1 - too much details.
Morning stand up.
Manager: what's the status of that new feature.
Senior Engineer: well I tried to call that service but it was timing out so I spoke to Bob and he said to check with DBA on why that stored procedure but it's so slow and turns out index is missing so we tried to add it but mysql and varchar something fckn something...
Dude couldn't you just tell it's delayed due to DB and then expand if needed.
Example 2 - insufficient details
I return from the meeting and discover avalanche of emails, chat messages and urgent meeting invite, all with same topic - "Blah service fails, we are blocked" but no details apart from that. On the call I get description of the problem - blah service fails and how everyone is blocked and how infinitely critical it is and what ETA for resolution would be.
What endpoint? How does it fail - timeout, connection aborted, 503 response, 200 response but with error message?
Having links to actual code, like GoDocs have it, is something I appreciate too.
And communicating gets even more important the higher up you move in your seniority.
I thought they were going to go the direction of "he's an asshole" and was ready to accept that, but this particular criticism is actually disturbing. People with strong visions can often appear to be "making it up as they go along," when really they are just subpar communicators.
Short story, I am helping my current company switch from a monolith to service-oriented architecture, and in the process have built a framework for spinning up new fully-deployable services from scratch that gets engineers 90% of the way there (minus the business logic). I have a strong vision for how it works and had a dozen-page RFC accepted by the engineering team for it. Yet there are engineers who think I am making it up as I go (I have been asked this indirectly), without any vision guiding the pieces into place. I have chalked up this feedback to me needing to improve my communication of the vision.
So the post's response of "I realized that he's making it up as he goes along like the rest of us" is disturbing because it makes me realize just how difficult communicating a vision is... if this hero that the poster paid $5k to go see can't even convince one of their fans, what chance do normal people like you and I have of convincing people that we're not making it up as we go?
>EDIT: I realize that I am posting under the assumption that the person's hero does in fact know what he's doing. If he truly is making it up as he goes, the above doesn't apply.
Then there's the times where you think you know exactly what you're doing and after going down a road you realize it's the wrong way. Failing to learn and see the signs and make the embarrassing declaration that you got lost and need to turn around is never good. But some people just keep driving. It's hard though when there's a big line of cars behind you that think you actually know where you are headed.
Sometimes it's a hybrid of knowing what you are doing but not knowing the implementation specifics. You know you need to connect high-level pieces A, B, and C with specific constraints, but it won't be until you get into the low-level implementation that you'll know if it is indeed possible. I think that's an example of both having a vision and also improvising as you go.
I am concerned about how to effectively communicate visions to people, because it gets everyone rowing in the same direction. If nobody thinks that you have a vision, when you do, there is no reason they should choose your direction vs just do their own thing.
Maybe it is communication related. Does everyone know that this is the way they should do something? But they still don't? Have you created docs and edicts around these areas? Have you been assertive in code reviews? Have you been proactive and requested design sessions before a lot of work was done?
Has a decision been formally communicated? I see this step not happening enough, people are hesitant to be authoritative after a discussion on architectural concerns. If you are the lead, that needs to happen.
Most of the time it's simply a case of them not knowing how to do it. People are afraid to show their lack of skill and knowledge and ask for help. They get deadline pressure and deliver their default way.
Have you provided a feature or cut through the system that shows this vision for people to follow? Maybe example code, resources on the web that go very deep into the ideas and tactics? Have you paired with them to help them get started or get over obstacles. Perhaps pair your most senior person with juniors for a while to get them on the same page and capable with this vision.
For myself as an opinionated person in a devops role, the vision that I try to communicate to my colleagues is mostly broad principles like configuration as code, helping people help themselves, consolidation of systems, and then some more pointed specifics like don't touch prod, don't make changes without a consensus, start by understanding why it's the way it is before changing it, etc.
I used to have visions. Now I have collaborative design discussions driven by some starting designs. I found that if people don't contribute to the overall design then they have little impetus to actually understand it. This tends to lead to a better design and a more engaged team so a win on all counts.
Nothing if you're good at it. But if you're hoping to learn something from someone, it is pretty disappointing. How to make it up as you go along is far less teachable.
I take this really just to mean "everyone has faults".
People often idealize heroes and think of them as beyond human. If you do that and met your hero then your illusion will often be shattered. But the problem is just that you were putting them on an unreasonable pedestal.
Of course some people are frauds and some people have no idea what they are doing but manage to make people think they do. But I didn't read this as being one of those situations. Just someone they saw as beyond human being only human.
I "know" guy who's conference speaker, so he knows other conference speakers, drinks vodka with them and so on.
He says there's significant amount of bullshit aka things that work nice on slides, things that only work cool in theory, but in practice they arent as great
I think that's what OP meant
I think people like to believe this because it helps them cope with impostor syndrome, or maybe they think it puts them on even ground with people who do in fact know what they're doing.
The foundation tends to be pretty simple, deeply held, and unchanging, while the higher levels are increasingly fluid and specialized. The higher you get on this stack, the more "making it up as you go along" it becomes, but every improvised part is perched on something more stable.
They key to "knowing what you're doing" is organizing this hierarchy well, having the right supports in place to successfully guide improvisation and course-correction while steadily fortifying the foundation.
Your assessment of why people like to believe this seems spot on.
Most of this stuff comes from students / junior developers, where yes, they probably visit stack overflow every 20 minutes
And it is bad to be a newb? And even for experienced devs to go to stackoverflow regulary ... isn't it productive, to not always reinvent the wheel?
I can solve allmost everything on my own. But if I have a new problem, I assume someone else already had - I would be stupid, to figure it out on my own, when I could get a working solution in 5 min googling.
But I actually programm without internet connection most of the time, as I like being outside, away from noise (and wifi)
That said, stack overflow can be a great source and I’ve written plenty of code with a comment pointing to a SO link to further explain a pattern or snippet for a future reader.
Yeah sure. A stupid programmer will remain a stupid programmer, even if he reaches a certain productivity by living off of stackoverflow ...
I do see two kind of people doing that. One group learns from the code in order to become better, and can use it over and over to be more efficient. The other group doesn't care how it works and just wants to have a snippet that works.
The second group usually misses a curiosity, of which the effects show up in many more places than just copy pasting from stack overflow. They also tend to have a flatter learning curve. I don't want to generalize, but in this group you will encounter people who don't care about the difference between a list and a set, or think that code works when it compiles. In both cases the juniors know very little, but one grows and the other one doesn't (or less)
There is space for both in the world, but I prefer the first group in my team.
How do you distinguish between what you do and "making it up as you go along"?
In sports terms it would be something like a baseball pitcher being able to throw a great curveball, a great fastball, and an all right slider, and knowing roughly what situations to use them in. There will still be a high degree of randomness and mistakes will be made.
I would much rather work with people who have a good track record of making it up as they go as opposed to people coming in with a fixed idea of how something should happen and are more likely to misapply whatever lessons led to those views (probably someone elses anyway).
I'm guessing that this was based on first-hand experience building such services and witnessing engineers struggle getting new services up. And not so much that you've had specific training or past experience in developing bootstrapping frameworks. This would be my definition of making it up as you go and is great way to do it. Another way is learning how to make bootstrapping frameworks and applying it wherever you can which doesn't go as well.
(I work at Red Hat, and my programming heroes are also my colleagues.)
That's a weird one. I don't know anything about that subreddit, but HN comments are frequently great. I submit stuff because I want there to be HN comments on it for me to read. I typically read the comments first and only bother opening the link if they were interesting.
>HN comments are terrible. On any topic I’m informed about, the vast majority of comments are pretty clearly wrong. ...
>And yet, I haven’t found a public internet forum with better technical commentary. On topics I'm familiar with, while it's rare that a thread will have even a single comment that's well-informed, when those comments appear, they usually float to the top. On other forums, well-informed comments are either non-existent or get buried by reasonable sounding but totally wrong comments when they appear, and they appear even more rarely than on HN.
It is also a reason why I dont want to mention or see HN links in mainstream media. Although I think most reporters sort of know this as well and tend to not mention or link to HN as source.
Otherwise it’s best to assume everything you read here is full of it.
But to be honest I rarely find pearls here. Even many of the "pearls" in the article are just well-written articulations of stuff that is... kind of obvious. Some are inversions of ordinary wisdom for the sake of inversion. Only some carry new information.
I find the most valuable stuff here tends to be arguments where someone knows their shit but is going against the grain, and that person will usually be flagged into oblivion. But there aren't many places you see someone like that responding in context to the mainstream dogma.
Interestingly though, reading through it I explicitly remember a lot of the comments he quoted, e.g. the FedEx Airport one (which was really interesting). It's kind of crazy to think the site is small enough that we're all reading the same good stuff.
Unfiltered stories from those who were there are often useful, even if their memory of how it transpired may not be entirely accurate.
Full stack compresses two jobs into one. It's purely for cost-savings. They're paid so little because companies revert to "well, you can still only do 8 hours, so you do half as much of each", but really that's just them trying to weasel out of paying for knowledge. They also blur the lines by putting full stack along-side other devs, even though the other devs may not have invested the same time to gain as much knowledge as full stack.
When you take a full stack job, you undervalue your knowledge (and the time invested) and are selling it for roughly half of what it's worth.
Not having to wait for some other engineer to make the backends made us a lot more efficient. It definitely was rewarding to be able to complete products end-to-end.
Hiring standards and pay were the same as for any engineer, at least in FAANG.
 Yeah, occasionally we had to optimize some SQL queries or whatever but we're competent engineers, we can figure it out even if it's not what we do every day.
This is 100% accurate in my experience too, and it's also true in the other way around: "full-stack" means backend developer who can put a basic SPA using React/Vue.
From the frontend perspective: they call themselves full-stacks for knowing how to spin up a NodeJS HTTP Server powered by Express with MongoDB inserting JSON in the database. But they are missing:
- AWS/cloud computing: not necessarily creating the infrastructure (although that's a must on more senior levels), but how to orchestrate the different components together.
- databases: why SQL/NoSQL, beyond basic SELECTs, knowing how to properly add indexes, debug why queries are slow, modeling, understanding locks implications and transaction levels, and so on.
- tooling: how to set up a bundler, linter and formatter, testing, CI/CD. This overlaps a bit with the responsibilities of the DevOps engineer, but a full-stack should know at least on intermediate level all of those things. I can't say how many times I've seen "senior full-stacks" that had no clue about how webpack worked at all.
From the backend perspective: they call themselves full-stacks for knowing how to spin up a React/Vue app that does basic CRUD operations using forms, powered by a UI framework like Material UI. But they are missing:
- CSS: most will find CSS hard/annoying and won't bother understanding at all how it works even on a fundamental level, will defer to hacks most of the time to make things work, especially when it comes to adjusting for edge cases like responsive design or cross-browser support.
- the DOM: normally they don't understand it at all, or to a very limited extent.
- Web vitals: how to measure and makes things faster and performant—not really including here overly optimized, but just making sure your app is 60fps or close to that most of the time. Usually when things get slow either on the network side or in the app itself, those engineers will blame is the framework/library, not their misusage of it.
Those lists are definitely non-exhaustive, as I didn't even mention more advanced stuff like protocols (how HTTP works? most can't answer), caching, etc etc, but you can get the point I'm trying to make:
The problem with the term full-stack is that only very few engineers really are sufficiently great on both sides of the stack and could say that they mastered both, simply because there's just too much to learn! Frontend has become so much more complex with SPAs compared when it was about rendering static HTML with some CSS and basic behavior with jQuery. Same for backend with the advent of cloud computing and several types of databases.
I've been coding professionally for a decade and I've met only a single engineer that I'd consider him full-stack (he checked all those boxes I mentioned and more). I think I would also include myself, because I spent 50% of my career spent as a front-end engineer, became senior, then I transitioned to back-end engineering because it pays the same or more and it's less stressful (for most of the regular companies that most of us work at). My current title is "principal full stack engineer", but in practice I only do backend/devops, I don't actively code front-end but I keep up with the industry by following the new trends and testing things here and there in personal projects.
Ultimately, I believe for being a full-stack engineer you have to be first a front-end (or back-end engineer) then learn the other, what we have today is most people doing the same from the very beginning of their careers and they either go deep on a single one or in none of them.
The front or backend bias you and your parent's comment talk about are team specific things so you don't give a hard front-end task to someone who is biased towards the backend. That still means they can take the less difficult tickets.
You can replace full stack for your example with "extremely good/talented/gifted engineer" and I'm almost sure it has nothing to do with the full stack label because it's the only example in a decade you've found.
But also the jobs that they get feed back into what they call themselves. So even if that's your example, it isn't how the business defines it, which IMO is ultimately why it's just a cost savings label to get people to work harder for similar amounts of pay as front or backend people.
My point is that they should be paid poorly, because they fail to be excellent at every part of their job, but rather than: Yes, this should in most cases not even be a job title. If you find someone who can do all of this well, you almost can't overpay, but are you really sure that you want to tie everything up on one person anyway?
Full-stack developer knows some frontend, some backend, some sql. They are paid good money because they are convenient, not for their knowledge. A dev who does only one thing knows way more about it that a team of full-stack devs.
Full-stack devs earn a lot and will probably earn only more in the future.
This is mostly a result of tech advancements like cloud infrastructure, tools etc that takes all the hardest things from you - like managing a DB, implementing security, managing infrastructure, deployments etc.
You don't need deep experts because of it, generalists are perfect for quickly shipping new features.
You can make doctor level salary at the ceiling if you move to a FAANG as a full stack dev.
The fact that boot camps even exist shows that it’s not that difficult of a job.
Other industries boot camps don’t even exist.
It sounds great because it is great. I don't make that much and I work on boring systems.
I guess those numbers also explain why the author can recommend maxing out the 401k. People supporting a family on less than $100k don't have $19.5k per year to put into it.
- OP uses the phrasing senior engineer.
- Never worked for FAANG. This is relevant because $120k + bonus/benefits is basically a FAANG new grad. Fairly normal for SV tech companies.
- Those numbers likely provide a solid standard of living, but as a senior engineer you are likely underpaid.
- Esoteric and proprietary knowledge means if you choose to go elsewhere, you will be at a competitive disadvantage compared to using industry standard tools. There are of course tradeoffs and lots of general learning that comes with experience, but all other things equal it's a disadvantage.
> I guess those numbers also explain why the author can recommend maxing out the 401k
Yes, but again fairly typical for the target audience I think. There are even startups trying to target this market to optimize the flow of money from salary -> 401k -> post tax contributions/megabackdoor -> other investments, brokerage accounts, etc., e.g. https://www.helloplaybook.com/
Just as a sanity check, BLS suggests the median SWE wages work out to ~$110k.
It sucks that I suck. Oh well.
Are you on LinkedIn? Do you ever speak with recruiters about other opportunities? That's a great way to get a feel for the 'market rate' for your skillset in your area. When's the last time you changed jobs?
I recently made the transition but it was extremely difficult. I ended up with 1 offer after ~10 or so interviews and ~50 or so applications.
The real problem is I deal with this sort of stuff. I started doing analysis about modifying a system to provide a new field to another system for the purpose of reporting. After spending a day looking at it, they pulled the story because they didn't actually need that field. And this isn't a one time thing - pulling back work. Then they give me BS stuff. They wanted me to increase the code coverage on an app that we were going to transfer to another team. The target percentage - 100%. It was already at 97% line 98% branch. Why am I wasting my time on this miserable task?
You are almost assuredly more desirable in this market than you think. Consider making finding a new job your new hobby.
The past couple of jobs I've had I came in at the upper end of Glassdoor's reported salary, for the specific company even, even when I had relatively few years in the role, and without negotiation on my part. And Glassdoor didn't at all represent bonus and equity properly. Levels.fyi did a much better job of it (but has fewer data points for non-tech companies).
I have never worked extra hours to come up to speed (and in general haven't put in extra hours, though I've sometimes had to work weird schedules due to working with people across timezones), and have pretty consistently been a high performer.
I'd still recommend just doing some searching. Worst case, you validate that your current comp is the best you can get. Mediocre case, you find you could get paid better, but not doing anything you feel comfortable taking. Best case, you find something that is interesting and exciting and will pay you better.
What kind of job do you have? I thought extra hours were normal in tech?
I have looked around. This area (Philly region) seems to be pretty terrible for tech jobs. There are some higher paying ones, but they tend to be niche.
Every place I've worked I've seen the same thing. Work/life balance is stressed as being important, BUT you will totally end up working extra hours if you cave to implicit pressures others set on you (oftentimes the business, product, etc). Someone will try and schedule you for a meeting at 5 PM, or say "we need this by next week", or whatever. And every place, I've said "No". Sometimes it's "I can't make that, I have to be home", sometimes it's "That isn't what we committed to this sprint" or even "That's what we committed but the sprint has been broken because (other thing)". Occasionally it's even been "Hey, we ran into something unexpected; even though we committed to that it isn't going to happen by that date".
On call pages happen; I always take time off the next day. As a manager, I -tell- my team to take time off the next day if they get paged.
Even where I am now, where I will have meetings scheduled at 7 AM, and 6 PM, routinely, I just close my laptop up in the middle of the day. Sometimes I'll even book time pre-emptively just to prevent people from trying to schedule me straight through (and thus leading to a > 8 hour day).
Etc. The perverse thing is that by doing this people actually get a -better- impression of me. There's an element of confidence; couple that with the fact I do deliver, and they don't question it or push back on it. And ultimately even if they wheedle enough I say "okay, fine, yeah, I have to talk to the guys in China; 6 PM meeting it is", I just book time 9-5 for my own stuff, or (when in the office), leave at like 3 so I can get home, unwind a bit, have dinner, and take that call, and still only work 8 hours.
I can't speak to Philly; you might be right. If you're tied to the area, consider remote. If you're not tied to the area, consider looking for things outside of it.
I've found that if I only work 8 hours they will say that I'm not getting things done fast enough. It makes sense if they are comparing you to people who work more. I know one department where the tech leads all work 10 hours consistently.
I was once in a discussion about how to get to a senior dev position (after filling the role of senior dev for a year and tech lead for another year). I was told I had to work an extra hour per day. That's a 13% increase for a 7% raise and a role with higher expectations...
First, a great deal of research shows that working past 8 hours of day for long stretches actually -reduces- overall output compared with working 8. Now, to be fair, the bulk of this research was done on physical labor, not mental, but interestingly, what research -has- been done on mental found the same thing, except the actual hours was more like...6.
Second, you're working at a place that actively -encourages- you to work unpaid overtime. That sees it happening and rather than saying "what can we do to prevent this and not risk burnout" instead says "good. Keep at it". If you're salaried, you're effectively getting paid a lower hourly rate for the work you're doing. So...not to beat this horse again...but...look for another job. It doesn't sound like the pay is great, it doesn't sound like the environment is great. The only thing keeping you there is a belief you can't get anything better; maybe that's true, maybe it isn't, but it certainly will be true if you don't at least look.
Get yourself on Linkedin. Find a tech focused resume writing service to help you with verbiage, both for your LinkedIn profile and your resume. Let recruiters know you're looking on LinkedIn (https://www.linkedin.com/help/linkedin/answer/67405/let-recr...). Start looking for roles in your area, and remote (and in any area you'd be willing to relocate to). That sounds like a better use of an hour a day than giving it, for the same price (free), to your current company, in the hopes of a 7% raise and higher expectations in the future.
Programming languages are all the same, so learn 3 or 4 new ones and discover that you can probably write in any language for an interview (then do some in relatively unfamiliar languages for kicks and giggles to practice).
Tech is all the same. Take data in, poop data out. That’s the whole job. The formats and protocols change, but once you starting thinking about your stacks as data-in, data-out, they all start looking the same.
Make video games or hardware drivers from scratch, those are the hardest things to make. Video games from complexity overload, and hardware drivers from interface complexity.
I don't have any interests in games or drivers. Those aren't applicable in my company either. I am currently working on an Angular site. I will host it on S3 with a Lambda and maybe SQS for a marketing email list. This is tech that we use at my job, and many other places.
They'll likely run a background check and make sure your resume isn't obviously lying (they'll confirm start/end dates and title, basically), and make sure you're not a criminal, MAYBE do a credit and/or drug test.
There were a few post on HN recently about fresh grad asking for / being paid $200K and anything lower they felt they were feeling lowballed.
For those of us outside US we could never quite grasp whether something is true or not. Salary across the pond is just incomprehensible.
$200k total comp, absolutely. A standard new grad offer with no negotiation might be something like $120k salary, $25k cash, $120k options vesting over 4 years with 6 month periods. So in the first year, no negotiation, you're taking home $177k pre-tax. I know people who have gotten up to $60k cash bonuses (split over two years) out of college, simply by having another offer.
This one has started to dawn on me. I'm never going to love my job as much as I love my personal projects, so a job that I don't get very excited about but doesn't drain my energy is better than one that I get somewhat excited about but does drain my energy (not that it's impossible to have both, but it's rare)
Life is extremely, infinitesimally short. If it's at all possible for you, you should try to spend as much of it as you can doing things you love. I know many people can't, but it's bleak to just give up and permanently settle, I think. (Especially if you don't currently have any dependents who rely on you; it changes the equation if you do.)
It seems to me that many if not most people I meet fall into one of two camps.
The folks that just don't care at all and do the minimal amount of work they can get away with not to get fired.
The other side is folks that care so much about it that they spend so much energy on as to ultimately be unhealthy for them and also making it worse for the rest of us (not taking vacation, unpaid overtime etc.
It is very rare to find people that care 100% while at work. But after ~40 hours each week that's it. Sure I'll stay later for that one meeting that's sorta important and needs to start at 4:30p.m. But you better not try that every day from now on. Want me to be on call for a fixed rate that has nothing to do with my salary? I can't log an hour for a 5 minute call at 1am and I can't take time off in lieu? Good luck getting me on that rotation!
Can confirm this and it was surprise for me when I discovered it. I assumed they would try to maximize $$ in my offer in order to maximize their profit. Yet most of them were trying to get offer accepted as quick as possible. Seems like more offers with less money in each is preferred over less offers with more money in each.
There are “rules” to how the game is played, and how the information is revealed, and a lot of it seems hidden (in fact I have seen agents that don’t understand the perverse incentives!). A seller’s agent will usually need to make sure they have deniability, and they will prefer options where the vendor remains happy (so the vendor will use them again, and recommend the agent to other sellers).
I have only a very little experience, and none in the USA, but that was what I noticed in my own country.
It surprises me how little effort people put into understanding the game, given that playing it well can easily make the same difference as many years of income.
This would likely be more difficult for an average person to pull off, but can be a great mutual win if you pull it off.
For the sellers, maybe, but I don't think I'd be at all comfortable selling a home without a (good) agent.
I'll 100% agree with you that most realtors are bad. The good ones are gold though.
Why is that sad? Have you ever heard of speed dating? Similiar concept.
There's a well known study from OkCupid about this, women are way more selective so men only have very few matches a week (and some of those don't even respond) unless they try a lot.
I guess there will be more on their way.
Couldn't agree more
There is still to this day the idea that we're supposed to plan to future like oracles or pad our resumes on the company's dime.
For me, 90% of day-to-day algorithms use is not writing O(nm) or O(n^2) code, and knowing which data structures to use to avoid it (usually a hashtable, occasionally a heap or balanced tree).
However, to the point of the person you've replied to: It's good to know that stuff to be able to assess the situation in the first place.
First draft: don't be stupid with asymptotics as PP says.
*only if there is actual problem in prod or whatever: bust out the concrete measurements.
Amen! Not just junior engineers. PMs, analysts, testers. We all have a lot to learn from the rest of the team!
The other ones are quite obvious but this is the one that really resonated with me. People, with experience, tend to get more pragmatic and less "academic".
I got older, stopped using python for exactly the reasons above (I just really felt that it was wasting my time for trivial reasons), and found mypy which made it bearable again.
Every time I see an example of a bug that TS would solve, it's something that I routinely find in 2 seconds by looking 10 degrees to the left at my second monitor and noticing the screen is white and there's some red text in the dev tools. "Compile time" doesn't mean anything if it consistently happens 0.5s before "run time".
I believe typescript also has options for compiling statically.
Have you ever considered PHP?
The last time I had a job was on a Clojure team, and that team has since abandoned Clojure and some of my former colleagues have come to me and essentially said that they got tired of fighting with runtime errors.
Strong-typed languages (especially in the damn browser) make no sense to me. It's not like we need to manage memory or anything.
20+ Years of experience here, and I hate TS with the passion of a thousand suns.
I don't see people commenting on this one! That may be a good thing because it means people are not questioning it :)
Just put all the expected input/outputs in a big table test and iterate.
TDD is awful when you don’t have a clear idea about the interface and need to mock a whole bunch of internals that are likely to change.
I really should
When someone says "lisp is the greatest programming language" it's these principles that they refer to, most of which Clojure discards so it can play nice with Java and promote very specialized ways of solving problems in order to best fit a particular niche.
The best way to discover the essence of lisp is to read SICP and learn Scheme.
Clojure has very tangible and definite downsides and tradfeoffs (to name some: weaker REPL, though not as weak as some Schemes. JVM required. Heavy interop reliance), but it has served well as the flagship functional lisp.
If you can get by mostly writing Clojure code (either by wrapping the Java libraries that you will use on helpers, or by using third-party libraries), it is a great language, even if in the end it is a very different from any other Lisp (but I'd argue that the changes are for the better, for example first instead of car, thread macros, protocols, immutable data structures). But yeah, for sure Clojure is much more optionated than any other Lisp.
Now, if you need to interop with Java code constantly, yeah, Clojure can be a pain. A good chunk of the code you will write goes to appease the alien structure that is the concept of Class on a FP language.
Highly recommend a journalism class at local community college.
It does help but I am not sure I would describe _my_ education as a panacea.
The main reason it was good was because it helps frame how you think of writing,
The most important things go first. Statement of fact, then you go into what the implications are, then you start introducing less and less relevant elements.
when you’re writing you keep the 5 “W’s” in mind, make sure you answer them. (Who what where when why), for instructive documentation you add: How.
Obviously an education bakes this into you in a better way than I can convey here.
What I learned is probably only decent for writing overviews.
Personally I find structure to be the biggest bottleneck/difficulty when making documentation.
What do others normally struggle with that don’t have this education?
Learning how to write proper instructions is the gateway to proper documentation.
Sometimes fixing the problem will require special access to Production which you don't have, or even a specific role with that extra bit of initiative.
Otherwise i agree 100%.
Operation, and on-call staff fixes broken systems just enough, that they will work until 8:00AM when the developer is back at work.
Just this week I talked to a developer, and he asked if I could switch the phone numbers, so issues would get routed to him first. My question: Why? You can't really do much without me being awake as well, so maybe I get the first call, and I call you... IF I need to?
If it's unfixable, that's when you quit
Maybe... none of this applies to people who own meaningful portions of the companies they work for.
But this one stood out for me!
> Titles mostly don't matter. Principal Distinguished Staff Lead Engineer from Whatever Company, whatever. What did you do and what did you accomplish. That's all people care about.
To say nothing of a lot of brain.