Hacker News new | past | comments | ask | show | jobs | submit login
You've heard of “yes men”. Now learn about “no men” (2011) (rachelbythebay.com)
188 points by derwiki 71 days ago | hide | past | web | favorite | 144 comments



I definitely felt this decades ago when I started in the industry.

It's 2019 now though. only 8 years later, but a lot of things changed.

The industry is growing fast, faster than knowledge can be shared even with all the power of the internet. It's very bottom up, with newer engineers severely outnumbering those that have been around. People are making fat salaries and companies are catering to their whims. They/we're more spoiled than they've ever been. People go to work hoping they'll be find a reason today to try the latest pattern they saw on Twitter or try the trendy new tech or library their friend just talked about.

If you don't have people saying no, it quickly (very quickly) get out of control. As organizations grow, you need to start saying no. A lot. Else within 0.2 seconds of the new intern joining or the new grad coming in, or the lateral hire who was the king of their 5 people startup will be pushing some stuff that isn't mature enough or doesn't follow established patterns. More often, they'll be reintroducing something problematic that you just finished getting rid of 6 months ago.

It sucks. Sure, there are some power hungry people who get off on power trip by being gate keepers. But overall, it's just people desperately trying to keep things in one piece and the company going, and that means making hard decisions and enforcing them. It just doesn't feel good to say no, arguing all day with people who have way more energy than you do, but it has to be done.


Tim Cook agrees - "The hardest decisions we make are all the things not to work on. There is lots of things we'd like to work on but we can't do everything great. Most companies do larger and larger portfolios because its so easy to add. It gets hard to stay focused. Yet we know, we can only do our best work if we stay focused"


There's a big difference between saying "no" to the right things (valuable) and a borderline oppositional personality disorder (not valuable). There's an art to saying no. Ideally you look for ways to say yes, and try and distill the nuggets of value and work them into your future plans while acknowledging that now may not be the right time to pursue them. Or nerd-sniping them with other ideas to leverage their enthusiasm and bright-eyed/bushy-tailed attitudes. Likely they're just looking for impact and don't know how to get it.

Otherwise you're kind of a dick, and nobody wants to work with that. Unless you're in charge and they can't get away, I guess.


That is why you write that in a standards document, along with the reasons why and have a consensus about it. That way it becomes impersonal.

Same thing with auto formatters and linters sidestepping a lot of pointless bickering and bike shedding.


That works for things you want to DO, but is hard for things you DON'T want to do.

As I mentioned in another post, as your org grows, there's going to be an infinite amount of things people push, and pushing back is hard. If you have to stop and write a consensus document for every single one, you'll be doing nothing else.


With things you don't want to do, you make it concise with a whitelist. Like 'we only use these 2 languages for our backend and here is why:'.

If you have the authority to tell people no in the first place to make it stick, you have the ability to write that in a wiki page. You can go full FAQ mode for the miscellaneous stuff as it comes up.

Documenting the why is really important, so you don't get zombie decisions self perpetuating in your org and allow for change.


How hard is it to say "these are the types of things we want to do. anything must look like these types of things in terms of (insert x y and z criteria: size, scale, opportunity, profitability, growth, etc)

spoiler alert: it isn't hard and this is what we do at my startup that is currently experiencing hyper growth and is on multiple "unicorn" lists


I can't even tell if it's satire. But if it's not: I haven't see it be done easily over several hundred engineers. If you have that kind of charisma, hat off to you!


Charisma is certainly something I have never been told I lack. Maybe we are lucky and have the right people as well.


[flagged]


You could have just refrained from posting.


Linters are probably the single biggest productivity boost an organization can engage if they don't yet have one.


It's still personal: it's "authors of the document" vs everyone else. Often the "everyone else" have not been hired yet.


> Often the "everyone else" have not been hired yet.

And they may never get hired and you still have to fight with them.

I'm, honestly, TERRIFIED of what "thought leaders" may post on hacker news/twitter/reddit tonight, tomorrow, next week. The moment I see one of them suggest something that just wouldn't work for us, I -know- I'm going to have to fight 50 people at work soon who drank the koolaid.


If something is written down before you were even hired and it says no, then you know that the people are saying 'no' not because they don't like you or some other petty interpersonal political reason, but because of the actual technical reasons or some overarching political reason unrelated to you, personally.

When something isn't personal, it diffuses a lot of emotional tension. Our monkey brains jump to taking things personally way too easily probably as some sort of early tribal instinct for survival.

I don't really understand what you mean by personal otherwise? I was implying 'personal' was specific person personal.


I've had the opposite experience. Working for a "no guy" who was hired fresh out of school, stayed at the same place for 8 years or whatever, and then blocked anything anyone learned from working somewhere else.

We started to wonder if his blocking of everything was just pathological because the tech stack wasn't all that great.


I have to say "no" a lot in my role. Thank you for this post.


> it quickly (very quickly) get out of control

javascript


JavaScript is indeed a very good example of this. Not just in the ecosystem, but even just the standard itself. People are pushing for more and more stuff. People on the committee can say "no", but there's an infinite amount of bullshit to push, and only a limited amount of time to push back. So sooner or later, bullshit makes it in.


What are examples of bs in the javascript standard?


Not in the standard yet, but good incoming examples are decorators and private fields. They are not bad in all languages, but the way they are getting pushed in JS is awful.


String (capital "S") class.


I think that it is good. These functions for primitives (String, Number, etc) are good, one thing sometimes to call it, and also because of the prototype. Also, there is such thing as String.prototype.constructor and Number.prototype.constructor and so on, which I have used for parsing command-line arguments (to replace the default values with the user values, converted to the correct type).


Why don't you like Strings? All prototypes (Number, Array etc) have a capital letter.


Automatic semicolon insertion


JavaScript


arrow functions

curried functions


Even before I became the manager, as a Lead I felt half my time was explaining to younger, less experienced engineers why "no, we should not" introduce something.

And it really was an explanation, not a condescending or dismissive refusal. That part is key.


I wish more places understood this. Change of leadership where I am is causing so many issues because there isn't any substantiated reasons why something is a 'no'. Its just 'no', and attempts to get clarifications and how to proceed in the future are met with little clarification (and sometimes ignored altogether)


All while you're the "no wizard" ;)


"Chesterton's fence".


The opposite. In this case people saying no are the ones understanding, and people complaining about them are the ones who don't have the full context. Of course the job of the people saying no becomes to document this, but at large enough scale its hard to get people to read the documentation...(and even when they do, its hard to internalize it).


Chesterton's fence is about taking away things we don't understand.

There's a big gap between that and fighting cargo culting.


CF is about, generally, changing a thing without understanding why it was set a given way in the first place.

Where it is necessary to change (or to make changes) without full understanding, at the very least, instrument and observe for some time.

https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence


Yes, about changing the status quo.

Aping someone else's fancy, new solution is not the status quo. You have no way of knowing how badly they have steamrolled Chesterton's Fence.


But you really don't know if it's cargo culting until you know why the fence is there.


A powerful "no man" can be a valuable asset sometimes.

I was involved in the design of an ASIC. This was going to be used by people from all different areas of the company, and they all had various demands. Most had never worked on a chip before, and didn't realize the cost of what they were asking for. The easy thing to do would have been to ensure that every requested feature was added But that would have burned area, power, and huge amounts of design and verification time.

Luckily we had a senior guy who had the clout to say "no". Over and over again, he would tell people how much things would cost in terms of money and time, and ask if the features they wanted were worth that much. Or he'd point out that what they wanted could be done with firmware plus existing features. Overall, he probably saved millions of dollars and caused the project to be delivered on time.


I have an unofficial board of advisors for my small company, and one of them is a "No Man". I knew that when I asked him to join. Every time the company has an idea, this guy will shut it down immediately. He forces us to defend every single decision.

Do we always listen to him? No. Does he always win? No. Is he always right? Absolutely not. But I keep him around because he forces everyone in the company to defend every idea. If you still feel good about the idea after defending it from him, then go ahead. If you've lost faith, then I'm glad we didn't waste any time on it.


Every company really, really, really....really needs someone like that.

Imagine how much better the Star Wars prequels could have been if only Lucasfilm had a strong No Person to stop them from doing stupid shit.


George Lucas’ ex-wife was the no-woman on the original trilogy. She was his editor!


He needs to remarry her.


I'll admit to being a semi-frequent "no" person and I'm fortunate to work for someone that values that.

I've learned some people can take personal offense to having their ideas challenged, particularly when...ahem... There are some glaring holes in them they frankly should have caught. So over time I've found ways to try and take the people out of it, and also to be enthusiastic about the act of having ideas, because I'd rather have people have a bunch of ideas that mostly don't work, with occasional nuggets of gold than people who aren't capable of or engaged enough to think of new things.

Personally, I start considering most ideas, particularly my own, from the standpoint of "why will this NOT work?" It helps me sift through things quicker by identifying potential deal breakers earlier in the process, and helps me make sure I've done my due diligence on important decisions.

The net result is that I kind of enjoy poking holes in things all day, even if others see it as "not optimistic enough."


What a lovely team member to have. Everything should be questioned. This is how we get smarter. This is how we learn. Vocalizing your arguments helps convince yourself and others and validates (or invalidates) an idea.

Arguments put forth should be criticized and challenged. However this becomes difficult when egos come into the picture. There are stubborn engineers that will never accept that the possibility that they might be wrong. With this in mind, the "no man" cannot be the decision-maker, the "no man" should be playing the role of a lawyer.


Having someone who will (or can) justify their nos can be quite useful. Having a cadre of people who simply emit a nonending stream of nos for, erm, no reason, and who cannot be reasoned with (see the parallel response by freehunter about the "no" board for a counterexample), is directly harmful.

There's also a time and place for experimentation, and a time when it's got to be made clear that there's only so much time (or other budget) for research, and that that's going to be tracked and burned down (though the discretion may be put on the R&D team itself to decide how to allocate that).

Much of Rachel's essay is more about management-by-mythology than about the saying-of-nos, specifically.


I don’t get it.

1 - there is a Mr.No guy but, the author only knows him by reputation

2 - there is a technical problem that seems to revolve around memory

3 - the author has an idea of a solution, she actually tries as people don’t believe her

4 - her solution works, but she is told the “No” guy said “no swap” once a long time ago

Her reaction: “Did I care what he supposedly said? No.”

Why doesn’t she go talk to him ? The “No” guy could have had another solution than raising swap. He could have recognized being wrong, allow swap and learn from it. He could continue say “No” but now the author has an actual experience of it, and not just hearsay and reputation.

It seems like nobody communicates correctly in that company, the author included, and it must hurt in so many places.

Edit: fixed author’s gender, thanks


> It seems like nobody communicates correctly in that company, the author included, and it must hurt in so many places.

Depending on what you think of Google (or maybe Rackspace, but I think Google's more likely the source of that post), you might be surprised at what company you're talking about.

I think you're spot on but am curious if you'd write that if you knew where they worked.


I see Google as a super huge organisation with smaller organisations within (I never worked there so that's just an image).

It would be less surprising to have these kind of disfunctionments within giant corps I think. As a consumer, Google's product strategy also mirrors that image (the number of messagings platform they released that ironically don't talk to each other is legendary by now)

In particular, a very small company with a lack of internal communication would fail very fast, so it's less likely IMO.


Rachel's at Facebook


In 2011, a few months before the date of this post, she had quit google.


She left Facebook early this year, IIRC.


Early 2018


It probably was late 2018 then. I don't know the specific dates because I don't work there anymore. All I remember is that I left a couple of months after her (uncorrelated with her leaving) and I left early this year.


For the record, the author is a "she".


The way you deal with a "no" guy is to ask for forgiveness instead of permission.

Put the responsibility on yourself to make the decision instead of passing it off to others.


These where my thoughts, "No" guy could have a good reason for no swap. no swap might have been a fix for some other issue OP is unaware of, which OP has now reverted. Don't know until you ask...


>no swap might have been a fix

The chances of that are astronomically low. From the article:

>I later found out that all of our older machines had swap enabled. It was just the ones which had been replaced or reinstalled which had been popping up without it, perhaps as part of a policy change. That means it was perfectly valid for us to run with swap on, since most of the machines we owned were using it.


Just an FYI, the poster is female.


you caught like half the male pronouns, there are still a bunch left, it actually makes it more confusing.


It makes it sort of like the sci-fi book Ancillary Justice by Ann Leckie. You have no idea what the gender of the protagonist is. Perhaps future gender-neutral speech will sound like this instead of choosing a third pronoun.


It’s crazier in languages where adjectives take genders as well. Japanese is blessed in this regard.

(I fixed the remaining bits)


Yes! I listened to a good bit of that audiobook, but it was such a pain having to puzzle out who was doing/saying things. I eventually gave up.

I mean... it was an interesting idea, writing a book that way because the characters sexes shouldn’t really matter. But in practice, it was a mess.


It was easier to follow in the written form.


This is my experience too. Dealing with "no men" is usually far easier in person.


1. The Mr. No guy was clearly well known for actions that involved no personal action but were publicly visible, e.g. commit messages.

2. Said problem was OOMs on machines with swapoff

3. Yup

4. It's pretty easy to get someone's measure from reputation if you have multiple independent sources - enough to fully distrust Mr. No's judgment. If this person is high enough in the chain that she's getting his orders second-hand, it also may not be very easy or fast to get his attention or a face-to-face meeting. So she made her change, publicly stated she was making it, and allowed him to countermand her (with a snarky "he can have my pager" to force him to justify his decision if he wanted to reverse the decision).


I've found there's something even worse: Policies that aren't actually attached to any known person.

At least with the swap example, there was someone you could talk to, or ask your manager to talk to. But what if it was just an Onion in the Varnish and nobody remembered who said no swap in prod, it was just "That's the [company name] Way"?


Happened at my company, where we had some policy in response to an incident, put into place by a couple of senior leaders... the policy was overly broad, but kinda made sense for the problem.

After a while, those two leaders left, and we started seeing issues that the policy was causing. However, people still stuck to the policy. When I would argue that the policy wasn't really accomplishing what it was originally intended to do anymore, I was repeatedly told it didn't matter, it was the policy. There was no one to even argue against anymore, because they were gone, but the policy remained.


It is super important that policies are mutable, that they can change with changes in circumstances and knowledge. They should be as easy to change as the org chart.


It’s like the monkeys being shocked every time one of them goes for the banana.


I know exactly who she's talking about! When I was at the unnamed company I worked on a project that he retained control over and said no to a lot. We found that if we sent him a very well reasoned, documented and measured change he'd say yes.

I was there when the no swap policy was under lots of active discussion. It was there to ensure that application authors could have strong expectations on how long a memory access would take. Imagine trying to provide evidence that removing the "no swap" policy wouldn't affect any of the applications that were running across a very large infrastructure.

Her point really resounds though, this person was referred to all the time in conversation, "well, we probably couldn't get this past X". He was mythical. In general that was often a good thing. Making changes that could have such a broad impact should be well reasoned. But it's also quite annoying, having a nagging voice inside your head that just says "no", seemingly disconnected from the actual source.


What the app something that required deterministic latency? There are situations where your rather hit OOM and kill your app and go buy some RAM rather than have it operate slowly - executing trades too late, or pointing at targets that have moved, etc.


I'll take a stab at explaining what happened here:

"No swap in production" probably stemmed from someone who worked on the servers, and that person discovered they were swapping too much, which would eventually lead to hard drive decay. And, maybe for budget reasons or whatever, they decided to make it a policy to disable swap, rather than increase RAM or change the underlying software that caused the servers to use too much RAM.

And now, later, on different servers with different software and different specs, the policy still exists, because nobody really paid much attention to why it was created in the first place.


This reminds me of a previous company that paid consulting fees to have their Postgres configuration tuned. When the hardware was upgraded nobody knew to review the memory settings. After a series of production outages we realized that the new Postgres servers were utilizing 16 GB of RAM when we had 128 GB available.


Swap causes not disk decay but performance decay; the system gets unamanageably slow and would have been better off with a clean crash and failover.


Only if there's thrashing. It’s prohibitively more expensive to build a machine that has enough memory for way more than peak usage than to have some swap that can handle some level of overflow pages.

Like everything swap has to be monitored and there’s no need To disable it entirely


It depends. Sudden peaks of latency can get comparatively more expensive on a trading floor than buying an absurd amount of RAM.


It sounds like a documentation or communication problem. If somebody makes a technical suggestion, such as "don't do X", the reason behind it should be recorded somewhere.


This seems far less about "just say no" than it is about not understanding, or insisting on, long-obsoleted policies.

I've encountered something of a converse of this situation: a set of prod servers running a Java workload on which performance was frequently abysmal, which turned out to be due to a very poorly tuned set of JVM GC parameters. The feature(s) were poorly documented (lots of detail, little useful guidance), so it took some tuning and ongoing performance monitoring to see that 1) the problem had been solved and 2) that no new problems were introduced.

The latter being of particular concern as this was a long-lived project that had gone through generations of development and ops support (I was latter), few of whom were on current speaking terms with the shop in question.

The management feedback I got on my monitoring was (a rudimentary bit of instrumentation, and periodic checking) was that this seemed to reflect an excess of concern and/or attachment to the issue and/or solution.

Given the numerous other years-outstanding issues which I'd identified and (eventually) largely remedied, I found this response tremendously perplexing.


Isn't no swap in prod fairly standard? Kubernetes pre-flight checks check it, for example.

If enabling swap 'fixes' it, isn't the proper fix to provision more memory?


> If enabling swap 'fixes' it, isn't the proper fix to provision more memory?

Mr. No says all servers must match one of the existing configurations, none of which have very much ram.

If you're on the border where a little bit of swap gets you zero pages, and no swap gets you stuck machines, you probably need to figure out how to scale within your constraints, but a little bit of swap should be ok, in the neighborhood of 1GB gives you a little bit of breathing room without prolonging the inevitable for very long.


1. The article states that the older machines had swap enabled, so they weren't homogenous anyway;

2. At some point you do have to change configuration, and if you have insufficient memory on one you either have insufficient memory on all machines, or a badly distributed workload.


Wait what? How do machines running K8s exec processes when they are near their physical memory limit? Swap does more than just provide a place for pages to get evicted to, it also increases the virtual address space.


I don't think that's true, at least on Linux.

This program uses 1 TiB of virtual address space, and runs fine on a machine with no swap (and much less than 1 TiB of physical RAM).

    #include <sys/mman.h>
    #include <stdlib.h>
    
    int main()
    {
        size_t const len = 1024 * 1024 * 1024; // 1 GB
        for (int i = 0; i < 1024; ++i) {
                char *p = (char *)mmap(0, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON,
                        -1, 0);
                if (p == -1) exit(-1);
        }
    
    }
The space won't be associated with any physical object (whether RAM or swap) until you actually touch it.


Thanks for the repro case!

I wasn't precise enough. The case where we needed swap not to error out was in a big mapreduce process, consuming maybe 44GB out of a 48GB host. When we didn't have swap on the box, the big process would exec a small command line process and the exec would fail, there wasn't 88GB of address space available during the brief period between the call to fork and the overlay. Adding a swap device fixed our problem even though it was _hardly_ ever used (MB to single GB).

At least that was my analysis from 10 years ago. I never went full science on it, but I did read plenty of documentation on overcommit and virtual address space.


I'm a bit out of my depth here, but maybe you could solve this by:

    echo 1 > /proc/sys/vm/overcommit_memory
According to documentation, 0 (the default) means usually allow overcommits according to some heuristic (which was probably failing in your case), 1 means always allow, and 2 means don't allow.

Edit: It appears from https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... that things will go wrong if you try to allocate more VM space in one call than the total free RAM+swap available to the system.


Yes, one example where overcommit can be better than swap is Redis.

When redis is persisting its state to disk, it temporarily needs 2x the memory because it forks a child that does this in the background. Since fork uses copy-on-write, only the difference between child/parent is actually consumed as extra memory. The rest is "de-duplicated" in a sense.

The choice is then to

1) enable swap, possibly killing performance 2) enable overcommit, possibly killing the redis process

It is sometimes preferable for a service to fail hard and fast than get bogged down with swap. I believe this is the idea behind the "no swap" policy, at least on Linux.

Redis is a good example here because any amount of swapping will kill performance since literally all of its allocated memory is consistently accessed for the purposes of doing a dump. And Linux will always try to swap things out, even with vm.swappiness=1.


Yes it is pretty standard in container environment (which is what borg is too) so that example kind of proved the opposite point.


I don't think this is necessarily about people who are critical, it's that you can have someone who is just as stupid as a "yes man": they just say "no" just because instead of, you know, actually thinking.

I don't believe this "no man" is someone who is managing their attention or saying no for an actual reason. They're just contrary. And yeah, there's a lot of them around.

What I actually find even more frequently are "parrots". These are people who "learn by runbook". Instead of asking why something is done a particular way, they want to learn how to parrot some procedure. It's another variation of things: instead of saying "yes", or "no", they just parrot "process X".

I think this is really where the "no swap in production" example came from. I've seen _lots_ of stupid similar ideas in software development.


There are any number of these type of people at any large organization, and I often flip flop on their usefulness.

Usually, they are in the form of protectors of the "process". This wasn't vetted by so and so, or didn't go through the proper channels, or doesn't use the right technology. And 51/100 times, they may be right - they keep unnecessary scope creep, bad ideas, technical debt, overhead out of the process. Literally the only reason people bother following a process is because of them.

But at the same time they are a huge drain on time and resources, and can enable bad or inefficient processes by manually keeping things in line. They add nothing to innovation - and it's a huge bummer to work with/around them.


As one of the dreaded "process enforcers" at my company, it's a tough line to walk. Any organization with more than a handful of people needs some written and agreed upon process or it's just chaos. But you can't just blindly follow a process for its own sake. You need to be able to articulate why we follow it with examples from the past of what happens without it. And, you need to be willing to constantly look for ways to improve it. Anything I can do to automate processes or simplify them makes the developers happier and more productive and me less of the bad guy. An inefficient process where people are always waiting on something to happen is demotivating to everyone.

Most process is scar tissue from something that went wrong in the past. We have mandatory code reviews because when we didn't we let a severe security bug get deployed to production. We have legal review all changes to the terms of service so we don't get successfully sued over it again. We have a schedule with code freeze N weeks before release because when we didn't we could never release anything because...just...one...more...change.

I was hired at a small company once without any process for anything. No release process or schedule. No bug tracker. No testing plan. No code reviews. Nobody used source control. Builds from different workstations produced wildly different binaries. It was a total madhouse and it was amazing that anything got done at all. The CEO simply thought the engineers just weren't competent enough, but they were--it was just a circus because nobody stepped up to formalize anything. Once we got a bug tracker and SVN set up, things turned around pretty quickly.


This doesn't have much to do with saying "no" and everything to do with not understanding the fundamentals of a policy. If they guy who said "no swap" had a reason, then you could have discussed and potentially changed the policy.


You ran with swap on because you saw a bunch of other machines with swap on and decided the policy was invalid. Thanks for making real security jobs hell, this is the #1 developer excuse for avoiding security requirements.


Wow, I had multiple very similar experiences with the same person she's talking about.


Microsoft used to be like this in 00's (I don't know about now). You had to run everything by Legal, and the default answer from Legal was always "no", no matter what you're asking. Want to use zlib in your product? Even though Windows is using it in like 3 different places already, the answer is still "no". JQuery? Out of the question. Anything more exotic, might not even ask. I'm pretty sure Microsoft wasted hundreds of millions of dollars reinventing a wheel, and doing it badly, over the past few decades.


I'm a bit of a "no man", and at a company I worked at in the past my (pseudo-technical) manager told me to spend hours creating some sort of extremely purpose-specific test environment to aid in resolving a ticket that would only take me 30 minutes to resolve directly - so I just resolved the ticket instead and closed it, and got on with the rest of my work.

Eventually I was dismissed from this company a couple of days later and this situation was raised as one of the points. I was told "you can't refuse a mission given to you by your manager, regardless of the circumstances, that's ridiculous" and when I replied "I have different views to you on this" it was considered to be rude. The person who dismissed me is more or less the management boxing bag - a certified "yes man" who nobody respected.

Of course this hasn't changed my attitude, though. Still a proud "no man" and wouldn't go with the flow unless I'm sure everyone completely understands the consequences. At some point people have to realise that their loyalties don't go towards their manager and instead should go towards the shareholders and clients. There is a bit of give on this, of course, but eventually you have to do the right thing by the people that actually stand to gain and lose - not just being another cog in the machine that exists only to support other cogs in that machine.


Sounds like a toxic place. Instead of compromising or arguing a technical point and making the discussion an engineer effort, your manager ruled by feudal law. This is the antithesis of good engineer, in which good technical arguments and trade offs are made with everyone equal at the table.

Why your manager is making technical decisions is beyond me as well. Managers are supposed to be managing people, not doing engineering work.


The funny thing is we're all making trade-offs at various times. So there will be people who will point at someone else and say "He's a no-man" and at other points they will be the "no-man". I remember saying "no" a lot at some points of my life, and I think it was because I hadn't yet learned to say "the cost of that is this and in my opinion it isn't worth it" but there are still times I am speaking to people where they are unable to truly grok the cost given my current skill of explaining and then it probably looks like I'm a no-man.

Besides, come on, who even has dogmatic rules like "No Swap In Production". That's just short-hand for a concept. When it's in dispute you have to trace the reasoning again. There's no alternative. I cannot imagine doing something only because someone said so. I certainly recall a time when someone said "Person X isn't going to like this" and saying "Well, then, person X is going to have to find another way to solve your problem". I certainly hope that if I'm person X in someone else's story, they take the same approach.

And if you have to trace the reasoning more than once, it's probably worth writing down your argument. Not as a big RFC-style thing necessarily, but even just a Slack thread you can point to.


Have Slack threads risen to the level of documentation now?


A principle I have is that I lower the barrier for others to contribute the things I want to see more of. Rather than holding a Slack thread up as the apotheosis of documentation, I recognize that I am more likely to receive wisdom if I allow for a Slack thread to be sufficient. It's just a permanent link to a conversation.

This is reinforced by the additional fact that the people with the most I have to learn from are the people with the least time to teach me. So I minimize the effort I require of them to give me their best.

For instance, if I had a chance to ask Feynman stuff, I would happily accept answers in txtspeak with emojis and broken grammar. Sort of a personal Postel principle.


I'm a big believer in apply postel is interpersonal communications. Particularly in the emotional realm.


When trying to start a culture of documentation, permalinks to Slack conversations where people made decisions are often helpful.

Think of them as real-time meeting minutes and it may be easier to swallow.


One of the easiest strategies to protect your reputation in life and business is to never venture to say yes (because if it fails you are on the hook), and to let your discussion always revolve around an idea's failings, never its strengths. It gives an appearance of expertise and command, but risks little, and thus often leads to promotion. But I personally find it poisonous to the enterprise's progress: Even the best plans have weaknesses.


Er. This is awful advice.

You’re also on the hook for impeding the organisation.

I’m a “no” guy. My boss is a pure yes man.

I don’t say no to impede progress. I say no to avoid overworking my team which would reduce the quality of our work.

I say no, because it changes the scope of work in progress, and that requires sufficient justification because it means either throwing work away (which is both demoralising and wasteful), potentially increasing time to delivery or creates technical debt in droves.

I say “no” so that people don’t make their own standards (like not following logging patterns, or metric naming conventions) which would cause all kinds of gunk in the infrastructure.

It’s not about following a strict process. Although that can be important for streamlining workflow through an understaffed team like mine, it’s about ensuring that the quality of what we do is not at risk.

Quality is our only currency, without it everything suffers.


What you say is true, which makes the topic so very subtle because it depends on the context. Saying no to protect your people/process is a good thing. Saying no because it is personally profitable to be disdainful of other's ideas while offering up none of your own, irrespective of what is good for the organization or your team, is not.

If one needs a mental picture, think of the quintessential hipster:

https://sabotagetimes.com/life/hipster-irony-is-destroying-o...


Change for the sake of change is almost always bad, making decisions without considering externalities is worse.


It's better to list the risks in a matter-of-fact way rather than imply a project is summarily a bad idea. If you act like Chicken Little on everything, then your next complaint will have no credibility, like in the boy-calling-wolf story.

But there are times, such over ADA/accessibility and data security issues where I am somewhat insistent, but people ignore me. They seem to value a "cute" UI over these. I'm doing the right thing but get rebuffed. We are not in the cute-UI business. Our domain is not a toy or fashion store. We've been sued for ADA issues in the past, and our competitor has had data breaches, so they are relatively likely risks. It's frustrating to be dismissed the way I am.


Honestly saying no is one of the most important things either a startup or large company can have. Saying no gracefully, respectfully, and having the receiving person understand the rational is the hard part. This is one of the function of "fellows", "Senior Members of the Technical Staff", "Principal Architects" or whatever your org calls it and the people selected for that should be excellent at it. I guess the problem the author points out is that is the only thing a person might do and that is not good. Folks in this position should also know how to say yes and build buyin from business people/exec/program management.

Most organizations fail horribly at saying no. Instead they try to silo things to cope. This works to some extent, and you will always have silos as a company grows up, but you really should never end up with 12 programming languages, 5 RDBMS, 4 NoSQL DBs, 3 Javascript frameworks, 3 CIs, 3 cluster schedulers and then all the aforementioned also duplexed across organization units. That level of sprawl reduces margins, increases onboarding time, stresses even the most experienced staff, and makes adaption and business decision making murky at best.


I agree with the author's overall premise, which is that other people who don't own and who are not responsible for your system should not have authority over how you build and operate it. One of the things that I like about the current company where I work is that owners always have the final say about their systems, not any external technical authority. There is no such authority who could say no where I work (with rare and limited exceptions), and I'm glad for that; service owners make the judgment call about trade-offs.

> "so-and-so says no swap in production".

That said, while I understand the point of the article is something different, I broadly agree with this advice: generally no swap in production.

If you need swap, then that's probably a sign that the workload isn't matched well to the resources of the machine (unless the workload was specifically designed to employ swap, which most are not).

The problem is that: maybe most of the time the application's working set will fit within memory, and so it will perform well, but some of the time it will begin to swap and performance will degrade. The result of adding swap to a system is non-deterministic performance.

If you're building high-availability, high-performance production systems, then you don't want the potentiality of arbitrary performance degradation or non-deterministic behavior. Instead, you would prefer to solve the root cause -- such as by increasing the amount of memory available to the process or system, so that swap is no longer required.

An engineer adding swap to a system is like a doctor treating a patient's symptoms instead of curing their disease. Both are useful but one is preferable. If you encounter a situation where you could solve an application problem by adding swap, then an even better solution is to address the root cause -- ensure the application doesn't run out of physical memory. Swap is a band-aid that masks a risk better addressed directly. Hence: no swap in production.

(On the other hand, it might be fine to employ swap in production for services that aren't performance sensitive, and where it's preferable to accept the performance hit from using swap rather than pay more for better hardware. That's a fine trade-off if consciously made. In my experience, variations in performance are usually much more problematic to a business than changes to cost.)


> generally no swap in production.

This is a bad policy. The problem is that it breaks Linux's overcommit logic. Many applications allocate huge chunks of virtual memory far in excess of what they actually use. Due to the magic of virtual memory, this doesn't get mapped to physical memory until it's actually used.

The trouble is that there's no mechanism in any programming language I'm aware of that allows the OS to retcon an allocation. The only tool the kernel has at its disposal is the oom killer. So if you allocate too much, and you have swap disabled, the oom killer has to step in and kill something even if you're only using half your ram.

This is bad enough, but in addition to this, you have vfs cache to worry about. If you have 64GB ram, your applications are using 40GB, and you have 22GB cache, you and I might assume you have 24GB free, but the OS only sees 2GB free, and a few allocations might trigger the oom killer if they happen faster than the vfs can dump cache. (which is surprisingly slow) With swap, this isn't a big deal, because the pages don't actually get assigned until they are used, which happens later, so the vfs has plenty of time to evict.

The point of swap isn't to give you more memory. It's so the OS can keep its promises about memory allocation and overcommitting.


> So if you allocate too much, and you have swap disabled, the oom killer has to step in and kill something even if you're only using half your ram.

You can allocate all the memory you want, the OOM killer only steps in when you actually use all the the memory.


But not having swap will cause the machine to misbehave (https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConseque...). Swap in this situation acts like a backup, surely you would prefer temporal degradation then total system shutdown. The solution here is continuous monitoring of free memory (we use datadog) + swap.


> surely you would prefer temporal degradation then total system shutdown

Sometimes, but sometimes you want the opposite.

A slow service (e.g. a db server with a failing disk) might still pass health checks but degrade overall system performance.

It is easier/simpler to detect a total outage than a slow/flapping service.


I think what may be happening here is that at some point the Linux kernel's behavior has changed and things run more smoothly with swap than without, even if you aren't swapping appreciably. There might be some code path it hits when it needs to flush cache or something like that that causes issue.


I'm coming to the conclusion that many technical problems are actually morale problems. This especially goes for flamebait, where some people come down heavily on one side of the argument or another.

If you don't care, it's much easier to default to someone else's directive, or complain about a toolset you haven't bothered to learn.


Interesting combination of the "yes men" parroting the "no men"! Great example!


i was at this company when this happened, and oh boy, EVERYONE who knew the person in question and the 'new' kernel feature got some amazing, long-lasting and well deserved laughs.

btw, it was /proc/<idiot's username>, not /dev :)


I didn't recognize the story until you corrected the path. https://news.ycombinator.com/item?id=1242831


The issue here isn't with "Mr No", it's with those who mindlessly parrot his policies.

I've met "Mr No" at Google, and he is in fact quite a reasonable and funny guy.


I've heard the "kernel extension guy" story at my last job; it's fairly well-known there although it wasn't a "completely obvious change", it was heavily obfuscated and possibly done over several PRs in order to sneak it in. Hadn't heard any subsequent stories though, including the swap story of the rest of the article.


I used to have a “don’t know/can’t remember” man in my team.

Can you recall what this code you wrote previously do? “I can’t remember.”

Do you know where the files are placed? “I don’t know.”

Where are the documentations for xxx. “I don’t know.”

Do you know the difference between xxx in testing in production server? “I can’t remember.”

It was frustrating.

9 out of 10 questions were “I don’t knows” and “I can’t remembers”.

I’d much rather he said yes or no.


Of course it would be great if everyone remembered everything, but isn't an "I don't know" better than an 80% guess? The longer I do this job, the more likely I am to say I don't know, when I don't know.

> Can you recall what this code you wrote previously do?

What it actually does might be different from what I remember. It might have been changed since then -- maybe by me, maybe by someone else. I might have made a mistake in the first place, and it doesn't do what I think. "I'm not sure, let's look at it."


There's also the Maybe guy. All this guy ever says are things like "Maybe that's possible," "in theory that maybe could work," "I'm not sure if that will work."

The thing that the maybe guy differs from the Yes or the No guy is that he's always 100% correct. He's never wrong ever.


Most developers probably view sysadmnis as "no men". You want to do WHAT in production? flipoff


I still have an internal no-man that says "Don't use OR's in where clauses" in my head from years ago, then I have to stop myself and think "OK lets actually test the performance of this particular query rather than rely on a no-man policy in another company from years ago"!


I think of most in-house corporate lawyers as playing the role of "no men". There job is to highlight risks and stop things from happening. It's a rare lawyer who understands business goals and can balance them against risks.


That's not the lawyer's job - the lawyers' job is to present the risks, not to make the decisions as to whether the risk is worthwhile


In theory. In practice it often turns out that the decision makers think that any risk at all is unacceptable, giving the lawyers de-facto control.


In which case you have poor decision makers, not poor lawyers.

The benefit of domain aware lawyers is they can explain the specific risks and help the decision makers apply those risks to the business.


Good lawyers figure out how to say yes while minimizing legal risks.


Trust but verify. I'd bet if the author went to the original source, the "no man" himself, they may find that there is actually no such policy! Just a mishearing of a conversation parroted by "wrong people".


“Trust but verify” litteraly meant “Don’t trust” (as in “don’t trust the Russians” in context). Was it the nuance you intended ?



does it ?

> Reagan used the phrase to emphasize "the extensive verification procedures that would enable both sides to monitor compliance with the treaty"


But you have to look at why he used that particular phrasing–a clever bit of wordplay to quickly get the idea across to Russians in familiar terms. It's a Russian figure of speech in the first place.


“Wordplay” is too generous IMO. The original russian proverb is also double speak, as we would say now.

It’s obvious when applied to trivial situations:

“I trust my wife, but I take the time to verify her phone every night and check the private eye report of her day”

We wouldn’t call that “trust”, pretty much the opposite


One time my app started crashing when I added a large async routine. I ssh'd in and topd and noticed memory and swap were maxed out. Doubled the swap and it runs just like it did before.

Gotta love simple fixes.


I know this "no man", had to work on an approval from him once, and I actually saw the infamous code change that implemented this. I don't remember him reviewing it. IIRC it was sent to a mailing list, to which he might have or have not subscribed.


Even saying "no" should have checks and balances.


The most important part of a fast car is good brakes.


As opposed to stuck brakes, or ones that activate at random.


Sounds like that workplace is a no man's land.


Why not just buy more ram?


The extreme example of Chesterton's Fence.

Is this, then, Chesterton's Berlin Wall? That is, a completely idiotic waste based on idiot politics?


I suppose it's also worth noting that Chesterton's Fence is an (imagined) real physical thing which required time and resources to construct. Thus the odds are pretty good that it was made for some real purpose, as opposed to a policy which is ephemeral and does not intrinsically require a significant expense of time or effort to create.


May as well start off with an introduction to toxic masculinity.


How so? The author is a woman.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: