Hacker News new | past | comments | ask | show | jobs | submit login
“I’ll Finish It This Week” and Other Lies (arxiv.org)
306 points by lnwlebjel 39 days ago | hide | past | favorite | 115 comments



imo, the problem with estimates vs reality is that estimates don't include mental state. And just to clarify more about mental state, I am referring to being in the zone / flow, and individuals are healthy and experience normal stress / happiness levels.

We assume and expect consistent productivity from ourselves. Anecdotally, In my 20 yrs experience, I've never heard a software engineer saying to their manager "add 2 more weeks on top because my mental state is not at its sharpest these days". No SE in their right mind would put themselves in such a vulnerable situation especially when no one else talks about it.

The delays we naively anticipate are the obvious technical related ones and gotchas / cases we might not have considered for the problem.

This is a problem that is prominent in this field and unless I've been missing out on discussions, I've never seen it discussed.

There is a huge productivity difference between people that wake up in the zone vs any other zone. And I've yet to see an SE or any other human that can wake up in the zone every single day. It's impossible. Although this is something we can improve at an individual level by focusing on lowering the std deviation which will help us estimate better and imo it's one of the key factors why some SEs can give more accurate estimates than others assuming they have similar technical skillset.

Each SE deals with this fluctuation / "problem" alone. I've often seen SEs jumping on the management track when the gap between "being in the zone" start to consistently widen. Often after a burnout that never fully recovered.

You wouldn't see this being a problem in careers like HR/Management/Sales. Because an individual can still be somehow productive without being in the zone. But in software engineering, art, writing and the like, it's kind of binary and is very visible.


I strongly agree with your comment, with a single caveat:

> You wouldn't see this being a problem in careers like HR/Management/Sales. Because an individual can still be somehow productive without being in the zone.

I think it could actually be the other way around. That kind of work is very conductive to be "in the zone". In my few attempts at managing groups of people, I've learned that the work that's easiest to give you the state of flow is one where "next actions" are always obvious, and you get rapid feedback. Work where you can - and often should - substitute moving forward for thinking things through. Work where thinking gets distributed, outsourced to group interaction. That is sales and management, and I imagine lots of HR also.

So in a way, I think "people people" are "in the zone" much more often than we are, which is why they may not understand just how hard it is to get there when your tasks are brick walls requiring intense concentration.


You can totally crash manager flow.

I wouldn't recommend doing this on purpose, but if you say something that forces the whole room to stop and consider, you can often break the conversational flow and leave a meeting of managers dead in the water.

You can also do something that wipes an individual manager's mental cache and they'll spend days trying to remember what everyone is up to and what's happening.

I think your final sentence is close, but I'd rephrase it: for managers, those disruptions and conversations in your schedule are them dealing with things that require intense concentration. I think most of them understand what disruption feels like, they just don't associate sitting at the computer typing with that focus because it's not the mechanics of their role -- their focus manifests differently.

My managers only rudely interrupt me when they, themselves, are clearly still processing something their boss said to them and they need my input to unblock that process. Or some kind of thing I need to do now, again to unblock whatever they have going on.


I think you hit the nail on the head, I have definitely experienced a lot of this, even when dealing with managers I enjoy working with and that have an understanding of engineering-type workflows.


I'm not sure the metal state that you are referring to really makes a difference in terms of deadline estimation. The way I see it is that, in the grand scheme of things, if you are consistently missing the targets then either you get fired or, if the problem is systemic, the company folds.

In other words, if you are not getting fired and the company isn't folding, and that neither is happening anytime soon, then everything is just "business as usual" and missing a deadline here and there is just part of the natural statistical fluctuations that everyone have come to accept according to some unwritten rules.

I would go as far as drawing parallels between "being in the zone" for a software engineer and "meeting sales targets" for a salesperson. In that sense, not being in the zone every now and then is just as reasonable (or unreasonable, depending on your employer) as missing sales targets every now and then.


I would add that many "deadlines" are actually just arbitrary project management bullshit that exist to create a sense of "urgency" because the people that make the promises to customers don't trust the people that do the work.

But on an individual level, procrastination and slipping deadlines has a deep seated psychological basis. The best way an individual can deal with this problem is by addressing internal motivations and their own emotional state-- that's definitely NOT in the PM-BOK. (see https://www.nytimes.com/2019/03/25/smarter-living/why-you-pr...)


The problem is deadlines in and of themselves, it quite often leads to student syndrome[1].

I've seen it from the small up to the very big project. And I also believe it's also a little self feeding, after cramming so much for the first deadline you feel you deserve a little break, right?

We need to develop far more steady working practices harnessing intrinsic motivation rather than drive through fear derived motivation of deadlines and other anathemas of healthy work ethic.

[1] https://en.m.wikipedia.org/wiki/Student_syndrome


> in terms of deadline estimation

In my experience, deadlines are usually arbitrary to begin with. They are the result of annual/quarterly budget allocation much more than actual customer needs. Nothing of value is lost when they are missed.

I understand the need to divvy up budgets, especially in publicly listed companies. What strikes me as nonsensical is the particular inflexibility with these budgets and their poor correlation to actual customer needs. Perhaps this is merely a symptom of the modern business world, where "the business" refers to a class of people rather than the actual business.


I believe that OP is talking about estimating deadlines for yourself. If you estimate a deadline and then due to some reasons your mental state deteriorates, that makes it considerably harder to achieve it. Sales targets and deadlines provided externally would have very different connotations imo.


> if the problem is systemic, the company folds.

This is not the neccessary result. If a company is well established in its bullshit dealings it can survive for decades with inefficiencies and in some perverse cases even thrive.


> if the problem is systemic, the company folds.

Why do you think that? Imagine we have two similar companies, and one of them has mediocre deadline estimates, while the other would be mediocre except they also underestimate everything by 50%. Shouldn't both companies have similar success in the market?


> There is a huge productivity difference between people that wake up in the zone vs any other zone

There is my secret. I am never in the zone, and neither is anyone else in my company since they’re asked for status updates, meetings and ‘war rooms’ every few minutes.

I guess the one positive is that it makes estimation easier :D


I've only been coding professionally for a few years and I've too noticed dramatic productivity differences some days compared to others that aren't related to unexpected technical aspects ... it's just the volume, speed, accuracy of my coding all by itself it seems.

With something that takes a lot of focus I think mental state is absolutely huge.

We see athletes go through warmups and traditions before they start a game to get going not just physically but mentally ... makes me wonder if I should do the same.


Personally, it's much easier for me to be invested in a task when what I'm doing is 'interesting' to me.

Almost always and ironically, when something's interesting to me it's usually more difficult work and thus, contains far more unknowns, hence making estimates that much less reliable.


The solution for artists and writers is to work consistently. Obsessing about whether or not you're "in the zone" isn't productive. Instead, just get started, work every single day on your field. Seinfeld had a calendar and he had to write an X on every single day after he wrote jokes. I don't see why any high-functioning individual in any field couldn't learn something new every day.


This is also the solution for developers. You sit down and do the work. Some days are better than others. If you have rituals or strategies to maximize that productive time, that's great, but don't sit around waiting for the muse to inspire you.

I find it helpful to have:

- resumable tasks - I want to sit down and work, not sit down and figure out what to do

- multiple projects - if today isn't a great day for intellectual pursuits and I keep getting interrupted maybe I'll do an admin task or fix some simple bugs. Also works for when something is stalled.

- acknowledge you cannot control everything - if the progress is delayed in ways the estimate didn't anticipate, document them and keep working


> - resumable tasks - I want to sit down and work, not sit down and figure out what to do

I've found this indeed helps me get back to productivity. But at the same time, if I know what exactly needs to be done, it would be hard for myself to stop until it's finished. And that's when multi-day coding binge happens, followed by a few days of fatigue and low productivity. Kind a of a catch-22 for me...I wonder if there's anyway to be moderate about it.


I am not able to do that now. I have a family that needs to be there. When I was younger, sure, I'd pull all-nighters. But those days are gone. When I work now,I have to use my time wisely and be laser focused. I also found when I was working crazy hours, the quality of the code wasn't great. I learned to sleep on problems, then I'd solve them quickly first thing in the morning.


I'd say that not talking about context switches is weird.

Sometimes I'm jumping between 3 different projects and having talks with 2 people about $new_project within day and it feels like at least 1/3 of the day is wasted on context switch.


In my case, I'm a "wee bit on the spectrum," so to speak.

The minus is that I don't relate with people on the same level as most. It used to be fairly crippling, but I learned to compensate, and these days, it's just a bit annoying.

The plus is a "flow" that is incredibly deep and productive.

I long ago, gave up on "scientific" estimates. They make good catbox liner; not much else.

Breaking a project into layers and explicit "packages" (modules) helps. It allows me to evaluate complexity a bit better; but it always ends up being my "gut." I've been developing software for more than thirty years, so it isn't actually a bad metric. I am a bit of a cynic and a pessimist. That helps.

But the thing that works for me the best, is "JIT estimates." I won't (as in refuse to) give an estimate for the long term. If someone has a project plan in mind, I will try to plan a scope that I "feel" will be doable within that scope, then reduce it. If it is just me, I'll reduce it by about 25%. If it is a team, I'll reduce it by at least 50%.

I will then avoid specific estimates until I start on a package/layer. If I think it will take longer than I'd like, up front, I may reduce the scope. It's easier when it's a small[ish], discrete, atomic job, as opposed to a months-long aggregate effort.

That said, in my experience, the boardroom will always have a classic waterfall schedule, and they are the ones that sign the checks. They aren't always thrilled with my approach. That's why I have waited until I'm running the show to do it.

I'm working on a project now, that is fairly ambitious. It's the kind of thing that usually requires a team of five to ten engineers, and I'm doing it on my own. Thankfully, a really significant part (the backend) is done, tested, and released (for a couple of years). That puppy took seven months. I've been working on a native iOS frontend for about five months, but took a break for almost a month, as I needed to give a class.

In this effort, the folks I'm working with have no choice. They have to abide by my schedule, as I'm the only one doing the tech work. They are actually thrilled with the speed at which things are coming together. I credit my "always ship quality" approach for that. It almost eliminates "QC surprises," and gives unmatched transparency.


Indeed, this is a standard cause of procrastination: the belief that our future selves will be more in the "mood" to do the work, while our current selves aren't. Unfortunately, in the future we will still be our current selves.

So, similarly, we give estimations based on this rosy picture of our future selves, who will somehow work the rest of the week in unbroken eight-hour stretches of concentration and flow.


Huge thanks for talking about that. I thought I was the only one. I'm really unproductive when I have budget issues, can't stop thinking about it.


>And I've yet to see an SE or any other human that can wake up in the zone every single day. It's impossible.

I wake up in the zone every single time because I sleep what I need. I also do exercise and eat well. It is that simple.

In fact I wake up two times because I sleep in between hours of hard work. I copied that from tennis players and other elite sport players. Rafael Nadal plays a multiple hour match and goes to the Hotel to sleep(At 12 AM!!) after that.

I see no difference when the stress is mental hard work instead of physical. I am an elite player too.

I believe what you call "in the zone" is code for "having rested from an stressful situation" or not.

I see people trying to get by by using stimulants like coffee to work, but that is deluding yourself. It is just removing the signal from your body that you need to rest after stress.

You don't see this being a problem in HR/Management/Sales because the intensity of their work is usually very low. But those jobs could also become emotionally very stressful too (having to fire someone, pressure from above).

Most people can not even control or manage their own time and waste 2 hours commuting each day. Something has to give.


> I wake up in the zone every single time because I sleep what I need. I also do exercise and eat well. It is that simple.

Tried that, didn't work. Hell, at some point in my life, the only consistent way for me to get into the zone was to be sleep-deprived.

The zone isn't about stress per se. It's about focus. Low levels of stress can often help here (acting as rails that bounce you back onto the track if you start to veer off). High levels of stress will of course prevent the state of flow, but if you can kill off that signal somehow, you may have a chance to get back into the zone.

> I see people trying to get by by using stimulants like coffee to work, but that is deluding yourself. It is just removing the signal from your body that you need to rest after stress.

Thing is, people often have no other choice. The body doesn't have a "snooze" button - once it considers something an emergency, it doesn't understand the signal was received and noted, but there's nothing that can or should be done about it now. It'll just keep bugging you.


I recommend that you read about flow then.

The original idea of flow(in the zone) comes from this book and author, but is obsolete now:

https://www.amazon.com/Flow-Psychology-Experience-Perennial-...

I recommend this book and audiobook about gammification to understand flow better.

https://www.amazon.com/Reality-Broken-Games-Better-Change/dp...

Whatever breaks the game in your work, stops the flow. For example, instant feedback is very hard to get creating software unless you design for it(like using REPLs and reusing almost everything).

Other things that the book mentions break your game too(not having clear rules, being alone)...

Another important thing is thinking too much rationally. The logical mind is so slow getting results that you break the instant feedback.

Also working too much in low level has way slower feedback than working with higher level languages. I metaprogram lower level languages for this reason, I am orders of magnitude faster.

>The zone isn't about stress per se.

Sorry, I was not eloquent enough. when I talk about stress I am referring to stress in general: Mental, emotional, or physical stress.

That is, if you are doing something difficult mentally, you stress your brain.

For every stress you need time to recover.

You can do 10 hours without recovering if the stress is low. If the stress is high, you will only be able to do 4 hours.

It is very easy to understand, you can walk for 8 hours, but you can not run for 10 hours. A marathon is about 4 hours or less.

What I am telling you is that if you run for 4 hours hard mentally and do not recover, you will not be able to work another 4 hours of hard work, unless you rest.

Nothing will do, no technique will do unless you rest.

Of course you need to do right lots of other things too. But don't say it is impossible just because you can not do it.


ProTip: Human psychology and experience are sufficiently variable that One True Way responses for complex activities are exceedingly rarely universal.

What works for one person often does not for others. What works for a person at one stage of life often doesn't at others. What works in the short term often does not in the long.


Another anecdote. I'm someone who's made "sacrifices"[0], giving up alcohol, leave parties early to make sure I always go to bed at the same time (no exceptions), getting my diet in order, exercise etc etc. Literally prioritizing my sleep above everything.

I'm experiencing exactly what you describe. There's zero ramp-up time to get in the zone. I've gone from having 4 productive hours on a really good day, to having 8, at work, and several after work. I just get up in the morning (no alarm necessary), sit down with a clear mind, and start working.

I also do something similar to waking up twice. But it's more just lying down and staring at the ceiling for half an hour in the afternoon, if I feel it's necessary

[0]It's funny how the me of 3 years ago would consider these things huge sacrifices yet after I've done them it feels ridiculous to describe them as such


I've heard the theory that flow requires periods of struggling with learning or improving a skill, and that the intense learning is a precursor to having sufficient skill to be 'in the zone' relative to the challenge at hand. Does that jive with your observations? Is learning just another challenge that you have sufficient skill for, or do you find that cycle isn't necessary, or something else?


I am sure many people are like me in that there are constant meetings, questions and other interruptions to the point that you never actually get into the "zone".


I have adjusted estimates saying that exact thing to a publisher, but never in a traditional software environment unless it was a physical injury or sickness.


Regarding mental state: this study covers June 22, 2020 to the "present" (the paper is dated April 1, 2021, but I don't think it's an April Fool's joke). I suspect that people are to some extent calibrated to pre-pandemic expectations. I know I can't get things done like I could have in the Before Times, either because of mental state or because of more distractions at home than usual.

Figure 1(b) of the paper might suggest that people's estimates are worse on tasks that require flow, although I don't think there's enough data to be sure.


Energy.

I’ve seen a couple ugly conversations where someone didn’t understand why an engineer couldn’t do 2 tasks that take half a day/week/sprint in the same time interval, and it comes down to both of those tasks being very high maintenance, or requiring a high degree of concentration. It may only take 20 hours to complete but it sucks up 30 hours worth of energy. You need something brainless for the other 20 hours, not a second task of that ilk.


Overriding priorities has nothing to do with "bad mental state". It only has to do with changed priorities and scope.

Missed deadlines have to do with need to fix other more urgent stuff.


An important thing I've learnt is that people over-estimate what they can do in a short time whereas they under-estimate what they can do in a long time.

You probably wont finish that feature tonight, and that's OK.

But you will have built something good in a year if you work on it a little bit consistently.


I feel like people underestimate the time something will take but overestimate the actual effort. What you say seems to be a consequence of that.

We underestimate the time, because in reality we often worry about things, and procrastinate on decisions. But in the estimate we don't think about it, we assume all decisions will be made immediately.

We overestimate the effort, because we sometimes do not know how to solve certain problems or work efficiently and in the estimate, we only know the hard way, not the easy way. But with practice, the hard way often becomes easy.

It might also come down to us valuing future work more than the work in the past, so looking back things are less effort than looking forward. But in the case of time itself (represented by hit/missed deadlines) we value these as more important with respect to the past than to the future (they already happened, while the future is uncertain).


That's one of the big problems I see with enterprise agile development. Breaking up things into small independent parts never allows you to gain momentum on something. Pretty much everything I did that I am still proud of were done after focusing on the an issue for weeks or months until I/we had a very good grasp of the issue and built up the mental and code infrastructure to do cool things. You can't achieve this in 3 week sprints.


"Never commit to deadlines"

                   -some software engineer

Software has an unbounded set of problems, like in the creative fields. The difference is that you can fill in a book with random thoughts (if there is no time), but you cannot just fill in functional features in the codebase you did commit to. The same issue exists in the very large and complicated builds (eg skyscrapers) although you can plan ahead really far due to the existing physical constraints (site size, machinery, materials, avg. labour output).

Over in the software shop I work for, we just prioritize each week. Works really well. And we cannot commit to the deadlines or even estimate, as there are too many moving parts, too many products. To make customers happy the product managers comfort them with words (just like sales people promise mountains of gold for a penny).


I interacted in a workshop with a person who taught about personal organization, and we had a phone session.

I thought that it was important to fill your planning and calendar with fixed things.

Instead she advised me to just plan week by week. I'm a freelance and honestly, it sort of feel better to be part of a bigger, organized group that tells you the things that are to do.

When you're the only decider of your own planning, it feels really weird: do I just have to fill up my planning with whatever things I deem of importance?

Just like Sartre said, we are shockingly free (existentialism etc), and we still crave and aspire to be part of an organized group with goals we can adhere to, to feel safe.


I agree; regimenting your day is a poor substitute for short-term planning. It looks very orderly to have items like 9:00am, read email, 9:10am, Make sure Jira tickets are in correct states, 9:20am, respond to email, etc., filling your calendar from now until eternity, but it makes more sense to spend ten minutes at the end of every day planning the start of the next. Maybe you know your Jira tickets and email are in order and you can sleep an extra half hour. Maybe you have an important presentation in the afternoon and it would be best to spend 8am-9am nailing down the answer to a question you know will come up.

It's very hard but very important to stop working a few minutes early at the end of the day. You want to keep coding (or whatever) right up until the minute you need to stop, but that leaves you unknowingly disoriented at the start of the next day. You will either pick right back up with the coding, forgetting about everything else, or start aimlessly and waste the first hour of the morning trying to restore your awareness of priorities.


Honest question. Say you have a project. Your boss expects it can be done in two weeks. Let's say you are good at estimating your own velocity, and you estimate four weeks for that project.

Do you:

a) Tell your boss that you effectively suck and it will take you 4 weeks instead of 2?

b) Tell your boss it will take two weeks and then blame delays when the deadline passes?

or

c) Tell your boss it will take two weeks, then cut corners to deliver a finished project in two weeks, but one that is so lacking that someone (possibly not you) will need at least 4 more weeks undoing and fixing stuff at a later stage?

(Note: I have NEVER seen anyone do 'a'. It's usually a creative hybrid between a and b)


Not quite what you are asking but related: Many years ago my boss handed me the notes that he had made in a meeting with a customer and asked me to work out a project plan to create the hardware, drawings, software, etc.

I did that and gave it to him. The next day he called me into his office and told me that he and the customer had already agreed a delivery date and that I had to edit the plan so that it would be delivered on that date. That date was two months earlier than my estimate, 20% less elapsed time.

I tried to argue that we didn't have the necessary resources (software developers, hardware development, circuit board design and manufacturing, prototyping, etc.) to do it any faster but he insisted. So in the absence of any way to cut the time in a way that really made sense I just cut about 20% off each of the items on critical path of the Gantt chart and gave it back to him.

He seemed happy then but he was much less happy when we delivered on time and under budget according to my original estimate.


I go with a) every time. I would find anything else to be unprofessional.

I don't "effectively suck", I disagree with them on the timing. I'll explain my reasoning, and if they cannot accept that, then it's out of my hands; they have chosen to cut corners.


I go with a) as well, and I add a very generous time allowance on top of my estimate. But you can't really do much when you're told to 'get it done this week'. First to go out the window is TDD, of course.


Have seen companies fire people for that. Sometimes you can't afford that.


If a company chose to fire me for that, I would not want to work at that company anyhow.

I know this is somewhat naive; not everyone is in a position where they can pick and choose what companies they wish to work at. But still, such a company would be a terrible company to work at.


I (as a manager) always get a push-back when I try to impose my timelines on my team. Though we do have a healthy conversation as a result and we end up uncovering a bunch of unknowns which I then take it to my boss to inform them.

When an engineer says they need 4 weeks instead of 2 it's my responsibility as a manager to get into the details, not as a way to micro-manage but to help my team be more confident about the 4 weeks timelines. Often times the engineers have a "gut" feeling but aren't sure about "why" so I push them to spell out that "why".

To answer your question; "b" is just about the worst possible option. It not only makes you look bad but also the whole team will get labelled; remember that your boss also has someone to answer to and it'll make them also look bad. Never keep your boss/manager in the dark. Push back with data and educate them as to why it takes 4 weeks instead of 2.


> when I try to impose my timelines on my team

As a senior developer, I don't like this sentence. If you want an estimate on a certain job, you ask your team. If you want to know what can be done in a fixed time-frame, you ask your team.

I don't take "imposed" timelines well. In such scenario's, I make it very clear who is at fault when deadlines are not met (hint: it's not me or my fellow developers). When you push back on this, I will ask who made this unrealistic estimate. Ah? You mean the people who know the least of the work made the estimate? Well, then it's normal the deadline was shit, no? Maybe next time, ask the developers, they probably are the best at making estimates of their work.

When you say "gut" feeling of developers, you probably mean the instant reaction they give when you tell them the timeline. Well, it can't be "informed" because you didn't ask them to make an estimate did you? Developers need time making an estimate. "Gut" feeling is probably your developers thinking "WTF?!?!?!?".

I always educate my managers on who makes the estimates, how much time it takes to make the estimate, and how estimates and deadlines are not the same. I hope you also have a senior developer under you who can educate you on this topic ;).


My "aha" moment on this was realizing the difference between estimations and commitments. If there is some probability distribution of how long a task will take, an estimation gets the median of that distribution. On the other hand, a commitment gets at least the 95th percentile of that distribution. Estimations are useful for general scheduling and resource management, while commitments are useful for having others waiting to get started as soon as a task is finished.

The biggest problem I run into is people not being clear (or not knowing themselves) whether they are asking for an estimation or a commitment. "How long do you think X will take?" sure sounds like a request for an estimation, but if they follow it up with "Great, I'll tell the customer it will be done by then.", they clearly were looking for a commitment instead.


Real question: do you actually believe this to be positive more so than negative most of the time?

Many people will feel resentment over what is essentially continuous doubt and having to spend time to push back that could be spent thinking about the problem, diving in and making a better idea later. Additionally, your way of presenting the argument seems harmless and emotionless, yet reality is often far away from this. A slew of social dynamics take place which make developers say things to get managers out of their hair more than truly having a discussion on how long it takes.

The far majority of managers also impose these deadlines without clear rime or reason ("because the customer wants it"/"because sales said so", at best). What's really going to happen if we estimate higher and end up overdelivering at the end? Are you truly going to lose customers, or are you just trying to pressure people in "being accurate", which often devolves more into having one's subordinates take the blame and the fall for underdelivering?

With all due respect, it feels like we are normalizing things for the sake of manager's security without much reciprocation (only make-believe that it is being reciprocated).


It's the managers duty to create the trust needed for having a constructive discussion of task breakdowns and timelines. If the engineer is "just saying things to get the managers out of their hair" the manager has failed at this.

You seem to dismiss entirely that doing a proper task breakdown is a way to think about the problem, revealing unknowns, clarifying requirements and making sure as many steps along the way as possible are clear and actionable.

In an ASIC project, if you miss your tape-out deadline, your time to market will be delayed by several months until the fab can find a new slot to fit you in. This can cost many millions of dollars. Accurate development estimates are important.


I'm not dismissing a work breakdown. I'm dismissing the idea that pushing a quick 30 minutes to make people defend themselves is really as fruitful as a lot of people, non-technical managers especially, like to claim. When reality is, we hardly have any data to back this up, and there are many counterarguments against this idea.

A proper work breakdown may cost a lot more than 30 minutes. In any ticketing system, this can be inserted as subtasks which immediately grants an, admittedly superficial, level of transparency into how far the entire thing is done. If you truly want thinking to be done, surely you would agree talking it over for 30 minutes doesn't imply a whole lot of thinking, as much as it implies a whole lot of talking. At that point, just take a step back, chill, tell the developer they need a strong defense and teach them to provide that defense in whatever way suits them best that still makes the point come across.

Software development as a field has made such a knee-jerk reaction against failed projects, it now overemphasizes "security" and "correctness", and tons of red tape, on top of methodologies which inherently should already fix most of the problem. Aren't we supposed to work iteratively so managers can make a proper forecast? Let the actual data speak for itself and don't bind too strongly to the words of the individual. The far majority of people do not work on projects that truly require tight deadlines, yet almost every software company out there is trying to emulate this high-pressure environment without clear insight as to why people join a low-pressure environment to begin with.


While having the discussion could definitely be unpleasant and cause problems depending on how it goes, "having to spend time to push back that could be spent thinking about the problem" is a bit much. Half an hour isn't going to affect the deadline. The odds of "making a better idea later" aren't going to change significantly, either.


It isn't just half an hour. It's half an hour of being almost forcefully yanked out of the zone, which may lead to hours of lost efficiency. The idea that we just measure by the time it takes to talk to people plus a minor overhead, is part of why this is being normalized so often. Yet that idea has never been proven, and there is more suggestive evidence of the opposite. Nor has the value of such a "more accurate" estimate ever been shown, beyond a few actually critical deadlines (extremely rare in the field of software dev). Especially considering so many projects even under harsh estimation rules go overbudget and overtime anyway.

That's before going into the long-lasting effects of such a dynamic, and potential escalations. The most obvious one: dragging in part of, if not the entire team, for every little thing.


This discussion is part of planning that is already happening as part of the pre-work. There is absolutely zero need to take someone out of the zone to do it.


As a manager it's my duty to create a safe space and frame the challenge in a manner that doesn't create friction in the team. When an engineer says it takes 4 weeks I well and truly trust them, all that I seek is why. Because whey they then get down to one or two level deeper details they typically uncover more unknowns and for all you know it may in fact end up as 6 weeks.

Also, none of this happens on the spot; as in "tell me right know why it takes 4 weeks, let's do a 30 mins white board session". Having been an engineer I totally understand interruptions and, as a manager, interrupting my team is like shooting myself in the foot. When I say "conversation" it happens asynchronously over several days. I encourage them to spend a day or half a day to work out task break downs and dependencies because it takes a different frame of mind to "plan" a task as opposed to "working" on them.

> Are you truly going to lose customers, or are you just trying to pressure people in "being accurate",

It's a combination of both though I wouldn't phrase the latter as "pressure" or "being accurate". It's a way to discover information that will increase my team's confidence in delivering the project. As we all know most of the project failures are not because people slack but because they uncover a dependency too late down the line or they don't validate an assumption or a new constraint is discovered. Most of which could be accounted for with a simple planning process.

I do recognise though that most engineers don't like planning or task breakdown and so I spend quite a bit of time to take much of the heavy lifting so that they don't spend more than 10% of their time on it. As a manager I am held accountable for my team's delivery and I absolutely want engineers to be spending most of their time on writing/delivering software. One example is I don't ask for plans/task-breakdowns for anything less than a week. In fact there's not even a debate; on the contrary if an engineer says 3 days I ask them to round it up to 5 days (a week). Only for tasks that are longer than a week I may get into the details and for more than 2 weeks it's almost mandatory.

> developers say things to get managers out of their hair

This seems like a well and truly dysfunctional team. Engineers and managers are partners and only when they work together can they deliver something meaningful. I would address this mis-trust or friction first because if there's no transparency within a team then all bets are off. We'll end up with a series of failed projects, burnt out engineers, operational nightmare and what not. What I've noticed is these social frictions end up manifesting as "estimation problems" and people try to address the symptom by trying out different project management processes each of which fail spectacularly.


Thanks for the answer and being understanding.

>I do recognise though that most engineers don't like planning or task breakdown

I don't think planning or task breakdown themselves are the issue. Rather, it is a strong binding towards what is estimated when in the back of the head, there is always the looming threat of a missed dependency in a piece of legacy code one forgot about. Individuals can capitalize on missing ETAs by bundling them and still serving a track record which is largely coherent with the total estimate. They lose that power once a second or third party shifts the weighting and places a disproportionately high focus on missed deadlines over overall productivity and crucial deadlines being hit. That's when people start inflating estimates to give themselves security. Which shows up in the statistics as "perfect estimation", as unlike an underestimation, an overestimation can masquerade as a perfect estimation as long as someone doesn't investigate deeply (which itself is costly).

>This seems like a well and truly dysfunctional team. Engineers and managers are partners and only when they work together can they deliver something meaningful

Disclaimer: I'm moving the goalpost of the discussion here. I agree with the idea that managers and developers should work together, not against one another.

Unfortunately, this is the part where idealism and reality clash often. I question whether the majority of places execute this way, even if they claim to value the same idea. Given the inherent power imbalance between management and developers, it is just too easy for management to force its point of view and normalize away any dysfunctionality. Most developers won't quit on the spot: they still need to eat, don't perceive an abundance of jobs, etc.. The entire thing sets itself up for a frog in boiling pot situation, where as long as the management layers don't massively mess up, developers will continue normalizing away more dysfunctionalities.

I've seen this happen both with technical managers who no longer have to work with the tech themselves, and non-tech managers who have zero understanding of the entire thing. Power is intoxicating. The required empathy to deal with these gaps in perspective is a trait the far majority of people don't possess. The aforementioned power imbalance and other traits (maybe not in Silicon Valley, but here, management is both more prestigious and pays better) causes people without that trait to pour in, often selected by people who were lacking that trait to begin with. We get massive amounts of practices with zero support, because that same gut instinct that developers get berated for, many managers will often use to claim their way is better, disregarding any counterargument to their own whimsical feelings. If not gut instinct, it's some snake oil salesman selling a practice which, again, has zero support.


> When an engineer says they need 4 weeks instead of 2 it's my responsibility as a manager to get into the details, not as a way to micro-manage but to help my team be more confident about the 4 weeks timelines.

As a manager, isn’t your job to understand the well-documented problems with direct time estimates of knowledge work even when the work is being estimated by subject-matter experts who regularly perform the type of work being estimated, and understand and coach your team on use of the known techniques with that mitigate that somewhat (among which “managers getting down in the weeds to ‘provide confidence’ is not particularly prominent), so that you develop as-decent-as-practical estimates with, over time, enmpirical error bars without unproductively wasting everyone’s time and morale in exercises that can be assured to, in general, fail to.refine them meaningfully, rather than engaging in self-deluded pop-management-culture self-ego-stroking approaches that let you feel useful, even though they are known to make estimates worse.


Please don’t go into details. You’ll end up with people just doing c because they don’t dare anything else. Did you come up with data for your time hypothesis? Do what you want others do to you.

Programming is a million details codified, each code-line/detail is easy but the whole is complex.

If you really want to know become a team lead instead of a team manager and just code alongside them (and preferably take the easy parts to give the glory to those under you). I’ve tried it and the colleagues where over the moon.


I always do a, and that's why people think that I am slow and I don't have a great career...But at least I deliver stuff that works.

At my previous client, I had a "IT supplier team lead" that kept promising stuff delivered in x days even tho that x months later, the stuff wasn't working. He was seen as a god. Nobody called on his bullshit but me, later on, management was telling me that I wasn't a team player, good stuff.


yes, alas, that's exactly my point. The only peoply who are conscientious enough to go for (a) are also typically the same people who will pick up all the bullshit from people doing (b) and (c).


Tell your boss that you effectively suck and it will take you 4 weeks instead of 2?

No, I tell my boss that he sucks and it will take me 4 weeks instead of 2.


hahah, I like this.


Never seeing A is IMO a bit of a red flag.

A+C is probably the most common where I'm at (enterprise SaaS), though its usually with management's knowledge/understanding. We wind up rushing something out to meet the client's direct need to hit their deadline, then figure out generalisation afterwards (which admittedly does drag out and has its own issues).

As a genuine question, is this approach abnormal or bad?

I'm on the technical management side of the whole thing - am I screwing my staff by encouraging this?

We want to shorten that generalisation process but that (internal) pressure to just close it out seems like the lesser of two evils when compared to the sorts of pressure that clients paying significant $$$ apply


Where's D where we talk it out and why we currently disagree on the expectation difference?

Isn't that how it's supposed to be?


For me, it's not even that. I've never had a boss tell me what it'll take. I've always been asked, even though I've repeatedly told them I'm horrible at estimates. Thought sometimes they'll suggest a timeframe, hopefully.

When I tell them that they were too optimistic, their usual reply is that I need to get back to them as soon as anything changes for the worse. Otherwise, truck along and do it.

If I say 'nope, 4 weeks' right off the bat, there's usually a step where they consider if it'll be worth it in that time (or possibly longer) and they tell me when to ditch it and move on, if it's really going to be that long.

And it happens sometimes, even on projects that I really wanted to do.

The opposite happens, too, though. Sometimes they think it'll take longer than I think, and I tell them in that situation, too.


a) happens routinely. It's not just that you effectively suck, your boss also sucks and needs to learn from it.

Basically a) is your chance to recalibrate the expectations, and if you don't do it you'll be stuck forever meeting random deadlines (well, even if you do it you're still not out of that cycle, but at least you're setting precedents)


I think enough folks have covered the fact that you should be able to discuss how long things take with your boss.

However i want to make a point about cutting corners - in my experience, its a jusdgement call on what's essential, and whihc corners can be cut. I've seen engineers spend loads of time on things that in the end did not matter, etc. So I would say that's an inportant conversarion to have.


> a) Tell your boss that you effectively suck and it will take you 4 weeks instead of 2?

Assuming the company hired me because they had a gap to fill that I fit in and also that I delivered so far in reasonable velocity: if it's really so pressing yet unpredictable to deliver, I'd take the time to analyze the task and explain the boss what parts would take how long and why. If it's a good boss, he or she will either prioritize sub-tasks or talk to the other stakeholders and change requirements, i.e. adapt requirements to reality.

If you work for a good boss/team/company, the effect afterwards will be that everybody will be more happy. Also on the long-term the project will be successful. So if making such a statement implies "you effectively suck", consider changing workplaces because this is unreasonable and even toxic. (Although even the most toxic places will give in if you have the patience and confidence to explain. If not, they'll definitely make sure to blame it on you.)


Old timer here, and I struggled with this when I was younger. But I figured it out by working at a lot of different places. I always get great reviews and am considered a highly valued programmer, so you can depend on this advice :).

When your manager (never say "boss", because you are your own boss, they are the manager) says "2 weeks", at that point it's their responsibility. So as long as you don't say "I can do it in 2 weeks", it remains their responsibility.

So an easy way to respond is "I'll do my best", or "I'll see what I can do". In your head, you say "YOU said 2 weeks, NOT me. So if it takes longer, it's YOUR fault, not MINE". And then you work on the project like any other project. It's not about you being bad at your job, it's about the person being a bad manager. Keep the manager up to date on the progress.

Next step is to educate your manager. Don't fall in the trap of making estimates in 5 seconds on the spot. In this case, you could respond "2 weeks? I don't know, I haven't made an estimate on that." This way you indicate that YOU are the person who should do the estimate, not them. You could follow up with "Do you want me to make a detailed estimate?". If no, then indicate "ok, I'll see what I can do" as above. If yes, make a realistic deadline (NOT estimate, see below).

Bad managers come with all kinds of tricks to pressure you, like for example "The customer expects this to be ready in 2 weeks". At that point, you really need to educate them and show how reality works:

"Wow, I hope for your sake that you made a correct estimate. I didn't estimate the work, so I have no clue if this is possible. But you might be in a lot of trouble when you made the wrong call here". See what I did here? Managers try to push problems on you, don't let this happen. Always make it clear that the responsibility is always in their camp as long as you didn't make the estimate. If the 2 week deadline is not met, it's their fault of not asking you for a proper estimate in the first place, secondly for making a wrong estimate themselves, and never because you are bad at your job and should work overtime. Don't let bad management be your fault, because it isn't.

So when you can make estimates yourself, make sure you know the difference between an estimate and a deadline. Most managers think it's the same.

An estimate is "I think it takes around 4 weeks", which basically means "Best case it takes 3 (which it never will), and something might go wrong so it could be 6 weeks". A deadline always has a buffer built in. It's a worst case scenario. Your manager will ask for an "estimate", but 99% of the time it will be a deadline. So make sure your "estimate"=deadline includes a buffer of worst case scenario. If you estimate 4 weeks, they will promise it to the customer in 4 weeks. This is wrong of course, since there is no buffer. So say 6.

And this is how you keep your sanity as a developer. Educate your managers, because a lot of them are not really knowledgeable and experienced in dealing with proper software development.


Usually this has a tendency to propagate, in my experience. The deadlines come from above and/or externally dictated. So even if you have balls to say a), your boss (or your boss' boss, etc) may not, or they might have different agenda, that has nothing to do with the quality of the deliverables.


At my job, our PMs don't suggest any time frames, they ask all the devs for the expected effort instead. More discussion and agreement then by the people actually qualified to say. Of course we do still have some deadlines but that's also not in the PMs control as they're external factors


Yeah, the most helpful answer is a, but with a follow up explaining what you can do in two weeks, and you let the boss decide if that’s good enough, if there’s more resources available, or if you should work on something else with that time instead.


d) Tell my boss that THEY effectively suck at estimating and it will take 4 weeks. If they want less, I'll not sign under unprofessional work and they can get somebody else to do it.


My manager sometimes makes commitments to stakeholders on the spot when he agrees to take on the work, without consulting with the developer (me, in these cases). Often the requirements are not even clear yet.

It's not comfortable to do, but I will jump in and veto his commitments right there if they are clearly not attainable.

The stakeholders tend to be accepting of the fact that things are more complex to implement than just: "make the software do X", but my (technical) manager always seems a bit baffled. In the long run, I know he values this about me, but it frustrates me to no end.


Yeah, I get that blind enthusiasm sometimes but I've learned to control it. I show enthusiasm on my face -- and it's genuine! -- but I don't rush into promising. I found that to be the best balance because (a) people like it when you genuinely care and (b) you still don't come off as a pushover that accepts any demands for budget and schedule on the spot.


> Note: I have NEVER seen anyone do 'a'. It's usually a creative hybrid between a and b)

So, you've never seen someone being a professional ? I've told my boss plenty of times his estimates are wrong, and that it will take longer. It's my job to know how long things take. If my boss thinks it can be done shorter, he better 1/ have the same work experience that I do and 2/ do it on his own.


I have seen the effects of not saying 'a'. People even asked to leave the company because of simply agreeing to finish as asked. When we agree to it, our head will roll unless the company never cares about missing deadlines.

Another approach which I follow nowadays is, agree but tell them that most probably I will miss the deadline.


As someone who has been in the "boss" role on and off for a long time I have a policy of never trying to argue developers down on estimates - I usually argue up as, at least in my experience, most developers are fairly optimistic at what can be achieved in a given time.


d) Tell your boss what can be done realistically in 2 weeks and what timeframe it will take to do the full "Project".

You have two things... a request and a time frame. If they don't match? Admitting you suck isn't a realistic option. It's a lie to start with and it helps no one. Lying to your boss ("Sure, I'll get what I said would take 4 months done in two weeks") helps no one either.

Helps if you have some "street cred" to back up your claims... and you won't get that by saying "I suck" or "Sure thing boss".


d) Explain to my boss why I think it will take four weeks rather than two.

We'll then have a discussion and come up with a hopefully more realistic estimate. This might include finding a different solution to the problem.

Sometimes it has to be done in two weeks, and then the issue is what we can do in two weeks. Other times it's just a bigger problem than what my boss assumed. And yet again maybe I'm overthinking things and it can actually be done in two weeks.


d) because nobody knows how much non-trivial project takes until its done. Its well known that developers missestimate time needed by the factor of 4 to 8.

So d) which you miss, is about 4 months. Anything else is not professional and you cut corners. There is no such thing as 2 weeks in enterprise - hello world would take more with adequate docs, ci/cd, metrics, proj. management, optimizations, authorizations etc.


d) Tell your boss it will take 6 weeks, deliver in 4


I see you're from the Montgomery Scott school of estimation.


0 - your boss is a smart one and actually knows how to use Pi properly as an estimation multiplier.

About your honest part - your boss should really have a lot better estimation of your skills (or he sucks at that too). I suppose you're preoccupied with their opinion of you rather than getting the job completed.


a) has the implication that you suck if you can't do it in two weeks. It's pretty sad that we can't state the cold, hard facts but have to lie.


Over the years our team has gotten good at grooming and splitting up tickets, at estimating individual tickets and keeping a steady Sprint velocity.

Each 3 week Sprint we deliver what the team commits itself to.

However on a macro level we fail spectacularly. Initial Roadmap estimation of how long new features take are way off. A new feature should be ready in 3 sprints.. make that 6. This milestone should be reached in 2 sprints.. Make that 4.

It often comes down to hidden requirements that are discovered midway or prototypes that get into feedback loops with stake holders and take much longer then expected. Compared to those problems, we Rather seldomly we hit unexpected technical problems that prolong a project (but of course this happens too)


This here is why requirements need to be use case oriented, not code oriented. “I want to be able to look at my user profile” not, “I need to click on my user profile picture and navigate to a page displaying the following ten items...”

There needs to be room to deliver the value that can take on that uncertainty. It’s a good thing when an engineer can take a look at the use case and have options for how to deliver it, if you hired for that.


Does the roadmap include 1 sprint blocked off for unknowns for every 1 sprint that is full of known tickets (actual ratio should be determined empirically for your team)?

If not, I think I've spotted your problem.


If you focus on short-term, it's somewhat unsurprising that you fail at long-term.


it's all fat tails, they're all fat tails, assume correlational structure and whack peeps in the head if they say anything about bell curves

when you play the game of code you play a demonic St. Petersburg martingale. you start with one time period. flip the coin- heads, you finish. tails, you double the time period. at the end of that time period, flip it again: heads, you finish. tails you double the time period again. and so on: the expectation is infinite.

"well, that doesn't sound so bad, really" - there's why bernoulli was scratching his head.


> I said a 8 but I don’t really know why

> Could be a 1 or it could take my whole life

https://www.youtube.com/watch?v=sh7A8UChBTI


I didn't know this existed. Thank you


Hmm, who knew that if I had just paid good enough attention at my team's standups, I could have gotten a publishable paper out of it.

This isn't sarcasm. I'm actually glad someone did it.


> So, what did we learn?

1. If you are coding or writing, multiply the expected time of any specific task by ∼ 1.5x (the real completion time will now probably still be 1.5x longer than your new expected time, but you tried).

1.5x matches my real-world SW dev experience.


I read somewhere that a rule of thumb is to double the number and then scale it up by one order of magnitude.

Example: 1 day == 2 weeks, 1 week = 2 months, 1 month = 2 years. More than a month = Let’s not even start.


Estimates are here to fail, but they're nonetheless crucial:

* set an estimate, you'll deliver in 1.5x your estimate * don't set an estimate, you're losing total control over delivery (rarely for the better, even though that may happen)

In other words, estimates are not a tool of delivery planning, but of risk management : they are here to help limit time dilation.


I will paraphrase a former Japanese manager who taught me a lot about designing for production:

"Plans will always fail but we must always have plans."


"Plans are worthless, but planning is everything" is the version I've heard.


All models are wrong but some are useful.


If you can escape from the constant firefighting of interrupt-driven work, you can finish things on a predictable schedule.

Unfortunately, you log on to work on X, and it turns out customer A had a sev 1 outage, and youve got to scramble. Maybe once that is resolved, you can get back to X tomorrow. Except tomorrow, customer B reports some other bug, and you're dragged into triaging and reproducing that, while its still fresh. Now it's Wednesday, and you get one day to work on X. Thursday rolls around, and your day gets chewed up by another handful of meetings and status checks with A and B, also C came in with something. On Friday, you're worn out and reeling, and you realize project X never really got off the ground.

Rinse and repeat until things just fall through the cracks.


I wish they had plotted those time-series graphs with simple lines, instead of with overlapping filled curves that hide the initial history for data plotted "underneath".

Has MIT changed from a technical institute to a business school, where pretty colors matter more than the conveyance of information?


Estimates are bad because everyone is preconditioned to lie and don’t want to hear the truth.


Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.


I like the use of “spoiler” in the abstract. I mean, that’s basically what an abstract is.


reminds me of "cheap, fast, good; pick 2"


This has been formalized in a triangular graphic. Why not provide this to a project manager, on a corkboard or magnetic board, along with pins/markers to show original plans and current estimates ?


Doesn’t non-fast imply non-cheap?


"Cheap" == billable time * per-worker capital and support, not elapsed time.

Obviously, if a given workforce on a time-based compensation consumes more billable time units, they will cost more. That isn't the general case envisioned by the GFC triad, but rather that additional resources (bodies, brains, greater talent, tools, extended workdays or weeks) be applied. Classically, The Mythical Man Month. Accrued technical debt is another (often hidden) cost.

Good / Fast / Cheap is a possibility / optimisation frontier. When you are at the frontier, you cannot optimise any one third factor without sacrificing one of the other two.

An obvious countertactic in negotiation is to argue that the task is not yet at the frontier / optimum. In practice, this may be true, though the frontier may still lie far closer than such tacticians generally seem to posit.


If I understood your question, cheap + good + slow is generally when you do everything yourself.


I guess it depends on your salary. ;)

The way it makes more sense to me is when you spread out the same amount of effort over longer elapsed time, quality increases just my not rushing things and being able to reflect more.


It can imply junior engineers/interns doing the work as well hence cheap and slow.


There's another dimension which is "scope" and is also often negotiable.


> "Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law"

- Gödel, Escher, Bach: An Eternal Golden Braid. 20th anniversary ed., 1999, p. 152.

https://en.wikipedia.org/wiki/Hofstadter%27s_law


Ha, you beat me to it. Though you quote it wrong. That was the whole sentence in the G.E.B.. Without the quotes.


I updated the the quote. Thanks!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: