“Aim for at least half of all commits to be refactorings”.
I feel like this is the end game of scrum and most agile methodologies - endless refactoring on a treadmill with no off button,
I like to be introspective, and I am human so my code is far from perfect. But if I was refactoring half of my time I would go more than a little crazy.
The good systems I have worked on have converged on designs that work for that space. Both developers and users see and value the stability.
The bad ones have had the kind of churn the article mentions. Developers are constantly rewriting, functionality is subtly changing all the time; stability doesn’t exist.
The real key here is to understand Netflix’s business, and also many social media companies too.
These companies have achieved vast scale because correctness doesn’t matter that much so long as it is “good enough” for a large enough statistical population, and their Devops practices and coding practices have evolved with this as a key factor.
It is not uncommon at all for Netflix or Hulu or Facebook or Instagram to throw an error or do something bone headed. When it happens you shrug and try again.
Now imagine if this was applied to credit card payments systems, or your ATM network, or similar. The reality of course is that some financial systems do operate this way, but it’s recognized as a problem and usually gets on people’s radar to fix as failed transaction rates creep up and it starts costing money directly or clients.
“Just randomly kill shit” is perfectly fine in the Netflix world. In other domains, not so much (but again it can and will be used as an emergency measure!).
It’s about pleasing upper management. Those are not the same things. The article even says it explicitly - if your users hate it and the market laughs, but executives like it, you’ve shipped. If you’ve given users the best software ever but your executive management doesn’t know, or even actively hates it, you haven’t shipped.
I think the article does a very good job of defining the term "shipping" in the context of large organizations, where it you genuinely do need to please upper management - if you want further career advancement, in any case.
It's a bit cynical, but that doesn't mean it's not true.
If you want to ship software without worrying about upper management opinions, go work somewhere smaller.
You can still please end-users AND upper management at the same time. That's what I would aspire to do in that situation.
It's much easier when 'upper management' is 3 individuals you can have a normal conversation with, than a hierarchical hivemind above you that does inexplicable things for random reasons.
People are also usually suddenly much more reasonable and cooperative when you're discussing something face to face with them and not through third parties :).
It's also like this in small companies, like the one I work for, and I think rightfully so. One of the mobile app projects I was involved in got bogged down for months due to the lead engineer's need to refine and polish the perfect UX and architecture which kept breaking features we'd already completed. This cost us a lot of money and the product didn't look or function to an end user that much different. Management had enough and pulled the guy off the project because it just wasn't getting shipped and what he was doing didn't align to their goals. After he left i was able to tie up the loose ends and ship it live within two weeks. The only way i was able to do that is I knew every component of that system, our deployment infrastructure, our support staff, the real goals of our management team, etc.
Job advancement rather. Career advancement is possible by making users happy and telling executives at other companies "I did that". It's a better story if you had to fight for it.
If end-users do like it upper managment notices it but it will take time, generally you iterate faster against upper managnent. Same kinda applies when end users dont like that stuff so you cannot lean too heavily to opticks game.
I think this falls under the category of "Don't hate the player, hate the game."
I agree with you: It totally sucks that this is the way the business world works.
But if you want to work within the confines of this flawed system, you need to know how to navigate it well, and I think this post does a good job of giving you the guided tour.
There is a failsafe, however. The post points out that if the project launches and users love it and it makes the company a ton of money, then upper management is going to find out about it even if you don't tell them, and they will be pleased that you have made the company a ton of money.
There’s a hero element here that so think is what irks me.
The reality is, most of the time when you “ship” your software, executive management neither notices nor cares. Even if you get the “attaboy” from a manager, they will forget it 10 minutes later.
You ship your software because you care about work, and maybe because you care about your users. In a big software company, shipping is not going to get you noticed unless you are literally the guy delivering gmail or google maps.
If a ship happens in the forest and nobody at company notices, did the ship really happen?
To be clear, the pathway to management noticing is not you going around the building loudly blaring that you did a thing, it's that churn went down, or ARPU went up or CAC went down or uptime went up or something management cares about.
But the essential thing to also understand is that this doesn't just happen for free. Figure out what metric your ship is meant to affect, put monitoring in place and proactively notify those above you about how your ship impacted that metric. If you don't do this last step, you didn't ship.
> The reality is, most of the time when you “ship” your software, executive management neither notices nor cares
your reality might be different than mine, but things that got shipped definitely get noticed because they are used as a bargaining chip for promotions, funding and for visibility of the entire team, that's one of the main concerns of a manager (unless they've checked out and are searching for another job)
Not necessarily. You can have upper management chieftains, who view the rise of a "new" one- as a threat and will actively try to prevent "upstart" new sources of income, to protect the dependency on "whoever" pays the bills for decision making.
You will not get a product thats not search off the ground at google.
You will not launch something else then windows / now cloud at microsoft.
Even if it makes money. The old chieftains domain has to wither and be in retreat first, before something new can be started with all the resources available.
Its really exotic, if cross domain success is achieved. Amazon logistics and then AWS is such a example. If you look under the hood, the companies that allowed for that- are often more conglomerates, basically smallerish independent companies within a brand-trenchcoat pretending to be one large company. Youtube is also such a thing.
My pet theory is, that its company internal value chains, the chieftains depend upon that allow for such things to form. Like search is advertising and needs data. And data comes from the engagement guys down the hall in the other building.
There was a time, upto 10-15 years ago when tech company culture was largely anti-corporate.
Now corporate-y things like playing office politics is order of the day in medium to large tech companies.
My (maybe uninformed) hypothesis is that tech made so much money that it brought in the MBA and HR types who brought their corporate culture with them.
Even the smaller tech companies are now mimicking big tech culture as the standard.
>My (maybe uninformed) hypothesis is that tech made so much money that it brought in the MBA and HR types who brought their corporate culture with them.
This is some strange engineer's fantasy, that every company was ruined by "MBA types". Meanwhile, as evidenced in places in this thread, "software types" don't even understand basic business principles. They like to believe that if they were just left alone they'd churn out brilliant product, as if a large business isn't far more complex than that. Someone needs to count the beans (MBAs) and someone needs to deal with the people (HR).
I have seen the opposite happen a lot though, even if a project is successful you will get numerous people coming to you asking “who approved this? Why was I not involved in the decision?” And the ballooning aftereffects are why it takes months to push a simple button.
Big companies are like government bureaucracies now.
> Big companies are like government bureaucracies now
Large organizations are, by definition, bureaucracies regardless of being public or private. Big companies were bureaucracies 100 years, and they will be bureaucracies in 100 years. I never understand why people expect private enterprises to be free from the inefficiencies inherent to coordinating lots of people and things
Generally the game is created by external forces, not the players. If you’re an independently wealthy individual you can afford to work for pleasure. The rest of us? We like to eat and live in comfortable dwellings and take care of our families, so we play the game of pleasing our superiors in return for the money to live a comfortable existence.
Then you would expect to see inversion of behavior at the higher levels, especially at big tech, when one does become independently wealthy, but that's pretty obviously not the case.
I see your reply as strawman levels of uncharitable. They never said any such thing for either of your claims and the only reason I see for your statement is that you are playing.
It's funny because this is the absolutely tell of someone who spent their entire careers in large organizations. I alternated between large and small, and it's incredible how resigned and compliant people who have been in larger orgs can be. At some point the absence of actual survival pressure creates a dichotomy between internal and external success, and only the former is important, as this article relates. It's so hard to fight the flow, that either you go with it, or you get jaded and desperate, and just leave for better pastures.
To go and call that "shipping" is a stretch in my opinion, it is only correct as far as your cynicism, semantic tolerance, or resignation to mediocrity can stretch. But with reduced perspective and a slight ego, it's hard to tell that things can be different.
how can it be any different? Someone else is paying you to do something; you either do it (aka, you please them), or you don't (aka, don't ship, and you stop getting paid).
If you dont want this, you'd need to become your own boss - in which case, you ship when you feel you've shipped! You can argue that this makes you more aligned with your customers. And i'd agree - you've made the customers your boss directly. So now, they determine if you've shipped or not.
So the only situation in which this is different (or still the same tbh), is if you're your own customer.
> how can it be any different? Someone else is paying you to do something; you either do it (aka, you please them), or you don't (aka, don't ship, and you stop getting paid).
The good situation is when you get paid to do the thing, because you're getting paid by someone who cares about whether the thing is done. The bad situation is where you get paid to make a Potemkin version of the thing because the person paying with you cares more about the appearance than the reality, consciously or otherwise. You can redefine words and say well obviously when they told you to do X they were just doing an elaborate LARP and there's no deception in doing Y and telling them it's X, but that's just sophistry.
Working for arms-length counterparties helps, as that naturally forces communication to be more explicit and honest. Working in small businesses where reality tends to intrude more quickly helps. Of course neither approach is perfect.
You can love it or hate it all you want, the important thing to internalize is that this is a feature, not a bug of doing things at scale.
There's no way to engineer it such that you don't get to live in this reality. Be it tech companies, marketplaces, politics, revolutionary movements; once you reach a critical mass of people, all meaningful change looks like this.
That is before a smaller and nimbler movement, company or entity surpasses the now large, slow-moving and fossilized incumbent, which then starts the gradual process into irrelevance.
People get hung up on this stuff. Getting things done in different environments is about understanding the constraints. The objective isn't to religiously follow some "ship" cult. The objective is to get things done. Or at least that's my cult. I have a friend in Amazon's logistics department. One day he told me how he was planning something that involved buying up the entire capacity of a local airfield. The thing with a long enough lever and a fulcrum is that these guys are moving the world.
Love it or not (mostly not), shipping software in large organizations requires navigating what the business and its leaders demand. You can try while ignoring leaders, but you'd be unlikely to ship anything.
You'd be in good company.
In my experience, there are plenty of people at BigCos who are more interested in covering their ass to ship nothing or the wrong things.
The article actually presents strategies for shipping, not changing the hearts and minds of management, or making great products. Shipping implies neither of those things.
> If you ship something users hate and makes no money, but your leadership team is happy, you still shipped. You can feel any way you like about that, but it’s true. If you don’t like it, you should probably go work for companies that really care how happy their users are.
I once tried to ship a product and nobody else was ready. Marketing assumed the product would be late and wasn't geared up to market it. The CEO couldn't point to a marketing campaign so didn't want to mention it at the conference. Support hadn't bothered to ramp up on it since marketing had told them it wasn't actually going to ship until next quarter.
It's not about "pleasing upper management" is about understanding that a product isn't "shipping" just because you can launch it the software. The entire company needs to be on board.
> First, you have to get clear on what the company is looking to get out of the project.
You are not just pleasing the executives. Assuming the company described in the article is a good company. Executives are not idiot that making impulsive decisions.
You need to be able to sense the company direction, communicate clearly, manage the risk, build the trust. This is not an easy task in a big company.
Such an awesome article IMO that explains the art of shipping in a big firm. I believe if any engs can master the skills described in the article, you will get promoted to a leadership role very fast. Of course, this might not be the path everyone loves.
There’s a really interesting article that I’d like to read about the “first” - ongoing stakeholder management through the lifecycle of the project, the product and your time there all allow you to ship effectively (as far as the org sees it at least).
Focusing on the tech and even product UX minutiae doesn’t typically get the headspace you need from senior management for rolling down tech debt, the headcountb you need to really ship useful and market leading software.
Yet, that's how you sustain your job, get promoted and have more responsibility over time – large number of people don't seem to get this simple fact that your customer is whoever pays your salary and it's your job to make them happy.
It could be as well called “doing your job”. “Shipping” your job/work doesn’t seem like a stretch - especially that definition is clearly stated in the article. It’s also nice play of words that emphasises the fact that many people seem to get the rules of this game wrong.
The article also made it abundantly clear that if you find yourself in that situation, and you don't like it, you should find another job.
The article isn't about the ethics of the situation, career choices, ect. The rest of the article focuses on more important details about shipping software.
Give the article another chance. I skimmed it twice yesterday and now that I'm not reading my phone on the throne, I gave it the attention it deserved.
---
I should also point out that I've had a lot of career success in situations like this by disagreeing and committing. Turns out that a mild degree of patience in situations like this often means I get to focus on very clear customer pain points and build industry-leading software. (I just don't get to focus on them the exact moment I want to.)
There used to be something called beta testers. You could actually see if users hated your software before releasing. For a small subset of users, of course.
There also used to be alpha testers, i.e. your own staff. There's another subset of users and one that probably still can tell you if they hate your software today.
Just the use of the word "shipped" implies that software is an interchangeable commodity that rolls off a conveyor belt in a factory.
That's not the reality with any software that actually matters, no matter how hard M.B.A.s try to abstract the human factor out of it.
Good software isn't manufactured. It's created. It takes thinking, and planning, and crafting, and all those things that are the opposite of a plastic widget factory that "ships" products to its customers.
Don't let then dehumanize the profession any more than it already is.
Most commercially written software is closer to manufacturing than art. That doesn't satisfy everyone, I know, and I'm not here to judge people who don't want to work on integration #5 for widget #3 in order to unblock a million dollar contract from Foobar Inc. But that's where a substantial majority of the market demand for software development lies. The MBAs are paid precisely for their ability to channel the creative process of software development into the manufacturing slots where it's needed.
(Even art, of course, is in practice pretty close to this. MBAs are paying you to sublimate your creative energies into a snappy video on how to sign up for Robinhood or whatever, and there are substantial business constraints on both its content and delivery date.)
It's closer to engineering than either art or working on an assembly line. Software tasks just aren't fungible in most companies, and neither are they open-ended interpretable works designed to please or strike fear into the human soul. The average codebase is pleasing to me in the way an engine block or an oil refinery is pleasing. Q_rsqrt is pleasing in the way a mathematical proof is pleasing.
I’ve worked in engineering businesses, and SaaS scale ups and the “engineering” that gets done by SWEs had little to nothing in common with any of the engineering disciplines I’ve worked in apart from the E in the title.
Little comprehension about cost engineering, maintenance, safety, durability and resilience. Half-baked bodies of knowledge. CV driven development. Fads and critical production systems held together with spit, tape and hope. It’s like Aristotle’s Cave in our field.
When you ship it is arts and crafts which usually leans heavily to a small group of people Then if you want it to be commercial succesfully it needs some manufacturing practices because otherwise, you end up with legacy software.
I don't know why this post is getting down voted but the MBAization is truly it.
MBA practices cannot deal with uncontrollable production. But software engineering is utter chaos. So they try to come up with bullshit units that can be easily managed in their opinion. But it never works - aka, a nimble competitor always eats the giant in software.
The MBA style practice does work in factories and warehouses. But in software, a nimble startup with own the incumbent just be providing higher quality value.
Maybe the author didn't feel the need to state the obvious - that once the executives like and understand what's being shipped, then they can confidently pull the trigger on expensive processes like marketing. Otherwise the feature or product could sit hidden and forgotten.
If the customer isn't using the software, it hasn't been shipped to them.
I think a better definition of shipping is ensuring your project meets the definition of done. At big companies, for many projects, I think there's an implicit requirement in the definition of done that management is aware of and somewhat happy with what you shipped. The resources used to ship a project (the time, money, and people) are, after all, their resources.
I feel like you're getting at the fact that you can build a crappy product that management loves and build something customers love that management hates. I don't disagree with that at all, but to me that's kind of tangential to whether you've shipped a project or not. To me at least, shipping isn't about it being good or bad, it's just about it being done. I think what the article is getting at is that if management doesn't agree that it's done, then it's not really done.
Well, of course. Because you can only ship the software your management wants. If you want to ship a software users like, go to a company that cares about it's users. (Or turn your company into one, but you cannot do that by shipping software your management doesn't like).
Even worse, it misses the point.
Shipping means you realize benefits for a target audience.
Enable someone to hit a revenue target, compliance measure, etc. if leadership doesn't care about that or can't work to that kind of framework; they deserve to go the fuck out of business.
Treating it as "please management" is incredibly self serving. You please management by often making them money; sometimes despite themselves.
Almost all of us here has worked on a ticket that made us say "The fuck is this shit im working on, who will use this? Who the fuck came up with this?"
And we just work on it, take the paycheck, and move on to the next one lol.
Yeah, if you spent your life like this you kind of wasted it. I’ve noticed that HN has fewer and fewer hackers day after day. Not sure when they will change the name.
The number of "hackers" who think it's better for devices to be locked down by Apple because that keeps them secure (and freedom to install unvetted software is dangerous), and who think AI will replace us because we are all "statistical parrots" and just regurgitate what we read somewhere anyway is really astounding.
Also lots of "hackers" who measure success by how much money has been made, be it by a company or person.
I mean it may not be impossible to be a hacker and believe those things, but it should at least be a rare thing.
It's about the customer recognizing the product as something they are ready to buy, which is associated with the dictionary definition for shipped "(of a product) be made available for purchase."
It falls apart slightly in that the customer technically starts buying the product on day one, while it is still just a glimmer in someone's eye, but "shipping" referring to the point where the customer says "Yes, that is what I wanted" is close enough to stay within the intent of recognizing something available for purchase methinks.
> If you’ve given users the best software ever
Of course, in context, the users aren't your customer. They may be someone else's customer, but that wouldn't be shipped by your hand. Your delivery is limited to your customer.
Going back more than a quarter century ago (!), I was one of those zealots in favor of checked exceptions in Java. It took a good 15 years to finally conclude the “checked” part is more trouble than it’s worth.
Given that there is inevitably a root context for any given language invocation (main() for a command li program, top of a thread for a multi threaded app, etc), unchecked exceptions and a top level handler have proven to be more than enough.
I won’t comment on Python backwards compat, will just shake my head.
What you cherish on your death bed is often enabled by plain old hard work. You may not love the work, but for most of it is the path to those happy death bed memories().
Skip the work, and for most of us it will be a short, miserable life scrabbling for food and shelter.
() The big asterisk of course is tech salaries are completely out of whack with effort and complexity. There are a lot of us out here who get enormous salaries for doing comparatively little in the grand scheme of things. These lucky folks are skewing the narrative for the rest of the world.
For me, I will always cherish memories of vacations, my son’s first varsity touch down, my daughter’s vocal solo for a Christmas show, hanging out with friends in the woods with little more than tents, firewood and beer. I will also be quite honest that it’s been enabled by a lot of luck at work and the incredibly high salary I earn in software - which is less than half of the FAANG salaries I see mentioned here.
Life isn’t about living in the now. Or in the tomorrow. Or ignoring the long term, or short term, or whatever term.
Life for most of us is finding balance that works in our situation,
If I had a nickel for all the clients I’ve seen with micro services everywhere, and 90% of the code is replicating an RDBMS with hand coded in memory joins.
What could have been a simple SQL query in a sane architecture becomes N REST calls (possibly nested with others downstream) and manually stitching together results.
And that is just the read only case. As the author notes updates add another couple of levels of horror.
I think the overall thesis here is accurate. I can fill in the blanks where Fowler is light on anecdotal evidence. I have done many architecture reviews and private equity due diligence reviews of startup systems.
Nearly all the micro services based designs were terrible. A common theme was a one or two scrum team dev cohort building out dozens of
Micro services and matching databases. Nearly all of them had horrible performance, throughout and latency.
The monolith based systems were as a rule an order of magnitude better.
Especially where teams have found you don’t have to deploy a monolith just one way. You can pick and choose what endpoints to expose in various processes, all based on the same monolithic code base.
Someday the Internet will figure out Microservices were always a niche architecture and should generally be avoided until you prove you need it. Most of the time all you’re doing is forcing app developers to do poorly what databases and other infrastructure are optimized to do well.
> If it was optional, they would whither away over time.
In France unions are optional (in the sense of adhering to one, their presence is highly regulated and a company cannot do anything against that). They are quite strong (depending on the industry), and representatives are elected.
If a union does not do anything then it indeed whither away.
Now - in our case, some situations are a parody (especially in some places like positions in the administration, the famous fonctionnaires where everybody is on the street when the post office is required to be open outside the times "10:30 to 12:30, except on Mondays", for which their ancestors fought :))
We have a love-hate relationship with unions (which of the love or hate depends on whether you need to take a train and you forgot that there are only 126 days in the year when they are not on strike :))
In the US, labor unions are required by federal law to negotiate pay and benefits on behalf of all employees at a particular workplace, rather than just their own members.
Because of this requirement, some people argue in favor of permitting mandatory union membership, on the basis that without it, individual workers can free-ride, enjoying the superior pay and benefits the union negotiates on their behalf without joining and paying union dues.
Others argue that forcing people to join a union and pay dues is illiberal, and should be outlawed.
This is the crux of the "right-to-work law" debate, where "right-to-work" describes laws that make it illegal to require union membership as a condition of employment.
Personally, I find it odd that federal law requires labor unions to negotiate on behalf of non-members, but I am also not particularly well informed on the topic.
In France all unions negotiate joinly for all employees. It is strictly forbidden for an employer to have different treatment of an employee depending on whether they are members of a union. Membership is not public anyway.
About 10% of the workforce is unionized and only a few percent of the unions funds come from the members. The rest is from companies (mandatory fund), the region, ...
Unions encode the rather insane idea that if you're stuck negotiating against a monopoly then you should become a labor monopoly yourself. It's stepping backwards for short term comfort.
Historically, here in the US, things like 40 hour weeks, two-day weekends, overtime pay, safety protections, etc. came about because of union actions and the political climate they created.
English allows your "must be" to be interpreted two common ways.
1. A comment about how in general there should be a way to do that, but not engaging in the topic, ceding all power to the corporations which would be glad to take it, giving none back, and just mildly sneering at anyone foolish enough to join such a basal endeavor as a union.
2. A comment about how we would need to accomplish that goal and that establishing the working process should come before unionizing as it is the proper solution to the problem, ceding all power to the corporations which would be glad to take it until we solved the problem that is effectively created by corporations to be unsolveable.
Collective action works, it solves problems, it gives workers space to live and have basic rights, and we don't have to throw up our hands and say "there must be a better way!"
Hmmm...can you elaborate? Because I've always thought the "right to work" (ie, not be forced to join a union in a unionized industry) is an example of sacrificing short term comfort. In that case, weakening the union provides a short-term benefit (no paying dues) at the cost of long-term benefit (higher wages, better benefits, improved safety etc.).
I'm trying to find a generous interpretation of your comment, but it comes across as one of those overly libertarian theoretical perspectives that tends to break down in practice.
If the labor market functioned correctly you wouldn't need unions to enforce any of this, which is obvious, because these benefits exist in almost every non-unionized industry as well.
The only reason to engage with a union is because the employer is so large they not only monopolize the consumer market but the labor market as well. The correct solution is to break up the monopoly.
Labor markets are difficult. If low wages go down further, people will offer more Labor, not less, and your microeconomic equilibrium goes out of the window very quickly. Also, while demand for work is not monopolistic per se, the employer has usually a structural advantage when closing a contract with a single employee, and in some industries only a few very big employers compete on the labor market.
If every employer would be allowed to hire exactly one worker, then you could probably do without unions, but that would yield a totally inefficient economic system I wouldn’t want to live in. And therefore modern economies allow unions as an exception from the rule. Otherwise, at least in some trades, wages will go down to the absolute bare minimum, which will lead to political instability, which leads to less economic activity.
See my previous comment about theory breaking down in practice. There are all kinds of asymmetries that make economies not function correctly in a theoretical sense and we (as a society) develop policies to mitigate those dysfunctions.
I don't know that the scales operate in ways that allow for every corporate (pseudo)monopoly to be broken up and still have viable industries. Especially in countries that succumb to regulatory capture. That means there would need to be some countervailing balance to that power, and that's where unions come in.
I started training for recreational long distance running in my late 40s. This is a good list for any age. What I would add in is - find training programs and follow them. The original article obliquely references this in #1 talking about "acknowledging reality is critical". At any age, people need to learn how to work, out and how to train. Failing to do so can cause short- or -long term injury, which can really set you back. My first attempt at a marathon lead to IT Band issues and severe plantar fasciitis. My second was much smoother, and I was able to complete the marathon at better than target time without injury.
The moral is - part of listening to your body is learning how to train within your age / ability band.
Older runner as well and have also had to take sporadic time off to deal with plantar fascitis, IT band, achilles, etc. Agree with listening to your body and what I'm hearing is slow down to stretch out my total running lifespan/days. I'd rather run 5K every other day on gravel/dirt paths indefinitely than risk some really long term injury running 26 miles on asphalt (seen others have to hang up their running shoes after doing the big races).
I feel like this is the end game of scrum and most agile methodologies - endless refactoring on a treadmill with no off button,
I like to be introspective, and I am human so my code is far from perfect. But if I was refactoring half of my time I would go more than a little crazy.
The good systems I have worked on have converged on designs that work for that space. Both developers and users see and value the stability.
The bad ones have had the kind of churn the article mentions. Developers are constantly rewriting, functionality is subtly changing all the time; stability doesn’t exist.
reply