Hacker News new | past | comments | ask | show | jobs | submit login
Engineering Principles and Values (zenpayroll.com)
112 points by edawerd on June 15, 2015 | hide | past | favorite | 62 comments

Great stuff, looks like you've come a long way.

I'd caution against the term "over-engineering." It is a straw man[1] — no one is pro-over-engineering — so it is only ever used to discourage long-term thought. It's a good thing to try and avoid, but maybe pick something neutral like "predicting the future."

Hacking, on the other hand, is sometimes advocated for (e.g. Facebook) and can be good or bad depending on circumstances.

[1]: http://seneca.systems/values#what-constitutes-a-value

I have quite strong feelings on the subject of over-engineering.

I've had to argue against over-engineering many times in my career. I wasn't about discouraging thinking. I was fighting against actually implementing specific features that attempted to solve poorly-defined problems whose details had not yet come into focus. Developing those features would significantly impair other parts of the project.

For example, the problem might be "What if our database becomes overloaded one day?" Choosing to implement, say, sharding, could lead to an over-engineered system which gets in the way of solving the many different future performance problems that could crop up.

I think it's quite reasonable to use the term "over-engineering" in this sort of situation.

I definitely agree that over-engineering is a problem. My feedback was meant to be about how to communicate that balance.

In your example, if you tell the other person that their database sharding is over-engineering, they can say one of two things:

a) No it isn't. b) Yes it is and I think over-engineering is a good thing.

I have never heard someone say (b); if they thought it was over-engineering then they wouldn't have advocated for it. The negative quality is part of the definition.

What you're likely disagreeing on is how far in the future you're team should be thinking or how much data is necessary to make a confident decision.

I don't think it was your intent to stop long-term thinking, but if someone says "we shouldn't over-engineering things", I don't think that solves any ambiguity because no one was intentionally over-engineering before.

It's akin to saying "don't do bad things." People never do bad things, only things that are bad under a different set of principles or based on other values.

My feedback is that this document should define what over-engineering means in their context (since it, like what is "bad", is context-specific) and lay out what engineers should look out for.

Thanks for explaining, I see what you mean now!

If you agree it's a problem, I'm not sure why you think it shouldn't be talked about. Your (a) and (b) responses would come from an engineer who is not open to dialog, and I'd say from an engineer I'd prefer not to work with. I would add another option to your set of possible responses: (c) Let me now think harder about whether we really need database sharding and justify why I think it's needed, and allow this further analysis to possibly lead to a revised, simpler design. That's the route that the open-minded, discursive engineer will take.

I think the problem with a loaded term like "over-engineering" is that you are assuming they haven't fully through their idea through, and that's why they don't see things your way. I've seen much better results come from explaining why you think database sharding isn't necessary yet (even if you think the reasons seem obvious), and then asking your colleague to help you understand what you might be missing.

That's a reasonable perspective but the assumption I would be making isn't that they haven't thought it through, it's that they're using a higher bar for what's necessary, and are thinking it through from that point instead. The dialog is then about coming to a shared understanding of what the actual requirements really are. Therefore it's not critical or something to be taken personally about one's proposed implementation, it's the symptom of unclear requirements and the need to clarify them.

> Therefore it's not critical or something to be taken personally about one's proposed implementation, it's the symptom of unclear requirements and the need to clarify them.

Absolutely, and that's why I agree that the term Overengineering is an unhelpful shorthand that implies criticism to an unnecessary extent.

I'd caution against using the term straw man. I don't think the audience is ignorant but I think stating that over-engineering is frowned upon is a good thing in my mind. I have worked in several organisations where bright up and coming developers try to raise their profile by being "clever" and as a result there is always over-engineering. I agree no one is pro over-engineering but as you point out yourself it's sometimes something that isn't acknowledged because either the engineer doesn't consider they are over-engineering or are prepared to defend their work.

Stating that you are against over-engineering makes it explicit and as a value it is something you expect your engineers to think about. "No over-engineering" should sit at the heart of any good set of coding practices and it should be there because it means engineers have to think about and understand what it means.

"there are two ways of constructing a software design: one way is to make it so simple that there are obviously no deficiencies; the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - Tony Hoare

I think it's important to make a distinction between over engineering "features" and "properties".

I completely agree that over engineering that takes the form of "this feature will be cool, lets build it" without validation or demonstrated need from the market is generally a bad idea.

However taking time to understand the fundamentals of a system and build a robust core on strong principles is almost always a good idea.

> "over-engineering" [...] is only ever used to discourage long-term thought

Not exactly.

The term is used to discourage long-term thought that almost certainly won't pay off. As such, it usually will cause more trouble in the long term. So it's not just a failed investment with a missing long-term win, but an "investment" that will cause a long-term loss.

Over-engineering usually happens when people either over-estimate their ability to predict the future. Or, when they just learned a bunch of exciting technologies or patterns, and drift into some kind of hammer/nail thinking.

>no one is pro-over-engineering

It is when you hand them a blank check and allow more engineering ot justify writing down a bigger number.

A huge missing part of the "Employees" section — which is great as a focus, by the way — is specifying how your management works.

Especially in engineering organizations, a consistent problem is moving people into management positions based on their skill as an individual contributor or tenure. Both are orthogonal to people-management skills and, though good to have from a team respect standpoint, far less useful.

Is it a priority of yours to avoid those problems? If so, it would be great to see it listed out. How do you organize teams to be optimized for Employees?

Especially as you grow, I would imagine managing people towards their goals and those of the company is going to be a larger and larger part of that employee happiness.

edit: There is also nothing on employee growth. If that is a something you value, it would be cool to see it in there.

Most places I've worked had no career path above Senior Software Engineer that did not involve moving into management, which as you point out, is not necessarily a skills match with "that smart individual contributor who you'd like to reward".

I've heard some larger companies "solve" this by providing some title advancement past "3rd Senior Engineer from the left". But is it satisfying to you folks on these tracks? When you go from Senior Software Engineer to Staff Software Engineer to Super Staff Software Engineer to Super Duper Staff Software Engineer, do you consider it to be real career growth? Do you find that this is truly a _parallel_ path with management, or does it eventually plateau (i.e. the management path ultimately ends up reaching farther)?

At the teams I have managed at larger companies (Stamps.com, TrueCar) the technical track had official equivalents to management roles (Sr. Engineer -> Manager, Architect -> Director, Sr. Architect -> VP) for determining compensation, equity and any other role defined "perks."

Each of those roles also came with leadership expectations: Architects were responsible for the overall technical health of the systems they were responsible for, with Sr. Architects being responsible for multiple. They were empowered to reject system changes and were expected to review code and designs on a regular basis. Really good engineers that weren't system thinkers or didn't want responsibility for other people's code, could increase in compensation but weren't considered part of the leadership team.

What does it matter? If you're having real impact on the decisions you care about, and your pay goes up who cares about "career growth"? What is a career these days? There's no loyalty, no gold watch coming.

The titles and the career track matter because that's how you get to have real impact and that's how your pay goes up. If you don't want to be thrown a spec and told to just implement it we need more developers who have better job titles and who can play the political game well.

If you're a dev you're more likely to have to bow to pressure from marketing or other departments and won't have any real impact.

While the idea of separate IC track looks like common sense, I don't think it plays well with traditional top-down structure (i.e. the structure of every single company out there). In top-down structure "resources" are allocated to "projects". The nature of the "projects" and details of allocation are carved out by "management". If you work as an IC on somebody else's project then he is your boss and you can't get much freedom beyond freedom in technical matters (usually Senior Developer title already implies this). If you work on the project of your own choosing that means you are a manager with really constrained resources and either you get more resources (usually it starts with a junior developer to help you with chores) and gradually become a real manager or your project risks to slide into irrelevance as you are outmaneuvered by real managers with big headcounts and budgets.

What I would like to see is a working scheme for more fluid resource allocation scheme than fixed tree-like structure. This holacracy thing promises exactly that but I am not sure if it is really viable.

Is it literally just the titles that you're talking about? I don't see why companies couldn't max out the title with "Senior Software Engineer" but continue giving rewards like more money, vacation, creative leeway, etc.

I've heard (second hand) that VMWare has a good parallel track.

That said, it makes sense for a company in which individual contributions can revolutionize the business.

There are cases where being an individual contributor is only so valuable (though that "so valuable" number may be very high.) There comes a point where there is only so much impact you can have on your own, which is why _great_ managers absolutely deserve higher compensation.

It's hard to individually make a larger contribution than a manager making every one of their 5 teammates 10% better.

So to answer your question, I believe it plateaus. Where it plateaus is relative to how much individuals can impact the course of the business.

At Boeing (and the other major aerospace companies), they have 5 levels of engineering, then the Technical Fellow track with 3 levels. Level four requires you be a team lead, and level 5 even higher level of responsibility. That was a more recent requirement, and we were told they corresponded with manager and senior manager, which slowed down promotions significantly.

The Tech Fellow track was created to grant executive level compensation to retain subject matter experts. Beyond a graduate degree, becoming a Tech Fellow requires demonstrating your subject matter expertise within the company, with a key measure being that other groups seek out your help, so it does require some politics and networking. At the highest level, you pretty much have to be a national authority. A lot of work, but executive compensation at a megacorp is significant.

That said, unless your specialty is avionics or CFD, I don't recommend aerospace companies for software engineers.

Outside perhaps 'architect', what does a reasonable progression look like? I struggle with this personally so very curious what others think.

While Management usually the only thing equated with "soft" (hate that word beeteedubs) skills, moving up the IC ladder also requires good inter-personal skills.

Above senior you are allowed and expected to go "heads down" (by yourself or a small group) and knock out solutions to technical problems. However you are also expected to identify cross-cutting problems, rally the right people, coherently frame your arguments, etc to move the "implementing solutions to technical problems" needle (even if you don't directly write a lot of that code).

Management also get's the "fun" things like "let's talk salary!" or "you need to improve along X, Y, Z axis!" :P

Just my two cents :)

Oh, I agree with everything you just said. I was just wondering what it looks like from a career progression standpoint. If it's just titles, it's tough.

What you said about those cross-cutting problems and all that is spot on and that is basically what I see sr. engineers and architects. However, how do we or should we even further delineate career progression and responsibilities? And what does that look like? What is beyond those things?

I don't know :) titles are inherently political. They don't exist in some platonic vacuum. So some part of the progression is about broadcasting the status change to the larger organization. Beyond that I think the ladder becomes about company wide cross-cutting concerns, enough success to be an internal rallying flag. Things like that.

But I've never been in the position to either be promoted or do the promoting to that level :)

What' I've encountered:

Senior S/W Engineer -> Tech Lead/Specialist -> Architect -> Senior Architect -> CTO/VP-Technology

It's definitely a priority here, and more so as we continue to grow. This particular engineering value was intentionally left a little vague because prioritizing the people in the company can mean so many different things, and we wanted to purposefully leave it a little to interpretation and our best judgement.

It sounds like the purpose of these values is to provide guidance. From your piece:

> It’s a framework that we can turn to whenever we’re faced with a difficult decision, whether it be about how to best architect something, how to prioritize our time, who to hire, or how we interact with each other.

Why would you then leave a growing priority vague and up to interpretation? Isn't that exactly the type of miscommunication and lack of guidance you're trying to avoid?

P.S. I do think the exercise is great and don't want to sound unnecessarily critical. Especially for such an important part of companies (people) and common issue in engineering organizations (management), though, it seems like one of the first places you should put a stake in the ground in terms of action, guidance, and specific commitments.

Yet another team that seems to think the programmers should QA their own code. I learned that lesson 30 years ago, good QA people are worth their weight in gold. QA is a set of skills separate from the skills that programmers have; expecting a programmer to be good at both is asking for trouble. I never would hire an architect to wield a hammer and saw. You can't be good at everything.

I used to ask in interviews if they had a QA team, because it was on the Joel on Software list http://www.joelonsoftware.com/articles/fog0000000043.html

I stopped asking because the answers were always some variant on "that encourages programmers to be lazy, because they know QA will be there to catch it" and because it seemed like lot of companies treated that question as a culture fit red flag. The responses were pretty hostile, like I'd asked why none of the engineers were wearing suits.

I have no idea whether it's a good idea, but the current tech culture seems pretty opposed to it.

I once ran a small team with a specific QA guy, he was awesome. He had more technical knowledge of the product than the remainder of the team (3 other programmers) combined.

As a new manager, he was who I leaned on the most to find out things, and who saved my ass on a number of occasions.

He deserved more respect than he got.

If the expectations around QA aren't clear and you have a bad manager they'll actually force you to rely on QA since QA should be vetting things before they get to the customer. I've been told by one manager that I should not waste time writing unit tests and that I should rely on the QA person to test.

If the QA person has skills along the lines of a software test engineer (like at Google and other places) and can actually create tools and automate things and write some unit tests or acceptance tests, then they're half-a-programmer and I'll be happy to rely on them and share the work. Otherwise I assume the QA is there just to bow to management/marketing and give the OK for deploys.

Because having a QA team allows for a very nasty blame game. It larger organizations it can get pretty nasty. The problem only grows as greater amounts of bureaucracy separates the two sides.

ZenPayroll CTO here. At least for the stage that we're currently in, having high unit test coverage, high-level integration tests, a strong code-review process, and individual accountability for bugs has worked very well for us so far. So well, in fact, that we don't feel we need a dedicated QA team.

It's also worth mentioning that when coming up with our values/principles, we started with the understanding they can and probably will change over time, especially as we grow. So while we don't have dedicated QA team today, we'll keep an open mind do whatever works best for us in the future.

> Yet another team that seems to think the programmers should QA their own code.

That phrasing is a bit different than what I think they meant. Programmers familiar with the problem domain should QA the code of other programmers is how I took it.

I've never seen the value in "pure" QA people largely because you need to be a developer to be an effective QA person [writing automated, repeatable tests] at which point you are just looking to use the developers that are more meticulous than the others to write those tests.

They are still ultimately software developers / programmers / whatever. Its just the problem domain they spend half their time on is QA instead of Ops or whatever else.

Maybe I'm just used to being part of an IT department embedded in a larger company where we tend to mix a couple things in a single position. :P

I've always argued that a QA engineer's job is to augment the skill set of the feature engineers to help them in writing good tests and automation suites that ensure the user requirements are actually met, as well as to build and maintain the CI server, QA and staging environments, and so on. Basically QA as a support for the engineers, not in contrast/conflict with engineers. Note that I said "QA engineer" there; manual QA is a different role entirely that should occur outside of engineering frameworks, whose goal is to really evaluate UX and who really answers to and works with product and design teams, not engineering. /2c

The tests+code review make sure people are doing basic due diligence on the code they're contributing. Meanwhile, SRE roles are defined to take up a bunch of the brass tacks of making software storm-worthy... Are there aspects of good QA that you feel aren't hit by this combination? (Genuinely curious.)

I applaud such a clear document; this is how company culture should be defined. When people make decisions, at any level, they should be able to reference a document like this to ensure it is within alignment and is proper.

But these are engineering principals, not company principals. And "employees" is last, not first. Nor is it particularly clear in what sense employees matter. Saying they matter does not describe in what way, it's a blanket statement. Do they matter that you will pay them at or above market rate? that you will help them with training and certifications? that you'll take care of them when times get tough? You have to spell it out. Really spell it out. Otherwise, it's too ambiguous and it'll get swept under the rug.

each page is 75% art, 25% text. I think this ratio needs to be flipped.

Still, this is something. I don't think it's "done" but it's more than most have.

I'll write more about this in my next post, but we started by going on an engineering retreat and came up with a pretty massive document. The hardest part was distilling all the text down to a point that was easily consumable. I actually feel that the amount of text is just about right.

I agree and think you did an excellent job here. Distilling down such critical information shows how much time and thought you put into it.

It's easy to ramble on for 100 pages, but capturing the essence of your values in a few slides with a couple of tidy examples each is really hard.

sorry, but... really? these are bullet points. I could come up with these points in literally 10 minutes. They aren't THAT original or revolutionary. What matter are the details; the push come to shove stuff.

Maybe this is good for a TL;DR, but if you don't spell out the details you leave room for ambiguity. Details help everyone think the same and gives them all the same reference points. They all have examples they can pull from already built in, so they are more likely to make the preferred decision.

My first reaction to this... the document says "We are owners of each and every feature we build", but who is "we"? If I write a feature, am I expected to own it throughout the lifecycle?

It's easy to hide behind "We", and easy to say "we isn't me". I might suggest something like "I own everything I build and everything my teammates build". That eliminates a whole class of avoidance and excuses for bad code.

Overall it's a nice distillation of general corporate engineering teamspeak but I didn't grasp anything new or enlightening. This document would be pretty easy to ignore if I were in that group.

What we meant by "we" is that the individual engineer who wrote the code is ultimaetly responsible for it. From building/modifying the backend APIs, frontend code, testing, QA, and ongoing maintenance/bug fixes.

I guess "we" really means "you"

That will never work because economic returns to a piece of code will not go to the engineer. Both management and company owners will want most of the returns of the code, leaving no profit motive for the engineer to be ultimately responsible for it. The engineer will move on and the owners and management are stuck with rotting code. Or management gets rid of all the engineers because they want to keep all the revenue and discount both cost of maintenance and need for engineers to profit.

I believe you described Marx's theory of alienation[1] as applied to software engineering.


Yeah, which is why companies should stop going on about 'owning' things. If I'm going to own its creation and operation, I'm going to own all the revenue it produces. All of it. I have all the means of production (no 19th century power looms needed to create software) and all I really have to do is survive while it is being produced.

This is actually what Philip Greenspun did at ArsDigita. Each project had its manager/lead own the profit-and-loss:

From Founders At Work:

"I'm organizing this company like McDonald's. Each restaurant is going to be managed by a few people, and they're going to have profit-and-loss responsibility. If they make a profit, they get to pocket half of it. If they make a loss, we're going to know who's responsible, and we're going to go there and fix it, and there are going to be consequences for those people." People have all the right incentives to make their customer happy, to do the thing on time, to take the customer's money, deposit in the bank, and then move on to the next one and get their bonus at the end of the year.

And apparently at one time Accenture had this philosophy too:

At the time Anderson Consulting (now Accenture) didn't have any salespeople. They always had the people who were executing the project sell it. "You eat what you kill" was the phrase at Accenture. You don't have a salesperson go out and tell the customer, "We can do this", making promises and then handing it off to a programmer"

What about sales, marketing, management, QA and other staff? What is their responsibility?

Probably outside of the scope of this document.

Can you imagine if a Sales team came up with a set of values and part of that document specified how engineers should do their jobs? There would be revolt!

I completely agree with you. Also to me accountability goes to whomever is currently working on the code and whomever manages that individual.

I think what they meant was that when you code something, you have to treat your work like you completely own it so you do a good job but to me that's self explanatory and is only repeated to enforce rules that are obvious.

> I didn't grasp anything new or enlightening

Repeating the "obvious" is very important. There is a constant fight against entropy, miscommunication and attrition of experience in any team, and things like these add up in a huge way. The more years I spend in the field, the more I realize the need to "over-compensate" affirming and syncing up on stuff like this.

And aggregating stuff like this into a list is tougher than it seems. This is one of the most balanced values summaries I've seen (for my subjective taste!) - not being too vague, but also not "hard-coding" specific methodologies or buzzwords.

Quick question for the ZenPayroll folks lurking here:

How do you handle situations where there's a lack or breach of trust? If an engineer comes to you saying "After $INCIDENT, I find it hard to trust $COWORKER", what do you do?

Trust is very important & I'm glad to see you've given it an entire slide. I particularly like "There’s little benefit to setting hard deadlines if you know everyone is doing their best".

That has definitely happened here. And I've personally had my fair share of breaching people's trust. We're human. It happens.

There's no magical solution to fix it. I think the best thing to do is acknowledge it to the person you've violated and tell them you're going to do your best to earn their trust back over time (and actually do it).

I always like these types of books not only for the information but for the idea that they will bring a team together around a joint identity. Instead of people feeling like they gotta ask Alice or Bob about the right thing to do, they can refer to the book and be guided toward it.

It would be helpful to have an indication on these types of career sections on what a companies stance is on remote employees. Many support it but never give hints, many don't and keep quiet. Would be nice to get that out in the open as a company and for specific positions.

I like this a lot having a set of values that are created and owned by an engineering team can only serve to improve software quality. I'd be interested to know if these principles are something you expect to evolve and how you plan to manage that evolution?

Trust, Competence, and Autonomy. This is not specific to software engineering. If you have one of those missing then there will be dysfunction.

no critique of the content here, but that slide deck is like a case study in how not to slide deck. no useful graphics, all the information content crammed into a tiny sidebar with squished text.

its an essay in slide deck form. why make it a deck? just write it as an essay.

engineering good. abstraction bad. my rough rule of thumb.

I know that's too high level of a description. but what I'm saying is... that you always pay the price for poor quality, rushed work, whether you pay that price in an hour, tomorrow, next week, you always do, somebody does. so better to get quality right.

however, I see lots of astronaut architecture and too-much-abstraction -- especially in all-Java/enterprise shops -- where, I see folks spending lots of time on things which objectively don't have any business value and don't objectively increase quality (reduced failure rate, reduced bug rate, etc.) instead arguably seem to increase it, or increase the complexity and quantity of moving-parts which can fail or otherwise bite people later. Quality good, quantity bad.

Simple-but-perfect is better than bigger-but-shoddy.

It's a good idea but the whole document comes off as really patronising. I mean - do good engineering teams need to be told to write tests and code review each others work? Do good engineers need to be told that the customer pays their wages?

Good engineers are always focused on delivering value to the customer, and good engineers will always select stable systems rather than be swayed in some kind of hormonal feature porn frenzy.

On top of that, in my experience, humility is not something you have to ask for in an engineer. He or she will be painfully aware of every mistake they make, it's the only thing driving their development.

tl;dr if the head of accounting thought you were a rowdy bunch of code monkeys, then read the agile principles you'd end up with this

We're all always on our way to becoming good engineers. Most of us really aren't good engineers in one way or another. These kind of principles help guide the whole team towards a set of high standards, in which individual team members may be more or less deficient.

Do you need to be so negative? Even if everyone agreed that these principles were gold, and I'm sure I know people who don't agree with everything here, it would still be more than worthwhile to say "this is what we think being a good engineer means." I honestly don't know why you would view that document with such disdain.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact