Hacker News new | past | comments | ask | show | jobs | submit login
Why Remote Engineering Is So Difficult (learningbyshipping.com)
74 points by go37pi on Dec 31, 2014 | hide | past | web | favorite | 40 comments

It's easy to say that communication is key, because it is, but all else being equal, if your team can't execute projects in a distributed, remote environment as well as it can in a centralized place, then it's not as good at communicating as you think it is. Dragging a failing remote team into an office isn't going to magically make them successful, and scattering a successful team across the world isn't going to suddenly make the project fail.

I personally find the very disconnected nature of the teams I lead in my consulting work actually forces us to communicate in an open, concise, discoverable way that would be good for any type of team, but that co-located teams get to cheat out on because a weak team member can always ride his stronger teammates' organizational coat tails with nosey interruptions and annoyances.

We have to think things through and write clear specs, because we can't use body language to clarify and count on the people involved in the meeting to just remember the attitudes expressed. We have to be good about change management, because we can't just turn around and ask someone to "get out of a file" or do whatever we want to the database whenever we want to. We have to keep meetings to a minimum, because you just can't have a productive, two-hour phone call.

These are all good things for non-remote teams, too. But they cheat and don't follow good practice.

This and the top comment nails it down that if your team is not delivering the stuff in remote model then they most probably wont deliver in collocated space as well. I am not very sure why people still refer to year 2000 time of offshoring difficulties till now. For last 14 years I have been in working in the offshore(software and infra outsourcing) model. And I can vouch for the model to be very workable while it will have some extra work for either USA partner or for India partner (Mostly on Indian side as they may not say no even if USA side offer to take extra efforts). In my opinion the following has made the life much better for me and hopefully for my partners.

1. Have the key folks (leads/project managers) travel and stay with team for at least 4 weeks in a common place. 2. Do not force video (leave the mode of comms to peoples preference (video / email / call)) 3. Allow decision to be made over text chat and have it added to project decisions in weekly meetings 4. Identify couple strong people per 15 members team and have relationship where they will tell you before deciding like 6 months in advance and allow people to leave. They will come back later. And wholeheartedly develop them even if you know they will leave, with one condition that they will develop one more person like them for the team. 5. Have good leader in India who can speak open with out fear of a westerner (I find this a most difficult one). This is a real deal maker between peace of mind and micro managing and going crazy. 6. Be open t listen to same idea that your team thought and decided to ditch from your offshore partner. (we can go for full blog post on that)

I think this would take the team long way. Please remember either USA side or India will have to lose 3 evenings a week, irrespective of SRS, Comm tools and so on if you need a successful delivery.

EDIT ADD: There is no additional issues in managing remote worker than it is with regular in office . It is just kind of issue, it is definitely manageable.

My experience is that engineers often:

    * hate meetings
    * prefer slack/hipchat/etc even if seated adjacently (myself included)
    * work asynchronously
So the main friction is in the interface between engineering and other disciplines. Remote works well so long as non-technical leadership has a local lead they can interface with.

The communication skills of other non-engineering team members can often suck in the digital domain. Empathy by text message can be hard to have. A combination of on-site and remote often works best. Learning the language of other domains can take a long time. The role of people who interface and translate between domains is important.

I totally agree with this statement.

> My job is in sales and I am teaming up with "sales" engineers. The bottom line is that they have the technical knowledge and I just have to manage/prioritise the relation with the Customer.

>>> I am doing the meetings instead of them >>> They don't like to be disturbed while producing, or move from their chair >>> hipchat is the best option for me >>> they are dedicated on different tasks/projects

So, when they are working remotely is fine for me as long as us don't loose the intimacy [common language]

I have always worked remotely. My company works this way, my employees work remotely.

IMHO it is not hard or difficult per se, but you need to design your company around it, like with other things.

It is also an area that needs more study in order to develop fully.

Probably if I had a conventional company and I had to transform or convert it to remote, it wouldn't work. There would be vested interest against the conversion on some people, some people would not believe it is not possible(people tend to believe that if you don't look busy to others you are not working) or managers will fear their employees slacking.

When you create a company that makes money from day one remotely, you have not this problem, as you organize around it and it is not optional.

In some ways it is weird. If you go out of your house to get your children from school because you did not spend 1 hour commuting to work, 1 returning from it, society could believe you are not working at all.

They are often surprised to see you have money, and much more money than they have.

But I suppose it is not different from an old farmer watching people sitting in a chair in the city call it "work".

>> In some ways it is weird. If you go out of your house to get your children from school because you did not spend 1 hour commuting to work, 1 returning from it, society could believe you are not working at all. They are often surprised to see you have money, and much more money than they have. But I suppose it is not different from an old farmer watching people sitting in a chair in the city call it "work".

Oh man, I run into this constantly. I think my in-laws may have finally gotten over it, as they probably figure their daughter couldn't have afforded three international vacations this year on her own.

I don't tell people I've cut my hours back to 20/wk. Most of the people I know just aren't ready to hear "I work less and still make more than you."

Very much like a service-oriented architecture: fairly easy if you design that way from the start, but difficult to refactor into after you have a giant monolith.

The more interesting thing about this story is how all of the examples are tied together in an already well known way stated by Fred Brooks in his 1975 book, "The Mythical Man-Month". You may have heard the adage of how adding developers to a late project only makes the project more late. The reason is communication overhead. One developer can produce some number of lines of code in a given day, so you'd expect adding more developers would result in more lines of code being written -- essentially O(N). Unfortunately, there is communication overhead and it increases complexity at O(N^2) as well as being an even bigger drain at the start when bring new developers up to speed with the existing group. If you take a step back, all of the remote working patterns presented in the article as working are essentially ways to eliminate the need for communication.

In a former life as a corporate software developer, I distinctly remember having to get up at graveyard-shift hours to infrequently coordinate with some of our Indian team members.

Some minimal time-zone overlap is never overrated. I think it is hard to beat real-time communication over chat or voice - asynchronous delays break up any flow in collaboration and turn it more into throw the requirement over the wall and listen for an echo the next morning.

I found that up to +/- 4hr works , other makes life really hard For example we have team in Singapore 12 hr diff , every communication is so delayed that is really painful

I believe the key to making remote work work is that everybody has to work remotely, or as if they were remote. If you have meetings and other communications happening face to face, then the remote people, to a greater or lesser extent become Cowboys, through no real fault of their own.

Every so often I have to take myself out of the office without warning the team and see just how far we have strayed from our remote communication ideal. It's generally pretty humbling.

Very good writing. A few comments:

"as a product you’re betting that your API design is robust enough that groups can remotely work at their own pace or velocity. The core question is why would you want to constrain yourself in this way? "

I am not certain which constraints are implied here - to me an API that facilitates such a work is simple and understandable which to me are admirable qualities.

Perhaps the challenge then is to split the architecture into small enough testable units which communicate through the simple APIs? Once again, this sounds great.

But in practice simplicity, understandability and good architecture are reachable by experience or trial and error so I suppose it would require either a situation were a known pattern is reapplied to new business requirements or one where there is ample time for designing and prototyping with these qualities as explicit design constraints.

"how to balance resources on each side of the API."

I suppose the best situation would be then an API that supports a directed graph of rensponsibilities. Like a plug in system, for instance?

"...however you find you can divide the work across geography at a point in time, it simply isn’t sustainable. "

So I suppose to make remote working work software architecture and team management need to be linked on a very intimate level? This would be nice in any situation, I think

Very interesting food for thought here, but the difficulties of remote engineering are not fundamentally any more challenging than any other collaboration obstacles. Obviously having all the stakeholders sitting in the same room is the platonic ideal of a team, but to me this is a secondary concern after getting the absolute best people we can.

I think the author is a bit out of touch with how startups operate. Personally I'll take a crack team made up of five best people sitting in San Francisco, New York, Berlin, Bangalore and Tokyo over a special team with a powerful corporate mandate ensconced in Microsoft's Redmond campus.

If everyone is good at what they do and has their eye on the ball, then communication can be managed from anywhere, but if you're dealing with the political reality of operating in a major corporation then you need all the help you can get.

Agree. It's interesting to compare the remote worker debate with the open office configuration debate- they seem like different degrees of the same problem.

I find it interesting that I myself am all for remote workers, but have had nothing but horrible experiences with workplaces where everyone has their own office. Pondering this a bit I think it's because every workplace I've been in with individual offices (Microsoft and others) also seemed to be very old school in terms of communication software.

Could you please elaborate a bit on what sorts of horrible experiences you had with the individual offices arrangement? How frequent were your team meetings? Did you guys use Slack-style chat softwares?

> If I had to sum up all of these in one challenge, it is that however you find you can divide the work across geography at a point in time, it simply isn’t sustainable. The very model you use to keep work geographically efficient are globally sub-optimal for the evolution of your code.

I think this is true of any organization, geographically divided or not. These are the challenges of organizational structure, and I've seen them affect organizations in a single location just as much as organizations with remote engineers.

In my mind, the single most important thing to do is to re-evaluate and adjust the structure early and often. The way you divide up work today will almost never be sustainable - regardless of geography. The trick is to help everyone understand and expect those changes to occur frequently.

What I always wondered with remote work is how do you prevent angry ex-employees to "share" their code-base with the rest of the world (dump it on The Pirate Bay) or with your competitors (in this case you may not even know that they did it).

Of course if they live in a first world country you can probably sue them (and hopefully it could refrain them from releasing your code) but if they come from countries where the law is an abstract concept there will be no brake to exacting their revenge.

That problem exists for local work in the same way, no? If you can 'git clone' onto a dev machine, you can also copy that git repo to a USB stick.

Efficient dev environments can't be totally locked down, because often people need to experiment with some parts of the environment (for example to reproduce bugs that only happen in some environments).

Yes but unless the local employee anticipated to be fired, most of the times they won't have made copies of the work.

When you tell them they're fired you can block them from physically accessing their dev machine.

Of course if you work on an open source project this problem go away, every employee is mandated to release their code ;-)

As a general rule, when talking about risks, you want to evaluate the cross section of Likelihood vs. Impact. Sometimes, you ignore high-impact risks because the likelihood of them occurring is just so low that it doesn't make sense to do anything about it, especially if there isn't much you can do about it. Like, "a meteor could strike our office, killing everyone inside". There's just nothing you can do about it[1], and if it does happen, there won't be anyone left to pick up the pieces anyway.

So I think you're underestimating the likelihood of employees copying code (it doesn't take anticipating being fired), and overestimating the impact it would have (it really won't do a damn thing)[2].

Every job I've had, the project code has somehow ended up on my personal computer. It has been a combination of factors. Generally, it goes that I get sick and declare I'll work from home. Either the company had provided a laptop or they hadn't, in which case I would have to use my personal PC to work. If a laptop was provided, the hardware was slow, or the tools installed were not my favorites (or particularly bad), or issues with the VPN connection/antivirus/corporate spyware software made everything slow. So in almost all cases, I've ended up with everything copied anyway.

And when I left the jobs, I had immediately deleted it. But even if I hadn't, even if I had taken and used the code (illegally), it really wouldn't have impacted the company. It's not like I would have been able to approach the clients and gotten them to go with me: typically the client owns the code anyway, I wouldn't have needed to copy the code. It's not like I could have started a competitor all on my own off of the company's own code base. And finally, the vast majority of projects just aren't that novel: why would I potentially throw myself into a den of thieves and lawyers for what is probably a crufty POS that I could replace in a weekend of caffeine fueled mania?

[1] oh, except, you know, not forcing your team to be co-located.

[2] This leads me to believe you are probably more the businessy/MBA type, rather than the engineering type. It's usually the MBA type who is most fearful of someone stealing the IP. The engineers who actually make the IP rarely consider it a threat.

> This leads me to believe you are probably more the businessy/MBA type, rather than the engineering type.

Sorry to disappoint I'm of the engineering type. I'm also of the paranoid type though ;-)

all this WFH fud. its total bullshit. i work on a distributed team and i do video conferencing, pair programming (screenhero) and chat all day w/ them. I talk to these coworkers more than I ever did my onsite cohorts.

I think it's almost impossible given human biology to expect excellent architectural/engineering decisions from remote teams. You need the in-the-flesh discussions, whiteboarding, leadership, etc.

So then, how does FOSS fit into your notion of human biology?

Good model for maintaining an existing codebase, but not for designing good ones from the ground up. It works best with a singular leader (or geographically centered leaders) who can make the overall architectural and engineering decisions.

I'm not personally aware of any well-designed[1] codebases made by more than a handful of people. As far as I can tell, most good "architectural and engineering decisions" are either made by a relatively small number of people or emerge from a combination of necessity and attentive devs, regardless of whether they communicate in-the-flesh on a regular basis.

I'd be interested in seeing/reading about cases of good "greenfield" projects where major architectural decisions weren't made almost exclusively by a relatively small number of people, particularly if the number consisted of more than one digit.

[1] subjectivity aside

How about the Linux kernel? Essentially, it began as something very basic in 1991 when communication methods were much different than we have today (e.g. video conferencing, etc.)[0]. There are many other examples of high quality open source projects that were born or greatly improved on the Interwebs, e.g. Git, Apache, ...


[0]: http://oldlinux.org/Linus/911010.pdf

This is speculation, but I think if Linus had a team a tenth of the size of the kernel team in one physical location they could get just as much done.

I'd love to see negating or supporting data either way, it's just my opinion that if you want a team of engineers (as opposed to a singular dictator) to build you something amazing (C Language, SR-71, Saturn V) they need to be in the same physical location and see each other face to face.

The only thing about remote work that is different than local work is communication paradigms. Specifically, whereas you could normally poke your head around a corner and interrupt someone to ask them something trivial, with remote work it's harder to forcibly intrude on them and get instant feedback. And I think that's what people have a hard time grasping: how to 'work' without instant feedback?

Of course we all own a telephone, so the instantaneous-ness is always there. But timezones create a mandatory delay to availability which can't be easily worked around. So whereas people are normally very accustomed to working via a continuous flow of real time communication, they now have to learn to communicate via delays and compartmentalize decisions and production.

Some people can't deal with this. So they start to find ways of splitting up work by location. But eventually, things get out of sync, because they never learned how to communicate in a new way. This then creates things like a mandatory daily meeting just to figure out what's changed in the last 24 hours, or red tape designed to keep people from making decisions on their own, or a lack of any kind of tape, or a breakdown in leadership, or the inability to accomplish tasks independently, or lowered morale, etc. This lack of being able to cope with communicating and working asynchronously is, I believe, a sort of stagnating death spiral that (without seriously competent self-starting workaholics) just results in mismanagement and tepid progress.

As a contrary example I like to use the open source development world. They work on different parts of different projects in different regions all the time. They self-organize and come to consensus quickly, and are led by a small team of well established hierarchy that makes quick, decisive executive decisions. There is basically no bone-headed executive level muddling with your work, nor contemptible middle-managers fighting for power, nor tepid low-level engineers waiting for someone to make a decision. Everyone just gets shit done because they all have one goal: to ship a good product. That's all very nice because you aren't part of a corporation, and you feel like your contribution matters more.

But in a corporation, you have to deal with all the usual corporation bullshit, in addition to now working asynchronously. I think this is where the big breakdown occurs in practice; two different sets of problems (asynchronous communication, and corporate bullshit) add up to a more complicated way of working. If you're very lucky, everyone will be able to work the same way and things will go smoothly. But in practice, not everyone works the same way, and thus bugs crop up in the work due to process incompatibility.

So why is remote engineering difficult? It's not; it's just a specific skillset.

I worked at two companies where everyone worked remotely.

I think one of the big differences I noticed is that I never felt like I was that close to the rest of the team. We talked a few times/week through Webex or Skype, but it's just not the same as being there in person.

Communication was also a big issue. It's not impossible to have good communication remotely, but I've rarely seen it in practice.

It might be because companies use cheap tools for collaboration and not go for something like telepresence that gives experience very close to a real one, including eye-contact - which is criticial in trust and bonding.

Also, to the best of my knowledge nobody applied telepresence directly as a bonding tools - it's mostly used for straight work, because(at least the cisco versions) are quite expensive.

IRC has always worked well. Done this in several companies. IRC is both real-time and offline. Even your toaster can run an IRC client so it's completely platform independent which is a big thing with engineers.

In a work environment people stay on global and per-team IRC channels. You can a) chat online if there are colleagues online and b) when you get back from your coding session you can always review what others have been talking about since yesterday. You can also chat in a relaxed way as in replying back every 10 minutes or whenever you don't have more important things to finish. This means you can extend your online presence over the workday with a latency of checking back several times an hour while still not getting bounced by each and every message someone decided to send you.

The missing part might have been Empathy.

I keep forgetting to enable that feature!

I think you are onto something here. One of the things I remembered that really bothered me about my last remote job was that the boss would constantly change the scope of the project right before the deadline..and extend the deadline even further.

I was never part of many of the decision-making meetings with the boss and my project manager and I felt like the manager never had our back. After we met onsite for our first bi-yearly 1-week meeting, I realized that he didn't really have any choice in the matter and did in-fact fight for us when these decisions were made.

This might be good since it can prevent a lot of bullshit politics that come with offices.

It's good that it dulls the bad side of human interaction, but the converse is also true; and therein lies the rub.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact