
Why Remote Engineering Is So Difficult - go37pi
http://blog.learningbyshipping.com/2014/12/30/why-remote-engineering-is-so-difficult/
======
moron4hire
It's easy to say that communication is key, because it is, but all else being
equal, if your team can't execute projects in a distributed, remote
environment as well as it can in a centralized place, then it's not as good at
communicating as you think it is. Dragging a failing remote team into an
office isn't going to magically make them successful, and scattering a
successful team across the world isn't going to suddenly make the project
fail.

I personally find the very disconnected nature of the teams I lead in my
consulting work actually forces us to communicate in an open, concise,
discoverable way that would be good for any type of team, but that co-located
teams get to cheat out on because a weak team member can always ride his
stronger teammates' organizational coat tails with nosey interruptions and
annoyances.

We _have_ to think things through and write clear specs, because we can't use
body language to clarify and count on the people involved in the meeting to
just remember the attitudes expressed. We _have_ to be good about change
management, because we can't just turn around and ask someone to "get out of a
file" or do whatever we want to the database whenever we want to. We _have_ to
keep meetings to a minimum, because you just can't have a productive, two-hour
phone call.

These are all good things for non-remote teams, too. But they cheat and don't
follow good practice.

~~~
jsudhams
This and the top comment nails it down that if your team is not delivering the
stuff in remote model then they most probably wont deliver in collocated space
as well. I am not very sure why people still refer to year 2000 time of
offshoring difficulties till now. For last 14 years I have been in working in
the offshore(software and infra outsourcing) model. And I can vouch for the
model to be very workable while it will have some extra work for either USA
partner or for India partner (Mostly on Indian side as they may not say no
even if USA side offer to take extra efforts). In my opinion the following has
made the life much better for me and hopefully for my partners.

1\. Have the key folks (leads/project managers) travel and stay with team for
at least 4 weeks in a common place. 2\. Do not force video (leave the mode of
comms to peoples preference (video / email / call)) 3\. Allow decision to be
made over text chat and have it added to project decisions in weekly meetings
4\. Identify couple strong people per 15 members team and have relationship
where they will tell you before deciding like 6 months in advance and allow
people to leave. They will come back later. And wholeheartedly develop them
even if you know they will leave, with one condition that they will develop
one more person like them for the team. 5\. Have good leader in India who can
speak open with out fear of a westerner (I find this a most difficult one).
This is a real deal maker between peace of mind and micro managing and going
crazy. 6\. Be open t listen to same idea that your team thought and decided to
ditch from your offshore partner. (we can go for full blog post on that)

I think this would take the team long way. Please remember either USA side or
India will have to lose 3 evenings a week, irrespective of SRS, Comm tools and
so on if you need a successful delivery.

~~~
jsudhams
EDIT ADD: There is no additional issues in managing remote worker than it is
with regular in office . It is just kind of issue, it is definitely
manageable.

------
klochner
My experience is that engineers often:

    
    
        * hate meetings
        * prefer slack/hipchat/etc even if seated adjacently (myself included)
        * work asynchronously
    

So the main friction is in the interface between engineering and other
disciplines. Remote works well so long as non-technical leadership has a local
lead they can interface with.

~~~
illumen
The communication skills of other non-engineering team members can often suck
in the digital domain. Empathy by text message can be hard to have. A
combination of on-site and remote often works best. Learning the language of
other domains can take a long time. The role of people who interface and
translate between domains is important.

------
Htsthbjig
I have always worked remotely. My company works this way, my employees work
remotely.

IMHO it is not hard or difficult per se, but you need to design your company
around it, like with other things.

It is also an area that needs more study in order to develop fully.

Probably if I had a conventional company and I had to transform or convert it
to remote, it wouldn't work. There would be vested interest against the
conversion on some people, some people would not believe it is not
possible(people tend to believe that if you don't look busy to others you are
not working) or managers will fear their employees slacking.

When you create a company that makes money from day one remotely, you have not
this problem, as you organize around it and it is not optional.

In some ways it is weird. If you go out of your house to get your children
from school because you did not spend 1 hour commuting to work, 1 returning
from it, society could believe you are not working at all.

They are often surprised to see you have money, and much more money than they
have.

But I suppose it is not different from an old farmer watching people sitting
in a chair in the city call it "work".

~~~
moron4hire
>> In some ways it is weird. If you go out of your house to get your children
from school because you did not spend 1 hour commuting to work, 1 returning
from it, society could believe you are not working at all. They are often
surprised to see you have money, and much more money than they have. But I
suppose it is not different from an old farmer watching people sitting in a
chair in the city call it "work".

Oh man, I run into this constantly. I think my in-laws may have finally gotten
over it, as they probably figure their daughter couldn't have afforded three
international vacations this year on her own.

I don't tell people I've cut my hours back to 20/wk. Most of the people I know
just aren't ready to hear "I work less and still make more than you."

------
jcr
The more interesting thing about this story is how all of the examples are
tied together in an already well known way stated by Fred Brooks in his 1975
book, " _The Mythical Man-Month_ ". You may have heard the adage of how adding
developers to a late project only makes the project more late. The reason is
communication overhead. One developer can produce some number of lines of code
in a given day, so you'd expect adding more developers would result in more
lines of code being written -- essentially O(N). Unfortunately, there is
communication overhead and it increases complexity at O(N^2) as well as being
an even bigger drain at the start when bring new developers up to speed with
the existing group. If you take a step back, all of the remote working
patterns presented in the article as working are essentially ways to eliminate
the need for communication.

------
wallflower
In a former life as a corporate software developer, I distinctly remember
having to get up at graveyard-shift hours to infrequently coordinate with some
of our Indian team members.

Some minimal time-zone overlap is never overrated. I think it is hard to beat
real-time communication over chat or voice - asynchronous delays break up any
flow in collaboration and turn it more into throw the requirement over the
wall and listen for an echo the next morning.

~~~
cunac
I found that up to +/\- 4hr works , other makes life really hard For example
we have team in Singapore 12 hr diff , every communication is so delayed that
is really painful

------
ojbyrne
I believe the key to making remote work work is that everybody has to work
remotely, or as if they were remote. If you have meetings and other
communications happening face to face, then the remote people, to a greater or
lesser extent become Cowboys, through no real fault of their own.

~~~
Multiplayer
Every so often I have to take myself out of the office without warning the
team and see just how far we have strayed from our remote communication ideal.
It's generally pretty humbling.

------
fsloth
Very good writing. A few comments:

"as a product you’re betting that your API design is robust enough that groups
can remotely work at their own pace or velocity. The core question is why
would you want to constrain yourself in this way? "

I am not certain which constraints are implied here - to me an API that
facilitates such a work is _simple_ and _understandable_ which to me are
admirable qualities.

Perhaps the challenge then is to split the architecture into small enough
testable units which communicate through the simple APIs? Once again, this
sounds great.

But in practice simplicity, understandability and good architecture are
reachable by experience or trial and error so I suppose it would require
either a situation were a known pattern is reapplied to new business
requirements or one where there is ample time for designing and prototyping
with these qualities as explicit design constraints.

"how to balance resources on each side of the API."

I suppose the best situation would be then an API that supports a directed
graph of rensponsibilities. Like a plug in system, for instance?

"...however you find you can divide the work across geography at a point in
time, it simply isn’t sustainable. "

So I suppose to make remote working work software architecture and team
management need to be linked on a very intimate level? This would be nice in
any situation, I think

------
dasil003
Very interesting food for thought here, but the difficulties of remote
engineering are not fundamentally any more challenging than any other
collaboration obstacles. Obviously having all the stakeholders sitting in the
same room is the platonic ideal of a team, but to me this is a secondary
concern after getting the absolute best people we can.

I think the author is a bit out of touch with how startups operate. Personally
I'll take a crack team made up of five best people sitting in San Francisco,
New York, Berlin, Bangalore and Tokyo over a special team with a powerful
corporate mandate ensconced in Microsoft's Redmond campus.

If everyone is good at what they do and has their eye on the ball, then
communication can be managed from anywhere, but if you're dealing with the
political reality of operating in a major corporation then you need all the
help you can get.

~~~
randomfool
Agree. It's interesting to compare the remote worker debate with the open
office configuration debate- they seem like different degrees of the same
problem.

I find it interesting that I myself am all for remote workers, but have had
nothing but horrible experiences with workplaces where everyone has their own
office. Pondering this a bit I think it's because every workplace I've been in
with individual offices (Microsoft and others) also seemed to be very old
school in terms of communication software.

~~~
desdiv
Could you please elaborate a bit on what sorts of horrible experiences you had
with the individual offices arrangement? How frequent were your team meetings?
Did you guys use Slack-style chat softwares?

------
dperfect
> If I had to sum up all of these in one challenge, it is that however you
> find you can divide the work across geography at a point in time, it simply
> isn’t sustainable. The very model you use to keep work geographically
> efficient are globally sub-optimal for the evolution of your code.

I think this is true of any organization, geographically divided or not. These
are the challenges of organizational structure, and I've seen them affect
organizations in a single location just as much as organizations with remote
engineers.

In my mind, the single most important thing to do is to re-evaluate and adjust
the structure early and often. The way you divide up work today will almost
_never_ be sustainable - regardless of geography. The trick is to help
everyone understand and expect those changes to occur frequently.

------
jmnicolas
What I always wondered with remote work is how do you prevent angry ex-
employees to "share" their code-base with the rest of the world (dump it on
The Pirate Bay) or with your competitors (in this case you may not even know
that they did it).

Of course if they live in a first world country you can probably sue them (and
hopefully it could refrain them from releasing your code) but if they come
from countries where the law is an abstract concept there will be no brake to
exacting their revenge.

~~~
perlgeek
That problem exists for local work in the same way, no? If you can 'git clone'
onto a dev machine, you can also copy that git repo to a USB stick.

Efficient dev environments can't be totally locked down, because often people
need to experiment with some parts of the environment (for example to
reproduce bugs that only happen in some environments).

~~~
jmnicolas
Yes but unless the local employee anticipated to be fired, most of the times
they won't have made copies of the work.

When you tell them they're fired you can block them from physically accessing
their dev machine.

Of course if you work on an open source project this problem go away, every
employee is mandated to release their code ;-)

~~~
moron4hire
As a general rule, when talking about risks, you want to evaluate the cross
section of Likelihood vs. Impact. Sometimes, you ignore high-impact risks
because the likelihood of them occurring is just so low that it doesn't make
sense to do anything about it, especially if there isn't much you _can_ do
about it. Like, "a meteor could strike our office, killing everyone inside".
There's just nothing you can do about it[1], and if it does happen, there
won't be anyone left to pick up the pieces anyway.

So I think you're underestimating the likelihood of employees copying code (it
doesn't take anticipating being fired), and overestimating the impact it would
have (it really won't do a damn thing)[2].

Every job I've had, the project code has somehow ended up on my personal
computer. It has been a combination of factors. Generally, it goes that I get
sick and declare I'll work from home. Either the company had provided a laptop
or they hadn't, in which case I would have to use my personal PC to work. If a
laptop was provided, the hardware was slow, or the tools installed were not my
favorites (or particularly bad), or issues with the VPN
connection/antivirus/corporate spyware software made everything slow. So in
almost all cases, I've ended up with everything copied anyway.

And when I left the jobs, I had immediately deleted it. But even if I hadn't,
even if I had taken and used the code (illegally), it really wouldn't have
impacted the company. It's not like I would have been able to approach the
clients and gotten them to go with me: typically the client owns the code
anyway, I wouldn't have needed to copy the code. It's not like I could have
started a competitor all on my own off of the company's own code base. And
finally, the vast majority of projects just aren't that novel: why would I
potentially throw myself into a den of thieves and lawyers for what is
probably a crufty POS that I could replace in a weekend of caffeine fueled
mania?

[1] oh, except, you know, not forcing your team to be co-located.

[2] This leads me to believe you are probably more the businessy/MBA type,
rather than the engineering type. It's usually the MBA type who is most
fearful of someone stealing the IP. The engineers who actually make the IP
rarely consider it a threat.

~~~
jmnicolas
> _This leads me to believe you are probably more the businessy /MBA type,
> rather than the engineering type._

Sorry to disappoint I'm of the engineering type. I'm also of the paranoid type
though ;-)

------
utuxia
all this WFH fud. its total bullshit. i work on a distributed team and i do
video conferencing, pair programming (screenhero) and chat all day w/ them. I
talk to these coworkers more than I ever did my onsite cohorts.

------
rexignis
I think it's almost impossible given human biology to expect excellent
architectural/engineering decisions from remote teams. You need the in-the-
flesh discussions, whiteboarding, leadership, etc.

~~~
moron4hire
So then, how does FOSS fit into your notion of human biology?

~~~
rexignis
Good model for maintaining an existing codebase, but not for designing good
ones from the ground up. It works best with a singular leader (or
geographically centered leaders) who can make the overall architectural and
engineering decisions.

~~~
mkhattab
How about the Linux kernel? Essentially, it began as something very basic in
1991 when communication methods were much different than we have today (e.g.
video conferencing, etc.)[0]. There are many other examples of high quality
open source projects that were born or greatly improved on the Interwebs, e.g.
Git, Apache, ...

\---

[0]:
[http://oldlinux.org/Linus/911010.pdf](http://oldlinux.org/Linus/911010.pdf)

~~~
rexignis
This is speculation, but I think if Linus had a team a tenth of the size of
the kernel team in one physical location they could get just as much done.

I'd love to see negating or supporting data either way, it's just my opinion
that if you want a _team_ of engineers (as opposed to a singular dictator) to
build you something amazing (C Language, SR-71, Saturn V) they need to be in
the same physical location and see each other face to face.

------
peterwwillis
The only thing about remote work that is different than local work is
communication paradigms. Specifically, whereas you could normally poke your
head around a corner and interrupt someone to ask them something trivial, with
remote work it's harder to forcibly intrude on them and get instant feedback.
And I think that's what people have a hard time grasping: how to 'work'
without instant feedback?

Of course we all own a telephone, so the instantaneous-ness is always there.
But timezones create a mandatory delay to availability which can't be easily
worked around. So whereas people are normally very accustomed to working via a
continuous flow of real time communication, they now have to learn to
communicate via delays and compartmentalize decisions and production.

Some people can't deal with this. So they start to find ways of splitting up
work by location. But eventually, things get out of sync, because they never
learned how to communicate in a new way. This then creates things like a
mandatory daily meeting just to figure out what's changed in the last 24
hours, or red tape designed to keep people from making decisions on their own,
or a lack of any kind of tape, or a breakdown in leadership, or the inability
to accomplish tasks independently, or lowered morale, etc. This lack of being
able to cope with communicating and working asynchronously is, I believe, a
sort of stagnating death spiral that (without seriously competent self-
starting workaholics) just results in mismanagement and tepid progress.

As a contrary example I like to use the open source development world. They
work on different parts of different projects in different regions all the
time. They self-organize and come to consensus quickly, and are led by a small
team of well established hierarchy that makes quick, decisive executive
decisions. There is basically no bone-headed executive level muddling with
your work, nor contemptible middle-managers fighting for power, nor tepid low-
level engineers waiting for someone to make a decision. Everyone just gets
shit done because they all have one goal: to ship a good product. That's all
very nice because you aren't part of a corporation, and you feel like your
contribution matters more.

But in a corporation, you have to deal with all the usual corporation
bullshit, in addition to now working asynchronously. I think this is where the
big breakdown occurs in practice; two different sets of problems (asynchronous
communication, and corporate bullshit) add up to a more complicated way of
working. If you're very lucky, everyone will be able to work the same way and
things will go smoothly. But in practice, not everyone works the same way, and
thus bugs crop up in the work due to process incompatibility.

So why is remote engineering difficult? It's not; it's just a specific
skillset.

------
paulhauggis
I worked at two companies where everyone worked remotely.

I think one of the big differences I noticed is that I never felt like I was
that close to the rest of the team. We talked a few times/week through Webex
or Skype, but it's just not the same as being there in person.

Communication was also a big issue. It's not impossible to have good
communication remotely, but I've rarely seen it in practice.

~~~
onedev
The missing part might have been Empathy.

~~~
paulhauggis
I think you are onto something here. One of the things I remembered that
really bothered me about my last remote job was that the boss would constantly
change the scope of the project right before the deadline..and extend the
deadline even further.

I was never part of many of the decision-making meetings with the boss and my
project manager and I felt like the manager never had our back. After we met
onsite for our first bi-yearly 1-week meeting, I realized that he didn't
really have any choice in the matter and did in-fact fight for us when these
decisions were made.

