
Offshoring roulette: lessons from outsourcing (2016) - throwaway3157
https://www.troyhunt.com/offshoring-roulette-lessons-from-outsourcing-to-india-china-and-the-philippines/
======
cosmodisk
Pretty much my experience re Indian outsourcing. What I used to find most
annoying is that 'manager' mentality: I appreciate a lot of people like to
boss people around and,as the author states, it's kind of safer (and better
paid) step on a corporate ladder, but boy some of them are useless.. You talk
to the guy, he agrees to pretty much anything,even If I'd ask to build a lift
to mars.Then eventually he comes back with dev's questions.I provide very
detail and explicit info on what I need(For anyone who has any knowledge in
the tech they used it would almost be like asking to write a couple of
loops).Again lots of yes,yes,yes.Then silence and again even more questions.
Manager's function is absolutely useless in this case. Also, if the churn rate
is low, the companies usually have very capable people who tend to know their
shit.

~~~
crystaldev
It's probably because engineers are not good at dealing with customers.

Edit: Apparently nobody gets this reference anymore, and I'm old.

~~~
alatkins
Take my upvote fellow oldie. Office Space can't be mandatory school curriculum
any more :-)

------
tempestn
Whenever offshoring comes up I'm reminded of my experience trying out an
Indian development firm some years back. I decided to give it a try with an
extremely simple feature for autotempest.com, just opening up a modal window
with some advanced search features. Specified what exactly should be included,
and it was nice and encapsulated so I figured it would be a perfect test.

To summarize, the results were literally _comically_ bad. All kinds of hand
massaged spacing using nbsp's and weird off by one pixel issues and
such—things that to do wrong as they did must have taken an order of magnitude
longer than it would have to do them correctly.

So, I went through several rounds of pointing out really obvious issues and
asking for revisions. They would apologize profusely and say essentially that
the person who had been working on it previously had been a _junior_
developer, but they'll put someone much more _senior_ on it now. Never really
made a difference.

But here's the one that really blew my mind. The site color scheme at the time
was greens and whites. For some reason though they decided that the heading of
the modal box should have a thick orange underline. How did they create this
~4px underline? _By embedding an enormous image of a sunset and cropping it to
a 4px orange line._

No exaggeration. This was like a 2560x1440px image of a sunset, which they'd
cropped to a sliver in the top corner to show an orange line.

I have to assume the developer(s) got sick of my nitpicking and decided to do
something so ridiculous I'd just go away. It worked.

~~~
bowmessage
literally sunsetting a project. I like it.

------
cbg0
I work remotely from Europe for a North American company, so here's my two
cents on this topic:

Having cost as the reason for going with remote devs is a very bad idea in my
opinion, and it should only be seen as an added benefit. You don't want to
hire cheap devs because they usually suck, and great developers will cost you
quite a bit regardless of where they live.

While it may be difficult to compete with the huge corporations in places like
SF, NYC, Seattle, where these competing corporations offer huge salaries,
great benefits and have brand recognition, there's quite a lot of value to be
had with setting up shop somewhere remote, where there's lower competition and
where you can offer above average salaries to attract top talent in the area.
Ultimately, every company says that they only hire the best, but unless you're
paying top dollar in your area, you're most likely not hiring only the best.

I believe that most good developers will want to be part of a stable company
that is actually building amazing things, instead of being part of an
outsourcing company where they get passed around from project to project, and
their resume will perpetually read "Software Engineer at Outsourcing Company
Ltd." \- most would trade that in a heartbeat for one of the FAANG companies.

If you want to have the best kind of success with remote devs, don't
outsource, invest a bit more and set up an actual office and hire the people
directly, people that want to have a career with your company, and not a
contract for an unknown period of time, which they likely won't be able to put
on their resume.

~~~
numbsafari
I think this is a really valuable perspective. Would be curious to hear from
more folks on “the other side” about their experiences. I’ve often considered
doing the expat thing and trying to work in or setup such a shop.

~~~
kavalg
My experience is that it is pretty easy to find a remote job as a single
developer/freelancer, but a lot harder to find a decent project for a small,
well-knit team. I now live in Eastern Europe (EU) and I easily find offers at
EUR 450-600 / day for consulting gigs (e.g. on-site + remote work). However, I
often struggle to find a project for a team of 3 Java/JavaScript/Python/PHP
developers, event at rates as low as EUR 250 / day.

I guess lots of recruiters are only interested in hiring individual
contractors, but not teams.

~~~
burntoutfire
Managers want to expand their little empires, by adding people who report to
them. They're less excited about spending their precious budget on a service
from an external vendor. There are exceptions, such as managers who learned
that managing software development is HARD and want to outsource the headaches
(with the naive hope that the outside company will do a better job).

~~~
blaser-waffle
If I hire an employee I am constrained by employee-employer regs. Those can
sometimes be onerous, and the remote thing adds a few wrinkles -- are you
liable in your state/province/department, or theirs?

But you also have the power, and it's far easier to plug them into your
existing work structure.

Hiring a small team is effectively outsourcing a component or subsystem to a
foreign coding company, the laws change -- B2B contract vs hiring -- and there
are communications and workflows to deal with.

~~~
cbg0
> If I hire an employee I am constrained by employee-employer regs. Those can
> sometimes be onerous,

Undoubtedly, but compared to hiring through an outsourced company, you do get
more control over how these employees are treated, and can thus remove the
risk of the operation being run as a sweat shop under your own nose, assuming
that's something you wish to care about.

> \-- are you liable in your state/province/department, or theirs?

In most parts of the world you should be able to set up the equivalent of an
LLC in order to minimize your risk, though of course, this will mean some
extra bureaucracy to deal with.

------
shruubi
I just came out of being forced into running an offshore team from the
Phillipines for roughly six-ish months, my experience was that they were very
affable with good english skills and worked hard, there was a noticeable
problem with technical skill which caused a number of bugs to occur to the
point where I would just spend the time fixing them myself than throw them
back at the team where they would cause more problems.

The biggest problem however, was that no matter how many times I asked or
provided documentation/feedback, they would ignore established coding
standards and established processes and just do things their way, and when I
would ask about this, I would just get shrugs.

Finally, we got saddled with a "Project Manager", which I put in quotes as
their contribution to the process was giving daily excuses as to not turn up
to our daily progress calls, and the ones he did turn up to, he would visibly
fall asleep. When I eventually pulled him up on this his response was that "he
wasn't directly paid for but an included part of the cost so you can't
complain about what you get."

Admittedly all of the above is rather negative, but I think this highlights
the "get what you pay for" aspect of offshore outsourcing.

~~~
watwut
Out of curiosity, what are "established coding standards and established
processes" for you?

------
starky
My experience with working with Asia off-shoring is with physical products, so
this was interesting to see the software perspective where there are multiple
programmers working on something in parallel, whereas my experience is more
serial with the (English speaking) sales contact communicating with the
various engineering teams as the project gets passed along.

Definitely agree with the can-do attitude of China. Sometimes I'm pleasantly
surprised, other times a vendor thinks they are doing us good and screw us
over by completing work before we have agreed to it. This is mostly a
temporary issue until you build a relationship with the vendor (in China,
relationship is everything) and know each other's expectations.

While I've never worked with an Indian vendor directly, it is interesting to
hear about how people there have very narrow knowledge bases. This is
something I've noticed with people I've interviewed that are recent
immigrants, they know their area, but when asked about the rest of what we
would be asking them to do they have zero clue. This is also (lesser so) a
case with recent Chinese immigrants. Not great when we are essentially hiring
a mechanical equivalent of a full stack engineer.

------
discreteevent
On India: I worked on a project outsourced to one of the big companies. A
senior engineer there said that when people offshore, the person making the
decision is a bean counter. So they only look at price. They take the cheapest
and they get what they pay for.

But if they would pay twice that (still at most half their normal cost) using
a smaller higher quality company they could get a much better job done.

~~~
mathattack
Sometimes the math is “We can do it twice in India and still save 50%”

Anyone who has worked with the big vendors knows that the correlation between
quality and price is very loose. The lowest cost provider is rarely great, but
the high cost provider can also be selling you a lot of juniors.

~~~
raincom
Exactly this. Usually, they have a couple of guys who are really good in that
team, along with lots of juniors who are there for billable hours.

~~~
mc32
They demo the skilled people and once they have the account they turn it over
to the juniors...

But that probably works in most cases/places anyway.

~~~
makr17
Yeah, but so does every US-based consulting outfit I've ever seen... IBM,
Oracle, Accenture. The knowledgable Senior Dev answering your questions during
the bid goes away _quickly_ once you sign. They're off to convince the next
sucker...

~~~
bradknowles
That's why you write the contract to the outsourcing company so that you lock
in those experienced people for an extended period of time -- like years.

But then the price mysteriously increases to the point where you might as well
have hired them on-shore and you're not saving any money.

So, do you want to save money by offshoring everything, or do you want to get
good people to actually do the real work that needs to be done?

~~~
watwut
Accenture wont sign that. And those more experienced people might not like to
work on your project for years and there is nothing you can do to stop them
from leaving.

They got where they are by doing what they are doing. Spending years on your
project is something else.

------
PragmaticPulp
Hiring good offshore developers directly as remote employees is a much better
option, in my experience.

It requires more up front work to identify, hire, and retain the remote
employees, but you need far fewer developers when you can appeal to the best
developers who know what they're doing.

This doesn't work if your company has any attachment to headcount, though. It
can be a tough sell to hire 3 remote employees for the price of a cheap team
of 10 from an outsourcing firm if your company only looks at the numbers.

~~~
MarkMc
> It can be a tough sell to hire 3 remote employees for the price of a cheap
> team of 10 from an outsourcing firm if your company only looks at the
> numbers.

Can you expand on this? I would have thought it was cheaper to hire the remote
employees directly than hire the outsourcing company...

Edit: I think I understand now: the 'remote employees' are based in say the US
while the outsourcing company is based in say India

~~~
chii
The premise is that an excellent developer can command a high price, whether
remote or not. Some developers place a premium in remote friendliness, and
thus allow a slightly lower salary for it, but only up to some amount.

------
sulam
My classic offshoring story is of a team of people we brought in to do QA
(literally to write the unit tests that Troy's blog post no one ever writes).
This team was very bullish on getting to do more than "just" testing. My CEO
wanted to let them take a crack at it, and I had more than enough to do so we
tried it out.

I had written some domain models for dealing with users and I needed
equivalent models for dealing with notifications. So I asked them to take the
work I'd done and use it as a template for the new stuff. I got back code with
the method and variable names search and replaced, pretty much exactly like
what I expected. What they _hadn't_ bother to modify was the SQL that actually
operated on the data (yes, I had to write it out like a cave man). So I had a
bunch of notifications domain objects that operated on the USER table.

That experiment terminated early.

------
edoo
Very early in my career I worked for a dev shop that would literally hire 2-3
Indian teams to do the same project in the hopes one would turn out ok and it
was still way cheaper than domestic labor. I was often brought in to fix bugs
and performance problems. It was during this period I saw the worst code of my
life: Pulling every record from a sql database table with a 'select * from
table' then a strcmp on each numerical (and indexed) `id` field to find the
record. It had worked in testing but blew up in production. It was truly
unreal to see that.

The other interesting thing I got to see was a shakedown of our company after
we poached talent from a connected local company. We had the option (which we
took) to pay several years worth of the average per capita income to local
government henchmen to be allowed to stay in business.

------
AnimalMuppet
The worst I've ever seen: Not only did the contractor deliver a buggy, half-
finished product, he also used our printer to print out an offer to a
competitor to sell them our source code!

------
gentleman11
Lets assume that the situation changes over the next 15 years and these
outsourcing places get really legitimately good at making software.

What can we, the highly paid locals, can do to justify our high salaries at
that hypothetical future point?

(playing starcraft taught me a tremendous amount of respect for East Asian
creativity and work ethic)

~~~
cosmodisk
What differentiates US (most big software names come from there) from the rest
of the world is their ability to sell. There are already enough countries on
this planet that can produce high quality software that could be sold as hot
buns.What most of those countries don't have is the huge VC industry+ tons of
people who know how and who to sell be it B2B or B2C. This, however is
changing slowly and you can expect much more level field over the next 10-15
years.

~~~
pm90
This is not accurate at all. What really differentiates the US is:

* fucking largest economy by far as your domestic market ie no import duties or import/export taxes

* truly massive investment by Government and Private sector into CS research

* high tech economy focused mostly on services which demands all tools it can get to improve business processes

* high disposable income via financial instruments for cheap debt (also less taxes than most developed countries)

* largest talent pool of Universities that keeps bringing in the best and brightest across the world

The ability to sell is far far less important than the systemic reasons listed
above that give the US an incredible edge in the software industry.

------
dang
Discussed at the time:
[https://news.ycombinator.com/item?id=12923925](https://news.ycombinator.com/item?id=12923925)

------
iamsb
It is quite interesting to re-read this after the zoom ipo. Zoom built a huge
team in China, effectively outsourcing most of the development to lower cost
countries, that gave them leverage in terms of being able to deploy capital
more efficiently.

There are certainly ways to work around most of the problems the author
outlines. They are definitely real problems, same as spending 5x for onshore
resources. Ideally you should build onshore teams, if you can spend the money.
But if you can not, then do not get discouraged by these problems. Employee
churns are not that difficult to mitigate, just matter contracting properly
with the employees and providing enough consideration in the contracts.

------
admn2
I heard Argentina as a popular place for "near-shoring", which likely wasn't
very popular when this was written 3 years ago. Anyone here been using
Argentinian tech talent?

~~~
jermaustin1
I've worked with a number of Argentinian developers through using Avanade and
Accenture on projects. The developers were great, even spoke decent English,
but since it was through Avanade or Accenture, after 3 years of the
managerial-interference, overhead, cost overruns, and contract disputes with
what is considered in-scope/out-of-scope, the projects were eventually
canceled. Had we just been able to hire the developers on an hourly basis,
everyone would have been happy... well except the sales managers, delivery
managers, development managers, and project managers.

That said, I didn't care for the way that our company was billed $75-150/hr
for a developer that makes $350-1200/mo, and that the developers made so
little when their managers were billing so high.

Early in my career, I was livid when I found out that my time was billed to
the client 3x what I made, so I cant imagine how demoralizing it would be
knowing you are paid 20x less than you are billed out.

~~~
cosmodisk
This, sometimes has very interesting consequences. A friend worked for a
company that basically moved parts of sales/customer service to a Northen
African country.They hired people on a few hundred dollars a month(if that
much) to sell high ticket hotel bookings sometimes going for thousands/day.The
team struggled to comprehend that anyone could get a room that costs 10 times
more their monthly wage.This highly affected their ability to service/sell.
Eventually the entire thing got canned and moved to Europe.

------
gwbas1c
I worked with quite a few overseas developers over the last decade.

In our case we worked with two outsourcing firms in Russia, EPAM and
Distillery. Our relationship worked very well but we had to approach it
correctly.

In our case we treated the outsourcing firms mostly as recruiting firms. They
pre-screened candidates and then we interviewed them just as if we were
interviewing and on-site employee.

Now, people are people and are imperfect, But the reality was that from both
EPAM and Distillery, The candidates that they brought to us were always the
highest quality. Something about how they work internally makes it a lot
easier to hire through them instead of the normal American way of hiring a
software developer.

One more thing: I was part of the team that chose EPAM over some of the other
contracting firms. we basically interviewed one of their engineers as if he
was going to be a new hire, and we did the same for other firms. The reason
why our relationship was so successful was because we we're methodical and
carefully chose who we used for outsourcing.

IMO, most of the complaints about outsourcing come from people who didn't
screen their contractors very well.

------
anthony_barker
He misses some stuff.

1) 4 hours of time overlap rule - you need to have overlap of 4 hours between
teams

2) Requirements need to be much tighter with outsourcing. Often you need a
full BA whereas with some devs you can work without a formal spec but some
wireframes etc

~~~
SamuelAdams
No, Troy specifically mentions differing requirements depending on the country
that the ousource firm is located in.

>Another pattern I found time and time again when outsourcing to India was
that they'd want really detailed documentation. This is always going to be a
contentious issue and there are many different views of how much should be
done under what circumstances. But more so in India than the other two
locations I'll talk about shortly, detail was important to them. There were
many occasions where we would make assumptions that a feature requirement
clearly implied certain things only to later discover they was deemed "out of
scope". Now that can happen in any project in any part of the world, but it
was extremely prevalent in India.

> On the plus side for China though, one of their strengths (particularly over
> India) was the ability to get down to work with minimal documentation.

------
freepor
My favorite place to offshore is Eastern Europe. Genuinely love working with
those guys.

------
Ididntdothis
I really don’t understand why US companies are outsourcing to India. The time
difference alone pretty much sets you up for fail unless people want to be
working at midnight all the time. I find it much easier to work with Costa
Rica it South America where you can quickly make calls during the day.
Especially with outsourcing you need to do a lot of handholding to get things
going so you need good direct communication.

The whole idea of sending a big pile of requirements and then getting
something useful back simply doesn’t work. You need direct communication
without middlemen.

~~~
lowercased
worked at a place in the south (think Georgia) that opened an Indian office.
about ~40 devs in the US office, they brought in.. 5? to the Indian office. It
was primarily for the time difference, and their primary role was to triage
and research issues during the 'off hours'. Anything they couldn't figure out
bubbled up to the US, usually after several hours of initial research. It
wasn't a horrible approach, and I think it worked OK because it was
intentional.

------
freepor
With China and India getting more and more product companies like
Baidu/Tencent/ByteDance/Flipkart/Paytm/etc, the average engineer left in
outsourcing gets worse every month.

------
watwut
I don't think all those people are leaving after a year to become managers.
That is highly unlikely. More likely is that those positions sux whether you
have passion for development or not - and that developers who have passion for
development will run away first from projects like this.

Like with Avni, if she really had a baby and went to have baby, the
relationships were nowhere near for anyone to know it. Which again, to the
extend people stay in positions due to liking people there, that sort of "keep
me there" relationships were not present.

I suspect this also because we had projects where more western company
outsourced their project on us - coders here, analysts, management and
decision makers there. The more skilled developer, the sooner they were
leaving or were very particular about what exactly they will work on. Even
those who stayed (because some perks and good relationships in our local
company) dont like the projects they work on. It is more about frustrations
and politics then about being proud what you do.

I can only imagine that it would be even worst in outsource only company that
aims for cheap and does not have perks. And it is even harder to find good
managers willing to work in such setup, they tend to avoid it too.

------
MarkMc
"I remember one case quite clearly where the project's dollar estimate was
identical to a local vendor in Australia yet the man hours were an order of
magnitude greater. Yet even then, there was organisational push-back on the
local vendor due to their high hourly rate"

That's very strange - I would have assumed that the code quality of the
Australian outfit would be higher

------
thrower123
I do a lot of work with a few of the big outfits, as we often get pulled in to
help setup a software product that they resell to Fortune 500 companies as
part of their package contracts.

I have never seen a case where they actually add value; it would always be
much easier if we could just get directly connected with the end-customer.

------
throwawayesteu
How to do the outsourcing business if you want to provide premium service?
Even if the field requires specialization, say machine learning / data
engineering, it’s hard to justify the price that is 40% more than average for
the area (say Eastern Europe). Example: I work for a company that does ML/DE
services, and I believe that everyone in the company (we’re small company in
Eastern Europe) tries their best to do the great job and do what is best for
our clients; plus seniority and talent is way above average for the market
(mostly due to specialization). But we’re still having hard time increasing
our hourly rate.

What is something that we can do that would say “these people are worth it,
let’s go with them even though they are more expensive”?

~~~
cosmodisk
Higher prices!= higher quality. Here's how your post reads: we have senior
devs+ we try charge 40% more,yet clients don't buy in. You know what? Every
shop in town has senior devs and most of them are passionate. Depending on
which Eastern European country you are based in, even the entire team full of
PhDs in ML with 10 years experience may not sound that extraordinary. You need
to find the reason why you are ACTUALLY better than your competition.Do you
have tons of white papers or cases studies on your website that clearly show
how and why you are solving these problems? What are you doing differently
compared to the guys from the shop across the road? You need to be very very
comfortable with saying " fuck yes, we are 40,50 maybe even 100% more
expensive than any other company you'll be speaking, but here's 25 reasons why
you search for ML experts stops here". These things are learnable, however if
there's no strong sales culture in the business,I'd advice to start looking
for someone who could deliver these messages to your prospects.My experience:
work for a company that operates in a low entry, saturated market and sells
services almost 3 times the average price in the country.

------
lmilcin
I have some experience with offshoring to India and China.

Pretty much as described in the lessons.

One important point, setting up the cooperation is critical and you can't hope
to get any positive results without considerable effort into setting up and
maintaining the cooperation and close vigilance over what is happening.

There is tendency for the quality to fall the moment the team figures out you
are not paying attention. Basically, anything goes unless you are constantly
showing you are caring and monitoring.

------
bluekite2000
Anyone interested in Vietnam? I spent tons of time there (Ho Chi Minh city and
Hanoi) recruiting Vietnamese software developers recently for my own projects
and am seriously thinking of parlaying that skillset into helping US companies
setting up in house engineering sites there.

------
obiefernandez
"Nearshoring" in Mexico is successful primarily (IMO) because of cultural
compatibilities. Plug: my firm [https://magmalabs.io](https://magmalabs.io)

When I was CTO of Andela one of my main claims to fame is pioneering the
"month one" curriculum, based on principles of Applied Improv -- taking
comedic improvisation education and setting it in a business context. Nigeria
is definitely one of those cultures where junior people are culturally
ingrained in them not to disagree with their superiors. Extremely important to
break them of that practice if they're going to succeed as software
consultants.

~~~
_JC_Denton
Heads up - I am getting an invalid certificate warning in your site. iOS
Safari.

~~~
cpeterso
And in Firefox:

"Firefox does not trust this site because it uses a certificate that is not
valid for magmalabs.io. The certificate is only valid for the following names:
m.magmalabs.io, www.magmalabs.io

Error code: SSL_ERROR_BAD_CERT_DOMAIN

[https://magmalabs.io/](https://magmalabs.io/) produces the error.
[https://www.magmalabs.io/](https://www.magmalabs.io/) does not.
[https://magmalabs.io/](https://magmalabs.io/) works in Chrome. Does Chrome
delay the certificate validity check until after the HTTP redirect?

~~~
cpeterso
I filed a Firefox bug:
[https://bugzilla.mozilla.org/show_bug.cgi?id=1617987](https://bugzilla.mozilla.org/show_bug.cgi?id=1617987)

Chrome has a heuristic they call SSLCommonNameMismatchHandling: if Chrome
tries to connect to a site like "[https://example.com"](https://example.com")
but the returned cert is for the "www." subdomain of the same domain, Chrome
will silently redirect to
"[https://www.example.com"](https://www.example.com").

------
lifeisstillgood
>>> Low hourly rates only make sense when you're comparing a similar number of
hours!

------
PouyaL
Thanks for sharing. Got almost the same experience in middle-east where a part
of culture within SW engineers came to low quality of development and higher
time spans. Though I guess the article wasn't consider the concept #TL;DR.

