
Lessons from Outsourcing to India, China and the Philippines - RmDen
https://www.troyhunt.com/offshoring-roulette-lessons-from-outsourcing-to-india-china-and-the-philippines/
======
anupshinde
Nice article and on the spot.

There is a difference between offshoring and cheap-shoring (i.e what I call
offshoring at very low costs). Most of the bad experiences you get from India
are a result of cheap-shoring. You get what you pay for. Cheapshoring
undervalues good engineers and overvalues a lot of bad engineers.

I have seen this for almost 15 years (from the Indian side), and I don't like
it. While some people see Indian IT as growing with the likes of Infosys, TCS,
Wipro, etc - I personally saw that as crippling good CS talent in India.
However, all that is coming to an end - probably sooner than later. These
biggies will either have to trim down significantly or collapse under their
own weight

If you want equivalently good engineers from India (as of now), be ready to
pay half(45-50%) of the pay in US. So if you are paying 120K in the US, the
same should cost you 54-60K USD when the engineer is in India. This will also
reduce friction. One may get lucky in the early days hiring at cheaper costs,
but sooner or later the good engineers will see their worth and will most
likely leave instead of asking for a raise. You have to do your due-diligence
selecting a worthy engineer - that is like finding a needle in a haystack.

If somebody chooses cheapshoring, they are planning for a higher cost that is
at least more than the 50% mark mentioned and in most cases if you analyze
retrospectively - it would have cost lower if you did not cheap-offshore in
the first place. The release may seem less costly, but the maintenance and
operating costs will ultimately outgrow.

~~~
ido
$60k puts you in the same order of magnitude as other western countries:

E.g. ~$65k gets you a senior developer in Berlin (=€60k) and similarly in
Canada (=90k CAD), so a lot more competition in that bracket.

~~~
dep_b
I'd say good programmers are relatively underpaid in Western Europe compared
to the rest of the globe.

~~~
slowmotiony
Absolutely - if they work as regular employees. In Munich a senior dev can be
offered around 100k. If he's single and without kids, the government will
immediately take more than half of that. Then the apartment will set him back
around 1000 euro a month. This leaves him with around 3000€/month.

I can make that in less than a week as a contractor doing the same thing in
literally the same place.

~~~
7fff00
We don't even pay nearly that much on our total income here in Denmark - this
chart says Germans don't either
[https://upload.wikimedia.org/wikipedia/commons/c/c1/Income_T...](https://upload.wikimedia.org/wikipedia/commons/c/c1/Income_Tax_Germany_2010.png)

~~~
slowmotiony
Google "brutto netto rechner" and calculate yourself. If you're single and
childless half your money goes to tax and social stuff.

------
sudhirj
One of the things you might want to remember when outsourcing to India: most
of the best developers tend to jump to product companies sooner or later. The
best Indian offshoring companies tend to charge around USD 50 / hour, out of
which the developers see about $3 (college grads) to $20 (10+ years) per hour.
The rest of it goes to managers and overheads. And this is the best companies.
The more mass market ones charge $20 to $30 an hour.

The better devs know that their possible value to the company is always
capped, and because the country has a huge number of engineering graduates
rubber stamped by a network of substandard universities and colleges,
developers are usually very easily replaceable cogs.

So Google, Amazon, Microsoft, Flipkart, Freshdesk, Hike, and a lot of smaller
not known Indian companies that develop specialized tech is where folks tend
to go.

~~~
fgpwd
In my previous company (a very big, reputable US Consulting company), I was
charged at 40$ an hour to my clients in the US, but I got only 2-4$ an hour
out of it. My monthly pay was fixed at around 550$, and I would have to work
8-12 hours everyday. This is a typical salary for a college graduate from a
fairly good (in the top 15) engineering college in India.

------
achow
One thing that jumped out to me reading the article was the author’s (or the
organization’s) aversion to writing requirement spec (“detailed
documentation”).

That sounded very worrisome particularly considering the fact that the
development was for a domain like that of Pharma, where rules & regulations
shadows heavily on what and how things are done.

I have some experience in developing software products for Pharma industry and
it was quite an effort in keeping track of FDA updates which would impact the
features that were under development or were planned for. Examples, basic and
mandatory feature sets like Audits, Logging, Security, Access management,
Reports etc. are heavily influenced by compliance regulations of FDA and
equivalent bodies of Europe, Canada etc. And added layer of complexity is each
regulatory bodies have different approach to regulation and compliance.

Considering the fact that Pfizer were outsourcing to ‘code shops’ who would
never ever get into that complexity; well thought out requirement
documentation which has taken into consideration current and future landscape
should have been number one priority. I would have sleepless nights if I was
coordinating a project where ‘can do’ developers start working from a basic
one-pager from ‘business people’.

I guess it was perhaps because the author was the go between the domain
experts (he mentions them as ‘business people’) and the developers.

It would have been great if the author would have shed some light on what
could client do from their end in terms of reducing ambiguity while working
with teams where communication and loss in translation is one of the biggest
risk.

I suspect that there is a whole missing dimension in this story.

~~~
ploxiln
The process of writing software involves some interpretation and making good
small-scale design decisions. Think of things like checking for buffer
overflows and error handling. If you completely specify how this is handled
everywhere you've practically written the software already.

One trivial and silly real-world example: I'm currently working on a C
codebase with a few developers in India. Sometime before I started the
project, the American company I now work for contracted out a security audit
of the code, and one of the obvious conclusions was that "safe string
functions should be used" \- strcpy() had been used to copy user input to
statically sized buffers such that it was vulnerable to classic buffer
overflows, all over the place. Requirement set, requirement followed - now
strncpy() and strncmp() etc were used _everywhere_ , but always either
_uselessly_ (25% of the time) or _wrong_ and unsafe (75%). Most typical was to
use the length of the source, calculated with strlen(). Now, I guess it was
not specified "use strncpy() _correctly_ , with the size of the _destination_
". Failure of specification, right?

For what it's worth, the root of the problems on this particular project are
that the original American lead dev was not that great and set some bad
examples, which the outsourced devs copy diligently and industriously.

(But then again, it's not enough to say "this is clearly wrong, it will
overflow if (expr)" \- they tend to need explicit examples to copy. Sometimes
they impress me by figuring the cause of a tricky bug. Sometimes they write
smart but actually useless code, like using floats and log() to figure out how
big a buffer is needed to print a uint16_t in decimal - 5 bytes will suffice,
no float math needed. Sometimes they write 20 lines of code to calculate how
long the string result of snprintf() will be, instead of using the size of the
destination, right after I've lectured on what strncpy() is for. They often
seem less thoughtful than a neural net. I've even seen "buf[strlen(buf)] =
'\0'" \- I just have no words sometimes.)

~~~
fgpwd
Some of these examples made me smile. But this cannot be taken as a
representative of all developers in India. I am an Indian developer, and have
met several very smart people. Most of them are presently working for big US
companies like Google or Adobe.

Often when I meet a programmer, I can tell if they are genuinely interested in
programming, or just happen to be a programmer because of circumstances or
because it's what they decided to do to make money. The latter when writing
code for you would just want to somehow meet the specifications and head off
for the next lunch. They would be the first ones to complain about their job
on social networks. As sad as it is for them, you really don't want to hire
them if your aim is to build high quality products.

There are still a lot of good, capable developers in India, available for hire
and at affordable rates. You just need a good process to find them and filter
out those without a genuine passion for writing code or for building products.
Hiring random people without a thorough selection process is always a bad
idea, no matter which country you hire from.

~~~
ploxiln
Agreed. It's not about India, it's about outsourcing. Good developers get
better jobs where they are not a fungible resource. (And sometimes they move
to the US or the UK.)

------
bkbaba
THIS! This is exactly what I hate about being an "Indian" developer living in
Bangalore.

It really hurts me that when someone is working with an offshore team they are
looking at me as a "cheap resource".

I don't expect the valley crowd to understand this, so let me suggest an
experiment. Go to your best developer, and tell her that you are there in the
company cause she is "cheap labour", and if it was possible in a fraction of
second you would get an engineer from Google and would hire him to do your
job! Repeat this constantly for 3 weeks and see if her productivity drops or
if she is still around! Its really painful to spend so much time, understand
your own workflow and identify shortcomings, being self critical, overcome
shortcoming with tips from HN / Stackoverflow, to concentrate on the problem
at hand and implement a feature and then have someone come in and call it
cheap labor. Previously I was working with a team in US and was reporting to
an American. I know he was happy with my performance compared with the rest of
my team members in the Valley as a lot of R&D work would be assigned to me. I
have worked with angualr, react, rails, java, chrome extensions, ios, android
apps. I am definetly not the best in any of this. But based on my previous
work I think I am fairly competent.

In enginerring its a huge challenge to bring down cost. SpaceX is celebrated
cause they have brought down the cost of launching satellites, when ISRO does
it, it hursts me to see the first person who says that they were able to do it
cheaper cause of the labor cost in India. I respect my work, and take pride in
having built it.

If you want to offshore work don't make them feel like that the only reason
you are giving them the work is because they are "cheap".

I know this is not what the article says but I wanted to share this even
though this is off topic.

~~~
krisroadruck
I feel what you are saying but... that _IS_ the reason people offshore. Why
else would they ship jobs out of the US, deal with funky hours, bad phone
connections, and often times very thick accents (which when combined with bad
phone connections often render conversations 80% understandable at best) if
not to save money? I would feel like crap if I was in that position too but I
don't know how you go about solving it and still have a valid justification
for offshoring that outweighs the negatives.

~~~
eequah9L
Shortage of local talent might be another possibility.

On my team, about half the people are not local--I'm a remotee working from a
different country, and a bunch of other folks are outsourced from India and
Ukraine. I don't have insight into how much the outsourcing costs, but a
figure of $50 per hour has been floated in this discussion. At that point you
are pretty much back to local wages, so my guess is it's not the money.

This may not be your typical outsourcing story though. Individual people are
actually hired like you would hire anyone local, with interviews and all,
except it's a manager traveling there to do a series of interviews and not
vice versa. The team is quite stable, the churn reported in the article
doesn't seem to be a problem.

~~~
jsymolon
> Shortage of local talent might be another possibility.

Only shortage is, is a person at the price that they want to pay.

------
ilaksh
If you are offshoring a project because you can't afford local prices then the
project may not be feasible at all within your limited budget, even at
outsourcing rates.

If you have a limited budget, realize that means your requirements will be
limited. Never select the vendor with the lowest rates. Always evaluate the
actual people on the team.

The idea of line items for features is unrealistic for software development.
It's X hours, days or weeks for each person, which needs to be re-evaluated
every week or two.

If you don't have a good developer you trust before aside from the outsource
team to look at the code on a routine basis then don't bother making the
software at all. If you don't have a budget that allows for automated tests to
be developed (20 percent of devs will do this automatically anyway) then
cancel the project. If there is no budget for refactoring then please cancel
the project.

If you plan on using thinly veiled racism to blame foreign developers for your
inability to specify coherent requirements or comprehend the complexity of
software development, then please cancel your outsourcing project.

If you follow this advice, 8/10 outsourcing projects would never start and
everyone could spend their time on better things.

~~~
user5994461
8/10 of all software projects should never start. It's not just about the
outsourced ones ;)

------
ensiferum
My experience (I managed a team of Chinese based devs for about a year) was
that:

a) people are indeed leaving. thats cause the main selling point (at least at
that consulting company was indeed price), so there was 0 commitment to
employees from the employers side, so it's normal that the employees have the
same and leave immediately when they get something they consider just slightly
better offer.

b) trying to "adjust" the agreemtents to their benefit. A use case example is
that the chinese org puts their better candidates as named engineers on their
offer. Once the project is signed they start to swap them out under whatever
bullshit. (Oh, we're very sorry, we have to swap John Holmes for Ron Jemery,
etc.) On that point I woudln't be surprised if some of these people only
existed on CVs and were totally fabricated resources simply used to try to win
projects.

c) the need for constant supervision and very precise instrumentation. If you
gave these chinese engineers a brick and said "see this brick here. it's nice,
but I'd like you find out how to improve it somehow. Maybe make it more
durable, cheaper to manufacture etc." Two weeks later the chinese engs will
get back to you and ask, "what do we need to do?" However if you give them the
blueprints for a brick factory and tell them to build you the factory and 1
million such bricks, that's a task they can do.

~~~
sumanthvepa
I'm curious. Have you tried asking a US engineer to improve the brick? I've
had that experience, and the engg. team came back to me and asked me to
improve the spec. I was an Indian product manager using engineers in the US
(for a very very very large US software company) I had to specify the exact
length of a text input field for a search box, because the team couldn't
figure it out for themselves! Idiots exists everywhere.

------
garagemc2
Really interesting and well written article. I don't think China will be an
outsourcing destination for long.

In India your slowly starting to see agencies that aren't merely code
factories but places where you can do actual product development. Its early
days though.

~~~
AvenueIngres
This is kind of frightening, our wages won't survive a world in which where
there's a surplus of talent coming from developing countries.

~~~
Roboprog
I have seem _some_ talent, at various levels, some of which is even quite
good. But I surely would NOT call it a "surplus" :-)

------
tropo
Nothing was said about the security implications of outsourcing to people who
are required to obey a different government. Are you going to open up your
network to them? Are you going to expose your trade secrets to them?

When you find a strangely perfect clone of your product on the market, do not
be surprised.

------
flanbiscuit
> What I'd typically find is people would specialise in one discrete part of a
> software component and that's all they'd do.

I work in pharma advertising as a dev but sometimes our client has their own
offshore dev team. I've dealt with Wipro many times. This right here has been
a common issue with them. Our account team will ask me if something is doable
and I'd say yes but Wipro would push back saying they can't. And our sites
fall into that brochureware category so these were mostly front-end trendy
things using history push-state or what not. We usually found that they had a
very limited scope and would not venture outside of it. I guess that's the
difference, you get what you pay for. Usually if something is handed to me
that I've never done before I do some research first and 90% of the time I
figure it out. We've had better luck with eastern European offshore team in
this regard. If I were looking for offshore teams I'd suggest there or South
America first

------
shagie
The article does generally reflect what I've seen - not only with outsourcing,
but also reflects the general cultural predisposition of people that I've
worked with, in the US, from those places.

A bit on India with the churn that was mentioned I've seen explained as that
it is difficult to get a promotion within the company - you need to get hired
by another company to move up. With one contract shop that I've worked with,
over the course of 18 months (we worked with them for three years) every
developer position was replaced with a new person (so twice across those three
years). This resulted in a lot of loss of institutional knowledge of the
product. You'd train up a person, they'ed work on it for awhile, and then move
on to another company and train their replacement. That training process was
always lossy.

The goal for a ((stereo)typical) Indian developer is to be where the pay is
best. The way that the companies are structured (and note other comments to
the effect of the pay scales) is that this is in management. One gets to that
by demonstrating management potential rather than technical skill. The highly
paid sr. developer path isn't as viable as the (comparably) highly paid jr.
management.

Another major point that didn't appear to be touched on in the article is the
intellectual property leakage and protection. Many managers here are afraid of
that (rightly?) and take significant effort to make sure that overseas
developers are strictly limited to what they are working on. Sometimes that
goes as far as "you have access to _this library_ - not the entire application
that it is part of" (or for that matter, even the integration test suite - a
build breaks because of a library change... and you can't provide the full
test to show how it broke) leaving the integration to the in house team. That
can cause some friction in the development process that is often unaccounted
for at the start of the project.

~~~
lake99
About churn, promotions are one of the issues, and the other is a good salary.
Many companies are quite exploitative (that's one way of looking at it). If
you can't play hardball with your HR, your freshly hired subordinate's salary
might end up being much higher than yours. One way that HR can operate is to
reward talent and productivity. The other way is to reward negotiation skills.

I can't empathize with the other points that the author made about India --
because I've never worked on an outsourcing job.

------
glaberficken
> "In the markets I'm discussing here, they almost exclusively want to move to
> a position where they're instructing others. There just seems to be less
> interest - less passion - in code and inevitably that's going to impact
> quality."

I think this statement misses the point completely, the difference isn't the
passion about coding, it's obviously the salary!

If you are one of the juniors toiling away in the front-lines, your pay is
still going to be really low. A programming role only allows to get your foot
through the door, if you are to make a decent living the only way is to shoot
for a promotion into management.

As opposed to "the West" where you can actually pursue a technical role
throughout your career because the pay is higher and there are many more
different types of roles you could be doing.

~~~
rawTruthHurts
In my experience, also in "the West" managerial positions are, on average,
better paid. Not that you don't make money on a tech role, but management will
still do better.

------
sriram_sun
Loved the Indian head shake video! Watch it if you can.

~~~
benmarks
I was _very_ fortunate to have a colleague familiar with this behavioral idiom
in a developer training I was giving to a team from Hyderabad. He kindly
interrupted me after I kept trying to explain a concept which it seemed to me
they didn't understand!

------
digi_owl
I really wish there was a neutral unit of account one could use for comparing
costs etc.

~~~
fapi1974
I like this idea. What do you have in mind?

~~~
digi_owl
Not quite sure. Something akin to the Bancor that Keynes suggested back in the
day perhaps.

[https://en.wikipedia.org/wiki/Bancor](https://en.wikipedia.org/wiki/Bancor)

------
_nalply
2001 I worked half a year in Pune, Maharashtra with Indian developers. In my
personal experience Indians are hard thinkers but undisciplined and chaotic
coders. This means they managed the mess they made rather well just because
they still understand what they did. Perhaps it was the CEO who just let them
do what they want provided it works after all.

The salary was one thirteenth of what I would get in Switzerland. The
efficiency was about one half, but only because the Swiss CEO himself lived in
Pune and later married an Indian woman. If he stayed in Switzerland efficiency
would have crashed badly. In any case, don't talk about the code quality.

I also noticed the extreme churn the article mentioned. In my half a year
about 5 to 10 percent of the people left and new people started.

------
AJ007
That was a great outsourcing article, except showing one picture of Manila
slums and calling the country poor.

Here is my suggestion -- don't use outsourcing agencies (acknowledging, easier
said than done.) With an agency you end up with invisible HR problems. When
something weird is going on at the office, you won't know about it. If senior
developers are slacking off and giving junior devs all the work, you will not
know. Hire the developers directly, give them the full salary, listen when
they ask for something, give them flexible time off.

If what they are working on is really important, take the time to fly and go
meet them in person, or if you have a team fly everyone to the same spot.

------
jister
"But conversely, the Microsoft stack that I worked in and we were increasingly
unifying around in Pfizer was simply much harder to find competence in"

That's probably because the developers that are strong in Microsoft stack were
already out of the country most likely pirated by companies in Singapore or in
other parts of the world. Majors banks in Singapore had been pirating IT
people in Philippines from QA to Project Managers for years.

~~~
_0nac
Did you mean "poached"? You can't pirate people...

~~~
rwallace
Let's not tolerate the use of "poached" either; it implies (in a "ha ha, only
serious") way that people are property, and it's been a long time since there
was any excuse for tolerating that idea. The proper words are "hired" or
"recruited".

~~~
bulatb
It also implies it's somehow _wrong_ to offer an employee of another firm a
better deal than they're currently getting.

Well-mannered firms don't do that sort of thing, you see. It drives the wages
up.

------
bewresu
the Philippines:

"one of the starkest patterns I saw outsourcing there was their strong
affinity to PHP. This isn't intended to be a derogatory... the LAMP stack is
really appealing to folks who don't have much cash... the Microsoft stack...
was simply much harder to find competence in. You'd go to a vendor and their
default position was 'Yeah, we can do that in PHP and MySQL'"

~~~
arvinsim
Java and PHP are traditionally the most popular introductory languages being
taught in CS courses (CE students are usually taught C).

I see some schools switching to Python as an introduction course. But
generally, the pace at which the educational system move to update their
curriculum is glacial. Heck, I even see some schools still teaching VB6!

------
kabisote
"Vendors don't build software, people build software"

Working with vendors is logical for big projects but an obstruction for small
projects. If we eliminate the middleman, outsourcers would pay less and
developers would be paid more. There'd be less bureaucracy. There'd be greater
loyalty.

I hope there was an easier way for outsourcers and developers to find each
other.

~~~
lisivka
It's called «freelancing». It's way cheaper and worse than outsourcing.

~~~
asenna
Any reason why you consider freelancing worse than outsourcing? I think if you
can build a good team of freelancers, you basically save yourself from a lot
of the issues this article mentions.

~~~
lisivka
If you CAN build good team of freelancers, you can start your own outsource
firm.

------
perseusprime11
Interesting article. Is the code written in China considered open source for
all practical purposes?

~~~
Roboprog
Yes, but it's a very specific "All your databases are belong to us" license
:-)

------
perseusprime11
As I was reading this article and watching the head shake video, I clicked on
this other video which I thought was hilarious:
[https://m.youtube.com/watch?v=k2W8aGgmn1A](https://m.youtube.com/watch?v=k2W8aGgmn1A)

------
elango
Well written and balanced. Clearly steered away from conflicting topics.

------
meganvito
The flow of capital tends toward a low hydrolic potential from a high end.
When the potential is lost, things turn to medicore, people just look around
for an exit.

------
quantum_state
Very nice article n much on the mark!

------
clifanatic
I can't believe he skipped over the real elephant in the offshoring room - the
time difference. All of those countries have almost no overlap with any part
of the United States. If we schedule an 8 AM CST meeting with India (which is
super early for some of the people here), it's already 7 PM at night there,
and they're completely exhausted and ready to go home. And that's _if_ you can
schedule a meeting far enough in advance that somebody else hasn't already
taken up that precious 8 AM spot... usually you'll have to go out later in
their day, sometimes as late as 10 PM there. Which really hurts when you
consider how many of them rely on public transportation... which stops running
before 10 PM. I've been told that some people have ended up sleeping in the
office rather than push back on a very late meeting request.

~~~
CountSessine
Here's a question. You're in the US and your employer has decided to out
source some work.

Why are _you_ , the American worker, tolerating your employer's request to
stay late to talk to their outsourced lackeys? Whenever I've been asked to
stay late for a time zone crossing meeting with outsourced employees, I've
refused. _WE_ are _THEIR_ customers, not the other way around. They can damn
well get up at 3am and WebEx-in if we're going to have a meeting. No way am I
going to stay late in the evening and miss my family so I can chat with our
offshore labour.

~~~
dmoy
Well, in some cases people will do it because otherwise their employer may
fire them. Living in the US most of us are at will, and don't have that much
luxury to give employers the finger.

~~~
CountSessine
What's your value to your employer?

If you're not willing to stay late to do a cross time zone meeting then I
agree that you're worth a bit less to your boss than you otherwise would be.

Personally I would rather accept a bit less pay to be rid of a nuisance like a
boss who thinks I'll inconvenience myself and my family for stupid shit like
adjusting myself to his outsourced labour's schedule. I _do_ stay late when
shipping a release or working on an urgent bug. But stay late so some delicate
flower off in India can work 9-5? Fuck that.

~~~
acd10j
So you Americans now are embracing being full-time assholes after electing
trump.

~~~
wavefunction
You seem to be asking if we're joining you?

The USA is a country of some 300 million people, and if you think the 20% of
the country's population that voted for him somehow represents everyone here
you're completely wrong.

------
kneegrown
His writing is God awful and a huge bore. I'm interested in the topic but
loathe his writing.

------
r3dn3r
What is he talking about? Indian with good english skills?? What? Good with
computers?? When people will end all those prejudices ???

