Lessons of Failure
Humans + Software Development = Always Interesting

TAG | outsourcing

Jun/10

9

The Outsourcing Low Cost Lie

If you walked into a store and asked to have someone make you a suit and you agreed on a price of $100 and week’s sewing time, a week later you’d expect to walk back in and be trying on your new suit after parting with a $100 bill (at least in America).

What if you went into a different store, after that initial price quote and they offered to make it for $25?  You’d think, “Great!  One fourth the cost of that previous guy!  I’ll take it instead.”

How would you feel if a week later you came back and they said it would be another week and $50 instead?  Think you’d be mad?

How much madder would you be after TWO weeks, and now the shop owner is telling your it will be $75 and one more week, but this time he’d definitely get it done.  And after you get it, you notice that the pocket is sewed on slightly funny and the trousers don’t fit quite the way you’d expect.  Would you be steaming mad now?

Yeah, I would too.

This is exactly what happens with outsourcing projects in most software companies.  If you ask companies why they do it, invariably they answer that outsourcing will save money AND time over local resources.  That’s an unbelievably huge lie. It’s time to tear that apart.

Forgetting the cultural problems you’re going to encounter with outsourcing for a minute, let’s say you have a project scoped by an outsourcing firm.  They bill their engineers to you at $25/hour.  Local contractors run you $100/hour, so you’re thinking to yourself, “Wow, 75% cost savings!  Sign this outsource guy up!”

Whoa, not so fast there, cowboy.  Let’s throw out some project statistics from the Aberdeen Group.  Unsurprisingly, reducing IT costs is the primary driver behind outsourcing for 82% of companies in the U.S.  But, as Aberdeen continues,

  • Nearly 50% of outsourced projects fail outright, or fail to meet expectations
  • 76% of companies said that vendor management effort and costs were much higher than expected
  • 30% reported ongoing issues with outsourcer management processes (e.g., inadequate governance and conflict resolution procedures)
  • 51% reported that outsourcer was not performing to expectations

In the end, the average cost savings for projects was a mere 26%.

That might sound like a reasonable number, but consider that first point more closely:  Nearly 50% of all outsourced projects fail outright or fail to meet expectations in the first place. Essentially, you’re taking the same gamble as red vs. black in Roulette about your project’s success right off the bat, and only then if you pass that hurdle, you’ll get on average, 25% savings over having it done locally.

So wait a second, where did my 75% cost savings disappear to? Let’s do the math:

Original project cost (outsourcer estimate): $10,000. (400 engineer-hours @ $25/hour)

Actual project costs: $30,000 (50% overrun on time and 100% overrun on people–longer than expected, 2x as many engineers as originally scoped, which is another finding of Aberdeen’s and other companies’ experience, so 1,200 engineer-hours @ $25/hour)

Unexpected onshore management costs: $6,000 (about 20% of project cost, from spending more time managing expectations, requirements, design reviews, documentation, etc)

Total actual project cost: ($30,000 + $6,000) =  $36,000.

Estimated project cost, if done locally: $40,000 (400 hours @ $100/hour)

Expected cost savings: $30,000

Actual cost savings: $4,000  (($40,000-$36,000)/$36,000) = 11% (for the clients I’ve worked with)

These numbers have been vetted by two recent clients of mine (due to NDA restrictions, I can’t mention either by name but one is a large entertainment conglomerate based in NYC, the other is a worldwide financial clearinghouse firm).  Both have substantial amounts of outsourced projects and both report little actual cost savings shown by front-line managers, huge management issues, and non-trivial project failure rates.  And yet, both continue to outsource because upper management believes they’re saving a lot of money through outsourcing.

Think about this for a minute: If you have 4 projects you outsource (let’s assume they are all the same:  400 hours each, and we’re using an outsource firm at $25/hour), their estimated cost of $10,000 each, and their actual cost $36,000, here’s what you get:   Your budget was for $160,000 and you believed you’d have $120,000 to spend on other projects (foolishly).  Here’s the harsh reality:

  • 2 of your 4 projects have high probability of completely failing (let’s assume that they didn’t expend the same cost above, but perhaps only half before they got canceled, so say you got halfway through each and canceled them, burning $36,000 total between them)
  • Of the remaining two, you get what you asked for but at a cost of $72,000 instead of the promised $20,000.
  • Total budget spent:  $108,000.  Expected budget spend:  $40,000.  (Difference:  +270% over expected costs)
  • Actual remaining budget for other costs:  $52,000.  Expected remaining budget:  $120,000.  (Difference:  -43% less than expected available funds)

Assuming you budgeted all your expected remaining project cash on other things, congratulations, you’re now over budget!  Welcome to every CTO and CIO’s nightmare.  And none of this addresses the other problems:  communication lags of several days to answer questions, lack of face-to-face interaction to solve problems quickly, miscommunication over simple requirements that you consider obvious but were missed in the implementation, and so on.  Are you really saving money here?

Let’s call a spade a spade:  Outsourcing just doesn’t save you the kind of money you think it will.

UPDATE:  Thanks for all the comments about my crappy math.  Yes, it’s fixed now.

· ·

Outsourcing:  The word American developers love to hate.  There are lots of stories out there about failed efforts that involve offshore development (“offshoring”).  I even have a few myself.  But this post is not about bashing outsourcing countries, the cheapskates that hire them, or the project managers who can’t control the resulting chaos.  This is about understanding why we have such a difficult time making offshore development work before any of those folks get involved.

Airline Disasters and PDI

What got me thinking about this subject was the book Outliers:  The Story of Success, by Malcolm Gladwell.  In it, Gladwell discusses the reason for a series of catastrophic airline failures.  Without repeating most of his excellent dialogue in the book, here’s the Cliff Notes version:

  • There was a study by Geert Hofstede, where he tracked the Power Distance Index (PDI) among selected world countries.
  • The Power Distance Index is an incredibly important measure of how a person in a country would generally react to an authoritarian figure.
  • Countries with high PDI would have more people willing to accept an authoritarian power figure in a paternalistic sort of way.  Like how you might defer to your father’s decisions, or to a king’s requests, for example.  People in higher PDI countries are less likely to question authority and more willing to accept instructions from those in higher positions of society.
  • Lower PDI countries are characterized by subordinates being more willing to question the orders of a superior.  Low PDI countries have people that tend to view themselves more like those in power than not.  In other words, you might judge yourself to be much like your boss in a low PDI country.

Crashes Caused By Power Differences

The NTSB regularly investigates airplane crashes to determine causes, but investigators were unnerved in the late 1990s to find a pattern of fatal crashes were found to be very airline (and also country) specific.  These two airlines with frightening records were Korean Airlines and Colombian Airlines.  The fatal crashes all had common attributes:

  • The pilots didn’t make any single fatal error.  They all started with several smaller errors that gradually built up to a catastrophic failure.
  • For any airline, the First Officers are always trained to double-check the Captain’s orders.  This is a safety protocol that prevents single-person failures from happening.  It is the First Officer’s duty to question and even override the Captain if the order he gives is unsound or improper.
  • In these particular cases, the black-box recordings indicated that the First Officers were hinting at problems, but were not strongly identifying them in a way that would make it very obvious.  Nor did they explicitly tell the Captain that he was acting against normal procedures.

Power Means Fear of Communication

In the aftermath, an astute researcher matched up the Hofstede PDI data with these two countries and found that both Korea and Colombia are high on the PDI scale.  What were the implications of this?

  • Clearly, a power difference would have existed in the cockpit between the First Officer and Captain.
  • The manner of communication between superiors and subordinates is very different if you are from a high PDI country than a lower one.
  • In a high PDI country, a subordinate must use the proper language, body posture, facial expressions and tone when communicating with a person of higher status or power.  In Korean, for example, there are no less than six distinct ways to address someone from the most formal to the least.  In the United States (a lower PDI country), we have a much looser style (“Sir”, which could apply to a General as much as a Lieutenant in the military, or even your father in some situations).
  • This disparity of power created hesitance on the part of the First Officer to embarrass his Captain when the Captain made mistakes (and in each of these cases, at least 3 mistakes were made).  So instead of using a direct method of communication (“This weather is really bad, we should turn back immediately or regroup for another airport”), he used a more subtle, formal and proper one (“Sir, look how it is raining outside.”).
  • The implication of a high PDI country is that there is a rich subtext going on between the two communicators.  But that subtext assumes that both sides are alert, paying attention and can clearly understand the implicit signals.
  • In the case of the airlines, the Captain was almost always sleep deprived and exhausted by the time the situation arose, making that communication impossible.

The researcher concluded that the pilots were inside a cultural framework that dictated how they should behave at a time when those behaviors turned out to be detrimental and outright dangerous to the safety and welfare of the passengers.  In other words, the fact they came from high PDI countries made it impossible for the proper communication to take place when it was most necessary to be plain and step outside the traditional power roles of these cultures.

Past Outsourcing Blames

What does all this have to do with outsourcing?  First, we need to understand why outsourcing has traditionally “failed”, according to both buyers (those who purchase outsourced services) and providers (those who perform the service in their local country, or send people to other countries to perform services).  Here’s a graph of combined data about a survey regarded failed projects from 2004.  The data are still relevant to today’s discussion.  (Source:  The Outsourcing Center)

Reasons of Failure for Outsourced Projects

Reasons of Failure for Outsourced Projects

Notice the even distribution of reasons once the provider & buyer survey data is combined.  This is interesting because neither party can clearly point to a single, differentiating causation factor in the failure of outsourced projects.  But I believe that’s because they asked the wrong questions in the survey.

The Real Issue with Outsourcing is Power Difference

If you have a buyer from a lower PDI country and a provider from a higher PDI country, there are already implicit consequences to your interaction that neither party will know about without prior outsourcing experience or natural cultural awareness(1).  And even with that experience, it’s not a given that they will understand the reasons behind the challenges of outsourcing.  Let me create an example from my own personal experience:

Suppose you had an American company (Buyer) and an Indian company (Provider).  The American company contracts with the Indian one to provide offshore outsourced software development at a fixed price per developer.  Certain key performance indicators are agreed upon by both parties and the game is afoot.  Let’s also assume the Indians agree to a six month project to write a content management system for the Americans.

A typical scenario of engagement might follow like this:(2)
  • The first month, everyone hammers out the requirements documents and in a great ball of fury, declares them sound and ready for implementation.  The American company at this point would typically reduce the daily oversight on the project to something more reasonable, like weekly updates.
  • The second, third and maybe even fourth months pass with little fanfare.  The Indian developers are quietly building the specified software and the Americans are receiving updates about it that are all positive and sound great.
  • At some point, the American company asks for a demo of the progress to date.  The Indians put together something after a bit of negotiation (since the Americans neglected to mention the demo as a deliverable before the end).  The Americans see the actual software and fly off the handle.  Performance is awful, the screens don’t look anything like what they want, and the software appears to be behind schedule.
  • Further code reviews by American developers indicate that the code quality is fairly poor, lacking in comments, unit tests, and filled with copy-paste blocks of duplicate code.  The Americans immediately demand the project be put under different management.
  • The project falls off of the rails somewhere after this.  It will either be canceled, brought back in house, or will be delivered extremely late after extensive modification to the original requirements.

There’s lots to pick on here on both sides of the table.  I would like to point out that the fact that I picked on Americans and Indians is actually irrelevant here. You could easily substitute “British” for Americans (3), and “Filipinos” for Indians with the same results.  But why are they so interchangeable in this fashion?  It’s because of PDI and the inherent cultural communication issues that come with it.

Dilbert

Dilbert says it best

Here is a list of the top 10 Outsource Providing countries in 2009, and their PDI scores.

  1. India (77)
  2. Thailand (64)
  3. Mexico (81)
  4. China (80)
  5. Indonesia (78)
  6. Malaysia (104)
  7. Philippines (94)
  8. Jordan (no data)
  9. Egypt (80)
  10. Bulgaria (no data)

For reference, the United States is 40 on the scale.  Western countries can run the gamut as high as Belgium (60) to as low as Austria (11).  The scale is from 1-120, where 120 is extremely high PDI.  You can see all the countries’ measurements in the original study on this colorful world map of PDI indices.  The gray countries are ones that weren’t measured.  India would be considered moderately high PDI at 77 (in the 61-80 range).

So what happens when you bring a low PDI buyer together with a high PDI provider?

In a word:  Disaster.

Cultural Context Matters In Communication

Each side expects a certain subtext to go on during a conversation because of their own cultural context.  Like this:

Low PDI Manager: So, is the new website ready for launch by Friday?

Low PDI Developer: No, and it’s going to be another 2 weeks because we need the new servers to arrive, for QA to finish with testing after they do, and then release the code.

Pretty straight question, pretty straight answer if you’re from a low PDI country like the United States.  There is little assumption about the subtext because a low PDI communicator is used to “speaking his/her mind” about it.  The information is supposed to be in the conversation as spoken words.  If it’s not there, it’s ignored.

But what if we change that a bit?  Assume the High PDI and Low PDI Developers BOTH have access to the same information and are equally competent:

Low PDI Manager: So, is the new website ready for launch by Friday?

High PDI Developer: Yes, it may be ready by then.  We are looking into it.

That seems like a bad answer if you’re from a low PDI country (mostly because we know the context from the first scenario), but it may be taken at face value because the Low PDI Manager expects straight conversation.  If there was a problem, the Low PDI Manager expected the developer would say something specific about it.  When they didn’t, the Low PDI Manager assumes that Friday will be the date.

And what about the High PDI developer?  He didn’t want to offend the Low PDI Manager, because that’s what you are careful to do in a high PDI country.  The High PDI Developer assumed that the Low PDI Manager would understand his subtext “may be ready” and either ask further questions, or understand that Friday wasn’t necessarily a realistic date.

This is just the tip of the iceberg.  If this happens on a simple conversation about a deadline, what about really big stuff like:

  • Requirements
  • Deliverables
  • Quality control testing
  • Development standards
  • Documentation

The implications are literally staggering.  In fact, I’d go so far as to say the fact that every outsourced project hasn’t failed is something of a miracle.  It’s a testament to having the right people who naturally and instinctually bridge these gaps through extra communication.

The Survey, In A New Light

Getting back to the survey questions, if you look at all of them and how they would be viewed relative to PDI, it’s arguable that PDI differential is the one, single, leading cause that relates to how providers and buyers have a hard time seeing eye-to-eye during the outsourcing process.  Of the eight named factors, I can see 6 of them that directly relate to PDI differential:

  • Buyer’s unclear expectations up front (buyer assumes he is understood when the provider stops asking questions, but that’s a typical low-high PDI interaction)
  • Poor governance (see my deadline example above)
  • Poor communication (again, the deadline example)
  • Poor cultural fit (again, the deadline example)
  • Interests become misaligned over time (you don’t understand each other’s communication needs and are frustrated)
  • Not mutually beneficial (you can’t work together because you don’t understand how to interact…)

Adding those 6 factors up, 72% of project failure reasons can be connected to PDI differential. If both sides understood that single factor going in to the process, everyone would be better served in the end.  Think I’m just making this all up?  It’s not just the buyers that complainHigh PDI country providers say the same things.

So what’s so hard about outsourcing?  It’s hard because of the cultural baggage we bring to the table on both sides, and neither side necessarily realizes it because of assumed interactions.  We need to be more aware of the cultural assumptions going in to projects like this, or we’re doomed to repeat them ad absurdum.

(1) I think it’s fair to say that most other countries would NOT say most Americans are blessed with “natural cultural awareness”.  🙂

(2) Before I get lots of angry comments from Indian readers about the interaction above, yes, there are other potential outcomes and perhaps you’ve been on projects where they are all successful.  I have a mixed bag of experience on this, and it’s not about bashing Indian developers.  Like American ones, they run the gamut–good, mediocre, and what-the-hell-are-you-doing-coding.  I’ve run into all three in about the same proportions as American developers, more or less.

(3) I’m sure one or more British readers are horrified at thinking they are interchangeable with Americans at this point.

Outsource Rank Country PDI
1 India 77
2 Thailand 64
3 Mexico 81
4 China 80
5 Indonesia 78
6 Malaysia 104
7 Philippines 94
8 Jordan
9 Egypt 80
10 Bulgaria
11 Hungary 46
12 Ghana 77
13 Pakistan 55
14 Chile 63
15 Poland 68
16 Czech Republic 57
17 Argentina 49
18 Romania
19 Ukraine
20 South Africa 49
21 Russia
22 Vietnam
23 United States 46
24 Israel 13
25 Canada

· · ·