Outsourcing Nightmare: Serial Communications

I worked on a handheld bar code scanner and built the SDK demo, support programs, and all of the product documentation. At one point I was building a serial communications interface and my protocol analyzer showed that something in the data stream was bogus. Earlier in the project the firmware and SDK was written and maintained by a senior in-house engineer. The company bean counters then foolishly outsourced the low-level engineering tasks to “senior” engineers in India. This is a tale of two engineering philosophies.

In-House Engineering

When the engineering was done in-house I walked to Dave’s desk (the senior engineer) and told him of the errant bogusness in the data stream. He walked to my desk, looked at the data on my protocol analyzer, and I explained the problem. He went back to his desk, patched the problem within about 3 hours, took another few hours to test and bullet-proof the code, and by noon the next the day the problem was fixed, verified, and demo programs were working. It took me 2ish hours to find and isolate the problem. It took Dave maybe 5 hours to fix the problem. It took me perhaps another hour to test and verify the fix. That comes to a total of about 8 hours. If our burdened rate is $80/hour then it cost about $640 to fix the problem.

Outsourced Engineering

When a similar issue arose with outsourced engineering the resulting engineering mess and kludging was truly astounding. This is a rough idea of how the problems generally went. The outsourced engineers did not have good command of the English language so I spent probably 4 hours detailing the problem. I had an old protocol analyzer that would not output data so I had to transcribe the data stream and explain the issue in extreme detail. I then sent the description to India. Of course they are halfway across the globe so it took 1.5 days to get a response. Of course they needed further clarification and I spent another 3 hours writing even more detailed descriptions of the problem. Another 1.5 days went by and I received a software “fix.” After 10-minutes of testing I broke the “fix” and spent another 4 hours describing why their fix did not work. After another day I got a response back asking for further clarification. After spending another hour giving them even more details they sent a fix a couple of days later. After testing I found that the fix worked. With outsourced engineering I might spend 12 hours with the descriptions, the testing, and the back-and-forth ($80/hour). They probably spent 8 hours doing software work ($20/hour). The total cost for the fix would be something like $1,120 which is about twice as much as it cost with the in-house engineer.

Conclusion

The in-house engineer got the problem solved the next day and took significantly less time than outsourcing. The outsourced engineer took more time, cost more money, and our customer had to wait over a week for a fix. Our customer was completely pissed off at us and we lost all future business. I was frustrated at having to explain, re-explain, test a fix that obviously did not work, and the company spent more money.

Outsourcing rarely works well. If you have a specific outsourced team that you have been working for a long time then it can work. If you want a quick job done then it normally works about as well as hiring an inexperience laborer to do fine workworking. If you want headaches then, by all means, do the outsourcing. If you want it done right then get an experienced geezer.