Jedi Software Training–Part 2

So far we’ve established a few things from the last post on Jedi Software Training:

  • We already implicitly and perhaps accidentally practice the Apprentice/Master model in software development today.
  • The route to Software Master is paved with ambiguity.  Most who achieve it can’t tell you how they got there.
  • A Software Apprentice often flails around in a sea of information without much guidance unless they are insanely lucky in their first job.
  • We need a systematic way to train apprentices and grow the craft in a reproducible manner.

Someone at this point might be saying, “Don’t we already have internships?  Isn’t that enough?”  Let’s distinguish an internship from an apprenticeship.



Exploring career option?



Short time frame?



Project-based opportunity for career exposure?



Employer is looking for permanent hire?



Provides training in all aspects of field?



The important difference here is about goals and time frame. The internship is about checking things out and deciding interest, the apprenticeship is about getting serious and learning the craft from start to finish with the intent of employment.  So no, the internship doesn’t address the need in the same manner.

To create successful apprentices, we need several things:

  • A road map to understand the most important aspects of the craft
  • A master who already understands the map
  • A place where the master practices and can freely train the apprentice

Who, what and where.  The how is being addressed here, and the why was already established in the last post.  Let’s focus on who now.

The Ideal Master

Taking a page from the book of Star Wars:  To train a Jedi Apprentice, we first must find a Master.  The master clearly must know software engineering inside and out.  They should be versed in every aspect of project development from soup to nuts:  analysis, requirements, design, development, testing, release and maintenance.  Ideally, they should have practiced for 15+ years and on a wide variety of projects–from small to large, from internal to highly public, and from successful to dismal failure.

A master should be fluent in many tools, languages and frameworks.  A good master is productive no matter what tool they use once they practice for a short time, given their vast prior experience on other tools.

A master must be able to communicate difficult concepts with simple and easy-to-understand analogies or other descriptions that even the newest apprentice can grasp.  True mastery of material means the ability to describe it in multiple ways.  The master should have infinite patience to describe things in various ways to give the apprentice many different ways to understand new concepts.

Most importantly, a master must be fluent in the business of software, since many software decisions are driven through the business, not the craft of software itself.  A master that cannot understand the balance of these two forces is no master at all.

The Path to Mastery

To turn an apprentice into a journeyman, we need a training program.  Electrical apprentices currently mix actual practice on the construction site with nighttime course work in electrical theory, such as the study of Ohm’s Law.  A similar program for software could be useful:

  • How to use the tools of the trade (e.g. IDEs, makefiles, testing scripts, automated build & testing tools, source control, etc)
  • Understanding the theoretical side of software engineering:  requirements analysis, functional decomposition, design practice, project scheduling, developer estimation, and so on.
  • Putting these theories into practice using a small scale, but real, project

Some of you may be thinking that this sounds precisely like a university program for computer science (at least in part).  Most computer science curricula are geared toward creating academics who are skilled in the field of computer science.  The fact that software engineers might pop out on the other end of the coursework is an afterthought.  It’s the same distinction between physics majors and mechanical engineers:  science vs. applied knowledge to practical situations.  Clearly we want the practical here.

The Master’s Workshop

There seems to be no perfect environment in which to train the apprentice in a modern setting.  This, to me, is the crux of the problem.  There are several potential possibilities, but each fraught with peril.  Let’s examine each of them individually:

  • Universities
  • Open source projects
  • Companies of various sizes

The University Setting

The first and most obvious is university setting.  But universities already have problems as a training ground for three major reasons:

  • Most professors are professional academics without the benefit of real-world software engineering experience in a business setting.  This makes them less than ideal Masters, even though they may be quite accomplished in their own right.
  • Their goal as academics is to create more academics, not to create masters in the field.  Generally, they prefer the theory to the practical aspects.
  • The end result of a professor’s research is a published paper, not a finished software product.  Often time, prototypes are more than enough for them.  Prototypes are not good examples of production grade software.

Dude, what about Summer of Code?

An excellent question:  What about Open Source projects such as  Google’s Summer of Code or the Apache Project?  There’s an unlimited number of them, creating limitless opportunity for apprentices.  They sure seem like a great place to start, but:

  • Most of these projects are working remotely, using distributed development techniques.  A master/apprentice relationship functions best when the master and apprentice are close in proximity to keep the apprentice from getting too far into the weeds.  SoC and other such setups do not offer that proximity and many apprentices are generally unprepared for that level of freedom, at least initially.
  • An apprentice is trying to develop confidence in their skills.  These projects often assume some self-taught skill with the tools involved.  Newcomers are often ridiculed and derided for lack of understanding of even basic tools.  That doesn’t foster confidence in apprentices.
  • Some are treated more like internships (e.g. Google Summer of Code).

Be A Company (Wo)man!

That leaves only one place left as fertile ground for training:  technology companies.  But which ones–large, medium, small, or startup?  Let’s make a case for each. futileLarge companies (more than a few thousand employees) seem to be a good choice:  well funded, plenty of good facilities, and larger numbers of job openings per year.  Yet I don’t think this setting would enable the craft of software to flourish and expand for several key reasons:

  • Large companies tend to hire large development groups.  Inside these large groups, mediocre and completely untalented engineers can reside undisturbed for years because of the inertia of large organizations to get rid of them.  This unseemly presence of incompetence could give the apprentice the wrong impressions of development techniques if left unchecked.
  • Large companies tend to foster painful bureaucracies that would stifle the creation or gestation of such apprenticeship programs.
  • CEOs and technical managers of such companies are highly focused on budgets and value-add for every person in the company.  Apprentices are the very definition of unproductive learners, taking up substantial resources up front in order to become productive.  Managers would have to place personal reputations on the line in order to start and maintain such programs in the face of budgetary pressure.
  • Large companies have very siloed groups, sometimes with company-specific processes, tools or standards, that would expose an apprentice to a very narrow view of development practices, making them less marketable to the broader economy as software engineers.

Medium size companies (say between a few hundred and a few thousand employees) would face some of the same pressures as large ones, but the right company with the foresight to create such a program might be successful.  It comes down to the courage of the management and the strength of will of the development team to make it happen in the face of current company policy.  However, in this current economic climate of 2009-2010, few companies would take such a chance, in my opinion.

I think the biggest bang for the buck will come from small companies (less than 200 employees, but more than a dozen) for several reasons:

  • Small companies because of their size must attract the best talent in their development teams.  Small companies don’t have time or money to waste with unproductive people. But they are usually willing to develop good potential talent because of the value proposition.
  • Small companies tend to be nimble in their thinking and can change their internal culture to support such a movement the easiest of the 3 described so far.
  • A smaller company can use an apprentice more easily across groups and teams, making more efficient use of the apprentice’s time and energy, and exposing the apprentice to more of the business of software.

But it’s not all sunshine and roses either:

  • Small companies fight tooth and nail for whatever money they get.  Budgets for developers are tough.  Add an apprentice to this and it gets even tougher.
  • Many small company developers are often over tasked in their current roles, making it harder for them to act in the role of a master developer.  A part-time master is a difficult and potentially inconsistent kind of teacher.

Finally, we’re left with startups (less than 20 employees).  While they have some of the small company advantages such as agility and talent attraction, I believe that the chaotic environment of startups is best left to more experienced engineers:  journeymen or at the very least, previously trained apprentices.  Startups rarely have the time to spend training new people in techniques beyond those that implement the vision of the founders.  And what meager monetary resources exist in small companies, even fewer are available with startups.

In short, the apprenticeship idea requires tremendous courage on the part of the company that fosters it.  Small companies are used to taking big risks with substantial payouts in the future.  I believe small companies are the ideal place to grow this notion and allow it to root.

The Bottom Line

Creating an apprentice program would require jumping a number of substantial obstacles, not the least of which have been enumerated here.  But the company that creates such an environment would become a magnet for the latest talent in the industry because so few places offer the right opportunities to train new graduates into real software engineers through a systematic approach.

Jedi Software Training–Part 1

Software engineering is perhaps the youngest of all the engineering disciplines (and some would even argue, we don’t practice an engineering discipline at all).  But like all disciplines, an engineer must be trained in order to achieve a level of competence to practice their craft with any proficiency.

The Fresh Out Of College Problem

Software engineers run the gamut–amazing, mediocre, and step-away-from-that-IDE-before-you-hurt-someone levels.  Some are naturals and some probably will never be able to program their way out of a wet paper bag.  In terms of how we get trained, it’s all very informal in most cases.  Universities rarely focus on any software engineering courses as a serious part of a Computer Science program.  Indeed in my own Alma mater, Computer Science was part of the Engineering School but did not constitute an official engineering discipline.  The focus is entirely on data structures, programming, algorithms, and nuts-and-bolts sorts of topics.

Don’t get me wrong–those topics are incredibly important, but almost every college graduate I’ve ever worked with directly out of school (within the first 3 years of their graduation) generally has poor knowledge of how to run multi-engineer, moderate-size software projects.  I’m talking about simple things that are bread and butter for a successful complex project: source code control, project module management, requirements definition, functional design, and documentation.  This seems like a huge gap in training, considering that everyone knows these things are absolutely essential when they get out of school.  Employers are frustrated because graduates are unable to take the reins of a software project in a meaningful way without starting out in (very) junior positions that are rarely available and only modestly tolerated by most companies.  Graduates are frustrated at their lack of opportunities because they don’t have enough ‘real world’ experience to qualify them for real software development positions.

The question is, why are we training our engineers so sloppily today?

The Apprentice Model

Use the Force, DudeEngineering training and Star Wars Jedi Knights have something in common:  they both start out as apprentices.  Let’s look at the history of this practice and see how it can apply today.

Back in the Middle Ages, if you wanted to practice a craft, such as blacksmithing, baking, masonry, or butchery, you needed to become an apprentice to a master craftsman.  That master had been practicing for years, had a demonstrated level of mastery, and often belonged to a guild of other masters who judged this master to be fit enough to earn that title of master in the first place.

Apprentices lived a hard life back then.  They were exploited as cheap labor for the master for a long period of time (5-7 years was typical).  The master provided tutelage in the craft, food, and lodging in exchange for their work.  After this period of time, the apprentice was shown the door and expected to fend for themselves as a newly minted journeyman.  Journeymen could either practice in solitude or work under a willing master (if available).  After some long period of time, the journeyman could try to produce a masterpiece of his or her craft in an attempt to demonstrate mastery, join a guild and attain the rank of master.

Some guilds were comically harsh in the training methods.  For example, up until 1791 in France an apprentice worked under a master for a long period of time through their journeyman rank.  If they failed to produce a masterpiece during their journeyman period, they were subsequently executed.  I, for one, am glad we don’t train people in this manner anymore.  The labor shortages this could create aside, the pressure to produce would be excruciating.  And in truth, not everyone is cut out to be a master.

Modern Apprentices

Apprenticeships are still used as a model for vocational training in many professions in a number of countries (in America, we still use this for plumbers, electricians, and carpenters among many other trades.  Across Europe, this is true as well), this training serves to bring new workers into the trade, give them baseline skills in a controlled environment under a specific instructor, and a route to advancement in the field.  But this formalization happened over years when best practices were easily formulated into codified documents and classroom formats, allowing the apprentice to reproduce a master’s work in a repetitive and formulaic manner.

If we characterize the differences between the three levels, you might see the following:

  • An apprentice can take a specific set of instructions, created by journeymen or masters, and faithfully reproduce the steps to create a result of lesser quality than a master or journeyman.  An apprentice is unable to work without a concrete framework of rules to abide by while practicing the craft.  Apprentices generally need constant guidance and intervention from others in order to complete a project successfully.
  • A journeyman is an apprentice who has completed their training period and achieved a baseline level of skill in the trade.  They are competent enough to work alone, but most often seek the continued education under a master to improve their skills.  Journeymen are comfortable in a wide variety of techniques as taught by the master, and many of these techniques are now second nature to the journeyman.  However, journeymen lack the ability to create new skills or perform the baseline skills in the effortless manner of a master.  Journeymen generally can do a skill, but have trouble expressing the exact reasons why one skill is preferable over another in a certain context outside of the rules given by his or her master.
  • The master is the culmination of years of practice into “effortless skill”.  The master’s abilities require no conscious thought to manifest.  Masters have the ability to see patterns in projects and skills, arbitrarily combining things into new and unique ways of using them, often pioneering new techniques as a result of this artful mixing.  Masters understand the rules to the point where they can break them at will, knowing which limits are completely arbitrary.  A truly excellent master has the ability to teach their skills to apprentices in a way that creates excellent apprentices.  These masters are exceedingly rare.

The apprenticeship method is a time-tested, battle-worn method for bringing people of low-to-no skill into a field and training them to a level of success. Dozens of fields practice this, and almost all formal engineering disciplines have a similar model (engineer-in-training and P.E. (professional engineer) certification).  Academics have similar models to produce scholars, with undergrads promoted to graduate students, then post-doctorates and finally becoming associate professors, full professors, and professor emeritus.

My question is:  Why don’t we do this with software engineers?

Apprentice, Journeyman, Master – The Path Less Traveled

The path from apprentice to master must be reproducible and documented through time in a formal manner by previous masters.  Developers today are largely left to hack out their own destinies at random with high variability in the results.  If they inadvertently become masters, they attribute it to their own skills rather than the lucky circumstances they managed to find themselves in early on in their careers.

Transferring that mastery becomes incredibly difficult when a master’s path lacks any formal system to follow along the way.  Software masters are revered for their superhuman abilities to swoop in to save the day on a doomed project through superhuman coding effort via nights, weekends, major refactoring on a level that ultimately undermines the morale of the rest of the team.  This “cult of the software hero” approach ultimately shrouds the master’s path rather than illuminates it for others to follow.

We know this model is still at work in software today, albeit informally.  We see the various levels at work in our own teams and workplaces.  Here’s a simple OO experience hierarchy that you probably can relate to:

  • An apprentice OO programmer will struggle with encapsulation and polymorphism, but can put together systems that have been well-specified.  They need highly detailed object designs to see the various interactions of a system come together with elegance.  Left without such guidance, they will inevitably create God Objects, Yo-yo Object Hierarchies, and other monstrosities.  Design patterns are something of a mystery to them.  Simple algorithms will require large amounts of effort to understand and encode into a language.  Their skills with the tools of the trade will be low to moderate (e.g. IDEs, source code control, automated build systems, bug tracking) and significant effort will be expended in using/learning them.  Design and architecture are generally beyond them.
  • A journeyman OO developer will understand the value of data hiding and be able to participate in interface designs for obvious parts of the system.  They are comfortable with the tools of their trade and know many time-saving shortcuts, along with several flavors of tools.  Journeymen are typified by rigid adherence to systems, tools, frameworks or architectures because of their comfort level with them (“The Golden Hammer” effect).  They will recognize some design patterns and have regularly applied them, but may not know all cases when they are applicable, or apply new ones with ease.  Journeymen dabble in architecture with increasing skill and ease, but complex problems are often met with complex, obscure solutions when designed by journeymen.
  • Master OO engineers can hear domain problems and suddenly see object models dancing in their heads before the description is finished.  Architecture is second nature and design patterns require little to no effort to apply or recall.  A master OO developer generally tends towards tool, language, and framework agnosticism because they understand their relative weaknesses and strengths, choosing only those tools that stay out of their way, make the most sense for the domain and allow for ease of creation.  A master’s trademark is the ability to create a simple and elegant solution to a complex problem.

So the $64,000 question becomes:  How do we create an apprenticeship model in software development that can succeed in cultivating better engineers? We’ll start to answer that in Part 2.

Software: Just Plumbing or Mad Science?

There seems to be a fundamental debate raging:  Is software more like mad science or plumbing?

This debate came to my attention via Mike Taylor’s article:  What Ever Happened to Programming? Mike’s argument is that programming is nothing more than plumbing today and it’s no longer fun.  He believes that there’s more fun to be had as a “mad scientist” developer, building everything from scratch with materials at hand.  So let’s look at the two major camps of this argument:

Software Developer as Mad Scientist

Mad Scientist

Reading the classics of software literature like Knuth, Brooks, et al, you get some notion of the programmer as hiding out in an arcane computer lab, late at night, pounding away at the keyboard as if to build some Frankenstein program.  Unlike Shelley’s creation, the outcome is far more benign and often even useful.  But the creation is always that:  pure construct from the mind of the programmer.  No assistance from the outside world aside from a few borrowed organs to create the Magnum Opus.

This image has historical truth in it–Knuth himself created TeX in a similar fashion.  Supposedly Woz and Jobs did the same with early models of Apple computers.  Every programming language we have at our disposal today clearly had some singular human force behind it:  Ruby, Haskell, Java, C, C++.  The list goes on ad infinitum.

These creations required intense and detailed knowledge of the hardware and operating system to create their monsters.  Whatever they required in their tasks, they often built from scratch by themselves.  They are the pioneers of our fields, the first wave of migrants on the digital frontier.

Without the Mad Scientists, we would be language-less, tool-less, and probably stuck with punch cards on ENIAC.

Software Developer as Plumber

I hear our job derogatorily compared to that of a plumber:  “We just put stuff together instead of build it”.  I don’t think that gives plumbing it’s due, nor does it really consider the rich history of the field.

So, I hear you've got a clog in your database...

Plumbing back in the late 1800s and early 20th century was a dicey business.  The entire practice was inconsistent, lacked any standard methods, and training was haphazard (for much more background, check out this article).

Appropriately, the National Association of PHCC (formerly the National Association of Master Plumbers), first met in committee in 1883 at the old Astor House, the hotel that provided the impetus to modern plumbing back in 1834. Many new plumbing inventions had appeared and too many plumbers were ill-prepared. Close on their heels would be the Mechanical Contractors Association of America, the American Society of Heating, Refrigerating and Air Conditioning Engineers and the American Society of Sanitary Engineering.

Wholesalers banded together, too, starting programs to prod manufacturers into standardizing such things as sink and basin outlets, faucet drilling, trap gauges, etc. The Central Supply Association, for example, was formed in 1894 and soon made contacts with the old Eastern Supply Association, the Plumbers Association of New England and the National Association of Master Plumbers. But it would take another 30 years to accomplish the standardization which everybody takes for granted today.

That means roughly the first 50 years of plumbing (1883-1925) was effectively like the Wild West:  Every man for himself and standards be damned.  The public often suffered as a result of this:

An outbreak of amoebic dysentery in Chicago during the 1933 World’s Fair was traced to faulty plumbing in two hotels. Tragic results were 98 deaths and 1,409 official cases. One year later, Major Joel Connolly, Chief Inspector of the Chicago Bureau of Sanitary Engineering, spoke these prophetic words:

“One of the lessons to be drawn from the amoebic dysentery outbreak … is that plumbing demands the very best, painstaking effort that thoroughly qualified, certified plumbers can give in every building, and especially where the systems are complicated and extensive, and where large numbers of people may be affected by contamination of water.” (emphasis mine)

Clearly standardization of materials, methods, and training gave plumbing a major shot in the arm for consistency, safety, reproducibility and public trust.  The plumber’s model started off as cowboy hacking of pipes in a haphazard way to a systematic method of standards, interoperability, training and licensing.  Along the way, there were glitches, problems, and issues.  Big surprise.  Sound familiar?

Our software legacy has taken us from raw register manipulation in assembly, through multi-generation languages (2GL, 3GL and god help us, 4GL), to huge amounts of frameworks, libraries and tools that give us unprecedented levels of productivity today that would be unheard of 10, 20 or even 30 years ago.  But the cost is that there is less of the low-level work to do, and more of the heavy-lifting at the business level.

We’ve complained, bitched, and moaned about spending too much time on things like low-level problems:  database connectivity, GUI frameworks, XML parsing.  And guess what?  People responded to those complaints by building libraries, tools and frameworks to make them happen. To give us what we always wanted:  the ability to focus on the business problems and not the lower-level constructs.  In essence, we’ve borrowed the plumber’s model.

You assemble the pipes, solder them together, solve local problems about how to route the sewer line around the funky wall shape, but you don’t get to set the pipe sizes, mold the elbows, or determine the ideal composition of solder for ease of melting.

You get to put things together for utility.  A large part of modern software development is nothing more than a utilitarian venture of “some assembly required”.

But even the first waves of migration to the American West by the military and trappers of the day had comparatively little impact to the settlers of the late 19th century.  And similarly, the mad scientists of software have a significant, but much smaller, impact in comparison to the plumbers of software.

So which are we?  Mad Scientists or Plumbers?

Both.  Neither.  It depends:  On who you work for.  On what you specialize in.  On where your interests lie.  There is a need for both:  mad scientists are the creators of new tools, frameworks, languages and OSes, plumbers are the integrators, the users, and the orchestrators.  Software requires both to survive.

This isn’t a question of what software is, but rather who you want to be.  But there are some facts that are hard to argue:

  • There are more plumbers jobs than there are mad scientist jobs.
  • The need for mad scientists seems to diminish over time, not because mad scientists are less important, but because more plumbers are needed once the mad scientists are done with their work.
  • It’s very hard to be a good mad scientist OR a good plumber, but they are vastly different skill sets.

Be an mad scientist or a plumber, but don’t complain when you’re a plumber but you really wanted to stay a mad scientist.  The choice is, and always was, yours to make.

Older Developers: Bad Habits Are Killing Your Career

OK, so my last post about five pervasive myths about older software developers was definitely getting a lot of:

Who do we appreciate?
Go old guys!

“Old guys! Old guys! Rah-rah-rah!”

in the comments.  And it wasn’t necessarily undeserved…after all, it was about debunking myths that have crept in as supposed truisms over the years.  But I left out a tiny little detail about something important.

Older developers are killing their careers from their bad habits.

Sorry, the sound you just heard is your jaw hitting the keyboard.  “What!?  But Dave, you said experience was valuable and…”

Yes, I know what I said.  And I meant it.  Every word.  But there is one distinct advantage the younger set has over us fogeys:  they haven’t formed as many habits yet.

I’m not talking about a $5,000-a-day-hooker-and-blow* kind of habit.  I’m talking about the practices that you’ve codified into your daily routines as a developer since you started.  Like your (in)ability to write clear, concise comments.  Or comments at all.  Your constant lack of communication with other team members when you’re making major changes because you don’t think it’s necessary.  Your refusal to write documentation.  Or your passive refusal to learn new technologies because you think you have enough information to do your job already.

These are all habits we’ve picked up over the years.  Some are good, like making sure you always have a bug tracking system in place, or using source control like a religion.  But not all of them are, like some I mentioned above.  If you’ve been developing for more than 10 years, you’ve got a mix of both.  Don’t kid yourself.  You get them out of sloth or complacency from doing the same things over and over.  You don’t bother changing them because well, they’ve worked just GREAT so far.  There’s no motive to change.  It’s Newton’s First Law of Motion applied to software learning:

Sir Issac Newton
Maybe I'll rest since I nailed that gravity thing...

“Any object in motion will tend to stay in motion; any object at rest, will tend to remain at rest, unless acted upon by an outside force.”

After you reach a certain level of competency, assuming you aren’t subject to the Peter Principle and haven’t been promoted out of your competence yet, your motives for advancement are reduced by your motives for maintaining the status quo.  You’ve been considered the Senior Software Engineer for 5 years now and you don’t want to become a Pointy Haired Boss anytime soon, so Senior Software Engineer looks like a happy place to stay.

Wrong. Dead wrong.

This kind of thinking is exactly what generates the age-based bias and discrimination on older workers.  An attitude of complacency gets you labeled as a slacker.  Being a slacker didn’t get you where you are today, so why would you suddenly think this change of strategy is a good idea?

As a younger worker, your mind was more a tabula rasa than the Statue of David.  Adding new habits was easy because everything was new to you–doing design, learning frameworks, figuring out how to estimate schedules.  You’re cutting a road in your mind with a wagon.  The first time is hard because the ruts for the wheels just aren’t there yet.  But every time you complete a project, your mind adds depth to the ruts in the road.  And after 10 years, that road is well traveled and harder to veer from.

Habits are hard to break, but not impossible.  Studies have shown that a new habit takes about nine weeks to take shape and really stick in your mind.  That means, on average if you’re really working at it, you can break around 5 bad habits a year, or add 5 new habits, assuming you want to focus all your extra effort into adding a single habit during a nine week period.

Think about it:  You can alter your habits such that every year, you spend the time to add 5 new technologies or practices to your repertoire, one about every 9 weeks. Been thinking about learning Agile?  How about picking up a new language?  Maybe changing source code repositories from CVS to Mercurial?  This is exactly how we can all stay relevant in the face of ever-changing technologies.

As far as the bad habits go, what kind of recommendation would you get from a colleague that saw you go from the least-documented code to the best in a 6 month time frame?  Wouldn’t that impress them enough to say, “Hey, that old dog can learn new tricks…I’ll be damned.”

If you’re under 30, don’t laugh too hard about these habit-breaking notes.  You’ll be here soon enough yourself.  Best to cultivate the good habits upfront so you can add more good ones rather than break the bad ones later.

OK, back to the cheering now…that was more fun anyway:

“Old guys! Old guys! Rah-rah-rah!”

* Although you probably want to stay away from the hookers and blow too.  I can’t see how that’s good for your career, either. No experience there, just sayin’.  🙂

Five Pervasive Myths About Older Software Developers

I recently celebrated my 40th birthday.  A friend joked to me, “Hey, guess that means you’re too old to program anymore!”  I laughed on the outside, but it gave me pause.  Age discrimination is nothing to laugh about in our field.  COBOL guys faced this problem years ago as Java guys like me were ascending the ranks, and we laughed heartily about legacy code and their inflexibility with new technology.

Now the joke’s on me.  Maybe you too.  And if it’s not now, it will be soon enough.  Still laughing now?  Yeah, I thought so.

Computer Science Degree Trends 1996-2008
Source: CRE Taulbee Survey, 2007-2008, pub 5/09
Computer Science Enrollment Trends, 1995-2008
Source: CRE Taulbee Report

Our field is ripe for age discrimination in so many ways.  We value hot, new technologies, the ability to absorb them at unheard of rates, working insane hours to push products out the door–all things attributed to the younger workers of our field.  And did I mention that younger workers are cheaper?  A lot cheaper.  But the trends of computer science degrees do not bode well for having a plethora of young, cheap workers at a manager’s disposal indefinitely.  In fact, all data point to one conclusion:  CS degrees enrollments have been declining or flat for almost a decade.  And if anything, the candidate pool for hiring is getting worse, at least according to Jeff Atwood.  You’re going to have to hire someone to write your next project, and with the backlash against outsourcing, who you gonna call, Egon?

If you’re thinking you’re going to avoid the “grey matter” of software development, think again.  There are a number of myths about older software developers that continue to be perpetuated in IT and software development that somehow put older, experienced workers at a disadvantage in our field.  But they’re largely crap and considering the degree trends, ignoring everyone 40 and over because we’re too old seems plain foolish.  Let’s debunk these myths one-by-one.

MYTH: Older software developers are more expensive than younger ones, making younger developers more desirable.

REALITY: The real reason experienced developers are labeled as expensive is because staff salaries are the #1 expense of any software organization.  The fact is, younger means cheaper.  But, while inexperienced, younger developers may save you budget, but they will cost you in the long run if that’s all you have on your team.  Younger developers haven’t taken the lessons of failure to heart.  They haven’t had enough time to learn those lessons yet. And guess whose money they’re going to be learning on?  Yours.  Think that won’t cost you money in missed deadlines and incomplete projects?  Think again.

Yes, older software developers have higher salaries than younger ones.  But what exactly are you paying for here?  With an experienced software developer, you’re paying for all the experience that comes with past project successes and failures.  Those are expensive lessons if you want to pay for them directly during your tenure as a manager.  But if you buy into an experienced worker, that’s like getting insurance against some of those classic mistakes in project management and software development that you don’t have to repeat.  Meaning you look better on your annual review because you hired smart people that know how to get the job done.

MYTH: Older software developers are less flexible and less capable of learning new technologies because of their legacy knowledge.

REALITY: It’s actually because of their past experience, that more experienced software developers can migrate to new technologies, frameworks, and systems more quickly and in greater depth.  For example, if you learn a GUI framework in C/C++, you have a mental model about message passing, event handling, MVC patterns used to design the system and separate presentation from back-end logic.  The very first time you learn a GUI framework, in addition to the syntax, the examples, and the quirks of the library, you also need to learn all the conceptual stuff.  And after two, three or more GUI frameworks, you notice that they all have very deep similarities outside of the syntax.  You might even notice newer frameworks have overcome substantial limitations that you used to have to work around with complicated hacks.  Those insights are lost on someone new to the framework.  And those insights can really boost productivity in ways you’re not able to directly measure.

MYTH: Older software developers are less able to perform the arduous tasks of software development (read:  work long, painful hours) because of family commitments and other attachments that younger workers don’t have.

REALITY: I think it would be fair to state that experienced software developers are less willing to work those long, painful hours because they’ve learned the hard way that there are productive limits to pushing yourself 80 hours a week for months on end.  It’s called burnout, and I’m willing to bet that anyone who has already experienced it in the past simply isn’t as eager to go there again.  But with that said, the supposed reason of “family commitments” is bogus.  High-quality, experienced software engineers are ruthless time managers, and those with families are even more motivated to get things done in allotted times.  They may have dance recitals and soccer games to attend to, but they also make up that time in off hours and highly focused work during the 40 hours they’re given in a week.  Good software engineers with families must become highly efficient with personal time management or they quickly get buried with the deluge of work coming their way.

MYTH: Older software developers are less mentally agile than younger ones.

REALITY: Aging does affect the brain and it is measurable to show that older workers think more somewhat slowly than younger ones.  But mental agility is only part of the equation.  Thinking faster isn’t always better.  What about judgment?  There’s an old expression:

Good judgment comes from experience, experience from bad judgment.

Lost mental agility is a poor excuse to not hire an older software engineer in light of the fact they’ve seen, done, and lived many more successes and failures than a younger developer.  Experienced developers have tons of past projects to draw from and assist in avoiding bad decisions today.  Younger developers have new ideas which are important, but often untested and unproven.  Having both perspectives on your team is of great value.

MYTH: Older software developers are more jaded and cynical and therefore, less desirable in the workplace than younger ones.  Younger developers are more enthusiastic than older ones.

REALITY: Anyone who believes this is probably someone who doesn’t like their ideas criticized by those who’ve been around long enough to see really stupid decisions put into practice again and again.  Experienced software developers smell crap a mile away.  They don’t buy your stories about how the product isn’t well received in the marketplace because they’ve been interacting with the customers for years and know you’re trying to cover up a future layoff in the company.  They won’t put up with managers asking them to work 80 hours a week because the customer wants the software next month and they already told you it will take 3 more months to complete with the features agreed upon.

Younger developers haven’t been in those situations as frequently and therefore, have less resistance to bad management practices.  The only desirable trait management wants here is naivete.  If you want a great team and great products coming out of it, having people that can call you out on bad decisions will save your bacon again and again.  But only if you have the courage to admit you don’t know everything.

And as far as enthusiasm goes, you can’t tell me age dampens enthusiasm.  If that was the case, Donald Knuth, Ward Cunningham, Bill Joy, Bill Gates and hundreds of others who’ve crossed the magic 40 barrier would be less interested in our field just because of age.  But they’re not.  Passion is passion. If you have it when you’re 40, chances are you really love that field.  That kind of love isn’t going to die overnight.  Younger developers still finding their footing in the world may have short-term passion, but that can be swayed in the face of obstacles and challenges in the field along the way.

In conclusion, let me be absolutely clear about a few things:  Young is not necessarily bad. Old is not necessarily good. And most importantly, anyone who can’t program their way out of a wet paper bag shouldn’t be hired, no matter how old or young they are. Keep your teams a vibrant mix of age and experience–where diversity exists, learning can take place.  But if you’re the person looking to hire someone, don’t write off the dude with the gray hair sitting across from you.  Let go of your age prejudice and see if they can impress you.

Someday that dude (or dudette) may be you.

Developer Lessons from Diablo II

Diablo II (DII) may be an old game but it’s a classic.  I’m one of thousands who gave up hours and hours of my life to it.  I have very fond memories of those times, staying up late with my friends on the phone, hacking evil monsters to bits, making the world safe again.  I think it’s a great form of geek therapy where no matter how bad your day was at work, taking out all the monsters on the Arreat Summit would somehow make things feel right and good in the world.

And with all that time invested in the game, there were a few lessons that are very applicable to real-world software development.  With Diablo III now generating a new wave of excitement for Blizzard’s series, I thought I’d share my insights from my favorite game of the early 21st century.

Take care of the monsters that spawn other monsters first

In the second act where you’re in the Kurast jungle, there are these incredibly annoying and potentially dangerous witch doctors.  Each witch doctor can summon more helpers that continue to attack you and drain your resources.  If you’re surrounded by 10 witch doctors and those incessant little critters they spawn, you’re in for a world of hurt, especially if you just focus on the spawned critters.  You need to hit the witch doctors first, ignoring the rest of the group, before you start attacking what’s left.

Lesson learned: Any problem that is creating additional, more painful problems, needs to be addressed early on in the process before worrying about others.  For example, if you have some serious bugs that continue to crop up and cause your team to spend cycles constantly nursing production systems along, but the fix is going to be hard and take away from your current development, perhaps the fix is worth the time considering the amount of time wasted in keeping the system healthy.

Brute force counts for a lot most of the time…

Subtlety is not really rewarded for the majority of DII.  Your best bet as any class is to pick the weapon, spell or skill that basically will take out the highest number of monsters given the situation.  That “silver bullet” is constantly changing depending on the Act or the area as monster vulnerabilities and resistances change.

Lesson learned: Pulling out your best code tricks for each and every part of the system can potentially be a waste of time.  This is just another way of saying that premature optimization is the root of all evil.  Use the simplest algorithm, framework, or idea that solves the problem.  Go ahead, put that brain dead String concatenation method in there…chances are, it won’t matter all that much in the larger scheme of things.

…But you need to be subtle around the Boss

When you face the Boss creatures on any level in DII, suddenly you’re forced to adopt a clear cut strategy.  Brute force won’t cut it, and even with a larger party you still need to work together and with great care to take down the Boss creature because he will most certain pwn the unprepared.  Certain character classes have advantages when facing the Boss depending on their vulnerabilities.  When I was a Necromancer, facing the Boss at the end of Act I was a real bear for me alone, but my Barbarian friend could easily take him toe-to-toe while I dumped damage into him from afar.  Together we made a big difference.

Lesson learned: You can’t always beat the code down with brute force.  Some very specific problems require a lot of thought, time and energy.  Pick your battles and choose your weapons to attack these problems carefully.  Sometimes, you’re not the best one to deal with the problem…maybe you had a hard time implementing the original solution.  Ask a colleague to step in and pair program it with your, or maybe even offer up their solution ideas for a fresh perspective.

There’s always a new shiny thing to go get

There’s always a better, bigger, faster, shinier, more powerful item to go get.  Whether you’re 10th level or 90th level, the quest for the Bigger, Better Mousetrap will always go on.  You will never have enough Elite Rare items, no matter how many of them you get.  There’s no winning this game but patience and persistence will get you what you seek if you’re willing to put in the time and effort.

Lesson learned: Newer technologies, frameworks and languages are always out there.  They will continually tempt you into thinking that your current technology isn’t nearly as good as What Is Out There.  Never let technologies drive your solutions past the original choices phase unless you’ve discovered a critical flaw in your original analysis.  For example, if you find that your database connection pooling solution simply cannot perform at the level you specified or tested that it would, it would be worth reexamining that particular choice.  Make a choice, stick with it for long enough to finish something with it.  Wait for the right moment to upgrade.

There’s no one best way to play the game

One of the great things about DII and other MMORPGs is the variety of characters you can play–Amazon, Necromancer, Druid, Barbarian, Sorceress, Paladin, Assassin or Monk.  Each has its own unique abilities, skills and fun factor.  I personally enjoyed the Necromancer and the Assassin–the former for summoning hordes of skeletons to do my dirty work and the latter for having the coolest looking claws to take monsters to the mat.  Anyone who says that <insert character here> is the only way to win the game, clearly hasn’t tried the others with similar zeal.

Lesson learned: Trying out new languages and frameworks is a great way to chase away boredom in your career.  Just because you did your last 5 projects with Java doesn’t mean it wouldn’t be interesting or worthwhile to try the same kind of project out with Ruby to see how it varies.  Maybe it will make you a better Java developer because of what you learned in Ruby.  Language snobs are boring and generally one-dimensional.  You’ll be a better developer for stretching your boundaries and limits in things you don’t understand as well as your native language(s).

Blind experimentation is a waste of time and resources, better to leverage others work first

The Horadric Cube was an essential piece of the game in DII but it could be a source of frustration if you didn’t know what really worked in it.  You had two choices with it:  Spend lots of time systematically putting things in and pressing “Transmute” to see what happened (which, depending on the items you stuck in, could be hours or days to re-find if you later discovered you made a mistake), or just look up the recipes online and then start hunting for the right items to make things work.

Lesson learned: Spinning your wheels experimenting with a problem when better and more complete information about it exists in other places is just a waste of time.  Reinventing the wheel might be fun, but it’s unlikely to be the best use of your time when the project clock is ticking away.  Stand on the shoulders of giants and leverage all that great information others have posted on the internet about your framework, language or technology.

Sometimes there is a secret cow level and it’s fun

If you found Wirt’s Leg and a Tome of Town Portal, put those in the Horadric Cube after you defeated Diablo on a level, there was a red portal to a secret cow level:  you got to go whack some cows for a change.  But not just any old heifer.  These were mean cows.  Big, bad cows.  Cows you wouldn’t be tipping anytime soon.  But damn, it was hilarious.

Lesson learned: Unexpected surprises are fun.  And never underestimate the Power of the Bovine.  Celebrate the absurd every now and then.

No matter how deep you are in Hell, it can always get worse

You made it all the way through Diablo II’s Normal level thinking, “Wow…that was hard.”  Until you turned on Nightmare and realized that you ain’t seen nothing yet. Those little rats from the first Act?  Wow, they’ve got serious bite now.  And Baal?  Well, let’s just say that Full On Bad Ass is an understatement.  You might have gotten away with quick and sloppy the first rounds with the Boss scenes, but that’s not going to cut it this round.

Lesson learned: If you think your performance problem is bad in production, consider what might happen if the entire server farm crashed.  Or if you’re dealing with a major crash, what would happen if you had 50% hardware failures instead of just a software problem?  The list goes infinitely deep, each more terrifying that the previous.  Complaining about the problem isn’t going to help.  You might as well be thankful your situation is not worse and deal with what you have.  If you did something that was sloppy during the product development, own up to it, fix it and move on.

There you have it.  Eight fun lessons from Diablo II.  Got a lesson that I missed?  Talk back below!

The Real Reason Outsourcing Continues To Fail

Outsourcing:  The word American developers love to hate.  There are lots of stories out there about failed efforts that involve offshore development (“offshoring”).  I even have a few myself.  But this post is not about bashing outsourcing countries, the cheapskates that hire them, or the project managers who can’t control the resulting chaos.  This is about understanding why we have such a difficult time making offshore development work before any of those folks get involved.

Airline Disasters and PDI

What got me thinking about this subject was the book Outliers:  The Story of Success, by Malcolm Gladwell.  In it, Gladwell discusses the reason for a series of catastrophic airline failures.  Without repeating most of his excellent dialogue in the book, here’s the Cliff Notes version:

  • There was a study by Geert Hofstede, where he tracked the Power Distance Index (PDI) among selected world countries.
  • The Power Distance Index is an incredibly important measure of how a person in a country would generally react to an authoritarian figure.
  • Countries with high PDI would have more people willing to accept an authoritarian power figure in a paternalistic sort of way.  Like how you might defer to your father’s decisions, or to a king’s requests, for example.  People in higher PDI countries are less likely to question authority and more willing to accept instructions from those in higher positions of society.
  • Lower PDI countries are characterized by subordinates being more willing to question the orders of a superior.  Low PDI countries have people that tend to view themselves more like those in power than not.  In other words, you might judge yourself to be much like your boss in a low PDI country.

Crashes Caused By Power Differences

The NTSB regularly investigates airplane crashes to determine causes, but investigators were unnerved in the late 1990s to find a pattern of fatal crashes were found to be very airline (and also country) specific.  These two airlines with frightening records were Korean Airlines and Colombian Airlines.  The fatal crashes all had common attributes:

  • The pilots didn’t make any single fatal error.  They all started with several smaller errors that gradually built up to a catastrophic failure.
  • For any airline, the First Officers are always trained to double-check the Captain’s orders.  This is a safety protocol that prevents single-person failures from happening.  It is the First Officer’s duty to question and even override the Captain if the order he gives is unsound or improper.
  • In these particular cases, the black-box recordings indicated that the First Officers were hinting at problems, but were not strongly identifying them in a way that would make it very obvious.  Nor did they explicitly tell the Captain that he was acting against normal procedures.

Power Means Fear of Communication

In the aftermath, an astute researcher matched up the Hofstede PDI data with these two countries and found that both Korea and Colombia are high on the PDI scale.  What were the implications of this?

  • Clearly, a power difference would have existed in the cockpit between the First Officer and Captain.
  • The manner of communication between superiors and subordinates is very different if you are from a high PDI country than a lower one.
  • In a high PDI country, a subordinate must use the proper language, body posture, facial expressions and tone when communicating with a person of higher status or power.  In Korean, for example, there are no less than six distinct ways to address someone from the most formal to the least.  In the United States (a lower PDI country), we have a much looser style (“Sir”, which could apply to a General as much as a Lieutenant in the military, or even your father in some situations).
  • This disparity of power created hesitance on the part of the First Officer to embarrass his Captain when the Captain made mistakes (and in each of these cases, at least 3 mistakes were made).  So instead of using a direct method of communication (“This weather is really bad, we should turn back immediately or regroup for another airport”), he used a more subtle, formal and proper one (“Sir, look how it is raining outside.”).
  • The implication of a high PDI country is that there is a rich subtext going on between the two communicators.  But that subtext assumes that both sides are alert, paying attention and can clearly understand the implicit signals.
  • In the case of the airlines, the Captain was almost always sleep deprived and exhausted by the time the situation arose, making that communication impossible.

The researcher concluded that the pilots were inside a cultural framework that dictated how they should behave at a time when those behaviors turned out to be detrimental and outright dangerous to the safety and welfare of the passengers.  In other words, the fact they came from high PDI countries made it impossible for the proper communication to take place when it was most necessary to be plain and step outside the traditional power roles of these cultures.

Past Outsourcing Blames

What does all this have to do with outsourcing?  First, we need to understand why outsourcing has traditionally “failed”, according to both buyers (those who purchase outsourced services) and providers (those who perform the service in their local country, or send people to other countries to perform services).  Here’s a graph of combined data about a survey regarded failed projects from 2004.  The data are still relevant to today’s discussion.  (Source:  The Outsourcing Center)

Reasons of Failure for Outsourced Projects
Reasons of Failure for Outsourced Projects

Notice the even distribution of reasons once the provider & buyer survey data is combined.  This is interesting because neither party can clearly point to a single, differentiating causation factor in the failure of outsourced projects.  But I believe that’s because they asked the wrong questions in the survey.

The Real Issue with Outsourcing is Power Difference

If you have a buyer from a lower PDI country and a provider from a higher PDI country, there are already implicit consequences to your interaction that neither party will know about without prior outsourcing experience or natural cultural awareness(1).  And even with that experience, it’s not a given that they will understand the reasons behind the challenges of outsourcing.  Let me create an example from my own personal experience:

Suppose you had an American company (Buyer) and an Indian company (Provider).  The American company contracts with the Indian one to provide offshore outsourced software development at a fixed price per developer.  Certain key performance indicators are agreed upon by both parties and the game is afoot.  Let’s also assume the Indians agree to a six month project to write a content management system for the Americans.

A typical scenario of engagement might follow like this:(2)
  • The first month, everyone hammers out the requirements documents and in a great ball of fury, declares them sound and ready for implementation.  The American company at this point would typically reduce the daily oversight on the project to something more reasonable, like weekly updates.
  • The second, third and maybe even fourth months pass with little fanfare.  The Indian developers are quietly building the specified software and the Americans are receiving updates about it that are all positive and sound great.
  • At some point, the American company asks for a demo of the progress to date.  The Indians put together something after a bit of negotiation (since the Americans neglected to mention the demo as a deliverable before the end).  The Americans see the actual software and fly off the handle.  Performance is awful, the screens don’t look anything like what they want, and the software appears to be behind schedule.
  • Further code reviews by American developers indicate that the code quality is fairly poor, lacking in comments, unit tests, and filled with copy-paste blocks of duplicate code.  The Americans immediately demand the project be put under different management.
  • The project falls off of the rails somewhere after this.  It will either be canceled, brought back in house, or will be delivered extremely late after extensive modification to the original requirements.

There’s lots to pick on here on both sides of the table.  I would like to point out that the fact that I picked on Americans and Indians is actually irrelevant here. You could easily substitute “British” for Americans (3), and “Filipinos” for Indians with the same results.  But why are they so interchangeable in this fashion?  It’s because of PDI and the inherent cultural communication issues that come with it.

Dilbert says it best

Here is a list of the top 10 Outsource Providing countries in 2009, and their PDI scores.

  1. India (77)
  2. Thailand (64)
  3. Mexico (81)
  4. China (80)
  5. Indonesia (78)
  6. Malaysia (104)
  7. Philippines (94)
  8. Jordan (no data)
  9. Egypt (80)
  10. Bulgaria (no data)

For reference, the United States is 40 on the scale.  Western countries can run the gamut as high as Belgium (60) to as low as Austria (11).  The scale is from 1-120, where 120 is extremely high PDI.  You can see all the countries’ measurements in the original study on this colorful world map of PDI indices.  The gray countries are ones that weren’t measured.  India would be considered moderately high PDI at 77 (in the 61-80 range).

So what happens when you bring a low PDI buyer together with a high PDI provider?

In a word:  Disaster.

Cultural Context Matters In Communication

Each side expects a certain subtext to go on during a conversation because of their own cultural context.  Like this:

Low PDI Manager: So, is the new website ready for launch by Friday?

Low PDI Developer: No, and it’s going to be another 2 weeks because we need the new servers to arrive, for QA to finish with testing after they do, and then release the code.

Pretty straight question, pretty straight answer if you’re from a low PDI country like the United States.  There is little assumption about the subtext because a low PDI communicator is used to “speaking his/her mind” about it.  The information is supposed to be in the conversation as spoken words.  If it’s not there, it’s ignored.

But what if we change that a bit?  Assume the High PDI and Low PDI Developers BOTH have access to the same information and are equally competent:

Low PDI Manager: So, is the new website ready for launch by Friday?

High PDI Developer: Yes, it may be ready by then.  We are looking into it.

That seems like a bad answer if you’re from a low PDI country (mostly because we know the context from the first scenario), but it may be taken at face value because the Low PDI Manager expects straight conversation.  If there was a problem, the Low PDI Manager expected the developer would say something specific about it.  When they didn’t, the Low PDI Manager assumes that Friday will be the date.

And what about the High PDI developer?  He didn’t want to offend the Low PDI Manager, because that’s what you are careful to do in a high PDI country.  The High PDI Developer assumed that the Low PDI Manager would understand his subtext “may be ready” and either ask further questions, or understand that Friday wasn’t necessarily a realistic date.

This is just the tip of the iceberg.  If this happens on a simple conversation about a deadline, what about really big stuff like:

  • Requirements
  • Deliverables
  • Quality control testing
  • Development standards
  • Documentation

The implications are literally staggering.  In fact, I’d go so far as to say the fact that every outsourced project hasn’t failed is something of a miracle.  It’s a testament to having the right people who naturally and instinctually bridge these gaps through extra communication.

The Survey, In A New Light

Getting back to the survey questions, if you look at all of them and how they would be viewed relative to PDI, it’s arguable that PDI differential is the one, single, leading cause that relates to how providers and buyers have a hard time seeing eye-to-eye during the outsourcing process.  Of the eight named factors, I can see 6 of them that directly relate to PDI differential:

  • Buyer’s unclear expectations up front (buyer assumes he is understood when the provider stops asking questions, but that’s a typical low-high PDI interaction)
  • Poor governance (see my deadline example above)
  • Poor communication (again, the deadline example)
  • Poor cultural fit (again, the deadline example)
  • Interests become misaligned over time (you don’t understand each other’s communication needs and are frustrated)
  • Not mutually beneficial (you can’t work together because you don’t understand how to interact…)

Adding those 6 factors up, 72% of project failure reasons can be connected to PDI differential. If both sides understood that single factor going in to the process, everyone would be better served in the end.  Think I’m just making this all up?  It’s not just the buyers that complainHigh PDI country providers say the same things.

So what’s so hard about outsourcing?  It’s hard because of the cultural baggage we bring to the table on both sides, and neither side necessarily realizes it because of assumed interactions.  We need to be more aware of the cultural assumptions going in to projects like this, or we’re doomed to repeat them ad absurdum.

(1) I think it’s fair to say that most other countries would NOT say most Americans are blessed with “natural cultural awareness”.  🙂

(2) Before I get lots of angry comments from Indian readers about the interaction above, yes, there are other potential outcomes and perhaps you’ve been on projects where they are all successful.  I have a mixed bag of experience on this, and it’s not about bashing Indian developers.  Like American ones, they run the gamut–good, mediocre, and what-the-hell-are-you-doing-coding.  I’ve run into all three in about the same proportions as American developers, more or less.

(3) I’m sure one or more British readers are horrified at thinking they are interchangeable with Americans at this point.

Outsource Rank Country PDI
1 India 77
2 Thailand 64
3 Mexico 81
4 China 80
5 Indonesia 78
6 Malaysia 104
7 Philippines 94
8 Jordan
9 Egypt 80
10 Bulgaria
11 Hungary 46
12 Ghana 77
13 Pakistan 55
14 Chile 63
15 Poland 68
16 Czech Republic 57
17 Argentina 49
18 Romania
19 Ukraine
20 South Africa 49
21 Russia
22 Vietnam
23 United States 46
24 Israel 13
25 Canada

Hiring Ren and Stimpy

Since we’re on the topic of hiring and interviewing, let me regale you with a story about two developers.  This is an entirely fictional tale, but drawn from real-world observation and experience over 17 years of employment.

Suppose you had two new junior developers at your company:  Ren and Stimpy.  Both have similar backgrounds, education and experience.  Both passed the interview process and were inducted into your team.  Both are bright, ambitious and self-starting.  Any resemblance to the long-running popular cartoon is purely coincidental.  Stay on target, Red Five.

Your New Star Developers: Ren & Stimpy

Ren started out on the first day setting up his computer, checking out the code and understanding what his first assignment was.  Within a week, he mastered it and went to his managers for more work.  After a few more assignments, he started to get bored and whine about how menial his tasks were, even though he was a junior developer.  He started to sulk to his manager about how he could “do so much more” than just writing unit tests for some “barely used modules.”

Ren would often interrupt have impromptu meetings with other senior developers and managers hoping to get more interesting and challenging tasks.  Often times, he’d get more of the same kind of work from them.  Instead of taking lunch and wasting valuable time, he would often work through lunch and stay late to get these tasks done.

Stimpy started much the same way.  He setup his machine, checked out the code and worked on his first assignments.  He too, asked for more work when it was finished, but he started recognizing a pattern to the work he was given.  The billing system testing he was doing involved some similar functions and classes that could easily be scripted.  He worked on this side project and then let his manager know that all future tests were now written.  This made the manager’s job even easier, since the senior developers could now use this script to auto-test things as they came out with new requirements and designs without further delay.  He spent his lunch hours socializing with members from other teams to get to know them better.

Stimpy also started looking around at other things going on with the team.  He found the SVN repository was a complete mess.  He started asking other developers how it ended up that way and got a broad perspective on how the company organized code over time.  He spent extra hours creating a new repository that completely reorganized the code in the current taxonomy which made everyone’s lives better.

Stimpy noticed that the company’s development document files were just sitting around on a shared directory.  He decided to create a wiki (after first talking with the network admins to see if one already existed, and finding the ideal machine on which to install it) and upload all the docs by project.  He published this to his manager, who in turn shared it for the very first time with Marketing and Sales, so they could see the progress on engineering projects.

Stimpy was asked to sit in on several design meetings to get his input because he was able to grasp the historical significance of past development based on his repository migration (with the benefit of hindsight, he had a unique perspective that even the original developers lacked), so they wanted to know what could be salvaged from old projects and what needed to be rewritten.  Ren was still working on his unit tests and complaining to his manager about it.

It’s pretty obvious which of these developers is going to make a positive, lasting contribution to the team and which might be more of a drag on productivity.  Five things really separate these two individuals:

  1. Understanding how things work in an organization. Stimpy took it upon himself to learn about the “other infrastructure” besides his tiny module of code.  He got the bigger picture, which made it easier to understand the importance of little things and which things are more critical than others.
  2. Knowing the roles of others and how/why they do their jobs. By interacting with others (like the network admin and the other developers outside of his immediate scope), Stimpy understood how all the parts fit together to make a unified machine.  And he saw where things in that machine needed maintenance.
  3. Finding where gaps exist, and taking the initiative to fix them. How many times have you heard, “Well, we don’t have time to do that right now.”  (Sounds a little like Habit 5:  Fix It Later, don’t you think?) Doesn’t matter what the question is, you’ll get that answer a lot from most people in any company.  If you’re the one person that is willing to do extra stuff (with a smile, of course) and still gets your own job done, you’ve already risen above the pack.
  4. Have no fear of repetitive, monotonous, or boring “dirty work”-be willing to get it done. Of course no one likes this kind of work.  It would be called something different if that were the case.  But honestly, what do you remember more vividly?  The guy who fixes the annoying SVN problems the entire team suffered with for 3 months, or the guy that created an awesome rainbow table for his code from scratch in a weekend?
  5. Stepping up and taking responsibility for things outside of your job description. Arguably, they both did this, but Ren was looking for the sexy, resume-filling work, where Stimpy was just getting things done, no matter what they were.  Never did he say, “It’s not my responsibility…”  If you want to get ahead, actively find things outside of your job description that you can do and that would have a beneficial impact on the company and do them.  Don’t whine about not being challenged by your managers (or teachers).  There’s only one person in control of your life:  youDon’t give that power up, ever.

No matter whether you’re starting your first job, or your 10th, always ask yourself:  What kind of  developer do I want to be? Ren or Stimpy?  And if you can sing the Happy Happy Joy Joy song to the delight of your team, so much the better.

Stop Dumbing Down The World with Bad Interview Questions

With the Holiday Dead Zone coming to an end and the global economy picking up, I’d like to talk a little about interviewing software developers since hiring season is now upon us.  Ever been to an interview where they ask some whopper like this?

“Can you ever write code in C++ where you say “delete this;“?”**

Or maybe one of these doozies from “Google’s Interview Questions to Make You Feel Stupid” (which, BTW, aren’t all from Google…but from Microsoft as well)

  • How many piano tuners are there in the world?
  • How many golf balls can fit in a school bus?
  • Why are manhole covers round?
  • How much should you charge to wash all the windows in Seattle?

I’ll be the first to admit that I went straight for this Stump-the-Chump mentality.  Along with the goofy questions above, developer interview articles abound with the kind of information you can ask about language trivia even the very authors of the language would probably be hard pressed to recall under pressure.

Uh, why are manhole covers round?

If you’ve never been subjected to such a grueling event, I commend you.  You must know some really awesome people to get past those kind of interviews.  But most developers know what I mean.  The kind of interview where someone just throws out obscure or impossible questions one at a time until the candidate softly whimpers for mercy, turns into a puddle of tears, and runs screaming out of the door.  Preferably with brown stains in their pants.

I started out as that kind of manager and interviewer.  My favorite question was the Microsoft classic of the Four People, a flashlight and a Bridge question, except I used a variant where the four Beatles were stuck at Red Rocks Amphitheater here in Colorado.  Inevitably, I asked that question to everyone I interviewed…project managers, developers, QA–I may have even tossed it out to a marketing guy once.  My reputation at that company was “bastard interviewer” (a direct quote from my friends, and not altogether undeserved).  Anyone who survived the ordeal always had a great collective story with a lot of the other employees to share at happy hour.  My interview tactics actually influenced other managers at the company to ask easier questions because they felt sorry for the candidates having been through “my interview”.

Years later, I realize not only how mean that was, it was also useless. Of the mix of people I interviewed back then, we hired great developers, mediocre developers and a couple bad apples.  These questions are supposed to be some sort of impeccable bozo filter–yet they all passed it.  Our team was not filled with little Bill Gateses and Linus Torvaldses.  While we were productive, we also didn’t launch our product either, so what went wrong?

When you attack someone’s encyclopedia of knowledge, these kind of questions barely touch on the darkest corners of experience if you’re lucky.  It’s somewhat like the SAT admission test here in the US for college admission.  But like the SAT fails to capture real knowledge and performance potential at a university level, the interview questions we use today are equally inept.

What you really want to know apriori is how this person will work on your team–do they share credit, steal it, or hide until the project ends?  You’d like to know what kind of code they write–is it atrocious, beautiful, disorganized, or anal-retentive?  You’d like to know how the interact with people–are they sociopaths, leeches, primadonnas or Mother Teresa?  You want to know if they’re Smart and Get Things Done, to quote Joel Spolsky.  Checking whether they know esoterica about C++ isn’t helpful when they’ll spend 90% of their time on Java.  Or hammering someone about parameters and exceptions to methods for a particular framework (Seriously, I saw this on several blogs asking people detailed questions about methods in Spring or Hibernate) when the answer is one Google search away.  All of this kind of interviewing is useless.

Now that we’re at the end of the first decade of the 21st century, I’d like to boldly propose that we do away with Stump-the-Chump interviewing.  Instead, why don’t we try this, adapted from this post about entrepreneurs where a VC has specific criteria for successful entrepreneurs:

  • Tenacity:  Is this the kind of developer who has really been through the ringer of a hard problem and crawled out of the sewer pipe with the answer raised in his or her hand, triumphant?  Or do they sit back and wait for others to answer questions for them?
  • Street Smarts:  No, I don’t mean do they know how to make a shiv out of the plastic pepper shaker in the break rooms.  I mean, do they know the rules and when to break them?  Are they creative in their solutions, or more conventional and straight forward?
  • Ability to Pivot:  Is this person going to whine or moan when the project suddenly needs to change direction and rewrite in Ruby?  Or do they take things in stride, take your instructions and start grabbing links for the entire team to bone up on the finer points of Rails, Grails and JRuby too?
  • Resiliency:  When the project reaches a point of just grinding out some mindless junk, are they the kind that complains about it or someone that finds ways to get past that with scripts they share or tricks from another job?  If things head south on the design meeting, do they bash the marketing team for the changes, or just take a deep breath and sink their teeth into getting past this setback?
  • Inspiration:  Does this developer constantly strive for new information?  Do they like to share blogs or resources that get them excited about their job?  Are they the kind of person who finds a new way to write a framework and creates a proof of concept just to show how interesting it might be for the next revision?

I’m not going to tell you which questions to ask, because really, that’s no different than another stump-the-chump session using my blog as the source.  Use their resume, ask leading, open-ended questions that get the person to tell you about their previous projects.  Listen to what they have to say, probe deeper into it, get a feeling for their experience.  Get past the bullshit, because we all know there’s a whole lot of training that we go through to “pass the interview”.  Call their references and get references from their references, because those people will actually give you honest answers about this person.  Find someone you know that worked with them.  Ask them if they’d work on the same team with them again.  Really, that’s the one question you need to ask.

And those are the kind of answers we’re looking for anyhow.

* UPDATE:  A great site popped up that allows you to perform remote coding interviews to help weed out the wheat from the chaff:  Try See[Mike]Code. They write, you watch. Live. Get some feedback before you even start the interviewing process.

* UPDATE (part 2):  An article from about Google’s ineffective hiring practices.

**Incidentally, for those uninterested or unwilling to look it up, the answer is yes, you can.  And there are some valid reasons to do so, but the question definitely shows just how much or how little you understand the underlying memory model and code execution of C++.  But I still think it sucks as an interview question for C++ developers.

*** Nothing new under the sun:  If you liked this post, try this gem from 2007 that mentions many of the same points but in a different way.  I found it AFTER I wrote this.  🙂

Habit 7: Principles Are More Important Than Results

Our last installment today is about the Great Ivory Tower of Standards and Architecture.  In case you missed the previous bunch:

Seven Habits of Highly Dysfunctional Enterprise Developers:

  1. Blame Everyone But Yourself
  2. Confuse Motion With Action
  3. Use Complexity To Demonstrate Intelligence
  4. Keep Important Information Secret And Safe
  5. Fix It Later
  6. Reuse Is Overrated
  7. Principles Are More Important Than Results

Principles Are More Important Than Results

Beware the Ivory Tower

Ever work on a project where some group (or some person) in your company espouses standards above all?

“You absolutely must follow our standards to the letter.  We’ll be having daily code and design reviews, anything out of compliance will of course, be refactored to meet the corporate standards immediately”

Principles and best practices are great things to engender on a project.  A highly respected architect friend of mine said to me over and over, “It’s better to be consistent than right”.  Consistency gives predictability to people looking at code for the first time.  They learn what to expect and it reduces up-to-speed time for new developers.  It also creates mini “code templates” in your work patterns to speed things up as you develop.

You can take this too far.  Projects where standards are used like a whip are demoralizing and turn action into motion purely for the sake of consistency.  Remember:  Your goal is to ship a finished product that meets the customer’s requirements, on time and under budget.  Following standards is part of that but creating painfully complex standards and then demanding absolute adherence in this fashion is just torturing your developers.

Have standards, but keep them simple.  Standards exist to create consistency, not some absolute right-or-wrong litmus test for code.  Balance is important–shipping a product ad-hoc without any standards is a recipe for disaster during your first bug fix or maintenance release, but following each and every standard down to the comma is just wasting cycles.  Don’t err on the side of extremes either way.

In summary:

A good developer knows that there is more to development than programming.
A great developer knows that there is more to development than development.

Got a habit that I missed?  Sound out below!