Lessons of Failure
Humans + Software Development = Always Interesting

Jan/10

4

Stop Dumbing Down The World with Bad Interview Questions

With the Holiday Dead Zone coming to an end and the global economy picking up, I’d like to talk a little about interviewing software developers since hiring season is now upon us.  Ever been to an interview where they ask some whopper like this?

“Can you ever write code in C++ where you say “delete this;“?”**

Or maybe one of these doozies from “Google’s Interview Questions to Make You Feel Stupid” (which, BTW, aren’t all from Google…but from Microsoft as well)

  • How many piano tuners are there in the world?
  • How many golf balls can fit in a school bus?
  • Why are manhole covers round?
  • How much should you charge to wash all the windows in Seattle?

I’ll be the first to admit that I went straight for this Stump-the-Chump mentality.  Along with the goofy questions above, developer interview articles abound with the kind of information you can ask about language trivia even the very authors of the language would probably be hard pressed to recall under pressure.

monkey-with-cymbals

Uh, why are manhole covers round?

If you’ve never been subjected to such a grueling event, I commend you.  You must know some really awesome people to get past those kind of interviews.  But most developers know what I mean.  The kind of interview where someone just throws out obscure or impossible questions one at a time until the candidate softly whimpers for mercy, turns into a puddle of tears, and runs screaming out of the door.  Preferably with brown stains in their pants.

I started out as that kind of manager and interviewer.  My favorite question was the Microsoft classic of the Four People, a flashlight and a Bridge question, except I used a variant where the four Beatles were stuck at Red Rocks Amphitheater here in Colorado.  Inevitably, I asked that question to everyone I interviewed…project managers, developers, QA–I may have even tossed it out to a marketing guy once.  My reputation at that company was “bastard interviewer” (a direct quote from my friends, and not altogether undeserved).  Anyone who survived the ordeal always had a great collective story with a lot of the other employees to share at happy hour.  My interview tactics actually influenced other managers at the company to ask easier questions because they felt sorry for the candidates having been through “my interview”.

Years later, I realize not only how mean that was, it was also useless. Of the mix of people I interviewed back then, we hired great developers, mediocre developers and a couple bad apples.  These questions are supposed to be some sort of impeccable bozo filter–yet they all passed it.  Our team was not filled with little Bill Gateses and Linus Torvaldses.  While we were productive, we also didn’t launch our product either, so what went wrong?

When you attack someone’s encyclopedia of knowledge, these kind of questions barely touch on the darkest corners of experience if you’re lucky.  It’s somewhat like the SAT admission test here in the US for college admission.  But like the SAT fails to capture real knowledge and performance potential at a university level, the interview questions we use today are equally inept.

What you really want to know apriori is how this person will work on your team–do they share credit, steal it, or hide until the project ends?  You’d like to know what kind of code they write–is it atrocious, beautiful, disorganized, or anal-retentive?  You’d like to know how the interact with people–are they sociopaths, leeches, primadonnas or Mother Teresa?  You want to know if they’re Smart and Get Things Done, to quote Joel Spolsky.  Checking whether they know esoterica about C++ isn’t helpful when they’ll spend 90% of their time on Java.  Or hammering someone about parameters and exceptions to methods for a particular framework (Seriously, I saw this on several blogs asking people detailed questions about methods in Spring or Hibernate) when the answer is one Google search away.  All of this kind of interviewing is useless.

Now that we’re at the end of the first decade of the 21st century, I’d like to boldly propose that we do away with Stump-the-Chump interviewing.  Instead, why don’t we try this, adapted from this post about entrepreneurs where a VC has specific criteria for successful entrepreneurs:

  • Tenacity:  Is this the kind of developer who has really been through the ringer of a hard problem and crawled out of the sewer pipe with the answer raised in his or her hand, triumphant?  Or do they sit back and wait for others to answer questions for them?
  • Street Smarts:  No, I don’t mean do they know how to make a shiv out of the plastic pepper shaker in the break rooms.  I mean, do they know the rules and when to break them?  Are they creative in their solutions, or more conventional and straight forward?
  • Ability to Pivot:  Is this person going to whine or moan when the project suddenly needs to change direction and rewrite in Ruby?  Or do they take things in stride, take your instructions and start grabbing links for the entire team to bone up on the finer points of Rails, Grails and JRuby too?
  • Resiliency:  When the project reaches a point of just grinding out some mindless junk, are they the kind that complains about it or someone that finds ways to get past that with scripts they share or tricks from another job?  If things head south on the design meeting, do they bash the marketing team for the changes, or just take a deep breath and sink their teeth into getting past this setback?
  • Inspiration:  Does this developer constantly strive for new information?  Do they like to share blogs or resources that get them excited about their job?  Are they the kind of person who finds a new way to write a framework and creates a proof of concept just to show how interesting it might be for the next revision?

I’m not going to tell you which questions to ask, because really, that’s no different than another stump-the-chump session using my blog as the source.  Use their resume, ask leading, open-ended questions that get the person to tell you about their previous projects.  Listen to what they have to say, probe deeper into it, get a feeling for their experience.  Get past the bullshit, because we all know there’s a whole lot of training that we go through to “pass the interview”.  Call their references and get references from their references, because those people will actually give you honest answers about this person.  Find someone you know that worked with them.  Ask them if they’d work on the same team with them again.  Really, that’s the one question you need to ask.

And those are the kind of answers we’re looking for anyhow.

* UPDATE:  A great site popped up that allows you to perform remote coding interviews to help weed out the wheat from the chaff:  Try See[Mike]Code. They write, you watch. Live. Get some feedback before you even start the interviewing process.

* UPDATE (part 2):  An article from Monster.com about Google’s ineffective hiring practices.

**Incidentally, for those uninterested or unwilling to look it up, the answer is yes, you can.  And there are some valid reasons to do so, but the question definitely shows just how much or how little you understand the underlying memory model and code execution of C++.  But I still think it sucks as an interview question for C++ developers.

*** Nothing new under the sun:  If you liked this post, try this gem from 2007 that mentions many of the same points but in a different way.  I found it AFTER I wrote this.  :)

Be Sociable, Share!

· · · ·

6 comments

  • David Thielen · January 5, 2010 at 8:00 pm

    I’m the person who wrote that blog. I think you missed the point of the questions. The goal was not to see if someone knew the answer. As I said in my post, the question is of little value for those that know this.

    The reason this question is great is most people don’t know, and their gut reaction is usually wrong. So the question is a chance to see how they investigate a question. Using your criteria above (which are good), it measures “street smarts” and “ability to pivot.”

    Please read the “why” I listed for the value of this question before dismissing it.

  • Tweets that mention Stop Dumbing Down The World with Bad Interview Questions | Lessons of Failure -- Topsy.com · January 6, 2010 at 6:51 am

    […] This post was mentioned on Twitter by Steve Levine, brimurph. brimurph said: #tech #interview – Stop playing Stump-The-Chump http://is.gd/5O4aw and favor candidates who Stump-The-Stud http://is.gd/5O443 (via @blue64) […]

  • Author comment by Dave · January 6, 2010 at 8:29 am

    @David: Thanks for your comments. Actually, no, I didn’t miss the point. I understand that in theory these questions should display how you approach problem solving. But in practice, my experience on both sides of the table is vastly different.

    There are several problems I see with these questions not already mentioned above:

    They can be studied for, and therefore, used to game the system. Meaning you find out nothing about the person’s thought process, only their ability to pre-Google questions.
    Even if you get a bad answer, that’s not sufficient to eliminate someone because you may just be missing their sweet spot.
    You are often limited in time, these questions consume a great deal of time, and therefore, if you use only these questions, you are missing crucial knowledge about the candidate.
    Even though you point out NOT to do this, these questions are still used by people as a “bozo filter” and it doesn’t work, period.

    Three cases in point:
    One and two, I hired two developers using these kind of questions. Both had impressive resumes, both answered the questions “well” and appeared to be good thinkers. Both developers turned out to be mediocre at best. One was always mired down in excess complexity to the point no one else could understand his code, the other was simply unproductive because he insisted on using different tools than the rest of the team and spent most of his time fooling around with them, rather than getting work done. Both were Smart, both were unable to Get Things Done. Had I spent more time on getting references of references, I might have discovered this.

    Three, another developer I hired (someone I worked with previously and knew about his amazing work ethic) went through this process and was asked these questions by others. They all told me that I shouldn’t hire this person because they believed he wasn’t right for the job. I hired him anyway, and he was my #1 most productive engineer. Beat the socks off the other two guys above. And I still work with him today. He failed the question because it didn’t really expose the engineering abilities we needed. Again, the question failed to cover the necessary attributes: Are they Smart? Can they Get Things Done?

  • Hiring Ren and Stimpy | Lessons of Failure · January 11, 2010 at 6:15 am

    […] Stop Dumbing Down The World with Bad Interview Questions […]

  • uberVU - social comments · January 13, 2010 at 10:04 am

    Social comments and analytics for this post…

    This post was mentioned on Twitter by blue64: In a #tech #interview should you be asking questions like “How many golf balls fit in a bus?” Interesting view here: http://bit.ly/4O8A1X

  • Greg Battle · January 13, 2010 at 11:49 am

    These type of questions reveal more about the interviewer’s laziness than the interviewee’s ability to answer them. They’ve been around and recycled for eons.

    Get creative with your questions, not didactic. The question should be an exercise in thought, not one of rote learning. The point is to ask questions that are adaptive, as in the solution can get as deep as the interviewee/er is comfortable going. Esoterica belongs in the land of RTFM, unless the candidate’s solution leads there.

    Unimaginative questions reveal unimaginative managers and coworkers. In practice, don’t give a brain teaser unless you authored it – and expect to find exceptions and alternative answers even if you did. No one has a lock on creativity and innovation.

    Just one man’s humble opinion.

<<

>>