DRAFT 3: This is a work in progress. Please direct comments / questions to problemboss@gmail.com

Puzzled by the interview process

I've always been interested in the reliance on logic puzzles and coding tests in interviewing software engineering candidates (Microsoft has a reputation for using this technique heavily in the past).

While I think I understand the desire to use such techniques, I believe that they create and reinforce a monoculture among engineering teams, and that the resulting monoculture may not be desirable.

What makes companies think that they can get consistently great engineers as a result of an interviewing technique that requires almost no preparation whatsoever?

Filters and monocultures

People need to be aware that systematic, sustained use of puzzles and tests act as filters, and should be acutely aware of the filters' attributes: What do they pass? What do they block?

The whiteboard/puzzle filter is valid and useful, as long as you are willing to accept that the filter will reject some talented and highly useful people. And, in my opinion: The filter favors people who are good at extemporaneous problem-solving (I'd say that's exactly what it was designed to do, from inception, even if current practicioners aren't aware of it).

I suspect that in most cases, it was probably chosen as a way to avoid having to spend a lot of time reading resumes and preparing for an interview (you are familiar with the typical "here's the candidate's resume, can you interview him at 1 PM?" style of interview prep, aren't you?) If I'm correct, in many cases, this technique is little more than a trick, a shortcut, a way to avoid devoting precious prep time when considering a candidate. And this is another flaw in the process - if used as a trivial rejection filter, you save some time, but fail to gain any real knowledge of a candidate's abilities.

Now, an interesting question: what is the end result of many years of hiring (mostly) people who pass thru this type of filter?

Do you build with an organization composed mainly of people who favor quick answers, or do you end up with mostly thoughtful, contemplative, and careful engineers? I don't know, but it'd be interesting to get an insider's view of the primary personality type at Microsoft (I would be surprised to hear that I'm way off base, that mostly, people are quiet, thoughtful, and contemplative.)

Bouncer at the door

These tests function in much the same was as the bouncers at Studio 54 - Do you have the right stuff to make it inside the door? Will you be allowed to mingle with the beautiful people?

Many great dancers can't get in the door when those filters are in place, and that's the frustrating part - they can't get onto the dance floor because the bouncers decide that the candidate will clash with the rest of the patrons, or won't project or enhance the desired image. In this case the filter has nothing to do with ability.

And that's the aspect of the extemporaneous problem solving filter that I think is harmful - it isn't really about engineering ability, and it fosters a monoculture that can become detrimental over time.

What's inside?

[ TODO: minimsft.blogspot.com selected posts ]

Altered states

I've had people tell me that the puzzle/coding test helps them to understand a person's thought processes - as if the observer effect and observer bias doesn't apply to this techique.

Try as you might, you can't throw a candidate into a stressful situation without altering his or her behavior. And there's no way to make a test or puzzle that doesn't create stress, when the subject is acutely aware that the outcome of their test may be a major deciding factor in whether they will be offered a job. Oh, except, perhaps, if the subject doesn't care whether they get the job or not. Is that what you are looking for?

Blink

These techniques favor the quick-answer, 'blink' mentality. The same people who have created a culture of rapid-response, quick-with-an-answer personalities now bemoan the ascendancy of those who are best at this kind of behavior.

Interviewing software engineers as if they are engineers

In part two, I'll propose an alternative to the puzzle/test technique.

Resources

Joel on Software "Sorting Resumes" article addresses some of the issues related to testing applicants. Points out that really good engineers don't have to take tests to get in the door.