share
Stack OverflowWhat is the best way to discern an excellent programmer in a job interview?
[+52] [21] Claudiu
[2008-10-27 06:02:16]
[ language-agnostic interview-questions ]
[ http://stackoverflow.com/questions/239185] [DELETED]

In the setting of an interview: What is the best way to reliably identify when somebody is an excellent programmer. By this I mean he is one of those that is 10-15 times more efficient / rapid / better than his peers towards the lower end of the spectrum.

Many of us have heard of the FizzBuzz Problem [1] as a way to weed out the weak ones. Certainly, taking 5-10 minutes to solve that problem is a serious indicator that an applicant is a weak candidate. I posit that a good indicator is being able to solve that as quickly as you can write. This doesn't seem sufficient, though.

Is it maybe something like giving him a moderately complicated buggy program, and seeing how fast he can grok it and identify all the problems with it?

The question assumes that it can be done reliably. - Anthony
not necessarily. a valid reply would be 'there is no way at all' - Claudiu
[+50] [2008-10-27 08:29:36] rboyd [ACCEPTED]

My apologies to anyone who doesn't care for lengthy answers, but I do think it's pretty important to qualify your candidates before you hire them. Anyone who's conducted a significant amount of interviews in this industry knows that most candidates won't last through the first 15-30 minutes of an interview, so most of this list won't be necessary. Just keep in mind how expensive it is (both financially and emotionally) to fire someone before you dismiss my list as overkill. I've tried to list my interview topics here by order of importance.

General Intelligence (brain teasers/logic puzzles)

Computer Science Knowledge

Programming exercises

Knowledge of object-oriented programming techniques and common design patterns [27]

Algorithm analysis [28] (run-time O(n) complexity [29] and storage requirements)

Usage of tools and methodologies

Knowledge of common security vulnerabilities and attacks

Basic Mathematics

  • Numeral systems [39] (convert from one base to another)
  • Probability Theory [40]
  • Distance between two points on Cartesian plane (Pythagorean theorem)
  • Square root (Heron of Alexandria, successive approximation)

Cryptography [41]

  • Public key cryptography
  • Symmetric-key cryptography
  • Hash functions
  • Cryptographic protocols (secret sharing, zero-knowledge proofs)

Discrete Mathematics [42]

  • Logic
  • Set theory
  • Graph theory
  • Information theory
  • Combinatorics
  • Proofs (like existence of irrational numbers, infinite primes)

You might also want to take a look at the book Programming Interviews Exposed [43]. It's a good reference on the topic.

[1] http://www.braingle.com/brainteasers/7824/100-closed-lockers.html
[2] http://puzzlepage.blogspot.com/2008/02/crossing-bridge.html
[3] http://www.redhotpawn.com/board/showthread.php?threadid=24129
[4] http://brainden.com/forum/index.php?s=69929cfdecc74efee37a45cb942067f7&showtopic=4801&st=0
[5] http://en.wikipedia.org/wiki/Data_structure
[6] http://en.wikipedia.org/wiki/Stack_(data_structure)
[7] http://en.wikipedia.org/wiki/Queue_(data_structure)
[8] http://en.wikipedia.org/wiki/Linked_list
[9] http://en.wikipedia.org/wiki/Tree_data_structure
[10] http://en.wikipedia.org/wiki/Algorithm
[11] http://en.wikipedia.org/wiki/Sorting_algorithm
[12] http://en.wikipedia.org/wiki/Search_algorithm
[13] http://en.wikipedia.org/wiki/Shortest_path_problem
[14] http://en.wikipedia.org/wiki/Euclidean_algorithm
[15] http://en.wikipedia.org/wiki/Hamiltonian_path
[16] http://en.wikipedia.org/wiki/Data_compression
[17] http://en.wikipedia.org/wiki/Recursive_algorithm
[18] http://en.wikipedia.org/wiki/Model_of_computation
[19] http://en.wikipedia.org/wiki/Finite_state_machine
[20] http://en.wikipedia.org/wiki/String_searching_algorithm
[21] http://en.wikipedia.org/wiki/Regular_expression
[22] http://en.wikipedia.org/wiki/Backus-Naur_form
[23] http://en.wikipedia.org/wiki/Greatest_common_divisor
[24] http://en.wikipedia.org/wiki/Factorial
[25] http://en.wikipedia.org/wiki/Fibonacci_number
[26] http://en.wikipedia.org/wiki/Tower_of_Hanoi
[27] http://en.wikipedia.org/wiki/Software_design_pattern
[28] http://en.wikipedia.org/wiki/Analysis_of_algorithms
[29] http://en.wikipedia.org/wiki/Big_O_notation
[30] http://en.wikipedia.org/wiki/Debug
[31] http://en.wikipedia.org/wiki/Revision_control
[32] http://en.wikipedia.org/wiki/Test-driven_development
[33] http://en.wikipedia.org/wiki/Code_coverage
[34] http://en.wikipedia.org/wiki/Continuous_Integration
[35] http://en.wikipedia.org/wiki/Buffer_overflow
[36] http://en.wikipedia.org/wiki/Heap_overflow
[37] http://en.wikipedia.org/wiki/SQL_injection
[38] http://en.wikipedia.org/wiki/Cross-site_scripting
[39] http://en.wikipedia.org/wiki/Numeral_system
[40] http://en.wikipedia.org/wiki/Probability_theory
[41] http://en.wikipedia.org/wiki/Cryptography
[42] http://en.wikipedia.org/wiki/Discrete_mathematics
[43] http://rads.stackoverflow.com/amzn/click/0471383562

(4) Phew, that must be some long interview. - Rick Minerich
(2) Damn I've seen so many programmers that apply for a senior developer jobs that can't even do a simple factorial in any language. I'd love to see someone who can answer only half of this. - Eric Hogue
(4) Today I attempted to solve "Crossing Bridges" with my ACM Programming competition teammate. The only difference is that we have to solve it for any number of people. It took us about 30 minutes to solve for any number of N people.. but in an interview setting I feel like puzzles are a poor metric. - Simucal
(23) In an interview, puzzles suck because the interviewee is nervous, not thinking straight, etc. In addition, many puzzels are a-ha! type things that really don't tell you anything about the candidate. - Robert C. Barth
(4) I find these mathematical problems of high difficulty. After you've finished your Computer Science degree for 10 years, how can you remember how to prove irrational numbers and such? - lb
(2) The trouble with puzzles is that most people haven't solved them; they've just seen the answers before. So the most savvy candidates will pretend not to have seen them and haltingly work out the answer they already know. If your goal is to hire smart people, not deceitful people, then puzzles are a bad choice. - Kyralessa
1
[+20] [2008-10-30 00:00:30] Domchi

Ah, the eternal question.

I did a lot of interviews this year (I have two candidates scheduled tomorrow), and in my experience, hiring is more about gut feeling and people skills, and less about technical knowledge.

  1. Take your time with CVs. Some CVs can be rejected in seconds, some take half an hour. Sometimes I think about candidate based on CV a lot longer than I interview him. A few times I've prepared interview questions specifically for that candidate, even though I typically don't have prepared questions.

  2. Technical knowledge - there is a minimum I want, and this is usually pretty easy to tell. If in doubt, during the interview, talk about projects he mentioned in CV, and go as deep as you need to. This is usually more than enough to tell you what he knows and what makes him tick. Education is not important, previous jobs matter, possible personal projects score high.

  3. Ask about what he wants to do and where he wants to go with his career - do you need what he has, and can you provide what he wants? Also, near the end of interview, I usually ask about a preferred salary. If he's out of my range, or if I won't pay that much for what he knows, that's where we end the interview.

  4. Most importantly, candidate must fit into team, and I must feel confident that we'll be able to work together. I don't need to like him, but I must be able to handle him, and he needs to be able to handle me. If that's not the case, I'll pass, because I won't be able to use his technical knowledge. On the other hand, if this is the case, and if he's a quick learner, his lack of technical knowledge won't prevent me from hiring him.

I've trained girls from HR to pass me any CVs as soon as they get them; I schedule interview personally, as fast as I can (ideally day after tomorrow after receiving CV for good CVs). Then he gets half-an-hour or one hour interview with me and at least one co-worker (usually my boss or team-member), where I get to know him and answer any questions. Even if I reject his application on the spot, he gets 20-30 min tour of the company and I talk about what we do and how we do it. Then I send him to HR for psycho test and a bit of really really basic paper coding/SQL. Both tests almost never play a significant role in my decision, it's more of a verification that I judged correctly in the interview. After results, it's 15-min talk where I make him an offer, and if we negotiate the terms we're both happy with, he's hired.

This is a process I had to fight for through company bureaucracy, after missing a couple of great candidates, and which works because I'm the one that decides about hiring (although I do listen the advice from both HR and co-workers, my word is final). More decision-makers, longer process. The longer the process, the more you have to be Google to get top of the crop.

As soon as I'm certain that is a no-match, I end the interview, he gets the company tour and it's over. This might be as short as two minutes on the phone while scheduling interview. Even if you reject a candidate, sell the company. If you did a good job, good hire can come through word-of-mouth from rejected candidate.

Also, one tip. Do send rejection letters (or e-mails) for each application you get. In my current company I usually leave that to HR (apart from those I tell during the interview), but at one point it was priceless getting delighted response from rejected candidate in the lines of "THANK YOU! You're the first company which actually responded instead of leaving me wondering if they'll reply one day!"


Psychological testing? - Ink-Jet
(2) @Ink-Jet: nope, psycho testing was correct - the candidate is asked to write code which will be maintained by a violent man who also knows his/her home address. - Grundlefleck
That's what I read it as the first time, to be honest. - Ink-Jet
@Grundlefleck - yep, that's about correct. :) - Domchi
2
[+15] [2008-10-30 00:10:29] Rich

This answer is a little outside the box, but I think it's a valuable point.

The very best programmers rarely interview. They don't have to. If your company is particularly world-changing, or excitingly shrouded in secrecy, or several programmers they respect have gone there, then they might apply, but ordinarily great programmers get jobs through their network of associates, not by sending in résumés.

So: the best way to tell an excellent programmer in a job interview is that he's not there.


so true... great point. :) - Arnis L.
3
[+13] [2008-10-29 21:33:41] Andy Lester

Any answer must include code samples. Hiring a programmer without seeing his or her code is liking hiring a chef without trying his or her cooking.


4
[+7] [2008-10-27 16:43:04] interstar

Possibly an "excellent" programmer isn't coming to you for an interview. You probably have to steal him from someone else.


doh! this answer seems to be becoming popular. Just as I have to start going out and applying for jobs ... - interstar
5
[+6] [2008-10-27 06:16:40] Jouni K. Seppänen

There are different time scales at which someone can be "fast": some smart people can solve difficult puzzles in seconds, but some smart people produce a lot of good code in a month, even though they might not be that fast at interview questions.

Ask the candidates if they are active in any open source project where you could review some of their code, and spend some time reading those projects' mailing list archives and commit logs. That will tell you a whole lot more than anything the candidates can demonstrate in an interview. (Of course this cannot replace the interview, since not all good coders do open source work.)


6
[+5] [2008-10-29 21:14:32] sergdev

The book " Smart and Gets Things Done: Joel Spolsky's Concise Guide to Finding the Best Technical Talent [1]" may help to find an answer.

Table of content:

  • Introduction
  • Chapter 1: "Hitting the High Notes"
  • Chapter 2: "Finding Great Developers"
  • Chapter 3: "A Field Guide to Developers"
  • Chapter 4: "Sorting Resumes"
  • Chapter 5: "The Phone Screen"
  • Chapter 6: "The Guerrilla Guide to Interviewing"
  • Chapter 7: "Fixing Suboptimal Teams"
  • Appendix: "The Joel Test"

The Article by Joel "The Guerrilla Guide to Interviewing (version 3)" [2] can be helpful as well.

And the article "Done, and Gets Things Smart" [3] by Steve Yegge on the topic.

[1] http://www.joelonsoftware.com/items/2007/06/05.html
[2] http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html
[3] http://steve-yegge.blogspot.com/2008/06/done-and-gets-things-smart.html

7
[+5] [2008-10-29 21:17:12] John the Statistician
8
[+4] [2008-11-04 17:18:36] Huntrods

One way to tell the passionate programmers from the "I just want a job" programmers is to ask them what book they are reading this week. Then ask them about the books they have read in the past weeks.

I have found that the passionate programmers are ALWAYS reading, and usually the list will include a few programming / Comp. Sci. books in the recent list.

It's not just about keeping "up with the profession" - passionate programmers have a desire and love for programming, and tend to devour material on a variety of topics - not just the whatever language they are using now, but methodologies, other languages (especially new or "weird" or ancient ones), other aspects of IT (maybe robotics, or AI, or games, or...)

If they don't have a recent booklist at all, then they are probably not much of a programmer, in my experience.

Cheers,

-R


(2) My recent book list is almost always fiction. My recent technical reading is almost entirely online, because that's more up to date. - Darron
Better yet, ask them what book they wrote this month. :) - brian d foy
9
[+4] [2008-12-20 01:04:25] MusiGenesis

Tell me what?


Well, I'll be the first to say that I get it. - eleven81
What took you a month and a half? :) - MusiGenesis
hahahhahaaaaa... ^^ - Arnis L.
Please explain "Tell me what". I didn't understand as i'm an idiot. - Mahesh
@Mahesh: imagine that I consider myself to be an excellent programmer, and you will get the joke. :) - MusiGenesis
good one :):):) - Jahanzeb Farooq
Tell you what: if it had been directed at you, you would have understood that the answer to your question is the best way; the OP is just asking what it is. - intuited
@intuited: huh? - MusiGenesis
@MusiGenesis: okay, well you didn't get the bonus question, but you still did pretty good. - intuited
@intuited: huh? (I can keep this up all day) - MusiGenesis
10
[+3] [2008-10-30 00:24:30] twk

You should mainly judge the work that they've already done. Any code or ideas that somebody generates during an anxiety-ridden interview are a poor proxy for what they can actually produce on a team.

To do coding challenges, use IM with something like codepad.com and let them do it from the comfort of their own house. Do you write much of your code on a whiteboard in front of your boss, with a 30 minute deadline and your bonus on the line? I don't.

So, is the interview pointless? No, but the emphasis should be on them explaining what they've done and exactly what they've contributed.

You are also going to be subject to all kinds of psychological biases once you meet someone face to face. Don't accidentally hire a programmer because they made better eye contact or are taller than someone else. To route around these, I would conduct as much of the interview as possible over IM/email before you meet them face to face.


11
[+3] [2008-10-29 21:37:38] ManiacPsycho

Ask them a series of questions which require them to code, and have the questions get harder. If they seem to enjoy the challenge, you probably have a live one.

If they can't answer the first easy question, like "write a for loop" or something stupidly easy, then you know this person can't code.


12
[+2] [2008-11-25 18:58:09] ManiacPsycho

Have them code on a whiteboard. Only way you'll be able to tell if they know how to write code.


Don't know why this got downvoted. If a programmer can't write code on a whiteboard, what makes you think they'll be able to write it on a computer? - Kristopher Johnson
(1) @Kristopher: If a programmer can write good code on a computer, what makes you think they'll be able to write it on a whiteboard? Those are considerably different environments. - David Thornley
The "whiteboard test" is not intended to simulate actual coding. It's a chance to see how the candidate thinks, whether the candidate can describe what they are doing, how quickly the candidate forms a solution in their head, etc. Someone who just stares at the whiteboard with no idea what to write will probably have the same problem at a computer. - Kristopher Johnson
13
[+2] [2008-12-27 08:02:29] community_owned

http://www.programmerinterview.com/ is a good resource if you're looking for some practice interview questions. It covers quite a few subjects, and has some clear explanations


14
[+1] [2008-12-27 08:31:18] Stephen Cox

The language doesn't matter. Logic does. I mean IDE's and compliers are so good these days that any good programmer can pick up any language (ok maybe not assembler) in a week; become decent in a couple of weeks and be very good in a couple of months.

It's his (her's) brain you need to confirm. And you do that my talking. I ask them to solve simple problems. Not by writing code but by stepping me through their logic to come to a solution.

But I admit, if he can't write a simple loop counting 1 to 10, you got troubles.


15
[+1] [2009-02-02 20:15:55] mdm

Having a good programmer present in the interview is best in my opinion.

Only an expert can judge if the applicant just knows a lot of interview questions or if he is actually thinking about the problems and can go into detail. Remember, you don't want to hire people to solve interview puzzles, you want to hire them to get actual work done.

Puzzles are to exclude people who don't get the basics right. If you want to test for skill, prepare a few things you (or your "good programmer") can go into detail about and focus on the one the applicant has to think about for a while. How does he approach problems that he doesn't immediately know the solution of?


16
[+1] [2008-10-29 21:41:53] Chris Pietschmann

First of all, there's one way you can get an idea before the interview even starts:

If they have a blog or contribute to one or more open source projects, just look at the code and articles they've written. First of all, if they've done either of these then they have initiative to get things done. Also, you can compare these things to the work experience they have listed on their resume and get an idea if they go home and learn more after work, or if they go home and forget about work after 5pm.

Essentially, Do they have passion about programing or not? That's the real question.


17
[+1] [2008-10-27 06:53:50] Ather

One criteria that I use is to see the 'kind' of languages and tools he has worked on, either in academic or in professional projects and what exactly he achieved. Has he always worked at the application level using standard libraries (always a C# or VB6 guy?) Or has he done a project using C in Linux dealing with hardcore stuff like pointers, memory management, recursion, process synchornization, mutual exclusion, events etc. If he has always used these core and fundamental concepts under some abstraction layer, I will be doubtful.

This is obviously in addition to making him write code. Nothing is a substitute for that. I do however cater for the fact that some people can write code faster than others, and people have different response time when in an interview limelight.


recursion, process synchronization, mutual exclusion. Those technologies are equally important whether you work with C#, VB.NET, C or assembly language. - erikkallen
18
[0] [2008-10-27 06:41:25] Yes Fish...

An excellent programmer will be able to work with those lower-spectrum peers too. As long as they can do the test and don't wallow in their ego then you have a good candidate, no?

That fizzbuzz test is kinda funny though. The solution I can think of uses the modulo operator. Which I only know from working out character-sheet mapping coordinates (never mentioned in school or college). Would the average programmer even know about that or have I had crap education?


I am surprised that you did not come across the modulo operator. I was introduced to it in the different languages I have learned over the year. - David Segonds
(1) If you majored in CS in college, Modulo operator is Programming 101 - Simucal
surprisingly stuff like bit-shift and modulo gets skipped over in college - Claudiu
I think it depends what kind of things they are trying to teach you at college. I don't think I've ever used modulo in a real-world problem, nor taught it explicitly. But it's very common in this kind of exercise (and in exam questions). - interstar
seems like my professor mentioned it in CPL: the magic What Implementors Do With Negative Parameters (this absolutely nobody uses) - Ellery Newcomer
Bit shifts and modulo were covered in the 11th grade. - eleven81
Actually, they're both generally taught in elementary school; at that stage they're referred to as "the remainder" and "multiplying by 10". - intuited
19
[0] [2008-10-27 06:49:59] Ronny
20
[0] [2009-02-02 20:30:38] mannicken

I don't think that you should talk about passion on the interview. Frankly, it sounds like a company that looks for 'passion' really means 'working for no money for the idea'.

Passion doesn't even warrant excellence. I mean I spend almost all my life programming, reading about programming, learning crazy languages like Erlang or Clojure and I don't get paid for any of it. Yet I suck at programming.

I think excellent programmer should have a track of successful projects they have been actively involved in. Thus making a programmer write anything above basic FizzBuzz is unnecessary in the interview. Talk about their past projects and what they did. Are you hiring programmers to solve Rubik's cubes and count marbles or work on long and big and exhausting software projects of more than 50 lines of coe?


21