Eight young undergraduates were paired up into four teams and were squirreling away on a technical test I had set them, the test was simply to create a normalised relational database for an online bookstore (think Amazon), and then present their findings to a small team of technical folks who would fire some random questions. There was no right answer, and similarly there were no metrics used to decide a winner - the winners would simply be those who we (the developers) felt were a good fit and naturally worked and thought the same way most of the team did. The problem here is that we didn't have the final say.
During the exercise a colleague, Carl, and I made ourselves available to answer any questions and provide guidance to the candidates since we couldn't be entirely sure if we'd provided all the necessary details. Often what would happen is the first ten minutes or so would be socially awkward for the candidates since they would soon realise that they're technically all working against each other, but they also had to work together. Sometimes individuals would break away and choose to develop a solution entirely by themselves whilst the other team member looked on them with puzzlement. Other times we'd have to split the groups into odd numbers if all the candidates didn't show up on the day, anecdotally, I found that in those scenarios we'd typically end up with more disagreement in the group and tiny bit of resentment over whoever was seen to be the leader. Three's a crowd.
Going into a room full of candidates and introducing ourselves was always interesting, almost immediately you could tell who-was-who - the ones with bravado, the quiet ones, the extroverts, the ones who shake your hand limply but are studious and intent on solving the problem. It was early on in the exercise, Carl and I were quietly trying to weigh up which team might shape up to be a winner - which is actually very difficult to determine by face-value. About fifteen minutes into the exercise the candidates were getting more relaxed and starting to become fully-mentally-engaged in the exercise and asking questions. It soon became obvious that a quiet young man of Asian descent was talking to his team-mate and getting his team-mate to ask the questions. To me this seemed really odd, typically candidates are bursting with enthusiasm to show that they thought of something before anyone else in the room in an effort to score extra points with Carl and me.
I decided to approach him directly and ask him what his thoughts were, his English wasn't great and I'd later find out that he'd only been in the UK for two years. But what he expressed was fairly advanced, he began talking about foreign-keys, their data-types and how that might affect index fragmentation depending on the frequency of certain operations - so he was getting his friend to ask questions like "How many people are using the product?", "What's the conversion rate?", "Can people upload their own books?" and "What's the commission split?". Questions that on the surface of it seem like odd things to ask, until you realise the underlying motivation behind asking them.
It didn't take me long to figure out that this kid could see the system from a thousand yards and it was fairly easy for him to mentally model and then optimise his mental model, he was moving at light speed compared to the other candidates in the room. Carl and I had a winner and our job had become easier, we now just had to filter out one remaining candidate from the remaining seven.
Just before our exercise had started the HR Manager had given them all a psychometric evaluation, it was a written test that asked a few questions to determine some key personality attributes of a candidate. I'd taken the test myself when I'd joined the company, I'd never seen my results and I actually thought it was an odd practice, when I asked about it during my interview the HR Manager told me "Don't worry, it's just to check you're a good fit for the company".
During our technical exercise (and the subsequent Q&A) the technical department elected two winners, so we went and conferred with our HR Manager and some guys from Senior Management. We gave our verdict and almost immediately I could see the colour drain from the HR Managers face:
HR Manager: Oh no. You can't have him, he failed the psychometric
Me: What do you mean by failed? I thought it was something that just determined certain personality attributes?
HR Manager: Well it does, there's no right answer to it, but he got 1% - which is uncharacteristically low
Me: If he got 1%, then surely it's possible for someone to get 100% - so there is a right and a wrong answer?
HR Manager: Yeah, but even if they got 100% it doesn't mean they're a good fit for the company
Me: So... conversely, surely 1% implies that they're also not a bad fit for the company?
HR Manager: No, he's a bad fit
Other Manager: Look Matt, let's just drop it and move on
Me: Normally I might do, but I think all the technical guys are in agreement that he's hands-down the smartest person in that room. Have we considered that the test we gave him was in English, yet English is not his first language?
HR Manager: No, but I don't think that would change the outcome - the questions are to determine his personality type
Me: Yeah, but the questions are in English?
Other Manager: Can we invite him back maybe?
HR Manager: Hmm, well we can't now - he's already seen the test and it would be unfair on the other candidates if we gave him special treatment
The conversation continued and essentially we had to reject him and find a replacement. The psychometric test was supposed to produce a "true" reflection of how someone saw themselves, and I was told it couldn't change over time - i.e. whatever it determined was fixed, immutable and infallible.
About a year later the company wanted to track the psychology of its employees and encouraged everyone to re-take the psychometric test, it's something they were going to do annually. I willingly took the test, not to disprove that it was infallible, but instead as a kind of self-test to see how much I'd changed over time. A lot of other people who took the test got largely the same result as when they joined the company - my results had worsened (by the HR Managers standards) - she later told me that I was anti-authoritarian and more likely to do what I thought was right rather than what I had been instructed to do. I am still baffled to this day about how that is an undesirable attribute, see Milgram Experiment and Nuremberg Defense.
She also told me that if I was re-interviewing at the company then I likely wouldn't get the job based on my psychometric profile, which was actually ironic since I was highly respected in my role and was one of only a handful of people that could convincingly debate technical alternatives with Senior Management.
I still look back on that young candidate who was rejected because of some bogus score on a discriminatory test that ultimately means little or nothing. I hope he's found rich and meaningful employment elsewhere that employs people based on actually having a good conversation with them rather than an arbitrary scoring mechanism designed to be filed in a report to Senior Management.
Additionally - if you want my opinion about technical tests to qualify candidates then I can tell you that I once had the pleasure of using a web-based automated system where candidate scores vary wildly, and sometimes in contradiction to their CV/Resume/Experience. When we decided to give the test to the development team (about 15 developers) - most of them got scores that were lower than our threshold (45%), despite them all being rock-solid developers. Also, there were some candidates who managed to get 95% and above - but would then just be absolutely awful during the interview - we would later discover that they were paying someone to complete the technical test on their behalf.
There is no substitute for taking the time to sit-down and talk to someone.
HN Readers: Thanks for reading - comments are here