Our IT Hiring Process: How and Why

Over the past two years, the MPC IT Core has more than doubled in size. All of this hiring activity is a big commitment of time and energy for our team, so we’ve refactored the way we hire staff, aiming for a process that is sustainable for us, fair for our candidates, and still allows us to get an accurate assessment of a candidate’s fit for our needs.

I remember looking online a couple of years ago for inspiration from other organizations’ processes. I discovered that relatively few have written about this topic. Now that our hiring surge is complete, I wanted to write about our process and share what we’ve learned along the way. It’s the kind of article I wish I had found more of when we were doing our research. While the primary audience for this article is organizations which are thinking about refactoring their hiring process, I hope it is also useful for anyone who is considering applying to work at the MPC in the future, as well.

Before I go any further, I should note that the vast majority of our positions in the IT core are for software development. As a result, this process is designed primarily to assess software developer candidates (although our process for assessing system administrators and other positions is quite similar). I should also note that what I describe here is a snapshot in time of a process which is constantly evolving. There’s no guarantee that this will still be our exact approach to hiring in the future!

The MPC IT Search and Hiring Process

First, the “how”.

Each of our searches is run by a search committee, which consists of one Search Chair and 2-4 other members of our IT and research staff. After an initial screening for required and preferred qualifications, we then use a three-part process for candidate evaluation:

1. Phone Interview with Interactive Coding Exercise. A 45-60 minute phone interview, usually conducted by the search chair. The phone interview starts with about 20 minutes of conversation about background, experience and interest in the job. We then switch to the coding exercises, using an interactive coding tool (currently codinghire.com). We ask a few questions which are designed to take a few minutes each and assess baseline programming competency (e.g. data structure use, conditional logic, looping, and other software development fundamentals). We also ask a few discussion questions which give the candidate an opportunity to demonstrate how they communicate about technical ideas and approach a novel problem.

2. At-Home Coding Exercise. Our at-home exercises are designed to require 4-6 hours of effort and ask the candidate to implement some sort of representative working system (e.g. an MVC webapp for a web developer candidate or setting up, configuring and documenting a web application environment from the ground up for a sysadmin candidate.) We ask candidates to complete the exercise in a three-day window of their choosing.

3. Onsite Interview. Our onsite interviews are a full day. The candidate meets with the hiring committee to present their at-home exercise and have a followup discussion. They also have individual and small group sessions with committee members, their potential supervisor, their potential employees (if it is a supervising position), the IT Director, an HR representative, the Center’s Director or Associate Director, and other researchers from projects the candidate is likely to work on if hired. Lunch with other members of the IT Core rounds out the agenda.

Although it can vary significantly, the process from receiving an application to completing an onsite interview takes about 3-5 weeks.

Why Do We Do It This Way?

Now, the “why”.

Candidate evaluation is essentially about determining whether the needs of the organization align with the capabilities, experience and interests of the candidate. In other words, both parties are trying to answer the question, “Is there a great fit between the person and what the organization needs for this role?” Breaking this down a bit further, in our experience we find that there are three possible outcomes: “poor fit”, “good fit”, and “great fit”.

[Author’s Note, April 2019: In the almost 4 years since I wrote this, the topic of “cultural fit” and how seeking cultural fit can be harmful has become an important topic in the IT industry. Here, I was not talking about cultural fit with the team members, but rather the fit between the person and the job that needs to be done - “does this candidate have both the technical skills and the soft skills necessary to perform the job successfully?” At ISRDI we are committed to building a diverse workforce and we aspire to evaluate job candidates in terms of cultural add, not cultural fit.]

Because candidate evaluation is a big investment in time and energy for both the candidate and the organization, your process should aim to answer this question as efficiently as possible. Each part of our process has been designed with this in mind, and has been tweaked over time based on past results. In fairness to both parties, your process should try to arrive at one of the poor/good/great fit conclusions as quickly as possible.

In our experience, the most common way in which we fail to have a great fit for a software development position is when our expectations for coding proficiency are significantly different than a candidate’s capabilities. Therefore, we felt it was important to assess this skill as early as possible in the process, and this is why our phone interview includes an interactive coding exercise. With just a few small coding exercises which have been carefully designed and vetted, we can quickly determine whether there is a poor fit between our expectations and a candidate’s abilities. We believe doing this up front is the most fair thing we can do for our candidates. It also helps keep the overall process efficient for the organization. We find that about 70% of evaluations end at this step, which is encouraging because it means that we’re quickly determining the leading cause of poor fit, a benefit to both the candidate and the organization.

From there we proceed to the at-home coding exercise. The goal of this phase is to get a strong and accurate reading on the candidate’s technical abilities based on a problem that is directly related to the type of work the candidate will do if hired into the position. For that reason, we design our at-home coding exercises using some of our real data and ask the candidate to build a system that is a simplified but representative example of the kinds of things we build on a regular basis. Another benefit of this sort of exercise is that it gives the candidate a great window into what we do at the MPC.

There is enough depth and scope to the assignment that we can start to get some insight into what makes this candidate tick as a developer. Code structure and style, framework choices, data structure preferences, dealing with problems at scale, documentation and testing - all of those things which matter in the real job - there is space for a bit of all of them in the exercise. And because we allow three days to do 4-8 hours of work, it gives candidates a chance not just to implement the solution, but to let it sink in, step away and take care of their other life obligations for a while, and then come back and refactor a bit. We feel like we are getting a candidate’s best work, and not their rushed, time-pressured work. And we feel like we’re getting a strong preview of how they might perform in our workplace with our actual work.

Obviously at this phase the level of time commitment to the process on the part of both the candidate and the organization has increased. The candidate has now spent several hours on the exercise. For the organization’s part, we have a minimum of three individuals grade each submission (and as an aside, these individuals are not the same as the hiring committee - the grading committees stay constant through multiple searches to ensure consistency in grading for any given at-home exercise.) Given the increased commitment, our hope is that the at-home exercise has fairly strong selective power, and the data confirms that for us - the success rate is somewhere around 50%. We feel that the at-home is the ideal next, more detailed step for both parties to move the “Is there a great fit?” question forward.

By the time we get to the onsite interview, we’ve verified that the candidate can probably handle the technical aspects of the job and the candidate has a pretty good sense of the type of work that we would ask them to do. The onsite therefore is primarily about two things: 1. giving the organization a chance to evaluate non-technical, or “soft”, skills, and 2. giving the candidate a chance to learn more about the MPC and see if it is right for them. Hiring is a two-way street, after all.

The onsite is the most energy-intensive portion of the search, but by the time a candidate comes for an onsite, there’s a non-trivial chance we’re going to want to make a job offer, so this is the right time for everyone to invest the most energy. Our goal for this phase is essentially “no stone left unturned”. After an onsite interview, we want everyone to feel like they have all the information they need to make a completely informed decision about whether or not we have arrived at a “great fit”.

What our process ensures is that by the time we’ve reached the onsite, we’ve already accounted for the most common causes of “poor fit” and taken care of those portions of the assessment which are easier to do remotely. We’re now free to really dive deep and see whether we can move from “good fit” to “great fit” by exploring all of those facets that really require the high bandwidth of the in-person interview, such as assessing chemistry with the team, whether the candidate can effectively communicate highly technical information to multiple audiences, and giving the candidate the chance to have access to many members of the organization.

At the end of an onsite, we feel like we have built a substantial relationship with the candidate and that we deeply understand their capabilities and interests, and candidates tell us that they feel like they have a really good handle on what the MPC is about and that they have a clear understanding of the role. Certainly it has been our experience that since we’ve started using this process for hiring, it’s typically been quite apparent when we’ve achieved “great fit” for both parties.

It’s Not For Everyone, and We Know That

We hear a fair amount of feedback from candidates and others about our process, most commonly about the at-home and onsite phases.

For the at-home, we’ve had candidates tell us that it was too much of a time commitment, that the assignment was too complicated, that it was an unreasonable to ask a candidate to do that much work as part of an interview process. We’ve even had candidates see the assignment and then immediately withdraw from consideration. While we agree that it is a significant commitment for the candidate, we have carefully considered alternatives and have yet to find an option that would be less of a commitment and yet still has the same selective power. Before we did an at-home coding exercise, we used to do a coding exercise as part of the onsite interview. However, we discovered that adding time pressure and a live audience (“coding as performance art”) predictably was not an accurate representation of real-world work ability. Viewing submitted code samples is problematic because it’s usually code from a different domain which is solving a very different problem. Plus we don’t know the problem context in order to evaluate the effectiveness of the solution. Simplifying the at-home exercise further (something we’ve looked at on multiple occasions) leads to an exercise which is too shallow to have good predictive power.

For the onsite, we often hear that it’s just too darn long. We’re a really flat organization at the MPC, and major decisions like hiring usually have input from a broad swath of stakeholders. Each step of the itinerary serves what we feel is an important, irreplaceable purpose. So, the all-day onsite is here to stay. We want to get to know each other fairly deeply before making such an important decision, since we’re hoping it’s a multi-year, career-building commitment we’re making to each other when we hire someone.

We know our process isn’t for everyone, and we know we’ve occasionally lost excellent candidates because they had easier paths to employment with other organizations. However, we’ve also won over many candidates-turned-employees because through the process they were able to get a real look at the MPC and our culture and values, and they came to understand the sort of investment we try to make in our team. As hiring is among the most important decisions an organization ever needs to make, we’re willing to make this tradeoff in order to ensure we arrive at the best possible choice. This process has led to a series of fantastically successful hires for the organization, so we feel that we’ve found the right way to do hiring for our organization.

To sum up what we’ve learned throughout this journey, I would encourage others to keep these three things in mind:

1. Make sure your process is fair to the candidate and to the members of your organization in terms of respecting everyone’s time
2. Design ways to uncover the “poor fit” cases as early as possible in the process for as many cases as you can
3. Your process should lead both parties to feel like they have all the information they need to make the right decision - if the process is working, decisions should mostly feel easy.

I hope you’ve found this article helpful - please let us know in the comments if so. Happy Hiring!