Skills Profiles as an alternative to Performance Reviews
Performance reviews have always been a sore subject for me…
Reviews are often a pain in the neck. If you write one, they are very time consuming and with questionable output. I have participated and heard of those reviews where people were not honest because the corporate structure does not allow you to be fully honest (eg. bonuses highly depend on reviews so people have every incentive to game the system). If reviews are not anonymous, one may be afraid/hesitant to provide honest feedback. If they are anonymous, people may not be committed to the reviews because of the overhead that it provides and if there is no way for anyone to check that you're giving it attention, one can get sloppy.
As a result, when you get a review from your peers/managers they may be too vague or not accurate. Add to the fact that many companies do a once a year review makes them not valuable since there is a very large gap between these discussions.
Another alternative to once a year performance reviews are 1-on-1's. Those small meetings, done every few weeks, or monthly or whatever other interval fits in your situation. Those are better because they are in person and happen more regularly so both parties are committed. My problem with those is that
It's pretty hard to give negative feedback so both parties have to be completely comfortable with each other (I'm guilty of this for sure). Depending on how comfortable you are with this, it can be a pretty awkward situation
You only get one opinion form your manager/supervisor. So you only get one opinion and it's likely that they do not know all aspects of your work
If the feedback is negative, there is nowhere to hide - both of you are right here in the same room or on a video call.
In my current role, I'm in charge of holding 1-on-1's with a group of developers and I found myself thinking about this problem a lot. We had both yearly performance reviews and 1-on-1's but neither of them solved the problems.
So I found myself talking outloud to the CTO and a few experienced developers in the summer of 2017 and what came out of this discussion was one of the best implementation that I've seen (subjectively of course).
Let's talk about Developer Profiles
Prerequisites
Team setup:
At the time of this system, we had 11 developers of varying skill levels in different skills,
All developers work on the same overall system/team,
There are different projects that developers work on, however,
Most projects live within the same (or similar) overarching ecosystem (eg. similar stack),
There are projects that are drastically different, but not enough to make a significant impact in overall evaluation,
From management point of view, all developers report to the CTO,
However, many day to day activities are handing by the Team Lead (happens to be me)
Developer Profiles - the setup
Inspired by a blog post from my co-worker, 4 developers and CTO came up with a gamified version of performance reviews. Rather than summarizing things into a board performance for a person, we decided that we needed to break things down further into specific strengths and weakness for various skills that were important for someone to be successful on our team.
We need to provide feedback to developers that was:
Easy to understand - we needed a way to provide an easy way for the team members to know where they stand in different skills
Actionable - provide concrete examples of leveling up
Informed - or less biases, feedback cannot come from one "authority", people providing feedback should have checks and balances
Well intentioned - the goal of the feedback is not to criticizes but to provide opportunity to level up
Well thought-out - it cannot be something put together in an allnighter
Well rounded - being a developer involves doing many things so feedback must cover different aspects of being a developer, technical and non-technical
Accountable - people giving the reviews must be accountable for their feedback
What we came up was a concept of skill profiles which is loosely based on game profiles in Dungeons and Dragons.
Implementation
Once we decided on the strategy, we got to work on implementation. Here is how we approached all aforementioned topics
Well rounded - We decided on different categories that are most important for our team. These range from hard technical skills (ex.server-side development and client-side development) and go all the way up to soft skills (ex.time management and influencing others)
Well thought-out - yeah, that was a tough one. We spent a lot of time thinking about what categories fit the bill. We started with a brain dump of all the categories that we could think of. And then one by one eliminated things that were too broad, too confusing, or not fully relevant at the time.
Well intentioned - For each categories we decided on how we would rate a given reviewee. We decided to go with a few steps
Novice - someone who is completely new to the the skill
Apprentice - person who has some experience with the skill, they can work independently in many cases but may require help
Journeyperson - someone who can get things done but may or may not be able to teach others
Master - someone who can use a skill for new set of problems and is able to teach/mentor others
Grand Master - People-will-remember-you-after-you-die kind of level
Idea of these levels to see everyone move up from one level to the next one in the next iteration of developer profiles
Informed - We have a total of five people in a room (in our case, virtual) duke it out for every single person being reviewed. Person leading the discussion would announce a category and everyone else types in one letter in Slack at the same time - N, A, J, M. We never had any GM's. If there is agreement, great. If there was a disagreement, people would voice their opinions on why they disagreed. We don't move on until everyone agrees.
This is especially important for the cases where a developer work on specialized projects that are not visible to the whole team so one reviewer may tip the scale by explaining a trait about reviewee that others do no know about
This is by far the most time consuming part of the process, but there is a good news. More on this later
In my opinion, this is also one of the most controversial parts of the whole system as 5 people in a room will inadvertently create bias so it’s important to have people who are not afraid to speak out and have variety of opinions.
Easy to Understand - the end result is a matrix with all the core skills for a team with grades and explaining comments.
So in theory this is easy to process.
In practice, it's not always easy to understand all the categories right away so it does require some team preparedness
Actionable - this is by the most important part. What makes the results of this exercise actionable is that the committee needs to come up with concrete suggestions on what a person needs to do to level up. This can include
Taking a course in particular topics
Pairing with someone when they work on a certain skill
Asking for projects that exercise a given weakness
Etc
Another important part is that after the developer profile is complete, it is easier (although not easy) to make parallels to the particular skill in everyday life
Accountable - given that all decisions are done within a group of 5 people performed live, people cannot be distracted and cannot slack off (you know, for the most part). However, second part is that in order to make these decisions more trustworthy, we made all developer profiles for senior developers (those involved in the grading process) public to the whole team. This has a few implications
This makes it easier to understand what the ratings mean by setting an example
Creates a bit of vulnerability since everyone making decisions is exposed and is on display, hence, hopefully adding more trust to the process
Exposes that one doesn't need to be an expert in all aspects. Many of the senior developers in the group had several ratings that were either a Novice or an Apprentice. After all, we can't be good at everything.
I think this a very important parts of this process and I believe that if we haven't done this, it would jeopardize legitimacy of the whole process. All other developers have a choice of whether or not to make their skills profiles public. In my mind, ideally all developer profiles are public. But that won't happen any time soon. In our case, just one developer made their profile public.
The Skill Sheet
In the end we came up with a matrix like this
Note that there are three columns about how a reviewer rates themselves, how their manager rates them and how other senior developers rate them. It is very possible that the columns do not match and that is ok. It will provide for some surprises (positive or negative, so be ready for both)
Roll out
In the beginning, going through each profile was a bit difficult.
We had to discuss all details
Discard categories that were too broad/confusing
Argue over the meaning of each ratings
Document all the categories (this will be important for the future as there needs to be a way for anyone come and review meanings of different categories later on)
And just in general, we had to find the flow - although we worked together for a while, this is a brand new type of exercises that we haven't done together
However, after we ironed a lot of details, things were much quicker. In took about an hour per reviewee. This is probably a bit more than one would have spent writing individual reviews but given the quality of output, well worth the time.
By far the most crucial part of rolling out was getting buy in from the rest of the team. This needs to be done before starting actively work on developer profiles.
We explained the intentions, which was to improve current review process. It helped that most of developers didn't actually like previous way of performance reviews
We made it clear that it is the first time we're doing this and it may not work but we will do everything we can to make it work.
Next hardest part was explaining the meaning of each rating. In our case, Journeyperson was a very broad category and we reiterated over and over that this is a very good place to be at. It's important to note that Master is a very high level of achievement, we should all strive to get there but it will undoubtedly take a long time/effort to get there.
After all developer profiles were done, our CTO met with everyone individually and discussed the results of the profiles.
Results
Developing this new way of doing reviews certainly took a long time. But was it worth it?
Initial feedback from developers was very positive. It seems that most people liked the idea. Which was great. But did they like the implementation. Answer to this is a bit more complicated.
One of the main elements of developer profiles was to provide constructive criticism. We needed a way to tell people where they stand. If they were doing well, they should continue the great job but the expectation is that they will help others to get there. If they are not doing that well, expectation is that they will catch up… or
It is a very viable possibility that a person who is not doing well will start re-evaluating their role in the team. This may ultimately lead to them making a hard choice of leaving. I don't think this is a bad thing. I don't view people leaving as an unnatural process. After all we should all reconsider what we're doing once in a while.
I believe that for most of the developers, this was a very positive experience, hopefully because we did a good job preparing them and making sure that this was done not out of bureaucratic need of reviews but out of genuine desire to improve.
For me as a developer, it was very valuable. A few times it was humbling but a few times it showed that I underestimated my skills.
What's next
Now that we went through all this work, does it stop there? I don't think so. We use these developer profiles skills in every day work. When there is an opportunity to point out how one can improve in a particular skill, use that. If someone did a good job in a particular job, point it out.
There are a few other changes that we'd like to do
Perform re-evaluation 6 months from the original experiment
Included non-developers (eg. QA, Product), this may need more work since the same categories do not apply.
Provide a way for 2-way feedback. Developer profiles took away the ability for developers to provide feedback back to the senior developers. And this is a big problem in my book. So we will be adding this back.
Challenges
Although for the most part, this worked out pretty well. I noticed a few things
Most obvious, is the time that it takes to create these profiles. Everyone needs to agree that it's a worthy exercise
As a team-lead, when I had 1-on-1's with developers after CTO met with them to discuss developer profiles, I needed to do some clarifications. My guess is that some of the categories were too deep that high level manager didn't have full visibility into.
Not everyone understood all the categories the same way. This is an ongoing problem and we're becoming better by communicating more.
Whether this system ties into compensation is still an unanswered question.
A few people were surprised by their results - some were negative, some were positive. It's not necessarily a problem from implementation point of view but it may cause strong emotions, so be ready.
We have a pretty small team, so it worked out ok. However, I don't know what changes would need to happen for larger team.
Half a year in, I do see some people are taking these profiles more seriously than others.
Was it worth it?
Yes, there are problems with using developer profiles. Yes, there is more to be done. But is it worth trying it out. In our team, absolutely! Only time will tell if this method of providing feedback is better than traditional performance reviews.
I'd be curious to know your stories of how you handle performance reviews. And if you decide to give this method a try, it would be great to find out how it worked out for you. Please shoot me an email at ilyakatz _ at _ gmail












