Is code an asset or a liability?
Code is an asset, but each line is a liability. Let’s explain.
McKinsey says they can measure developer productivity. Kent Beck says they can’t.
It's rubbish, says Grady Brooch. This kind of surveillance makes geeks feel less safe, says Kent Beck.
Rarely has something triggered such a strong response in the software engineering community when mentioning productivity as McKinsey’s new article titled “Yes, you can measure developer productivity”. OK, except for when Elon Musk took over Twitter and wanted to make layoff decisions by counting the lines of code (which engineers allegedly had to print out!).
The developer community was probably maddened just because someone dared to state that developer productivity can be measured. To add insult to injury, the company that supposedly cracked the methodology is not a tech company, and it’s unclear how many developers, engineering managers, or tech leaders were involved.
Is engineering too complex to measure or not?
The article says the time is high for developers to stop presenting their jobs as so creative and abstract that their output cannot be measured:
Leaders and developers alike need to move past the outdated notion that leaders “cannot” understand the intricacies of software engineering, or that engineering is too complex to measure.
Some of the influential voices in the industry were quick to react. Grady Booch simply posted “It’s rubbish”, and Kent Beck commented on Linkedin:
McKinsey claims it’s possible to measure developer productivity. The report is so absurd & naive that it makes no sense to critique it in detail.
Only to elaborate his critique more in detail because:
What they published damages people I care about. I’m here to help geeks feel safe in the world. This kind of surveillance makes geeks feel less safe.
DORA, SPACE, and McKinsey?!
The other annoying fact for the developer community was McKinsey claiming they have “developed an approach to measuring software developer productivity that is easier to deploy with surveys or existing data (such as in backlog management tools).”
This new approach, the article claims, has been implemented at nearly 20 companies, and the initial results are promising. All while the whole industry has been doing research for decades and not finding the definitive solution to measuring developer productivity!
The McKinsey authors do reference the two well-researched and industry-accepted methodologies, such as DORA and SPACE metrics.
DORA metrics – created by Dr. Nicole Forsgren and her DevOps Research and Assessment Team, acquired by Google – measure four key metrics to identify Elite, High, Medium, and Low performing teams:
- Deployment frequency (DF)
- Lead time for changes (MLT)
- Mean time to recovery (MTTR)
- Change failure rate (CFR)
- SPACE metrics, developed by GitHub and Microsoft in an effort to expand DORA metrics, stand for:
- Satisfaction and well-being
- Performance
- Activity
- Communication and collaboration
- Efficiency and flow
The McKinsey article also warns about common pitfalls and absurdities that happen when metrics such as the number of lines of code or the frequency of commits are used (some developers make and commit unnecessary changes to increase the number of commits). While listing some common sense remarks on developer work – the less time spent on less-fulfilling activities, the better, the three metrics they propose don’t bring much new to the table.
McKinsey authors propose companies should measure the following:
- Developer Velocity Index benchmark – an internal survey
- Contribution analysis – assessing contributions by individuals to a team’s backlog (starting with data from backlog management tools such as Jira and normalizing data using a proprietary algorithm to account for nuances)
- Talent capability score – a summary of a specific organization’s individual knowledge, skills, and abilities.
People will optimize for whatever companies measure
While admitting that the article’s title is misleading, Infobip’s VP of Engineering Damir Prusac comments that the proposed model leaves enough space to adjust measurements to each organization’s specific needs depending on the actual challenges.
For developers themselves, especially new generations, it is important to make a maximum possible impact.
Damir Prusac
The developer and engineering community has always been against focusing on hard-set metrics, especially since they are usually tied to performance reviews and career progression. People will optimize for whatever companies measure, which only sometimes correlates with what’s best for the company.
Measuring developer productivity is a delicate task, says Josip Stuhli, CTO.
One should not blindly rely on metrics, especially if they’re linked to rewards or compensations. If emphasis is placed on the number of commits or the number of changed lines of code, developers might make arbitrary minor changes, even if they are not necessary.
Josip Stuhli
He mentions DORA metrics as a good starting point and adds:
It’s essential to consider what developers value. Most of them prefer coding over attending meetings. By automating repetitive and monotonous tasks, not only are costs reduced, but developer satisfaction also increases, which is vital for maintaining high productivity.
Measure teams, not individuals
Another often-cited source of thought on measuring developer productivity is this blog post by Charity Majors. Charity points out that metrics are great for evaluating how efficiently easy problems are solved – discrete, self-contained, well-understood problems.
The more challenging and novel a problem, the less reliable these metrics will be.
Charity Majors
In fact, to the extent you can reduce a job to a set of metrics, that job can be automated away.
Charity also points out that some of the most impactful engineering work is sometimes invisible on any set of individual metrics, and that is why, when measuring developer productivity, companies should focus on teams, not individuals:
The right way to look at performance is at the team level. Individual engineers don’t own or maintain code; teams do. The team is the irreducible unit of ownership.
So, you need to incentivize people to think about work and spend their time cooperatively, optimizing for what is best for the team.
To learn more about measuring developer productivity, we recommend listening to this recent podcast interview with Dr. Nicole Forsgren, the researcher behind DORA metrics.
