DORA, SPACE, GSM, or DevEx: How to measure developer productivity?

With all the conversation around McKinsey’s article on measuring productivity, I thought it would be useful to summarise some of the industry-accepted methodologies. Measuring developer experience, SPACE metrics, DORA metrics, and GSM (Goals/Signals/Metrics) are a couple of accepted frameworks for measuring developer productivity.

DevEx: What actually Drives Productivity

In their paper called “DevEx: What Actually Drives Productivity,” from April 2023, Abi Noda, Margaret-Anne Storey, Nicole Forsgren, and Michaela Geriler, provided a framework for measuring and improving developer experience (DevEx).

They found that developers are happier when they feel more productive, and that they cannot deliver as much value as possible when they have obstacles. Different things cause a terrible developer experience, such as interruptions, poor tooling, unrealistic deadlines, working on low-value tasks, and more.

The authors identified the three core dimensions of developer experience that have a direct effect on it:

1. Feedback loops – the speed and quality of responses to actions performed
2. Cognitive load – the amount of mental processing required for a developer to perform a task
3. Flow state – a mental state in which a person performing an activity is fully immersed in a feeling of energized focus, full involvement, and enjoyment.

So, to create DevEx metrics, we can use their three core dimensions with two methods:

• Perceptions – Gathered through surveys
• Workflows – Gathered from systems

So, for cognitive load for perceptions, we can ask how complex a codebase is or the ease of understanding documentation. On the workflow side, we can check the time it takes to get answers to technical questions or the frequency of documentation improvements.

Example DevEx Metrics (Abi Noda et al., 2023)

In DevEx, we should capture developer perceptions and their workflows. Picking the right KPIs is essential to track overall success.

The SPACE Framework for Team Productivity

In a research paper by Nicole Forsgren and her colleagues, “The SPACE of Developer Productivity” (2021.), the authors defined a framework as a systematic approach to measuring, understanding, and optimizing engineering productivity. It encourages leaders to take a comprehensive approach to productivity, communicating measurements with one another and connecting them to team objectives. The five aspects are used to categorize engineering productivity, called the Space Framework.

What are these categories:

• S – Satisfaction and well-being
  Here we measure whether our team members are fulfilled and happy usually by using some surveys. Why do we do this? Because satisfaction is correlated with productivity. Unhappy teams that are productive will burn out sooner rather than later.
• P – Performance
  This is also hard to quantify because producing more code in a unit of time is not a measure of high-quality code or productivity. Here we can measure defect rates or change failure rates to measure it, which means the percentage of deployments causing a failure in production. Every loss of output will harm the productivity of a team. Also, if we count the number of merged PRs over time, it’s correlated to production.
• A – Activity
  Activities are usually visible. Here, we can measure the number of commits per day or deployment frequency, i.e., how often we push new features to production.
• C – Collaboration & Communication
  We want extensive and effective collaboration between individuals and groups for a productive team. In addition, productive teams usually rely on high transparency and awareness of other people’s work. Here we can measure PR review time, quality of meetings, and knowledge sharing.
• E- Efficiency & Flow
  With flow, we measure individual efficiency to complete some work fast and without interruption, while efficiency means the same but on the team level. Our goal is to foster an environment where developers may experience and keep the flow for the longest possible period each day while also assisting them in feeling content with their routines.

To implement the SPACE framework, the authors recommend aligning three areas with company goals and team priorities. First, when a team selects a measure, this reflects team values. Here we want to start from team-level metrics, and when we succeed, we can roll it out to the broader organization.

Example metrics (“The SPACE of Developer Productivity,” N. Forsgren et al., 2021)
 

DORA Metrics for Team Productivity

The other way to measure team productivity is DORA metrics. With these metrics, we are evaluating team performance based on the following:

1. Lead time for changes is the time between a commit and production. Elite performers do this in less than one hour, while medium performers need one day to one week.
2. Deployment frequency is how often we ship changes. Elite performers do this multiple times per day, while medium ones do it once a month to once every six months.
3. The mean time to recovery is the average time it takes your team to restore service when there’s an outage. Elite performers do this in less than one hour, while medium ones do this in a day to one week.
4. The change failure rate is the percentage of releases that result in downtime. Elite performers are 0-15%, while medium performers are 16-30%.

The lead time for modifications and the deployment frequency reveal a team’s velocity and how quickly they react to the constantly changing needs of consumers. The stability of service and how responsive the group is to service outages or failures are indicated by the mean time to recovery and change failure rate.
Comparing all four essential criteria, one can assess how successfully their company balances speed and stability.

Goals/Signals/Metrics (GSM) for measuring developer productivity

Yet, there are other productivity frameworks, too, such as “Goals/Signals/Metrics (GSM)” metrics from Google. In this framework, you first agree that there is a problem worth solving, then we set a goal on what you want to achieve and decide which statements, when actual, would note that you are making progress (signals).

Finally, we arrive at metrics that you want to measure but focus more on the desired outcome, not just the metric. E.g., the goal could be “Make sure that engineers have more focus time,” signals could be “Engineers report fewer cases of meeting overload,” while metrics could be “Engineer focus time.” For metrics, you can build a team Dashboard that will collect them in one place, so it’s easy to analyze them.

Learn more about the GSM method of measuring developer productivity in this video by Google.