The new paper from the authors of DORA metrics got me excited and… worried. When misapplied, frameworks like this can, in fact, harm productivity. The paper itself does not mention the prerequisites for when the DevEx measurements are a good tool for the job. Let’s take a second to consider when and (more importantly) why measurements like these can be used.

The paper

The authors of “DevEx: What Actually Drives Productivity” propose 3 core dimensions of developer experience: feedback loops, cognitive load, and flow state. For each, they define 3 categories of metrics: perceptions, workflows, and KPIs. Each combination of dimensions and categories yields several DevEx-related metrics.

First of all, I appreciate this style of “pallet of choices” guidance a lot. This specifically reminds me of the Heart framework described in “Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications”. Both allow the reader to pick and choose whatever he finds fitting his unique situation. And while every company is different, I think there are clear patterns we can observe.

Secondly, the dimensions make a lot of sense to me. Having worked as a developer, team leader, and product manager I’ve seen how long feedback loops, heavy cognitive load, and constant interruptions can make things grind to a halt.

Is the fact that those ring truthful enough of a justification to jump right in with an attempt to measure, learn and improve DevEx at your company?

Who is this for?

The first thing that is not spelled out, but visible in the paper, is measuring DevEx works better in larger organizations. The case studies presented are from eBay and Pfizer.

In my experience, the interest in developer productivity (or, more generally, in performance management) grows with the company size, driven by dissatisfaction with delivery speed. At the same time, people in executive positions get further away from front-line employees and lose the ability to observe the work being done directly. These forces combined are a sure-fire recipe for more supervision and “performance management”.

Is development velocity a problem?

No matter what you do, a 100-developer company will not build software 10x faster than a team of 10 developers.

Some of the more obvious reasons include:

• The size of the code base grows, and now every not-full-decoupled change requires a developer to (at least) consider the rest of the system.
• With an established customer base there is more risk to mitigate in releasing and running the software. More tests, more monitoring, and more focus on the actual quality of the code.
• With more people working together, the amount of communication required to “align” grows exponentially. Some say, if an organization does everything right, it will grow linearly.
• More mature products tend to target larger clients. And those, in turn, will require the software to be built by certifications like SOC 2. Those put a toil on the software development process.

DevEx metrics attempt to quantify those problems and guide toward possible improvements. How well they work will depend on how important those problems have become at your company. This is a long-winded way of saying: pick your battles and know what you want. The metrics are the most effective in driving change if there already is a clear problem and motivation to solve it.

Is DevEx worth improving?

The obvious dismissal would be: “Can you afford not to?”, but the question is real. Software development, especially web app development, requires sitting on the shoulders of two giants: open-source and cloud. Most companies cannot afford to build their own React or home-grow GitHub Actions. We pick and choose what we leverage and try to mash those things together. Every change in the tech stack generates huge costs and might not even pay itself back before you need another one. If the company is fighting against a current of constant, forced changes in the tech stack - can you afford to revolutionize your tech stack motivated by DevEx metrics?

On the other side, there are non-technical reasons for bad DevEx. Those might be ingrained in your company culture and even harder to change, or not worth changing. Some companies are sales driven or have amazingly long release cycles, or business requires the dev teams to be reactive (and responsive). This might be the nature of your business, and changing it would risk having no business at all. Even if you find a DevEx problem, it does not mean it’s worth solving.

Do you need more features?

The push towards DevEx is a push towards the output rather than the business outcome. None of the proposed metrics measures an impact on revenue or customer retention. If this is the right trade-off depends on the stage in the product lifecycle and it’s a long-term strategy.

There are at least two moments in a product lifecycle when delivering features faster is not the top priority.

• When looking for initial product-market fit, the pace of experimentation and learning tends to be a top priority. This might depend on your delivery speed but does not have to. And the distinction might make all the difference in the world.

• When the product starts to decline, there are fewer opportunities for expansion and less need for the type of software engineering work the company did a few years back. If your engineers are looking into making the product more efficient to run and easier to maintain with a smaller crew - this might not be the time to invest in DevEx.

One can argue that DevEx is a way to get “more bag for a buck” from the development teams. I don’t think that’s always the case, even if Product Management is doing its job. Too much (unintended) focus on the output can distract the team from getting the actual outcome you want.

Who will drive those DevEx metrics?

Let’s say that we see a problem, want to solve it and we are willing to trade off some of the outcome focus for it. Let’s even say that the metrics helped discover actual opportunities for improvement. Who will make the change? While it’s possible for grass-roots initiatives to improve DevEx - those initiatives do not need (or want) metrics. Where measuring DevEx pays off is in guiding platform teams.

Development teams building the user-facing product can get their bearings from the ground truth of business metrics like user engagement, retention, or straight-up revenue. Where things break down is an attempt to do the same thing in platform teams’ performance. There is no way to attribute revenue to improvements in CI/CD system in a meaningful way. One can measure stuff like build speed or deployment success rate but those sit closer to “outputs” than “outcomes” even from the platform perspective. Impact on DevEx is a good enough outcome of the platform team’s work.

Do you know how things work?

Metrics exist to inform about the internals of a system we cannot observe directly. We collect signals to infer what is going on in a larger whole. But, for this to work, a good understanding of the system is also essential. In the context of software development teams, it means understanding their way of working.

Attempting to improve DevEx top-down in a situation where there is not enough standardization will not get the expected results. Some teams might already have betters solutions for the problems you aim to solve.

The perfect fit

“DevEx: What Actually Drives Productivity” paper contains a great piece of advice for companies looking to effectively fight the delivery slowdown by investing in engineering platforms. But, as with every piece of advice, it’s not one-size-fits-all. Especially, investing in DevEx measurement might not be the right thing for you to do. Talk to the developers first. Ask about feedback loops, their cognitive load, and the amount of flow state time they get. You will find things worth improving.

The most valuable thing in the paper is the proven insight into what software developers need to be effective. Using it as a conversation starter does not sound like much but might be the exact amount of influence that your company needs.