Developer productivity is a touchy subject. Those who are putting in the effort don’t want to be treated as if they were a machine, and those who don’t meet expectations, no matter the reason, don’t want to be exposed or pointed fingers at.

But productivity doesn’t revolve just around output—it’s a baseline for general developer experience and satisfaction.

Yes, you can measure developer work output, be very exact with numbers and how many lines or commits they’ve done – but that won’t tell you a thing about the “mental engineering” required and if that commit really helped solve technical and user issues or was that code actually good and comprehensive.

Measuring difficult-to-measure things, including those that are not reflected in developer log data, is actually the most important aspect of developer satisfaction and productivity.

The Google’s approach

Google has approached its research on measuring Developer Experience with that in mind, and we can all learn from it. A recently published paper on the topic covers their findings from the Engineering Satisfaction (EngSat) survey, which they have used since 2018 to measure and understand developer productivity.

Having that insight helped them not only track the impact of extreme situations on developers—such as the COVID-19 pandemic—but also reduce technical debt, which a survey showed to be Google’s main productivity hindrance. It helped them pinpoint the problems more easily, and they partnered with internal teams to develop best practices, management plans, and initiatives aimed at addressing technical debt.

How did it all start? They’ve gathered a team, now known as the Engineering Productivity Research team, which includes software engineers and UX researchers from diverse fields, such as behavioral economics, psychology, public health, and more. The team’s main goal is to improve developer productivity through various research methods, such as diary studies, surveys, interviews, qualitative research, and log analysis.

How to measure: Speed, ease, and quality

Ciera Jaspan and Collin Green, who lead the Google team, shared their experiences in the Engineering Enablement podcast. They emphasized that no single metric can capture developer productivity.

So, for the research setup, they’ve decided to separate developer productivity into three main categories: speed, ease, and quality, and develop multiple metrics for each one. For instance, they assess speed using logs and developers’ perceptions of their own speed and validate this through diary studies and interviews.

We don’t rely on just one speed metric but use multiple metrics and validate them against each other.

The complexity of these metrics is necessary because their measurements have so many nuances. A great example is the measurement of build times. The team found that log data included automated background builds, which skewed the perceived build latency:

Well, it turned out that the log data included not just builds that engineers kicked off themselves but also a bunch of robotic builds that were happening in the background that the developer wasn’t even paying attention to.

Be mindful of…

Human factors inevitably complicate the relationship between objective metrics and subjective experiences. Using build time as an example, Colin noted that reducing build time doesn’t necessarily result in linear productivity gains.

If a build takes longer than a certain threshold, developers switch tasks, meaning that reducing build time from, say, 30 to 20 minutes doesn’t directly translate to a 10-minute productivity gain. The real impact depends on what else the developer did during the build and how long it takes to resume the original task. He concludes that these non-linearities and qualitative shifts, common in psychology, are crucial to understanding developer productivity and reactions to changes in build latency.

In addition, the aforementioned Google paper states two primary problems that are likely to affect any survey program: increasing survey length and decreasing response rate.

To address the survey length, Google decided to keep a subset of high-level questions focused on productivity, satisfaction, velocity, quality, and flow and occasionally shorten the survey by removing redundant or obsolete questions. This made a significant change for smaller developer groups’ surveys, removing more specific questions in favor of more general questions that better support company-wide decisions.

To combat decreasing response rates, they employed two strategies: sampling and transparency. By splitting the developer population into three random groups and surveying each group quarterly, they reduced the frequency with which each engineer is surveyed, ensuring a large enough sample size for statistical analysis. Additionally, they enhanced transparency and continued participation by distributing a widely accessible quarterly summary report detailing what is measured and how feedback is utilized.

How to get better at measuring dev productivity?

So, what are the key points to take from their experience?

1. Use mixed methods. As we already covered, use multiple methods to measure each aspect of developer productivity. No data source is the ultimate truth; it is necessary to use both quantitative and qualitative data to get comprehensive and validated results. Quantitative data includes logs and behavioral data, while qualitative data comes from diary studies, surveys, and interviews.
   
2. Enable collaboration and multidisciplinary approach. Make sure people who conduct research are equipped with knowledge about quantitative and qualitative methods. Also, enable them to consult freely with multiple other teams and internal experts when necessary. Checking in with developers is a must!
   
3. Watch out for discrepancies. Sometimes, even using different metrics to measure the same thing won’t help you if developers’ input or log data were off in the first place. You will, however, notice that something doesn’t add up in the data. Make sure to investigate properly when you see discrepancies. There is usually a reason.
   
4. Take reliability and scalability into account. Cross-check methods and their findings to ensure reliability across different data sources, especially for qualitative methods. Take advantage of quantitative methods, as these can be passively collected from all developers without additional effort.

The post How Google got better at measuring developer productivity appeared first on ShiftMag.

Like most open-source enthusiasts, Eddie Jaoude got into the field completely by chance and as a hobby. Now, 15 years later, he first made open-source advocacy his full-time job and then scaled it to be his business as the founder of the open-source community Eddie Hub.

Jaoude’s motto is that there is a place in the open-source community for everyone so, as one of the keynote speakers at Infobip’s Shift conference in Zadar, he shared with developers his thoughts on why open-source is still relevant, how contributing to open-source can help them in their careers and what they (don’t) need to do become recognized and good contributors in the OS community.

Open-source helped me a lot to progress as an engineer, I believe it will help you too.

Collaboration, not code

However, Jaoude pointed out that not everything is about code but about collaboration. That’s why one person really can make a difference with their contribution, even though it often does not seem so from the start. He also stresses that the open-source community is full of amazing engineers, and that’s what meant a lot to him.

You have the opportunity to learn from great mentors and gain exceptional experience, he adds, explaining that the experience is all the richer because you get different perspectives from all over the world. Of course, it is also an opportunity to expand your network and accelerate your career.

Contributing to OSS is like brushing teeth

Eddie admits that onboarding is not the OS community’s strong point, but once you’ve done the basics, you can contribute – it doesn’t matter what level you are, because senior members will jump on board. The only thing important is that you start somewhere. 

That’s why Eddie can’t stress enough that the code is only part of the project and that the context of work and contribution is also very important. You can also make a difference by verifying existing bugs, reviewing pull requests, or opening questions when something is not working.

Jaoude is now a recognized and awarded OS contributor and advocate. Of course, the one question he always gets is how engaged one has to be to achieve that. Or how engaged a developer has to be in the OS to even be able to say they’re an OS contributor.

Eddie compares contributing to open-source to brushing teeth:

You don’t even think about brushing your teeth anymore, it’s become a habit. Although it is short-lived, it is very important and you do it every day…

It’s the same with open source – you participate little and often, but you have to be consistent.

How to choose a good project and start contributing?

Of course, there are challenges that need to be addressed, adds Eddie. He is aware that it is difficult for newcomers in the community to find projects that suit them – to develop at a speed that suits them, to use a tech stack that they know, and, after all, to be legitimate and active. 

Likewise, he notices that newcomers are often afraid of screwing something up, so he advises:

If you broke something in the code, congratulations, you probably found a bug and now someone will solve it.

How to be a good contributor to open source projects?

First and foremost, it is important to choose a good project. Eddie advises starting with projects and tools you have already used and are familiar with and looking for OSS initiatives related to them.
 Of course, GitHub search will be the main tool to do that, and as an additional piece of advice, he says that most first-issue projects are, in fact, not a good choice for a first project. Instead, refine your search and look for projects labeled as “good first issue”.

Getting involved in communities is also a good way to find a good project, be it related to a hobby, some idea, or technology. However, he notes, regardless of what you choose:

When you decide to become an active open-source contributor,  focus on one project and one community.

When deciding on a project, pay attention to several things.

First of all, is it contributor-friendly, or is the community generally friendly? Check if they have set guidelines for behavior and contribution to the project and how well they adhere to them.

A good trick for this is to check the pull requests and see the accepted and rejected ones. Those that are closed, and not accepted – open them and see why they are closed. You will learn a lot from the way someone was rejected: was it done correctly, politely, and with an explanation.

Browse, participate, automate 

There is always something that can be automated in the code, Eddie says. Plus, other project collaborators will love you even more if you get to grips with test automation.

When it comes to reviewing pull requests, Eddie warns against approving them blindly; that’s the fastest way to get you removed from the project. 

And when it comes to his advice on how to be a good and valued open-source contributor is the following:

1. Before you get involved, read the project’s contributor guide
2. Never use the main branch of the project, and do not make large pull requests 
3. Search for existing issues before posting new ones
4. No need to DM those who maintain the project as soon as you have made a pull request 
5. Don’t chase the green squares

The post How to contribute to open-source projects with Eddie Jaoude of EddieHub appeared first on ShiftMag.

According to Shawn Wang, founder of Smol.ai, the role is that of an AI Engineer, as he explained in his keynote speech “Software 3.0 and The Emerging AI Developer Landscape” at the Shift conference. 

Wang bases his thesis on the fact that there has been a profound transformation in the AI ​​landscape, thanks to the accessibility of advanced models through APIs or, as he calls it, a once-in-a-generation “shift right” of applied AI. 

Bridge the gap between AI research and application

A wide range of AI tasks that took 5 years and a research team to accomplish in 2013 now require API docs and a spare afternoon in 2023.

But Wang is not one of the emerging prophets saying AI will replace developers in writing code and that all of them will have no choice but to become Prompt Engineers. The job of AI engineers will be to bridge the gap between AI research and application to deal with all the challenges in evaluating, applying, and productizing AI: 

• Evaluating models
• Finding the right tools
• Staying on top of the research and news

 And, according to Wang, it’s a full-time job!

Not trained in ML, but ship AI products

AI Engineers can be found everywhere, from the largest companies like Microsoft and Google to leading-edge startups like Figma, Vercel, and Notion to independent hackers. They don’t do research and are not trained in Machine Learning; some of them may not even be fluent in Python but are still using AI advancements to launch products used by millions in record time.

 When it comes to shipping AI products, you want engineers, not researchers.

Illustration: Shawn Wang

The birth of the AI engineer is fueled by:

Basic models and their possibilities 
Specific capabilities of the underlying models allow them to generalize beyond what they are trained to do. In other words, the people training the models are unaware of their capabilities. 

Non-LLM researchers have the opportunity to find and take advantage of these greater capabilities just by spending more time with the models and applying them to areas that are often undervalued (e.g., Jasper for copywriting).

• There are not enough researchers

Big tech and their big research labs have exhausted the talent to deliver us “AI Research as a Service” APIs. You can’t hire them, but you can rent them. There are about 5,000 LLM researchers in the world but about 50 million software engineers, Wang adds, and such supply constraints dictate that a “middle” class of artificial intelligence engineers will be created to meet the demand.

• The models have nothing to run on

GPU resources are critical to training and running deep learning models efficiently. Many startups and companies are investing in GPU clusters to support their internal AI initiatives, but there aren’t enough GPUs for everyone (nor are they cheap). These circumstances mean it will make more sense for AI engineers to excel in using existing AI models rather than training new ones.

• Fire, ready, aim

Instead of asking ML scientists and engineers to do painstaking data collection to train a model for a specific domain and verify that it is production-worthy, a product manager or software engineer can query an LLM like ChatGPT to build or validate the product idea even before receiving specific data for training.

• Programming languages

The data/AI industry is mainly focused on Python, and AI engineering tools like LangChain and Guardrails came from the same community. However, just as many JavaScript developers in the dev sphere are finally getting their own AI tools (e.g., LangChain.js and Transformers.js), thus expanding the appeal of AI development to other dev communities.

• Generative AI vs. Classifier ML

Generative AI is slowly losing its importance. While the current generation of ML engineers focused on security, fraud prevention, recommendation systems, anomaly detection, classification, etc. – the new generation of AI engineers will focus on application development, personalization of learning tools, and development of visual programming languages.

The software 3.0 era is here

In the so-called software 2.0 era, programmers relied on detailed, hand-crafted programming languages ​​that precisely defined logic. However, Software 3.0 marks a significant shift, as it relies on neural networks taught to approximate logic, Shawn explained. This change enables new generations of software to tackle more complex and intricate problems previously challenging for human programmers.

The current debate on human vs. AI-generated code and if software engineers will become obsolete or not missed the point, according to Wang:

As human Engineers learn to harness AI, AIs will increasingly do Engineering as well, until a distant future when we look up one day and can no longer tell the difference.

The post Shawn Wang on the rise of the new role: AI engineer appeared first on ShiftMag.