I spend most of my time maintaining our companies internal developer portal (IDP), known internally as Kazhoon (which is built on Backstage). Like any developer portal, it provides a lot of features built by a wide range of engineering teams.

Beyond my core team of maintainers, there are about 50 other contributors across the company who regularly develop their own plugins and features. All of these engineers push code to a single, giant monorepo.

As you can imagine, there are going to be a lot of challenges with this kind of setup. In this guide, I’m going to focus on solving one of them.

The Multi-Contributor Challenge

When contributors are working on some feature, before merging, they would like to test it out, usually in a staging/testing environment. A small team working on a project has no issues here. They will just internally coordinate who gets to deploy at what time. But when you have 5 teams simultaneously working on different features, you will get conflicts.

Our solution initially was to tell people to merge their “feature” branch into a “dev” branch that in theory should contain the features of all the other contributors as well. We would then deploy just this “dev” branch. In practice, however, we were still getting conflicts, just this time in a different place.

Contributors would have issues merging their changes into the dev branch due to several reasons. Sometimes the dev branch was outdated, missing the latest changes from the master branch. Sometimes the contributors branch was not up-to-date.

Sometimes someone doesn’t merge cleanly, which causes frustration to all the other people wanting to merge as well. Every few weeks we had to create a new “dev” branch from scratch because it just became impossible to fix the problems with the old one.

So how do we solve this problem? The answer came to me while experimenting with Kubernetes. I was in the process of migrating one of our projects from a traditional “Docker on a VM” setup into Kubernetes. The project in question, as you might guess, is Kazhoon, our internal developer portal (IDP). 

Before explaining the solution, it helps to first understand how we deploy applications to Kubernetes.

How do we deploy to Kubernetes

Apart from having standard Kubernetes clusters, Infobip also leverages ArgoCD. A continuous delivery/deployment tool that sits between developers and Kubernetes. It uses GitOps principles to continuously sync infrastructure configuration stored in a Git repository with the actual cluster state.

This way, developers don’t need to directly interact with a Kubernetes cluster but instead just tweak and update the files in git and ArgoCD will make sure the changes get reflected in the actual Kubernetes cluster.

Also, it gives us a nice UI to monitor the changes.

So how do we deploy?

Well…, let’s say we want to deploy a “simple” application. In this case it is our IDP Kazhoon. Keep in mind that our app has do be dockerized for it to be deployed to Kubernetes.

P.S. The files below are created in a separate git repository that is linked with our ArgoCD application. For more information check out ArgoCD docs. Parts of the files have also been cut out for clarity.

We will first create a file which describes how our application should be deployed. Important parts have comments.

apiVersion: apps/v1
kind: Deployment # Type of resource in Kubernetes that manages our pods (aka. instances)
metadata:
  name: backstage-deployment # Name of the resources
  namespace: infobip-kazhoon
spec:
  replicas: 2 # Number of pods to create/manage
  selector:
    matchLabels:
      app: backstage-deployment
  template: # this is where we describe each deployment of our application
    metadata:
      labels:
        app: backstage-deployment
    spec:
      imagePullSecrets:
        - name: image-pull-secret
      containers:
        - name: backstage-deployment
          image: docker.domain.com/infobip-backstage:1.175.9 # dockerized image on internal artifact repository.
          imagePullPolicy: IfNotPresent # Yes, Kubernetes will cache your images, so if it already existed, it will reuse it based on the name/version.
          ports:
            - name: http # new port called "http"
              containerPort: 7007 # port where our application is listening
          envFrom:
            - configMapRef:
                name: backstage-config # Environment variables defined in a separate file
            - secretRef:
                name: backstage-secrets #  Secret Environment variable defined in a separate file
          resources: # We don't need to provision VMs for our application, we just tell Kubernetes how "heavy" our app is and Kubernetes will make sure to give us enough power.
            requests:
              cpu: "1"
              memory: "1Gi"
            limits:
              cpu: "1"
              memory: "1Gi"

We also need some sort of internal load balancer for our pods to be able to connect to them. We need to create a Kubernetes Service. 

apiVersion: v1
kind: Service # Type of Kubernetes resource that acts as an internal load balancer. It also allows other resources to communicate with out pods
metadata:
  name: backstage-service
  namespace: infobip-kazhoon
spec:
  selector:
    app: backstage-deployment # We tell the service which pods it should find
  ports:
    - name: http #  we define a new port 
      port: 80 # this port gets opened on the Service 
      targetPort: http # resolves to the 7007 port of our pods (check the ports defined in the Deployment snippet)

As a last step we also want to allow for people outside the Kubernetes cluster to connect to our application. We need an Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress # Type of Kubernetes resource that allows us to expose our internal network to the outside. In Infobip we use Haproxy for this.
metadata:
  name: backstage-ingress
  namespace: infobip-kazhoon
  annotations:
    iks-external-dns.k8s.infobip.com/hostname: infobip-kazhoon-iot1.domain.com # special Infobip annotations that create a DNS record for our network
    cert-manager.io/common-name: infobip-kazhoon-iot1.domain.com # used for TLS
    cert-manager.io/cluster-issuer: vault-issuer # TLS
spec:
  rules:
    - host: infobip-kazhoon-iot1.domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backstage-service
                port:
                  number: 80
  tls:
    - hosts:
        - infobip-kazhoon-iot1.domain.com
      secretName: minimal-ingress-cert # stores the certificate in this certificate resource

Once we create all the files, we push them to our git repository and hit “Sync” on ArgoCD. If we did everything okay we should see something like this:

Since this is a simplified “network” view we don’t see all resources (config files, certificates, etc…). The pods/instances on our right are the result of our “Deployment” resource which is not visible here.

As you can see, with just a handful of files we managed to completely skip VM provisioning as well as avoid the need to click through yet another UI just to deploy.

And then, it clicked

Kubernetes wasn’t just solving our infrastructure problems. Its design made it possible to solve the contributor problem I described at the beginning.

Instead of coordinating deployments, managing shared environments, or deciding who gets to test changes and when, we could let the platform do the work for us.

Everything could be automated.
Every change could have its own environment.
No coordination required.

The solution?

Ephemeral Environments

Whenever a contributor opens a new pull request, we can create an ephemeral (temporary) environment specifically for that branch, using the steps described in the example above.

We deploy the contributor’s branch to this environment and provide a unique link where they can test their changes.

This approach means contributors don’t need to think about deployment at all – everything happens automatically.

Each temporary environment is fully isolated from the others. The only shared component is the database, which could be further improved by switching to an in-memory solution.

Each environment has its own ingress, ensuring that every contributor receives a dedicated URL for their branch.

Because of this, we can eliminate the need for a separate dev branch and deploy feature branches directly.

How it works?

Step 1: Pull Request Hooks

Initially a contributor will just write some code, test it locally, and when happy, open up a pull request. This is where most developers would usually deploy to some IO environment that is close to a production setup for additional testing.

Instead of having to worry about the deployment, a Bitbucket webhook which listens to the PR_OPENED event will trigger a Jenkins job. The same Jenkins job that is used to build the main Kazhoon application. This is done to prevent duplication of jenkins groovy code as well as archetypes, which I will explain later. Simple hook below.

Notice the additional hook at the top. This is the main hook used for the app. It triggers the same Jenkins jobs but under different conditions (PUSH_EVENT – when code is pushed).

All of this can also be achieved on Github as well, using workflows.

Step 2: The main Jenkins Job

Apart from the main logic for building Kazhoon and pushing a docker image (standard for most Infobip apps), we additionally need the setup for Kubernetes. Keep in mind that the Jenkins job runs every time new code gets pushed. We need a way to detect which code change is linked to a PR. In other words: Does the branch that is currently being built have an open PR linked to it?

Initially we figured we could just use the build parameter PR_ID as seen on the image above. This was working okay for testing, but then a problem appeared. How do we detect code changes? After all, we would like the most recent code to be deployed and not just the one that existed when we opened up the PR.

Sadly, the PR_ID parameter does not exist on regular code changes. We could create another hook using the PR_SOURCE_RESCOPED event, but that would mean that we will have duplicate Jenkins jobs. One for the newly added hook and one for the original “main” hook that always gets triggered when we push code. We need a different solution.

Luckily, it is possible to call APIs through Jenkins, so what we can do is call the Bitbucket API, check if the branch currently being built has an open pull request, and if so, initiate the Kubernetes setup. This logic can be done on the initial pull request open event as well as subsequent code change triggers. Here is how that would look like. Example written in a Jenkinsfile using scripted pipeline syntax.

def findPrForSourceBranch(sourceBranch) {
    withCredentials([string(credentialsId: globconst.GIT_HTTPS_TAG_TOKEN, variable: 'GIT_TOKEN')]) {
        def response = httpRequest(
            url: "${globconst.BITBUCKET_API_URL}projects/IAF/repos/infobip-backstage/pull-requests?state=OPEN&at=refs/heads/${sourceBranch}&direction=OUTGOING",
            httpMode: 'GET',
            contentType: 'APPLICATION_JSON',
            customHeaders: [[name: 'Authorization', value: "Bearer ${GIT_TOKEN}"]],
            validResponseCodes: '200'
        )
        def prList = new groovy.json.JsonSlurper().parseText(response.content)
        if (prList.size > 0) {
            return prList.values[0].id
        }
        return null
    }
}
node('infobip-docker-nodejs:v20-dev') {
    LoadJenkinsPipelineLibrary(this)
    if (!env.PR_ID && env.BRANCH_NAME && env.BRANCH_NAME != 'master') {
        def prId = findPrForSourceBranch(env.BRANCH_NAME)
        if (prId) {
            env.PR_ID = prId
            echo "Found PR_ID=${prId} for branch ${env.BRANCH_NAME}"
        } else {
            echo "No PR found for branch ${env.BRANCH_NAME}"
        }
    }

In the later stages we can just check for the env.PR_ID. If it exists then there is a PR linked to the branch. Here is how that would look like:

Stages:

1. Clone Kubernetes config repository which is linked to the ArgoCD application
2. Generate Kubernetes files from template – make sure we sanitize the values
3. Push to git
4. Trigger sync on ArgoCD and leave comment in PR with link to new deployment.

def K8S_CONFIG_REPO = 'ssh://git-instance.com:7999/project-name/infobip-kazhoon.git'
def K8S_CONFIG_BRANCH = 'master'
def K8S_CONFIG_TARGET_DIR = 'k8s-config-repo'
def featureName = env.BRANCH_NAME.toLowerCase().replaceAll(/[^a-z0-9_-]/, "-") ?: 'feature-preview'
def k8sSafeName = env.BRANCH_NAME.toLowerCase().replaceAll(/[^a-z0-9-]/, "-").take(40).replaceAll(/-+$/, "") // Kubernetes name limit
def dnsSafeName = k8sSafeName.take(24).replaceAll(/-+$/, "") // DNS name limit eg. infobip-kazhoon-iot1-feature-verylongbr.domain.com

stage('Clone Kubernetes Config Repo') {
	// A neat trick to skip jenkins stages. Left out in later stages for brevity
    if (!env.PR_ID) { 
        Utils.markStageSkippedForConditional(STAGE_NAME)
        return
    }
	// Clone the kubernetes repository where we put our initial files for deployment.
    sshagent(credentials: [GIT_CREDENTIALS]) {
        globlib.git.checkout(gitUrl: K8S_CONFIG_REPO, gitBranch: K8S_CONFIG_BRANCH, targetDir: K8S_CONFIG_TARGET_DIR)
    }
}

stage('Generate Kubernetes Files from Template') {
    def featurePath = "${K8S_CONFIG_TARGET_DIR}/feature/${featureName}"
    // Copy and customize template. The template looks similar to the 3 files we created for the deployment just with some parts containing placeholders instead of fixed values.
    sh """
        mkdir -p ${featurePath}
        cp -r ${K8S_CONFIG_TARGET_DIR}/feature/template/* ${featurePath}/

        # Replace placeholders in template
        for f in ${featurePath}/deployment.yaml ${featurePath}/service.yaml ${featurePath}/ingress.yaml; do
            sed -i -e "s|{{IMAGE_TAG}}|${dockerLabel}|" \
                -e "s|{{K8S_SAFE_IMAGE_TAG}}|${k8sSafeName}|" \
                -e "s|{{DNS_NAME}}|${dnsSafeName}|" \"\$f"
        done
    """
}

stage('Push Kubernetes Resources to Git') { 
    dir("${K8S_CONFIG_TARGET_DIR}") {
        sshagent(credentials: [GIT_CREDENTIALS]) {
            sh """
                git config user.email "infobip-ci@infobip.com"
                git config user.name "Infobip CI (Jenkins)"
                git add .
				// We don't want the script to fail if there are no changes
                if ! git diff --cached --quiet; then
                    git commit -m 'added feature ${featureName}'
                    git push -u origin ${K8S_CONFIG_BRANCH}
                else
                    echo 'No changes to commit'
                fi
            """
        }
    }
}

stage('Trigger Sync on ArgoCD') {
    withCredentials([string(credentialsId: 'kazhoon_argocd_bearer_token', variable: 'BEARER_TOKEN')]) {
        def response = httpRequest(
            url: "https://argo-instance.com/api/v1/applications/infobip-kazhoon-iot1/sync",
            httpMode: 'POST',
            contentType: 'APPLICATION_JSON',
            customHeaders: [[name: 'Authorization', value: "Bearer ${BEARER_TOKEN}"]],
            validResponseCodes: '200'
        )
    }
	// Utility function for posting comments to Bitbucket PR.
    postPrComment(env.PR_ID, "finish", """ Your **feature environment** is live!
    ![party](https://media2.giphy.com/media/v1.Y2lkPTc5MGI3NjExcjFyczl4anFvZXYzNGY1aGM2amIyZTk5Y3B6ZWJ4dHM1Y2ExamptYSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/a0h7sAqON67nO/giphy.gif)
     Access it here: [Preview URL](https://infobip-kazhoon-iot1-${dnsSafeName}.domain.com)
     Note that it might take a few minutes for the DNS to propagate and the service to become fully operational.
     If you encounter any issues, please contact [#kazhoon_contributors](https://infobip-support.slack.com/archives/C098ETXLL2U) on slack.""")
}

Step 3: Cleanup

After the PR is merged, the letfover deployment is automatically cleaned up using a separate webhook and Jenkins pipeline which I won’t go into detail for this blog post.

In short, when a PR is merged or deleted, a Jenkinsfile.cleanup  pipeline is executed to remove all previously created resources and trigger a sync in Argo CD.

The same Jenkinsfile.cleanup pipeline also runs periodically to remove stale resources from old PRs, helping avoid unnecessary infrastructure and electricity usage.

Conclusion

Kubernetes and Argo CD allowed us to significantly improve the developer experience for everyone contributing to Kazhoon. By adopting a GitOps-driven approach, we eliminated the need to interact with a UI and, with it, the mental overhead of figuring out how and where to test changes. The entire deployment process is now fully abstracted away from contributors.

The original “dev” branch which was a major source of confusion and friction no longer exists. Developers work exclusively on their feature branches, each with its own isolated, production-like environment.

This setup also removed a large portion of our support overhead. Everything is automatic, predictable, and clearly communicated directly through pull request comments, allowing contributors to focus on what actually matters: building features.

Resources & Shout-outs

Special shout-out to Matej Kern and his team for helping me throughout the entire process. From explaining Kubernetes concepts to setting up the infrastructure behind everything described in this post. This project would not have been possible without their support and guidance.

A huge shout-out as well to the maintainers and contributors behind these amazing projects and resources that made this workflow possible:

• Kubernetes
• Argo CD
• Backstage, where I got the inspiration from
• The cloud-native computing foundation

Hope you enjoyed this blog post. It’s my first official one (and definitely not the last). There’s a lot I didn’t manage to include, because it would’ve ended up being a whole novel, so if you have any questions or thoughts, I’m always happy to chat. Feel free to message me on X.

The post Solving the Multi-Contributor Challenge with Kubernetes and Ephemeral Environments appeared first on ShiftMag.

The Deployment Pipeline is a critical tool for ensuring software quality – and quality is a prerequisite for speed.

So, what is a Deployment Pipeline? How has it helped us improve both quality and speed? And what are, in my experience, the common problems in using and implementing deployment pipelines?

A Deployment Pipeline is a machine that helps us move from an idea to valuable software in the hands of users by organizing software development work. It enables us to go from commit to releasable outcome as quickly and efficiently as possible, in a repeatable and reliable way.

— Dave Farley, Continuous Delivery Pipelines: How to Build Better Software Faster

Before exploring how Deployment Pipelines have helped us, it is helpful to briefly introduce Continuous Integration and Continuous Delivery.

Continuous Integration

Continuous Integration is a software development practice that helps us create and maintain a high-quality codebase that is easy to change and to implement new features.

As Martin Fowler points out, the practices of Continuous Integration are:

• Put everything in a version controlled mainline​
• Automate the Build​
• Make the Build Self-Testing​
• Everyone Pushes Commits To the Mainline Every Day​
• Every Push to Mainline Should Trigger a Build​
• Fix Broken Builds Immediately​
• Keep the Build Fast​
• Hide Work-in-Progress​
• Test in a Clone of the Production Environment​
• Everyone can see what’s happening​
• Automate Deployment​

Kent Beck developed the practice of Continuous Integration as part of Extreme Programming in the 1990s. The book Continuous Integration by Paul M. Duvall, Steve Matyas, and Andrew Glover was published in 2007. While this is nothing new, many software engineers and teams still do not practice Continuous Integration and therefore miss out on the benefits it provides.

Continuous Delivery

Continuous Delivery can further help create and maintain high-quality software. The excellent book Continuous Delivery by Jez Humble and David Farley was published in 2010. and it describes the idea behind continuous delivery in detail, and it introduces “…the central paradigm of the book – a pattern we call the deployment pipeline“.

Continuous Delivery is the ability to get changes of all types – including new features, configuration changes, bug fixes and experiments – into production, or into the hands of users, safely and quickly in a sustainable way.

– https://continuousdelivery.com/

Continuous Integration is a prerequisite for Continuous Delivery.

The minimum activities required for Continuous Delivery, as defined by minimumcd.org, are:

• Use Continuous Integration​
• The application pipeline is the only way to deploy to any environment​
• The pipeline decides the releasability of changes, its verdict is definitive​
• Artifacts created by the pipeline always meet the organization’s definition of deployable​
• Immutable artifact (no human changes after commit)​
• All feature work stops when the pipeline is red​
• Production-like test environment​
• Rollback on-demand​
• Application configuration deploys with artifact​

Deployment Pipeline

The best description of deployment pipelines, and practical advice on how to implement them, I found in another excellent book Continuous Delivery Pipelines, How to Build Better Software Faster by Dave Farley. According to him, automation is the key… and is the engine that drives an effective Deployment Pipeline:

• Test Automation​​
• Build and Deployment Automation​​
• Automate Data Migration​​
• Automate Monitoring and Reporting​​
• Infrastructure Automation​

The simplest deployment pipeline should, after each commit, compile the code (if needed), run all unit tests, create a deployable artifact, execute acceptance tests, and enable deployment to production.

However, a deployment pipeline can do much more, such as:

• Static code analysis
• Enable manual testing
• Performance tests (e.g., latency, throughput, load)
• Data and data migration tests
• Security tests
• Reliability tests etc.

CI practices

I see many teams/companies have a Jenkins job that compiles source code, runs (a few) unit tests, creates an artifact – often a Docker image – and have a separate deploy job/button to deploy to the dev and production environments. And they will say they are “using” CI/CD. But most of them are skipping some of the CI practices and don’t have code in an always-deployable state, even as a goal, yet still think they are “using” CI/CD.

CI practices are not a menu where you can choose what you like and still have all the benefits CI can help you with.

Make the build self-testing

Test-Driven Development (TDD), or test-first development, is still an exception, and developers usually write tests after the code. Those tests are mostly unit tests (developer-oriented) and much fewer acceptance tests (user-oriented) that cover only parts of the functionalities. The test-after approach usually covers less functionality because the code is harder to test, and tests are complex to write and maintain. Such tests don’t provide fast and good-enough feedback to automate “releasability of changes” or “enable refactoring for sustained productivity.” Developers don’t trust such tests, so manual verification and team coordination are needed to decide when code can be deployed, which is never fully reliable, and more bugs reach production.

Everyone pushes commits to the mainline every day

Same as TDD, trunk-based development is still an exception, and almost everyone is using branches and pull requests with many PR reviews at the end of the sprint, so they are not practicing continuous integration by definition. They are missing the benefits CI brings, like “less time wasted in integration” and “enables refactoring for sustained productivity.”

Test in a clone of the production environment​

Some companies have a “static” copy of production, with test data already prepared in a database, where they deploy their applications and run tests. As a result, developers can’t run tests on their laptops, test preparation is complex and slow, and they regularly break other tests. Feature flags can’t be tested, and it takes a long time from commit to getting test feedback. This demotivates developers from writing tests, and over time, this negative spiral creates ‘legacy’ code that is hard to change.

Rarely does any company I know create, on demand, a “clone of production” – with the configuration needed for a suite of tests set up exactly as in production, where the tested application is deployed the same way as in production and then tested to determine the releasability of changes.

Usually, a Jenkins job starts the needed testcontainers during the integration test phase and tests parts of the application – or a “test” version of the application – that differ from the production application. These tests don’t cover the application artifact that will actually be deployed, nor do they test deployment, configuration, or dependencies (e.g., microservices) that aren’t properly mocked. Instead, some fake repository or adapter implementations are used that don’t exist in production.

The application pipeline is the only way to deploy to any environment AND The pipeline decides the releasability of changes; its verdict is definitive​

From what I’ve seen or heard, everyone has a workaround that allows deployment of artifacts that didn’t pass the Deployment Pipeline. Sometimes they create artifacts on their laptop and push them to the repository so they can be deployed. Other times, the pipeline itself allows deployment of code that hasn’t passed all required verifications.

This can happen when a “non-important” or “flaky” test fails, or when a dependency has a critical vulnerability, etc. Usually, the team says these problems will be fixed later, when there’s time, but the new feature is considered urgent and must be deployed ASAP. In practice, the team never has time to fix these issues, workarounds become the norm, and the deployment pipeline turns into a theater.

Keep the build fast

With microservices being so popular, build time shouldn’t be a major problem. Yet it’s common to see builds lasting half an hour or more.

In my experience (mostly with Java), if you create tests – unit tests, acceptance tests, etc. – that can run in parallel, it’s easy to have a deployment pipeline finish in under 5 minutes, excluding deployment. Just run everything in parallel: unit tests, acceptance tests, services with different feature flags enabled or disabled, and static code analysis.

A Deployment Pipeline must be built, not bought

A deployment pipeline automates your software development process and reveals both its strengths and weaknesses. When your practices are strong, the pipeline amplifies them, providing significant benefits. Conversely, if your practices are weak, the pipeline makes this very apparent – it can be difficult or even impossible to implement effectively.

This is why Continuous Integration and Continuous Delivery are so valuable: they encourage good practices and make problems visible early. A Deployment Pipeline is not a product you can buy or outsource; you must build it yourself, leveraging various tools.

In my experience, the best way to achieve both speed and quality in software development is to combine pair or mob programming with test-driven, trunk-based development, supported by a deployment pipeline that automates feedback on key aspects of the process. This approach provides rapid feedback, within minutes, on ideas and experiments through frequent commits and accompanying tests. It allows the codebase to evolve continuously through refactoring and enables the frequent, low-risk release of new features with minimal stress.

The post What It Takes to Fully Benefit from a Deployment Pipeline appeared first on ShiftMag.

It’s a dull, rainy Friday. You finish coffee with your bestie and head home with your work laptop. On-call isn’t great, but at least it pays extra.

As soon as you get home, the company phone starts ringing. You find it and feel a mild panic – it’s Opsgenie. You know something is wrong, and time is critical. You grab your laptop, turn it on, find the alert it triggered for, and start troubleshooting.

You get an alert for increased errors on the spam service, follow the runbook, and realize you can’t fix it, so you call the team. They handle it, and you still have the rest of the night free.

Has this ever happened to you? Well, it can if you have good runbooks!

We made our first runbooks, and oh, they were great

When reacting to alerts, stress is inevitable, causing you to miss clues to the root cause. More often than not, you end up going through the documentation of how something works, instead of looking for what broke.

So, let me take you back to the time when we created our first runbook.

When I first started as a Reliability Operations Engineer, this was a newly formed team, so we didn’t have any alerts or runbooks. But, we knew we were blind on many fronts and needed to catch platform issues somehow.

This led us to inherit some alerts from teams like SRE and start creating our own. We were proud of our alerts, cheering when they caught incidents. Looking back, they were mediocre, but at the time, we didn’t know better.

It was smooth when alerts came during our morning shift, as every dev team was online, and we could ping them for clarification on issues.

This is a very fruitful process we do almost daily – we receive an alert, notice it’s for a specific service, ping dev teams for information and clarification, and then decide how that information affects what we want the alert to catch.

Was the alert useful? Did it notify us of an actual issue, or did it alert us to normal behavior for that service/location? We can then exclude errors from specific locations where they’re expected or fine-tune the thresholds for the alert condition.

Responding to our first on-call alerts nearly gave me a heart attack

I still remember it clearly – we had just started on-call rotations, and it was my turn. The previous week was calm, so I expected the same. After work, I did my groceries and chores, keeping my laptop and company phone nearby. Everything was quiet, so I showered, got ready for bed, and slept. Then, at 1:30 AM, my company phone rang.

Waking up disoriented, I stumble in the dark, avoiding lights to not wake my girlfriend – she really values her beauty sleep. I grab the phone, see an Opsgenie alert, and mild panic sets in. I head to the living room, grab my laptop, and start troubleshooting.

The alert was named ‘Whitelabel CUP DOWN,’ and I had never seen or heard of it. Normally, I’d start troubleshooting and look for what went wrong, but here, I had no idea where or what to search for. The pressure was mounting, as reacting to this alert was my responsibility, and I didn’t know what to do.

For the next 45 minutes, I stumbled through outdated Confluence pages and old Slack threads, trying to find anything about Whitelabel CUP. I was lost, unsure which logs to check or if it was impacting clients.

At that point, I decided I couldn’t just keep searching the documentation; I needed to act. But it was past 2 AM – what if it wasn’t an issue, and I woke someone up for no reason? Who should I even call?

To my relief, the alert resolved itself. I was ecstatic it was over, but what if it happens again? Where are the logs? What if someone’s already looking into it? What if clients have complained? I had so many questions, so I wrote them down, closed my laptop, and went back to sleep, hoping it wouldn’t trigger again. And it didn’t.

How can we improve this?

The next morning, I admitted my struggles in the daily, feeling hopeless because I didn’t know how to verify if it was an issue or a false positive, where to find logs, or who to call. As a team, we decided to dedicate time to figure out how to handle this better in the future.

The root cause was my lack of knowledge about Whitelabel CUP and how issues with it manifest.

This can be addressed in two ways: as a team, we could all invest time to learn about it, but that would require significant man-hours, which isn’t the most efficient approach, especially when a new person joins. Alternatively, we could assign one person to learn the details, create a runbook, and then share it with the team.

This is how to create a runbook

The task I was working on started as documenting Whitelabel CUP but evolved into creating a runbook with clear handling steps. Where to check for logs and, most importantly, who we should call if we observe problems, so that once we receive the alert, we have a flow to help us troubleshoot it successfully and in a timely manner.

A prerequisite for good runbooks is having good alerts. Because if your alert is bad, the runbook you create for it will, at best, be just as bad. So, for the sake of this article, I’ll skip this part – I’m assuming you already have good alerts, right? RIGHT?

I’ll share some generic tips that worked well for us, though your needs may vary since every alert is different.

Runbooks should be written as clearly and unambiguously as possible – so even the newest team member can successfully troubleshoot the alert.

The first step in a runbook is to quickly assess whether it’s a false positive or a real issue. This might involve checking synthetic monitoring, seeing if the same issues appear there, looking at a different data source, or simply checking how production traffic is behaving.

After the ‘false positive check’ comes the juicy part of the runbook. You can direct people to check logs or Grafana panels to confirm whether a hypothesis is true or false. The best approach is to ask a question and provide the answer right in the runbook.

For example, the most common one: ‘Is there packet loss towards this location? Check here: grafana.com/packetloss.’ This way, you’re asking the question and providing an easy way to check, all in the runbook.

Next, ask, ‘Is only service X affected? Check here: all-service-logs.com.’ This helps determine if it’s just one service or multiple. For service-specific alerts, include mitigating actions like, ‘Find the affected instance here: list-of-instances.com. Redeploy the instance with the highest error rate here: instance-manager.com.’

The goal is to have the ideal troubleshooting process, or as close as possible, while making it quicker. By providing exact links, you save time on typing queries and searching for the right Grafana panel.

Good runbooks are an ongoing, iterative effort

Effort is appreciated in all things, and runbooks are no exception. Creating a good runbook seems simple at first, but once it’s used, you’ll realize there are gaps, confusing parts, or areas for improvement.

With experience handling that alert, you’ll be able to streamline things, adding steps or excluding ones you initially thought were necessary. Keep in mind, you’re not writing documentation, but a quick guide to help you get to the root cause and take action -whether that means mitigating the issue yourself or engaging the responsible team.

Good runbooks are an ongoing, iterative effort. We update them when something’s missing or outdated. Done right, they speed up issue detection and mitigation, and help onboard new team members. Just give them a runbook and let them ask questions – it’s a great way to gather fresh insights.

If you have alerts, try them out – they’ve made our lives easier, and they might do the same for you.

The post Good runbooks are a MUST – unless you want to risk a heart attack appeared first on ShiftMag.

This is one of the insights from DORA’s new 2024 State of DevOps report!

This year focuses on how AI tools are changing the game, the importance of platform engineering, and why understanding developers is essential for improvement.

These are some of the takeaways from the research.

Empower developers to work more independently

For the second year in a row, this research highlights that AI tooling is actually lowering software delivery performance. But the explanation may surprise you; it’s not just that AI code is unreliable.

The real link between AI tooling and weaker delivery performance is the tendency for AI-driven coding to result in larger batches of code.

AI simply makes it easier to churn out more code in less time, leading to more complex deployments.

Also, software delivery throughput and quality are becoming less aligned, and overall performance appears to be slipping compared to last year.

And to address this issue, organizations should invest in tools and processes that empower developers to work more independently – like better documentation and self-serve platforms.

While these developer platforms may slow down delivery at times, they ultimately enhance both individual and team performance.

We code faster, but meetings are still dragging on as usual

And here’s a surprising twist: while AI adoption boosts individual productivity, flow, and job satisfaction, it may also decrease time spent on valuable work. Yup, you read that right!

We may be coding faster, but admin work and endless meetings aren’t going anywhere.

Now, here’s where the real magic happens: documentation. It’s the secret weapon for leveling up your development game.

DORA’s research shows a solid link between good documentation and better performance. They estimate that a 25% bump in AI adoption could lead to a 7.5% boost in documentation quality – making it more reliable and easier to navigate.

Key to platform engineering success? Put developers first

This year’s report also digs into platform engineering and what it means for developers.

A key factor in the success is to approach platform engineering with user-centeredness (users in the context of an internal developer platform are
developers), developer independence, and a product mindset.

Here are the highlights:

1. Internal platforms can help individuals and teams get more done, but they might also slow things down and introduce some instability. Still, companies using these platforms often deliver software faster and perform better overall.
2. To make internal platforms work, you need to understand your developers. Think about their goals, not just the tasks they need to finish. The more independence they have, the better!
3. Good leaders make a huge difference. When they have a clear vision and support their teams, it leads to happier, more productive developers.

Overall, software delivery seems to be facing challenges in 2024, reflecting the tough conditions many companies are dealing with in today’s economic climate. Buckle up!

The post DORA: Only 10% of developers see big productivity boost by AI appeared first on ShiftMag.

“Building a successful software deployment process is akin to manufacturing a high-quality motorcycle,” explains Jose Javier Moya, a DevOps Ambassador at Amadeus IT Group, who spoke at the WeAreDevelopers World Congress.

Moya also highlighted how Git repositories act as blueprints, Jenkins automates consistency, and reconciliation agents ensure deployment integrity, drawing parallels between manufacturing and software processes.

This analogy not only underscores the complexity of modern software development but also provides a clear framework for understanding how these elements work together to ensure successful implementations.

Building the perfect motorcycle

Moya used a motorcycle analogy to highlight the need for detailed planning in software development.

Imagine your company as a motorcycle manufacturer. To build a motorcycle, you need more than just a basic idea. You need a detailed blueprint.

Moya explained that this blueprint includes specifics, such as the size of the tank and the type of tires. Similarly, deploying software requires more than a simple configuration file. It demands a comprehensive description of how the software should be built and operated.

The role of Git

Moya compared motorcycle blueprints to Git repositories.

In our motorcycle factory, the blueprint acts as a contract between the manufacturer and the buyer.

Similarly, a Git repository serves as the blueprint for software. It stores all necessary configuration files and documentation, ensuring that every aspect of the software deployment is well-documented and controlled. Just as a motorcycle blueprint covers every detail, Git repositories provide comprehensive details for software projects.

Use automation as your secret weapon

Then, he highlighted the crucial role of automation in software deployment.

Imagine if your motorcycle factory were fully automated. Once you receive a blueprint, the factory will automatically build the motorcycle according to the specified details.

Similarly, automation tools play a crucial role in software development. For instance, Jenkins and other tools act as gatekeepers. They check to ensure that all requirements are met before deployment. Using automation, developers can build and deploy software consistently and accurately.

Quality assurance is automation’s secret superpower

According to Moya, automation extends beyond just building and involves quality assurance.

Jenkins performs various checks, including static analysis, security scans, and operational checks.

These checks ensure the software is free from vulnerabilities, does not break existing functionality, and meets operational requirements. This rigorous scrutiny helps prevent issues in production, much like how rigorous testing ensures a motorcycle is safe and reliable before it hits the road.

What is the role of reconciliation agents?

Moya described how reconciliation agents like ArgoCD manage the deployment process.

After passing all the checks, the actual deployment is handled by reconciliation agents.

These tools continuously compare the desired state, as described in the Git repository, with the actual state of the deployment. They correct deviations, ensuring consistency between the intended and actual configurations, much like ensuring every motorcycle built matches the blueprint.

Let automation and reconciliation take the lead

Jose Javier Moya also discussed expanding these principles to cloud environments.

As software and infrastructure grow more complex, the principles of automation and reconciliation extend beyond traditional applications.

In cloud environments, tools like Terraform manage cloud resources using the same configuration principles, adapting these concepts to manage increasingly complex infrastructure and services.

Welcome to the new layers of complexity

Reflecting on the evolution of operations, Moya pointed out that managing systems used to be more straightforward with mainframes and a few servers.

Over time, as systems grew and became more complex, managing them manually became impractical.

Virtualization and cloud computing introduced new layers of complexity, which tools like Git, Jenkins, and ArgoCD now help manage effectively.

The post Build your software like a motorcycle appeared first on ShiftMag.

Imagine yourself entering the DevOps world as a daring explorer, navigating uncharted territories, always driven by the pursuit of improvement. But amidst this excitement, how do you gauge your progress? How can you measure your commitment to the core tenets of DevOps? In this article, we’ll try to do just that!

In our quest for DevOps mastery, milestones serve as vital signposts along the winding road. They not only mark our progress but also signify triumphs, affirming that we’re on the right track toward mastering DevOps.

What do DORA metric measure?

One of the foundational frameworks for DevOps measurement—the DORA Metrics. This standard lays the groundwork for assessing DevOps performance:

1. Deployment Frequency: Picture a high-performing team seamlessly deploying code multiple times a day, like clockwork. Take Netflix, for instance, where changes are implemented swiftly, ensuring a seamless streaming experience for millions of users worldwide.
   
2. Lead Time for Changes: Imagine the thrill of seeing your code transform into a live feature in less than an hour. Companies like Amazon exemplify this efficiency, where new functionalities swiftly make their way from conception to reality, delighting customers with rapid innovations.
   
3. Mean Time to Recovery (MTTR): In the event of a glitch, envision a swift restoration of services within minutes, not hours. Google epitomizes this resilience, swiftly addressing any disruptions to its services and ensuring uninterrupted access for users around the globe.
   
4. Change Failure Rate: Consider the confidence of deploying changes with minimal risk—a testament to a low change failure rate. At Spotify, meticulous testing and robust deployment strategies minimize the chances of disruptions, providing users with a seamless music streaming experience.
   
5. Deployment Stability: Envision a deployment process characterized by a high success rate, ensuring smooth operations without disruptions. Slack embodies this reliability, with updates seamlessly rolled out to users, enhancing collaboration without interruptions.
   
6. Customer Feedback: Beyond technical metrics, imagine the satisfaction of users reflected in glowing feedback. Airbnb thrives on such positive endorsements, with users praising the platform’s stability, performance, and user-friendly interface, a testament to successful DevOps transformation.
   
7. Team Collaboration: Visualize the synergy between development and operations teams, seamlessly working together towards shared goals. At Microsoft, collaborative efforts drive innovation, with teams fostering a culture of shared responsibility and continuous improvement.

The realm of elite performers

Companies like Google and Netflix stand as beacons of excellence, achieving on-demand deployment frequency, lead times under an hour, and a change failure rate below 15%. Their relentless pursuit of perfection sets the standard for major and mature DevOps adopters worldwide.

What gets measured gets tracked

Now that we know about DORA metrics, how do we translate these into tangible markers for measuring DevOps adoption and excellence? 

The key lies in meticulous analysis and strategic action:

1. Set Baselines and Track Trends: Establish baseline metrics and monitor trends over time to chart your progress accurately.
   
2. Identify Bottlenecks: Dive deep into stagnant metrics to uncover bottlenecks hindering your journey toward DevOps excellence.
   
3. Drill Down for Root Causes: Explore the underlying causes of bottlenecks, whether technical limitations or process inefficiencies, to devise targeted solutions.
   
4. Implement Actionable Changes: Translate insights into action by implementing automation, refining processes, or fostering a culture of continuous improvement.
   
5. Monitor and Re-evaluate: Continuously monitor the impact of your interventions, adapting your strategies based on real-time data to drive sustained performance improvements.

The path to DevOps mastery is an exhilarating journey filled with challenges and triumphs. As Peter Drucker once said, “What gets measured, gets managed.”

By embracing frameworks like DORA Metrics and adopting a systematic approach to analysis and improvement, organizations can optimize workflows, enhance customer satisfaction, and propel their DevOps journey toward unparalleled success.

The post DORA Metrics: Mastering the DevOps excellence appeared first on ShiftMag.

There is something to unpack here, says Baruch Sadogursky as he takes the stage at the Miami Shift Conference. A conspiracy theory, even. 

And here it is:

“DevOps is a conspiracy by Ops people to make developers work harder.”

Jokes aside, Baruch urges us to look into who teaches us about DevOps. A quick Google search along the lines of “What is DevOps” reveals that the answer to this question can be found on the websites of AWS (an Ops platform), Atlassian (a DevOps company), Gitlab (Offering DevOps in the box), Synopsis ( Observability platform). Is it a coincidence that all sources that teach us about DevOps are (dev)ops or DevOps-adjacent companies?  

Why do they want to sell DevOps ideas to us so badly? Is it about the revenue that the industry generates? A DevOps salary is almost double that of a system administrator (with not that great of a difference in the job description); there are almost 20k open positions for DevOps engineers in us alone, and let’s not forget about the certification business behind it. 

One could easily make the connection between money and the push for DevOps – but only if we forget to go back in time to when DeOps first started. When it was first conceived and gained momentum, the money was just not there, Baruch notes. 

This looks familiar

If we look at the DevOps cycle, it seems pretty familiar – it is basically an agile development cycle, with an Ops cycle plastered on there, and suddenly it’s supposed to all be dev work! Even when we look at the main aspects of DevOps work, we hear Ops concepts: deploying code, going from code commit to running code in production, unplanned outages, and degraded services.

Baruch protests that none of them feature in the basic postulates of dev work: Developers do new features, refactor, and fix bugs; that’s the job. The code passes the tests; it works, and the job is done. What happens in production is beyond the scope of work, so there is no room for these ops concepts there, right?

Well, it’s not that simple. If we take a look at a software craftsman definition of done Baruch provided, the list is much longer:

The project is done once the dev understands what needs to be done; the code is simple, readable, and easy to deploy; non-functional requirements are met; no tech debt was created; tests pass & QA is happy with the code; team lead, PO and client are all satisfied. That’s a much longer list, and what stands out is quality.

Baruch leads with another difficult question here: What is quality, that we’re so concerned about?

Testing in production = caring about what happens in production

Quality is an ever-evolving concept. It’s certainly not what it was ten years ago, as Baruch illustrates with an example of his old bank website, which allowed you to check your balances and transactions but nothing more complex. It was nice to have but not critical, so when the website was out for 12 hours for update & maintenance to ensure quality of service, that was not a big deal.

That era is gone – we expect more of our software now. In terms of banking, we do it via mobile app and expect everything to work at all reasonable times, and to work quickly and efficiently. Code quality alone is not enough to cater to that.

Let’s just look at the size of the global datasphere – 175ZB. There is more and more data in more and more companies, which makes testing harder. Since we have so much data in production that we cannot, or at least it’s not worth replicating in staging. So, we test in production:

And if we’re testing in production, and being in production is what is needed to ensure the software quality that’s so important to us as developers, that means we now care what happens in production.

And if we care about what happens in production, that means we have to be there when we throw untested software on unsuspecting customers in production and observe the blast radius from our bunker. We have to be there in the middle of the night with the ops people because this is actually the first time we get to test our software.

The new scope of work

This new turn of events means that there are some more items on the developers’ to-do list:

• We understand how our software is going to be deployed
• The build is reliable, repeatable & fast
• The code is stateless & scalable
• It starts fast and dies fast
• It is observable
• It supports feature flags
• It’s backward & forward-compatible
• The code emits event streams

This is what you need to ensure the quality of your software in production, says Baruch, so starting now, you care about this stuff:

10 years in, developers are still not excited about DevOps. But still, we have to be there – because we care about what we’re doing. So how do we go from to :).

So, is DevOps a plot by Ops people? For sure, Baruch says, but it’s also an evolution and a means to an end. The end being quality, new features, lean software and security. DevOps just delivers what every business needs.

DevOps engineers are rainbow-farting unicorns

So now that you’re thrown into DevOps, you better get to know it. If we look at the traditional DevOps Venn diagram, we can see that DevOps is an intersection between Devs, QA, and Ops. So, are you supposed to master all those disciplines? If that thought terrifies you, Baruch has some good news: 

That person does not exist. It is a rainbow-farting unicorn, completely made up. DevOps is not about one person; it’s about collaboration. 

Instead, he suggests there are T-shaped people: People who know their job (coding) well but also understand what the Ops and QA part of the house means. This doesn’t mean developers have to become QA and Ops; they just take an interest in what they do. 

That’s not so bad, is it? Or, in Baruch’s own words: 

“DevOps is fine.”

The post Is DevOps just a conspiracy theory? appeared first on ShiftMag.

To excel in DevOps, the first step is to think like an architect – and yes, you heard it right!

This is the foundational rule that Nana Janashia, renowned DevOps Trainer and YouTuber instills in her students.

With her free courses and comprehensive bootcamps offered on YouTube and her proprietary training channel, Nana has facilitated the transition of hundreds of thousands of people into DevOps roles.

Additionally, her ‘IT Beginners’ course (for those without IT backgrounds) has attracted thousands of students as well, demonstrating her dedication to accessible tech education for people with different levels of IT expertise. Currently, about 5,000 students from all continents are enrolled in either of her premium courses.

Let’s delve into her approach and insights.

Thinking like an architect

First, the burning question: how do you define DevOps?

Nana: My own preferred definition of DevOps is this: It’s a combination of all processes and tools that remove any bottlenecks and slow-down points in the complete software release pipeline. And if we reverse engineer from that definition, we see the way to remove bottlenecks and things that slow down the process is to automate things, and that’s why at the core of DevOps are automation tools, everything as code concepts, etc.

As a DevOps engineer, you need to think like an architect first, examine the existing systems, and identify any inefficiencies, manual steps, long-running steps, blockages, bottlenecks, and any points of human mistakes or issues slipping into production. Then, redesign the systems to improve those inefficiencies, solve the blockages with automation steps, etc.

Of course, this is a rather challenging task, especially when dealing with complex systems involving multiple engineering teams. 

What is the biggest challenge when implementing DevOps within organizations?

Nana: The biggest challenge is that DevOps impacts the entire set of systems and the whole software development pipeline, which affects different engineering teams and roles.

And that can create a lot of resistance from other engineering teams, when they don’t fully understand the DevOps principles themselves and often see the implementation of DevOps tools as an interference in their already existing tasks and responsibilities.

The higher impact is on the Operations and System Administration side because the tools and processes DevOps uses to efficiently manage infrastructure, cloud platforms, platforms, etc., are more aligned with software development practices and less familiar with how Operations teams work.

That’s why the best way to implement DevOps is by introducing the changes step by step, smaller improvements, instead of redesigning larger parts of the systems. In parallel, educating other engineering teams about DevOps to promote collaboration.   

How a collaboration between development and operations teams should be promoted?

Nana: Firstly, it’s essential to ensure everyone is on the same page regarding what DevOps achieves and its implications for each engineering team. This helps to reduce resistance and misinterpretation. However, in practice, things may differ from theory.

When implementing DevOps, it’s important to understand that even though DevOps engineers are also hands-on engineers implementing processes they designed and architected themselves, they often need collaboration from other engineers to build fully automated processes across all systems.

To achieve that, especially in an organization that isn’t fully open to these changes, DevOps engineers should start with improvements that make the existing tasks of the other engineers easier, faster, and more efficient.

DevSecOps: a multi-layered approach

How do you handle monitoring and metrics in your DevOps practices?

Nana: As systems grew more complex, automated monitoring became essential to understand and manage them effectively. Since monitoring should involve multiple levels of the systems, the monitoring tools used for each may be different. Cloud platforms usually have their own services to monitor the cloud infrastructure and services, as well as access to the platform. In the Kubernetes world, tools like Prometheus play a big role in monitoring clusters and workloads inside.

With monitoring, it’s important not only to have insights but also to have automated alerting in place whenever anything deviates from a norm within the systems to proactively address any issues. 

How do you integrate security practices into your DevOps processes, especially in terms of code reviews and automated testing?

Nana: DevSecOps principles involve dozens of tools to integrate automated security checks in the DevOps processes.

Like monitoring, security is also applied on multiple layers. And since security is layered, it’s important to test all parts of the systems to identify any vulnerabilities. Therefore, the DevSecOps involves tools for testing security issues in the application code, in the third-party dependencies, in the image layers, dynamic security tests to tamper with the application to exploit it like hackers would etc.

Beyond those automated security scans we add in the DevSecOps release pipeline, we also have security on all other levels. Every tool and platform we introduce in the systems opens up a new field of security threats that can be exposed if they aren’t secured. That means security on the cloud platform, infrastructure level, security in Kubernetes, compliance as code, automated policy enforcement, etc.

And last but not least, probably the most important of all, the access management for all the systems used, from cloud platforms to CI/CD platforms, etc.

DevOps tools: AI-driven features and fewer tools

How do you design and implement disaster recovery and high-availability solutions in your DevOps workflows?

Nana: Many factors are involved in creating efficient DR and HA solutions in DevOps. Some key components I would highlight are:

1. Using Infrastructure as Code principles to define your infrastructure, networking, and configuration in code. Using tools like Terraform, AWS CloudFormation, or Kubernetes YAML files to enable you to recreate your infrastructure on-demand.
2. Deploying applications and services across multiple geographic regions or Availability Zones (AZs) to ensure HA. Many Cloud providers offer multi-region deployments for redundancy.
3. Regularly backing up data and systems to meet RPO requirements. Using automated backup and snapshotting tools from cloud providers themselves or third-party backup solutions. Automated is again a key here. 

In your opinion, what are the emerging trends in DevOps in 2024? Are there any particular tools or technologies you find especially promising or exciting?

Nana: My focus is rarely the tools and technologies but rather the underlying concepts that are either changing or being standardized. So I believe we will see many existing DevOps tools introducing AI-driven features or completely new AI tools for DevOps use cases specifically, which will add to the already vast eco-system of DevOps and Cloud tools. I believe these trends will continue into 2025 as well.

However, at some point, we will see the trend to standardize some of the technologies and concepts. So eventually, instead of a myriad of tools, we would have fewer tools that become established as the industry standard. But we still have a long way to go till then.

The post On implementing DevOps, too many tools, DevSecOps, and more with Nana Janashia appeared first on ShiftMag.

You’re in a DevOps, wait, I mean SRE… nope, scratch that… “Platform Engineering” team. People are coming at you to get Kubernetes up and running and then build some kind of platform on top of Kubernetes. Kubernetes may be new, but building and running developer platforms has been going on for at least 10 years.

That’s how Michael Coté, a senior member of technical staff at VMware, introduces the talk he will deliver at the Main stage of the Shift conference titled Lessons Learned from 7 Years of Running Developer Platforms.

Are DevOps, SRE, and Platform Engineering the same thing?

Of course, the first question we had to ask him was a follow-up to his first sentence. Is DevOps dead or alive and kicking, and are DevOps, SRE, and Platform Engineering actually all the same thing? Michael admits even he is not sure:

“I go back and forth on this a lot. The sum total is helpful – they’re all espousing helpful practices and changes that organizations can follow and make to get better at how they do software.

It’s fair to say that the first wave of platform engineer thought leadership was harsh on DevOps, but I think that the early “DevOps is Dead” take has dissipated.

The platform engineering community is doing a great job promoting the idea of product managing platforms, the notion of a platform itself.”

PaaS is for platform-as-a-service

However, Michael adds that it isn’t exactly sorted out yet what platform engineering means. He admits being unsure if he’s even part of the core platform engineering community, so he has given up on defining terms strictly.

When pressed for the definition of developer platform, Coté says he defines it as “everything above IaaS, what we used to call PaaS.“

“For whatever reason, people don’t like to use PaaS anymore, but it pretty much perfectly defines what a “platform” is. I mean, it’s right there in the name of Platform as a Service.

You could also throw in developer tooling like CI/CD pipelines and the collaborative sites/consoles/dashboards developers use. Internal developer portals are another category figuring itself out). If those tools are tightly integrated with the platform to make building, deploying, and running the applications better, it’s probably worth including them in the definition of “platform.”

How platforms work at companies with thousands of developers

Platform engineering and developer platforms have been in the spotlight for the last couple of years, but according to Coté, they have been a thing for at least 10 years. They’ve come to the spotlight now because most organizations have finished their first round of putting Kubernetes in place.

“It took several years to figure that out and start seeing more use in the mainstream. Once you get Kubernetes up and running, you need to start building a platform on top of it: you have to add all that other stuff developers use.”

What Michael is particularly interested in is how big companies with thousands of developers and usually operating in pretty regulated industries have been running developer platforms in the last five, seven, or even ten years:

“Companies like Mercedes-Benz, JP Morgan Chase, several militaries, and others have been running platforms like those for 5, seven, ten-plus years.

What I like to do in my talks is catalog the practices they’ve learned over those years and what works and doesn’t.

I’m especially interested in what very large, usually 30 to 50 or more years old organizations are doing with their platforms: how they make it work for thousands of developers.

As a community, we tend to dismiss the wisdom of teams like this because they’re not using the next great technology. But, the ways to run a platform are pretty much constant over time, especially in larger organizations.”

The developer platform is a product; developers are your customers

One of the most important lessons he learned from working with such companies and their developer platforms for years is to treat the developer platform as a product. And develop it as any company would develop a product by gathering user requirements, developing what they want, getting feedback, and iterating on it.

If you want to have a good platform you have to immediately start product managing it and think of application developers as your customers.

Whatever team is building the platform should be talking with application developers all the time (weekly or so) and getting feedback on what works well for them, what doesn’t work, and if recent changes to the platform have improved things.

You’d think this is what operations people who run this kind of thing do, but they’re usually more focused on the state and status of the system – if it’s running, if it’s secure, etc., rather than the platform’s usefulness to developers.

The post Michael Coté, VMware: Developer platforms have been a thing for at least 10 years appeared first on ShiftMag.

Viktor Farcic works at Upbound, the company behind Crossplane, as a – in his words – “developer advocate, product manager, maybe. Something like that”. He is also a member of the Google Developer Experts, CDF Ambassadors, and Docker Captains groups, a published author and a host of the YouTube channel DevOps Toolkit and a co-host of the DevOps Paradox podcast.

In his videos, Viktor often explains the concepts behind DevOps, compares different tools, or shares his knowledge. We’re not sure if he would approve of the term DevOps influencer, but he is an influential voice in the community and, hence, the best person to share his opinions on some burning questions from the industry. 

DevOps is about self-sufficiency

How would you define DevOps, anyway?

Farcic: Many people, companies, and teams misunderstand what DevOps is. Hence, we got roles like DevOps engineer, essentially the same as people did before.

I do not believe there is such a thing as a DevOps engineer.  If the whole point of DevOps is to enable communication and collaboration between people who work in Development and those who work in Operations, then what does a DevOps engineer do? That’s just a different title for the same job.

I believe that DevOps is all about self-sufficiency. It is about enabling teams to do everything they need related to their project. 

Suppose they work on a frontend application or backend application or anything else. In that case, they should be in complete control of the lifecycle of their application from the beginning, from the idea until it is running in production. 

That is much more efficient than throwing things over the wall: OK, I’m finished coding, let me send it to a different department that will do testing, and then they will send it to another department that will deploy it somewhere, and then they will send it to a different team that will do something else and so on and so forth. That is very inefficient.

DevOps is all about joining development and operational skills into a single team that can do everything.

Internal developer platform means enablement

What about all the buzz about internal developer platforms replacing DevOps? 

Farcic: Internal developer platforms are systems or platforms, as the word says, that enable everybody in the company to perform all the activities they need to do to do their job without relying on others and waiting for others to do that.

In practice, certain experts are codifying their experience into services. Hence, if you’re a database administrator, you’re an expert. Instead of waiting for somebody to ask you to create a database for them or to configure it, you can codify that knowledge, transform it into a service and plug it into that platform so that everybody else can do it themselves instead of asking you to do things for them. Click a button, fill in some fields, and create some YAML; whatever the system is, should be the mechanism for others to create and manage that database without you.

And you should focus on managing those services instead of managing requests from people to do something. That applies to everything. 

It applies to infrastructure, containers, building, testing or whatever that something is. If you’re an expert in something, you should codify that experience and expose it as a service to others.

When we combine all those services, we get a platform. We get what we call today the internal developer platforms. It’s a place where people can perform activities related to their work, but they’re not their core expertise because nobody can know everything. But we should be able to perform all the tasks we need for our project. Platform engineering is all about creating internal developer platforms.

Doing DevOps because of laziness

How come you ended up in the field of DevOps?

Farcic: The answer is simple, laziness.  Most of the time, I’m lazy, or to be more precise, I do not want to do the things I don’t want to do. I want to concentrate on things that bring value. 

Early on in my career, I spent a tremendous amount of time automating myself out of my job to let machines do the boring parts and me do the things that interest me. 

That had many forms in the past. I automated my tests, I automated my processes through CI/CD pipelines, and so on. Right now, that is in the form of DevOps. I’m finding better ways to enable myself and others to do things. 

Having more tools is good for the industry but exhausting for engineers

Many of your videos compare different developer tools or explain how a certain tool works. Do you sometimes feel like there is a new dev tool every day? Do you feel the tooling fatigue?

Farcic: Having more tools is not necessarily better for an individual, but it’s better for the industry as a whole. That’s how we advance. There is a group of people who think that they can do something better. They start a new project, they create a new tool, and that tool competes with other tools in that area. 

Without an increased number of tools, we would not have competition. We would be stuck with the tools from 30 years ago. 

I will never be against new tools because that would mean I’m against innovation. But that’s for the industry as a whole. For individuals, for developers, that can be overwhelming. That’s normal.

But that’s OK if people focus on what their expertise is. I expect a Node.js developer to explore new libraries for Node.js. I expect Kubernetes experts to explore new ways to create custom resource definitions in Kubernetes. I expect cloud managers to look forward to the new APIs in AWS for new services. That’s all good. 

The problem arises when companies assume that everybody will focus on everything, and that’s when people get overwhelmed with tools. 

But if people focus on specific areas, that shouldn’t be a problem because exploring new tools that help their core expertise improve and the job they’re doing improves is a good thing. Even if we spend one day every week exploring new tools that result in higher efficiency, we ultimately win. 

The disappearance of Kubernetes

Which new or existing trends or niches will be emerging as key in the next few years?

Farcic: It’s easier to say what trends would be emerging in a specific sub-niche or category like security, automation, building, or running stuff. But if I were pressed to answer for DevOps, the answer would be – Kubernetes is key. It feels like Kubernetes has been with us for a long time, but we are only starting now. 

What comes next is the disappearance of Kubernetes, but not in a sense that we will not be using it, but in the sense that it will become an implementation detail backed by higher-up structural layers. 

We will see companies, teams, and providers offering services on top of Kubernetes that will eventually wholly remove it. You will be able to deploy or manage applications or infrastructure very efficiently with Kubernetes without even seeing Kubernetes. 

DevOps is not a good starting point for a junior

Experts in the field often say there are no junior DevOps engineers. Do you agree? 

I don’t think that there is such a thing as a DevOps engineer, to begin with.

And if there is such a thing as a DevOps engineer, it couldn’t be a junior person simply because DevOps combines development and operations. Starting your career in that area would be silly because that means that you would focus both on infrastructure and systems and development from the very beginning of your career.

It makes so much more sense early on when you’re junior to focus on learning one thing: how to write code, use AWS, or focus on Linux. And eventually, over time, you will be able to extend your focus, to extend the number of areas you are an expert in, and then you can call yourself a senior. 

But if DevOps was a thing, if a DevOps engineer was a thing, which I don’t believe it is, that means that those engineers would need to know almost everything. That’s not a good starting point for a junior. 

You create videos, speak at conferences, are active on social media, co-host a podcast, and are the author of two books. Where do you get inspiration and motivation for creating content and sharing knowledge?

Farcic: I am an old person. I have been in this industry for a long time, and many people helped me when I started and through my career. In the second half of my career, I feel that it’s my responsibility to return the favor. It’s my responsibility to help and teach people younger than me. And when they become seniors and principals, they should be helping their younger colleagues. 

It is also gratifying for me, and I learn something when I teach others. I have support from the companies and organizations I am involved with. It’s my way of giving back.

The post Viktor Farcic: There is no such thing as a DevOps engineer appeared first on ShiftMag.