I Talk to Production Every Morning. It Talks Back.

I was good at math and algorithm coding challenges in high school, and I knew I would rather not be poor, if possible, and the salary of a software engineer is decent. I started working with a naive assumption that soon I will know how to do the job well, it would become a routine and then I would just cruise through it.

I soon learned that college did not prepare me for the job, got some kind of idea of the skills and knowledge required to ship production software, and figured out that the more I knew the less stressful the job would be.

So over the course of many years, much against my will, I became an expert. Then AI came into the picture, and nothing fundamentally changed, because the job of a software engineer is to help achieve business goals by leveraging their understanding of the technology, and now there was yet another technology to learn.

Get ahead of security issues

I have decided to focus on making the job less miserable for me. My original approach, once I learned how to use tools such as Claude, was to do the same amount of work as before and then just chill.

Nobody ever complained about my performance, so I figured if I just did the same amount of stuff in less time, I would have time for the things I enjoyed, like drinking coffee with other slackers on the balcony or writing pieces such as this one.

This felt good for a while, but still there were annoying things that needed to be done. One of them is upgrading packages to mitigate security issue. I don’t know about you, but I fall asleep each time I need to read a CVE Record.

So I get AI to write me a short horror story written from the perspective of a sarcastic and frustrated application security engineer whose warnings companies have ignored at their own peril.

The story always has some snippets that help me understand how this exploit might happen on services I am responsible for, and it always ends up with my company going bankrupt, this really makes the lessons stick and motivates me to fix the issues.

Sure I could completely automate the bumping, but where is fun in that, and what do I learn?

The production can be cruel

The other thing I don’t like is the production; the production is cruel. You surround your code with all kinds of tests, you create the production like environment with simulated traffic, you use feature flags, you deploy small increments often, and still production finds the ways to surprise you, and it has a tendency not to do that during business hours.

So you do what you must, add a bunch of observability, create runbooks and alerts. Now, with all that telemetry in place and a coding agent I can finally engage with production properly: not reactively when something goes wrong, but every day I can chat with the production, ask it about its concerns, about what is going on lately, if something feels wrong.

I do this by invoking the skill that gathers all the data and then reports it with a certain tone of voice. And I’ll tell you one thing, gaze long enough into production telemetry and the production gazes back.

Chat instead of jot

One last example, and I envy the juniors for having the ability to do this from the start, is the change in the way how I learn stuff. When I was starting, I used to jot down the things and concepts I did not understand, I would look them up in articles and books, I would go through half-baked internet tutorials, blog posts and man pages, and still come out with questionable understanding.

Now I start a chat session with an AI model, give him a personality, most often it is an old and grumpy but well-mannered distinguished engineer who explains topics with a lot of examples and then asks challenging questions while throwing causal insults, he asks me a challenging question and then roasts my answer before proceeding with the next lesson.

Hope you find some of these ideas useful, because I feel much of the talk has been on how to use AI to speed things up, and too little of the talk is about how to use AI to make software more reliable or the process of building and maintaining it more enjoyable.