<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Hrvoje Rancic, Author at ShiftMag</title>
	<atom:link href="https://shiftmag.dev/author/hrvojer-rancic/feed/" rel="self" type="application/rss+xml" />
	<link>https://shiftmag.dev/author/hrvojer-rancic/</link>
	<description>Insightful engineering content &#38; community</description>
	<lastBuildDate>Fri, 16 Oct 2026 10:05:56 +0000</lastBuildDate>
	<language>en-GB</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://shiftmag.dev/wp-content/uploads/2026/09/cropped-Shift-32x32.webp</url>
	<title>Hrvoje Rancic, Author at ShiftMag</title>
	<link>https://shiftmag.dev/author/hrvojer-rancic/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>I Talk to Production Every Morning. It Talks Back.</title>
		<link>https://shiftmag.dev/i-talk-to-production-every-morning-it-talks-back-12679/</link>
		
		<dc:creator><![CDATA[Hrvoje Rancic]]></dc:creator>
		<pubDate>Thu, 15 Oct 2026 13:22:52 +0000</pubDate>
				<category><![CDATA[Artificial Intelligence]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[artificial intelligence]]></category>
		<category><![CDATA[Developer Productivity]]></category>
		<category><![CDATA[Engineer Explains]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=12679</guid>

					<description><![CDATA[<p>A lot of people are leaving software engineering because agent-based workflows killed their joy. I don't have that problem; I never liked the job in the first place.</p>
<p>The post <a href="https://shiftmag.dev/i-talk-to-production-every-morning-it-talks-back-12679/">I Talk to Production Every Morning. It Talks Back.</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">I was good at math and algorithm coding challenges in high school, and I knew I would rather not be poor, if possible, and the salary of a software engineer is decent. I started working with a naive assumption that soon I will know how to do the job well, it would become a routine and then I would just cruise through it. </p>



<p class="wp-block-paragraph">I soon learned that <strong>college did not prepare me for the job</strong>, got some kind of idea of the skills and knowledge required to ship production software, and figured out that the more I knew the less stressful the job would be.</p>



<p class="wp-block-paragraph">So over the course of many years, much against my will, <strong>I became an expert</strong>. Then AI came into the picture, and nothing fundamentally changed, because the job of a software engineer is to help achieve business goals by leveraging their understanding of the technology, and now there was yet another technology to learn.</p>



<h2 class="wp-block-heading" id="get-ahead-of-security-issues">Get ahead of security issues</h2>



<p class="wp-block-paragraph">I have decided to <strong>focus on making the job less miserable for me</strong>. My original approach, once I learned how to use tools such as Claude, was to do the same amount of work as before and then just chill.</p>



<p class="wp-block-paragraph">Nobody ever complained about my performance, so I figured if I just did the same amount of stuff in less time, I would have time for the things I enjoyed, like drinking coffee with other slackers on the balcony or writing pieces <a href="https://shiftmag.dev/postmortem-the-boiler-incident-6986/" target="_blank" rel="noreferrer noopener">such as this one</a>.</p>



<p class="wp-block-paragraph">This felt good for a while, but still there were annoying things that needed to be done. One of them is upgrading packages to mitigate security issue. I don’t know about you, but I fall asleep each time I need to read a <a href="https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures" target="_blank" rel="noreferrer noopener">CVE Record</a>.</p>



<p class="wp-block-paragraph">So I get AI to write me a short horror story written from the perspective of a sarcastic and frustrated application security engineer whose warnings companies have ignored at their own peril. </p>



<p class="wp-block-paragraph">The story always has some snippets that <strong>help me understand how this exploit might happen on services I am responsible for</strong>, and it always ends up with my company going bankrupt, this really makes the lessons stick and motivates me to fix the issues.</p>



<p class="wp-block-paragraph">Sure I could completely automate the bumping, but where is fun in that, and what do I learn?</p>



<figure class="wp-block-image size-large"><img fetchpriority="high" decoding="async" width="1024" height="538" src="https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-3-1-1024x538.png?x32039" alt="" class="wp-image-12706" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-3-1-1024x538.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-3-1-300x158.png 300w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-3-1-768x403.png 768w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-3-1.png 1200w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>



<h2 class="wp-block-heading" id="the-production-can-be-cruel">The production can be cruel</h2>



<p class="wp-block-paragraph">The other thing I don’t like is the production; <strong>the production is cruel</strong>. You surround your code with all kinds of tests, you create the production like environment with simulated traffic, you use feature flags, you deploy small increments often, and still production finds the ways to surprise you, and it has a tendency not to do that during business hours.</p>



<p class="wp-block-paragraph">So you do what you must, add a bunch of observability, create runbooks and alerts. Now, with all that telemetry in place and a coding agent I can finally engage with production properly: not reactively when something goes wrong, but every day <strong>I can chat with the production</strong>, ask it about its concerns, about what is going on lately, if something feels wrong.</p>



<p class="wp-block-paragraph">I do this by invoking the skill that gathers all the data and then reports it with a certain tone of voice. And I’ll tell you one thing, gaze long enough into production telemetry and the production gazes back.</p>



<h2 class="wp-block-heading" id="chat-instead-of-jot">Chat instead of jot</h2>



<p class="wp-block-paragraph">One last example, and I envy the juniors for having the ability to do this from the start, is t<strong>he change in the way how I learn stuff</strong>. When I was starting, I used to jot down the things and concepts I did not understand, I would look them up in articles and books, I would go through half-baked internet tutorials, blog posts and man pages, and still come out with questionable understanding.</p>



<p class="wp-block-paragraph">Now I start a chat session with an AI model, give him a personality, most often it is an old and grumpy but well-mannered distinguished engineer who explains topics with a lot of examples and then asks challenging questions while throwing casual insults, he asks me a challenging question and then roasts my answer before proceeding with the next lesson.</p>



<p class="wp-block-paragraph">Hope you find some of these ideas useful, because I feel much of the talk has been on how to use AI to speed things up, and too little of the talk is about how to use AI to make software more reliable or the process of building and maintaining it more enjoyable.</p>


<figure class="wp-block-post-featured-image"><img decoding="async" width="1200" height="630" src="https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-2.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-2.png 1200w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-2-300x158.png 300w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-2-1024x538.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-2-768x403.png 768w" sizes="(max-width: 1200px) 100vw, 1200px" /></figure><p>The post <a href="https://shiftmag.dev/i-talk-to-production-every-morning-it-talks-back-12679/">I Talk to Production Every Morning. It Talks Back.</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Postmortem: The Boiler Incident</title>
		<link>https://shiftmag.dev/postmortem-the-boiler-incident-6986/</link>
		
		<dc:creator><![CDATA[Hrvoje Rancic]]></dc:creator>
		<pubDate>Fri, 21 Nov 2025 13:53:24 +0000</pubDate>
				<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Developer Experience]]></category>
		<category><![CDATA[Developer Productivity]]></category>
		<category><![CDATA[development]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=6986</guid>

					<description><![CDATA[<p>The following is an attempt to clarify the circumstances of a production incident that happened more than 10 years ago; it is also a lesson to all of us to decouple our production servers from boilers in distant Balkan countries.</p>
<p>The post <a href="https://shiftmag.dev/postmortem-the-boiler-incident-6986/">Postmortem: The Boiler Incident</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[<figure class="wp-block-post-featured-image"><img decoding="async" width="1200" height="630" src="https://shiftmag.dev/wp-content/uploads/2025/11/shifmag_final.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2025/11/shifmag_final.png 1200w, https://shiftmag.dev/wp-content/uploads/2025/11/shifmag_final-300x158.png 300w, https://shiftmag.dev/wp-content/uploads/2025/11/shifmag_final-1024x538.png 1024w, https://shiftmag.dev/wp-content/uploads/2025/11/shifmag_final-768x403.png 768w" sizes="(max-width: 1200px) 100vw, 1200px" /></figure>


<p class="wp-block-paragraph">A couple of weeks ago, while we were having a casual conversation over a cup of coffee, one of our veteran engineers said something that piqued my interest.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;This reminds me of the time when boiler broke down in the Zagreb office, and <strong>production went down</strong>.&#8221; &#8211; he made a casual remark.</p>



<p class="wp-block-paragraph">&#8220;I beg your pardon?&#8221; &#8211; I expressed my interest, and my left eyebrow rose slightly to signal disbelief.&nbsp;</p>



<p class="wp-block-paragraph">&#8220;We experienced <strong>major degradation of service across all products served over data center in Frankfurt</strong>.&#8221; – he continued in a neutral tone of voice, as if he was reading the airport announcement for an upcoming flight.&nbsp;</p>



<p class="wp-block-paragraph">&#8220;You are not being serious!&#8221; – I exclaimed in a shocked tone of voice, as if I realized I was late for a flight.&nbsp;</p>



<p class="wp-block-paragraph">“Of course I am serious!&#8221; &#8211; he said.</p>



<p class="wp-block-paragraph">Here’s the full story.</p>



<h2 class="wp-block-heading" id="boiler-in-zagreb-crashed-the-servers-in-frankfurt">Boiler in Zagreb crashed the servers in Frankfurt</h2>



<p class="wp-block-paragraph">&#8220;You remember the old office building in Zagreb, the skyscraper with the giant advertisement for toilet paper?&#8221; &#8211; he asked me.&nbsp;</p>



<p class="wp-block-paragraph">&#8220;Of course I do, I worked there for a couple of years. Very appropriate ad for the state the building was in.&#8221; I answered&nbsp;</p>



<p class="wp-block-paragraph">&#8220;Well, there was an <strong>old rusty water heater in the basement, and it broke down</strong>&#8221; &#8211; he said.&nbsp;</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">The next thing we know is that we are getting a ton of angry calls from our customers. I don’t remember the details, but it was connected somehow, a boiler in Zagreb crashed the servers in Frankfurt.</p>
</blockquote>



<p class="wp-block-paragraph">&#8220;Sounds plausible.&#8221; &#8211; I stated calmly, convinced he was pulling my leg. </p>



<p class="wp-block-paragraph">&#8220;Who can tell me more about it?&#8221; &#8211; and <strong>provoked him to continue</strong>, which he did without hesitation.&nbsp;&nbsp;</p>



<h2 class="wp-block-heading" id="the-authorization-server-went-down-first">The authorization server went down first</h2>



<p class="wp-block-paragraph">&#8220;I think the authorization server was the first to crash; it was handled by One API team. The team was disbanded years ago, but most of the people who worked in the team still work here, so you can try talking to them.&#8221; &#8211; he gave me names, and I jotted them down in my notebook.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">It made sense that an <strong>issue with the authorization server would affect all the products</strong>. It is an obstacle all aspiring requests must overcome, whatever their place of origin may be, if they wish to mature and become successful responses one day. </p>



<p class="wp-block-paragraph">But how does the boiler fit in the picture? Did we start as a water heater provider, and then pivot to communication platform space? </p>



<p class="wp-block-paragraph">The code of the authorization server I checked out provided no clues, so I sent a <strong>Slack message to the former One API team leader</strong>.&nbsp;</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="847" height="444" src="https://shiftmag.dev/wp-content/uploads/2025/11/image.png?x32039" alt="" class="wp-image-6987" srcset="https://shiftmag.dev/wp-content/uploads/2025/11/image.png 847w, https://shiftmag.dev/wp-content/uploads/2025/11/image-300x157.png 300w, https://shiftmag.dev/wp-content/uploads/2025/11/image-768x403.png 768w" sizes="auto, (max-width: 847px) 100vw, 847px" /></figure>



<p class="wp-block-paragraph">&#8220;I remember nothing!&#8221; she replied and moved on with her duties, ignoring further prompts, so I decided to continue my investigation elsewhere.</p>



<h2 class="wp-block-heading" id="boredom-led-me-into-the-dark-archives-of-jira">Boredom led me into the dark archives of Jira</h2>



<p class="wp-block-paragraph">As coding agents and other AI tools are doing my work now and I have nothing better to do, <strong>I started digging through git history and Jira archives</strong>. </p>



<p class="wp-block-paragraph">After a couple of hours I found a clue, <strong>there was a ticket with description</strong> <em>Reimplement the logging and email adapter, so that we avoid the issues such as the one we had when there was no electricity in Zagreb office.</em></p>



<p class="wp-block-paragraph">There was a link to a pull request attached to the ticket; the username of the author belonged to a prominent figure, so I sent a Slack message to our Engineering Director: &#8220;We need to talk about a <strong>pull request you made to the auth server ten years ago</strong>.&#8221; Shortly after, he invited me to his office.&nbsp;</p>



<p class="wp-block-paragraph">&#8220;It was a warm Spring day, the Sun was shining, the birds were chirping, the children were playing and production was down. We were drinking in the pub.&#8221; &#8211; he started reminiscing, &#8220;The production had a habit of crashing each time we went for a beer.&#8221;&nbsp; He confirmed that there was a serious outage and that it was connected to an email server; he <strong>couldn’t quite remember how the boiler fit into the story</strong>.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;Why was authorization server sending emails?&#8221; &#8211; I asked.&nbsp;&nbsp;</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">At the time we implemented a brute force attack detection mechanism that could ban or lock attackers. Sometimes a legitimate customer would get banned, so they would contact their dedicated account managers to help them. When a user, or a combination of a username and an IP address to be more precise, was blocked the authorization server would automatically send the data that might come in handy to the account manager.</p>
</blockquote>



<p class="wp-block-paragraph">&#8220;I see, so you mean something like <em>detailed</em> <em>description of our current algorithm, automation and telemetry that helps resolve the issue but edited out because of security reasons and not breaching NDA via public magazine article</em>?&#8221; &#8211; I asked only to show off my understanding of our security infrastructure in hopes he would be impressed and commend me to someone who is responsible for raising my salary.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;It was nowhere near that complex, we were not operating on this scale, and <strong>such issues would not happen very often</strong>, unless someone got too enthusiastic with pen testing&#8221; &#8211; he responded bluntly and with a look on his face that I interpreted as him having no intention to commend my deep understanding of our security infrastructure to someone responsible for the pay raise. </p>



<p class="wp-block-paragraph">Then he shared a couple of interesting production outage stories from the early days and gave me advice to <strong>talk to our first system administrator in the company</strong>, since that gentleman was maintaining the email server at the time.&nbsp;&nbsp;</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="846" height="594" src="https://shiftmag.dev/wp-content/uploads/2025/11/image-1.png?x32039" alt="" class="wp-image-6988" srcset="https://shiftmag.dev/wp-content/uploads/2025/11/image-1.png 846w, https://shiftmag.dev/wp-content/uploads/2025/11/image-1-300x211.png 300w, https://shiftmag.dev/wp-content/uploads/2025/11/image-1-768x539.png 768w" sizes="auto, (max-width: 846px) 100vw, 846px" /></figure>



<h2 class="wp-block-heading" id="let%e2%80%99s-pay-a-visit-to-our-sre-team">Let’s pay a visit to our SRE team</h2>



<p class="wp-block-paragraph">As I talked to the system administrator, he shared stories that can put this one to shame: duck taped disks, convincing people to put servers somewhere where rain won’t affect them, crashing the premises of a 3<sup>rd</sup> party vendor (euphemism for a guy who has a couple of servers at home and rents them out to small companies) who is putting server offline to make a firmware update during peak traffic time. </p>



<p class="wp-block-paragraph">He <strong>could not remember the boiler issue</strong> but recalled there was an unrelated air conditioner incident one summer in the same office building when an air conditioner cooling the centralized control system for other air conditioners overheated because it was installed on the rooftop.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">I decided to pay a visit to our <strong>SRE department</strong> as their lead engineer was a member of the One API team once upon a time. </p>



<p class="wp-block-paragraph">The SRE department is taking care that our global platform, spanning several continents and dozens of data centers, works reliably. They are working on <strong>early detection and quick mitigation of service degradations</strong> that can happen as we make thousands of changes to production every day. </p>



<p class="wp-block-paragraph">Surely, their lead engineer has nothing better to do than to talk to me about a production issue from a decade ago.&nbsp;&nbsp;</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">There was some electrical maintenance in the office building, and right before it someone reconfigured auth server to send e-mails to a standby instance in Zagreb office, twenty-one thousand four hundred and eighty-seven emails were stuck in the queue. Can’t talk now, sharks are eating network cables again.</p>
</blockquote>



<p class="wp-block-paragraph">…he said, hurrying off with his coffee.&nbsp;</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="808" height="699" src="https://shiftmag.dev/wp-content/uploads/2025/11/image-2.png?x32039" alt="" class="wp-image-6989" srcset="https://shiftmag.dev/wp-content/uploads/2025/11/image-2.png 808w, https://shiftmag.dev/wp-content/uploads/2025/11/image-2-300x260.png 300w, https://shiftmag.dev/wp-content/uploads/2025/11/image-2-768x664.png 768w" sizes="auto, (max-width: 808px) 100vw, 808px" /></figure>



<h2 class="wp-block-heading" id="boiler-blew-mail-server-died-everything-froze">Boiler blew, mail server died, everything froze</h2>



<p class="wp-block-paragraph">I have finally connected all the dots and <strong>found someone to confirm the theory</strong>.&nbsp;</p>



<p class="wp-block-paragraph">Our principal engineer &#8211; once a junior contributor to the auth server &#8211; explained what happened when he returned from parental leave:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">The same thread that banned the user also sent the email synchronously via RPC, with no timeout. When the boiler broke, it tripped the circuit breaker, cutting power to the floor. The mail server was down, and so were we &#8211; waiting in the pub for the electricity company. When customers complained, we were one thread dump away from realizing all request-handling threads were stuck waiting on the mail server.</p>
</blockquote>



<p class="wp-block-paragraph">&#8220;The circuit breaker broke, but there was no <a href="https://martinfowler.com/bliki/CircuitBreaker.html" target="_blank" rel="noreferrer noopener">circuit breaker</a> around the call to mail server!&#8221; &#8211; I joked nerdishly.</p>



<p class="wp-block-paragraph">&#8220;Exactly! After the electricity company fixed the issue, in order to increase reliability someone duck taped the circuit breaker. If we had those things back then, could you imagine what would status page and post incident review look like?&#8221; &#8211; he asked laughingly as we continued to entertain that thought.&nbsp;</p>



<p class="wp-block-paragraph"><strong>Status page &#8211; Frankfurt: Service degradation across all products</strong>&nbsp;</p>



<p class="wp-block-paragraph"><em>We are experiencing service degradation across all products due to boiler malfunctions. The plumber is on his way to the office building, and we expect to resolve the problem shortly. We apologize for any inconvenience this may cause.&nbsp;</em></p>



<h2 class="wp-block-heading" id="postmortem-boiler-malfunction-incident-420">Postmortem: Boiler malfunction (incident #420)</h2>



<p class="wp-block-paragraph"><strong>Date: </strong>April 1, 2010&nbsp;</p>



<p class="wp-block-paragraph"><strong>Authors: </strong>Alice, Bob&nbsp;</p>



<p class="wp-block-paragraph"><strong>Status: </strong>Complete, action items in progress&nbsp;</p>



<p class="wp-block-paragraph"><strong>Summary: </strong>The boiler broke down and all products experienced service degradation for 2 hours&nbsp;</p>



<p class="wp-block-paragraph"><strong>Impact: </strong>A large number of messages never went out, there was a noticeable financial impact, new valve on the boiler and 1000 dollars for one hour of plumbers’ work&nbsp;</p>



<p class="wp-block-paragraph"><strong>Root causes: </strong>Cascading failure due to a water heater malfunction in Zagreb office building, the floor lost electricity, and so the email server went down. At the time the authorization server in Frankfurt was configured to send emails over that email server. The calls to the email server were sent synchronously with no timeout, so all threads were blocked on the call to the email server, and there were no available threads to serve the incoming requests.&nbsp;</p>



<p class="wp-block-paragraph"><strong>Trigger: </strong>Malfunction in the boiler, waiting for the analysis from the plumber.&nbsp;</p>



<p class="wp-block-paragraph"><strong>Resolution: </strong>Authorization server reconfigured to use the email server instances in Frankfurt data center.&nbsp;</p>



<p class="wp-block-paragraph"><strong>Detection: </strong>The customers complained before we detected the issue.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph"><strong>Action items:</strong>&nbsp;</p>



<ol start="1" class="wp-block-list">
<li>Batch emails and send them asynchronously&nbsp;</li>
</ol>



<ol start="2" class="wp-block-list">
<li>Introduce a circuit breaker around RPC call&nbsp;</li>
</ol>



<ol start="3" class="wp-block-list">
<li>Duct tape the circuit breaker of the water heater&nbsp;</li>
</ol>



<ol start="4" class="wp-block-list">
<li>Put the electrician and plumber’s phone number on speed dial&nbsp;</li>
</ol>



<p class="wp-block-paragraph"><strong>Lessons Learned</strong></p>



<p class="wp-block-paragraph">Don’t couple the production traffic to the boiler.&nbsp;&nbsp;</p>



<h2 class="wp-block-heading" id="timeline">Timeline</h2>



<p class="wp-block-paragraph">2010-04-01 (all times UTC)&nbsp;</p>



<ul class="wp-block-list">
<li>9:45 &#8211; deployment of authorization server in Frankfurt, it starts using mail server in Zagreb due to a configuration mistake&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>10:32 &#8211; the boiler burns out and triggers an issue with electrical supply to the building, the rack with the email server also loses power&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>10:45 &#8211; the office building manager calls the electricity company&nbsp;&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>10:50 &#8211; One API team goes for a beer in the nearby pub&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>11:05 &#8211; a young and enthusiastic application security engineer hears about the new brute force detection mechanism and decides to test it&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>11:30 &#8211; electricians blame the plumbers&nbsp;&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>11:35 &#8211; <strong>OUTAGE BEGINS</strong> a lot of request handling threads are waiting for response from the email server, authorization server starts throttling incoming requests&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>11:45 &#8211; customers start calling the customer support, the line is busy we are trying to reach the plumber&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>11:50 <strong>INCIDENT BEGINS</strong> all of request handling threads are in waiting state, no traffic is processed&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>12:00 &#8211; in the pub One API team jokes about how the last two times they went for a beer production went down&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>12:05 &#8211; plumber answers the telephone and starts negotiating his hourly rate&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>12:30 &#8211; plumber agrees on an hourly rate&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>12:45 &#8211; One Api team notices missed calls from the customer support team&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>12:55 &#8211; One API team rushes back to the office and bumps into team of electricians telling them they can’t go in right now&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>13:20 &#8211; One API team makes a thread dump and figures out the issue&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>13:35 &#8211; the configuration of authorization server is changed&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>13:45 &#8211; <strong>OUTAGE MITIGATED</strong> the service stars recovering&nbsp;&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>13:50 &#8211; <strong>INCIDENT ENDS&nbsp;</strong>&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>15:55 &#8211; the plumber enters the building&nbsp;&nbsp;</li>
</ul>



<ul class="wp-block-list">
<li>16:05 &#8211; the plumber leaves the building after replacing the valve on the boiler&nbsp;</li>
</ul>
<p>The post <a href="https://shiftmag.dev/postmortem-the-boiler-incident-6986/">Postmortem: The Boiler Incident</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Things ancient Romans taught me about software development </title>
		<link>https://shiftmag.dev/things-ancient-romans-taught-me-about-software-development-5214/</link>
		
		<dc:creator><![CDATA[Hrvoje Rancic]]></dc:creator>
		<pubDate>Tue, 13 May 2025 12:28:48 +0000</pubDate>
				<category><![CDATA[Software Engineering]]></category>
		<category><![CDATA[Kent Beck]]></category>
		<category><![CDATA[software engineering]]></category>
		<category><![CDATA[The Agile Manifesto]]></category>
		<category><![CDATA[Uncle Bob]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=5214</guid>

					<description><![CDATA[<p>History is the teacher of life. There are those who have been shipping software for decades, and there is much we can learn from them.</p>
<p>The post <a href="https://shiftmag.dev/things-ancient-romans-taught-me-about-software-development-5214/">Things ancient Romans taught me about software development </a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="1536" height="1024" src="https://shiftmag.dev/wp-content/uploads/2025/04/software-engineering-philosophers.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2025/04/software-engineering-philosophers.png 1536w, https://shiftmag.dev/wp-content/uploads/2025/04/software-engineering-philosophers-300x200.png 300w, https://shiftmag.dev/wp-content/uploads/2025/04/software-engineering-philosophers-1024x683.png 1024w, https://shiftmag.dev/wp-content/uploads/2025/04/software-engineering-philosophers-768x512.png 768w" sizes="auto, (max-width: 1536px) 100vw, 1536px" /></figure>


<h3 class="wp-block-heading" id="si-vis-pacem-para-bellum"><em>Si vis pacem, para bellum</em></h3>



<p class="wp-block-paragraph">If you want peace, prepare for war. In other words, if you wish to handle traffic spikes, <strong>stress test the service</strong>. If it can handle twice the expected traffic, it will handle the expected traffic. A similar point was expressed by <strong>Kentus Becchus</strong>, who <a href="https://www.goodreads.com/quotes/10376041-write-tests-until-fear-is-transformed-into-boredom" target="_blank" rel="noreferrer noopener">famously wrote</a> <em>Scribe probationes donec timor in taedium mutetur</em>.</p>



<h3 class="wp-block-heading" id="aequam-memento-rebus-in-arduis-servare-mentem"><em>Aequam memento rebus in arduis servare mentem</em></h3>



<p class="wp-block-paragraph">Remember to maintain a calm mind while doing difficult tasks. The natural response to a production outage is panicking. <a href="https://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker%27s_Guide_to_the_Galaxy#Don't_Panic" target="_blank" rel="noreferrer noopener"><strong>DON’T PANIC</strong></a>. Troubleshooting production issues is just an ordinary task with the highest priority.&nbsp;</p>



<h3 class="wp-block-heading" id="mens-sana-in-corpore-sano"><em>Mens sana in corpore sano</em></h3>



<p class="wp-block-paragraph">A healthy mind in a healthy body. Software development involves a lot of thinking, but it also involves a lot of sitting, which is <strong>not a particularly healthy activity</strong>.&nbsp; To keep the mind fresh and sharp, sleep and physical activity are paramount, as <strong>Vladus Mihalcaeus</strong> <a href="https://x.com/vlad_mihalcea/status/1335981628322148354" target="_blank" rel="noreferrer noopener">has put</a> it <em>Somnus octo horarum optimus emendator est</em>.</p>



<h3 class="wp-block-heading" id="divide-et-impera"><em>Divide et impera</em> </h3>



<p class="wp-block-paragraph">Divide and conquer. Apart from being <a href="https://en.wikipedia.org/wiki/Divide-and-conquer_algorithm" target="_blank" rel="noreferrer noopener">a well-known</a> algorithm design paradigm, it is a sound piece of advice: when faced with a complex task that takes a lot of time, <strong>divide it into a set of smaller tasks</strong> and handle those one at a time.&nbsp;&nbsp;</p>



<h3 class="wp-block-heading" id="ad-praesens-ova-cras-pullis-sunt-meliora"><em>Ad praesens ova cras pullis sunt meliora</em></h3>



<p class="wp-block-paragraph">A bird in the hand is worth two in the bush. First implement the basic functionality<strong>,</strong> show it to the user, <strong>ask for feedback and iterate</strong>. The same idea is expressed in the old dictum from <a href="https://agilemanifesto.org/" target="_blank" rel="noreferrer noopener">the Manifestum</a> which teaches us to prefer working software over a mountain of requirements specifications. </p>



<h3 class="wp-block-heading" id="bene-vixit-bene-qui-latuit"><em>Bene vixit,&nbsp; bene qui latuit</em> </h3>



<p class="wp-block-paragraph">He who remained hidden lived well. Although some malicious hackers may draw inspiration from this one, it is also inspiring to infrastructure engineers: if you break something, <strong>it will not go unnoticed</strong>.&nbsp;</p>



<h3 class="wp-block-heading" id="discere-faciendo"><em>Discere faciendo</em> </h3>



<p class="wp-block-paragraph">Learn by doing. The mastery of software engineering cannot be attained only by speculative thought, reasoning from first principles, or by learning the sacred texts by heart, to learn how to build software it is necessary, but not sufficient, to build it.  </p>



<h3 class="wp-block-heading" id="panem-et-circenes"><em>Panem et circenes</em> </h3>



<p class="wp-block-paragraph">Bread and circuses. A great deal of software engineering job is maintenance; it is necessary but not appreciated enough. To silence the murmurs, you need to show some impressive dashboard, shiny UI, or AI-related feature to management from time to time.  </p>



<h3 class="wp-block-heading" id="cessante-causa-cessante-effectus"><em>Cessante causa, cessante effectus </em></h3>



<p class="wp-block-paragraph">When the cause ceases, the effect ceases. Unless you <strong>know the root cause of an incident</strong>, you cannot consider it resolved. Rebooting or scaling may resolve the issue temporarily, but this is just buying time to fix the real problem.&nbsp;</p>



<h3 class="wp-block-heading" id="verba-volant-littera-scripta-manent"><em>Verba volant, littera scripta manent</em>  </h3>



<p class="wp-block-paragraph">Spoken words fly away, written ones remain. <strong>Agreements from meetings need to be written down</strong> in meeting notes, architectural decisions need to be documented, tasks need to have descriptions, when possible, agreements need to be expressed in code as automated checks. You will forget things!&nbsp;</p>



<h3 class="wp-block-heading" id="periculum-in-mora"><em>Periculum in mora</em> </h3>



<p class="wp-block-paragraph">Danger in delay. Integrate code changes to main branch often and <strong>deploy often</strong>. The more you wait the harder it is to integrate the change and the riskier it is to deploy it.&nbsp;&nbsp;</p>



<h3 class="wp-block-heading" id="historia-est-magistra-vitae"><em>Historia est magistra vitae</em> </h3>



<p class="wp-block-paragraph">History the teacher of life. As <strong>Patruus Bobus</strong> <a href="https://blog.cleancoder.com/uncle-bob/2014/06/20/MyLawn.html" target="_blank" rel="noreferrer noopener">points out</a> majority of programmers are not very experienced, however there are those who have been shipping software for decades, and there is much we can learn from them. <strong>Study their advice.&nbsp;</strong></p>



<h3 class="wp-block-heading" id="non-omnia-possumus-omnes"><em>Non omnia possumus omnes</em> </h3>



<p class="wp-block-paragraph">We can’t all <span style="box-sizing: border-box; margin: 0px; padding: 0px;">do everything. It is easier to ship software in a<strong>&nbsp;cross-functiona</strong></span><strong>l </strong>team&nbsp;made up of<strong>&nbsp;</strong>people with different skill sets and roles.&nbsp;</p>



<h3 class="wp-block-heading" id="omnium-enim-rerum-principia-parva-sunt"><em>Omnium enim rerum principia parva sunt</em> </h3>



<p class="wp-block-paragraph">The beginnings of all things are small. Big systems evolve from small systems. Create <a href="https://dannorth.net/best-simple-system-for-now/" target="_blank" rel="noreferrer noopener">the simplest thing that works well</a> and build on top of that. This was succinctly expressed by <strong>Johannes Gallus</strong> <a href="https://www.goodreads.com/quotes/9353506-a-complex-system-that-works-is-invariably-found-to-have" target="_blank" rel="noreferrer noopener">who wrote</a> <em>Systema complexum quod operatur, ex systemate simplici quod operavit ortum esse inventur</em>.</p>



<h3 class="wp-block-heading" id="pacta-sunt-servanda"><em>Pacta sunt servanda</em> </h3>



<p class="wp-block-paragraph">Agreements must be kept. <strong>Take care not to break backwards compatibility</strong>. Take great care when designing APIs; if it exposes some functionality you did not intend to expose, do not whine when someone starts to use it. </p>



<h3 class="wp-block-heading" id="vinum-bonum-pax-in-domum"><em>Vinum bonum, pax in domum</em></h3>



<p class="wp-block-paragraph">Good wine, peace in the house. Since <strong>building software is a group activity</strong>, it helps if members of the group can get along with each other. Occasional team building activity helps to foster a friendly atmosphere. </p>
<p>The post <a href="https://shiftmag.dev/things-ancient-romans-taught-me-about-software-development-5214/">Things ancient Romans taught me about software development </a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>&#8220;Good enough&#8221; code is just not good enough</title>
		<link>https://shiftmag.dev/good-enough-code-is-just-not-good-enough-2844/</link>
		
		<dc:creator><![CDATA[Hrvoje Rancic]]></dc:creator>
		<pubDate>Tue, 27 Feb 2024 08:33:40 +0000</pubDate>
				<category><![CDATA[Software Engineering]]></category>
		<category><![CDATA[code quality]]></category>
		<category><![CDATA[Mob programming]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=2844</guid>

					<description><![CDATA[<p>Striving for excellence is not just a matter of professional pride; it is necessary to prevent the endless cycle of suffering.</p>
<p>The post <a href="https://shiftmag.dev/good-enough-code-is-just-not-good-enough-2844/">&#8220;Good enough&#8221; code is just not good enough</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="1200" height="630" src="https://shiftmag.dev/wp-content/uploads/2024/02/Five-stars-1.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2024/02/Five-stars-1.png 1200w, https://shiftmag.dev/wp-content/uploads/2024/02/Five-stars-1-300x158.png 300w, https://shiftmag.dev/wp-content/uploads/2024/02/Five-stars-1-1024x538.png 1024w, https://shiftmag.dev/wp-content/uploads/2024/02/Five-stars-1-768x403.png 768w" sizes="auto, (max-width: 1200px) 100vw, 1200px" /></figure>


<p class="wp-block-paragraph">My colleague Vedran recently stated that <strong>&#8220;it&#8217;s OK if your code is just good enough&#8221;</strong> in an article that was both well-received and well-debated. While he makes some good points, I&#8217;d like to share some objections.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph"><a href="https://shiftmag.dev/code-quality-good-enough-2034/" target="_blank" rel="noreferrer noopener">The article</a> provoked <strong>a heated debate on <a href="https://news.ycombinator.com/item?id=38234653">Hacker News</a>, <a href="https://www.reddit.com/r/programming/comments/181izxk/its_ok_if_your_code_is_just_good_enough/">Reddit</a>, and among my teammates.</strong> I was annoyed by it (the article, not the debate) and hastily proclaimed I would write a critical response by tomorrow. This proclamation was uttered in front of many people and, to make matters worse, written in a public Slack channel, so there is no going back. </p>



<p class="wp-block-paragraph">And here I am now, in my room at eleven o&#8217;clock in the evening: a hothead writing <strong>an opinion piece on a highly subjective and controversial topic</strong> to be read by thousands of people.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">Why did I make such a silly mistake that led me to this unfortunate predicament? The answer is simple: I have spent most of today and yesterday working on a project I do not usually work on. It is a project filled with what Vedran would call: &#8220;just good enough code&#8221;. <strong>Working on that project left me feeling miserable and annoyed.&nbsp;&nbsp;&nbsp;</strong></p>



<h2 class="wp-block-heading" id="good-enough-is-really-not-that-great"><strong>&#8220;Good enough&#8221; is really not that great</strong>&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">For those of you who haven&#8217;t had the chance to read the article in question, it states that good enough code is <strong>&#8220;a nice middle ground between implementing a feature fast and maintaining the code quality</strong>&#8220;.  According to the author, some forgivable shortcomings include unnecessary levels of abstraction, some unclear naming, some misuse of exceptions, some code duplication, and a few larger functions and classes here and there.  </p>



<p class="wp-block-paragraph">My first objection to this &#8220;good enough&#8221; mantra is that<strong> it promotes a culture of mediocracy</strong>. It preaches that it is OK to settle with barely adequate. This kind of attitude results in systems that inspire people to coin jokes like:&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;The difference between a tech enthusiast and a software engineer is that everything in a tech enthusiast&#8217;s home is wired to the internet and controlled over a smartphone, whereas everything in a software engineer&#8217;s home is entirely mechanical, except for a washing machine from 2000, but he keeps a shotgun close by in case it started to behave oddly&#8221;.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">My second objection to the definition of &#8220;good enough code &#8220;is the notion that <strong>&#8220;very good code&#8221; is so hard to achieve that it is not worth the trouble</strong>. I will challenge these notions further down, but before I do that, I&#8217;d like to make a few remarks on software projects in general.&nbsp;&nbsp;&nbsp;</p>



<h2 class="wp-block-heading" id="software-projects-do-not-exist-in-a-vacuum">Software projects do not exist in a vacuum</h2>



<p class="wp-block-paragraph">They are at the center of an ecosystem of stakeholders, and they are designed to satisfy their needs: <strong>the need of the customer</strong> for an affordable and user-friendly feature set, <strong>the need of the entrepreneur</strong> for a multimillion-dollar income, <strong>the need of the developer</strong> to earn a competitive salary by working with bleeding edge technologies in a collaborative, fast-paced, dynamic environment; to name just a few.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">It might not seem that software design details have a huge effect on the lives of the ecosystem&#8217;s inhabitants, but that is not the case.&nbsp; &nbsp;&nbsp;</p>



<p class="wp-block-paragraph">I&#8217;ll put this in context with an imaginary take on a classic game, Duck Hunt. Let&#8217;s call it Duck Hunt Extreme, a software that enables hunters to hunt from the comfort of their homes through digitally controlled weaponry.&nbsp;&nbsp;</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="780" height="438" src="https://shiftmag.dev/wp-content/uploads/2024/02/image.png?x32039" alt="" class="wp-image-2845" srcset="https://shiftmag.dev/wp-content/uploads/2024/02/image.png 780w, https://shiftmag.dev/wp-content/uploads/2024/02/image-300x168.png 300w, https://shiftmag.dev/wp-content/uploads/2024/02/image-768x431.png 768w" sizes="auto, (max-width: 780px) 100vw, 780px" /></figure>



<h2 class="wp-block-heading" id="unnecessary-levels-of-abstraction"><strong>&#8220;Unnecessary levels of abstraction&#8221;&nbsp;</strong>&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">One of the product manager&#8217;s responsibilities is to gather customer feedback and translate it to feature requirements for developers:&nbsp;</p>



<p class="wp-block-paragraph">&#8220;I would like the application to have support for hunting ducks with a chainsaw,&#8221; a customer optimistically makes a reasonable request.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">An enthusiastic <strong>product manager asks developers for an estimate.</strong> &nbsp;</p>



<p class="wp-block-paragraph">&#8220;Well, if it weren&#8217;t for these crazy levels of abstraction, we might give you an honest estimate, but <strong>the code is hard to understand</strong> and reason about, and consequently, it is hard to modify. The best we can give you is an estimate about when we will be able to give you an estimate,&#8221; the developers respond.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">The customer&#8217;s optimism and the product manager&#8217;s enthusiasm are significantly curbed. These <strong>supposedly harmless,</strong> unnecessary levels of abstraction increase cognitive complexity, increased cognitive complexity introduces comprehension latency, comprehension latency delays feature delivery and delayed feature delivery leads to customer dissatisfaction – <strong>a path to the dark side</strong>, that is.  </p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="538" src="https://shiftmag.dev/wp-content/uploads/2024/02/Chainsaw-1024x538.png?x32039" alt="" class="wp-image-2848" srcset="https://shiftmag.dev/wp-content/uploads/2024/02/Chainsaw-1024x538.png 1024w, https://shiftmag.dev/wp-content/uploads/2024/02/Chainsaw-300x158.png 300w, https://shiftmag.dev/wp-content/uploads/2024/02/Chainsaw-768x403.png 768w, https://shiftmag.dev/wp-content/uploads/2024/02/Chainsaw.png 1200w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /></figure>



<h2 class="wp-block-heading" id="some-unclear-naming"><strong>&#8220;Some unclear naming&#8221;</strong>&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">Intellectual labor that requires strong concentration and careful thinking is best done in peace and quiet, which is why modern software development is often performed in loud, open-space offices filled with chit-chat and background music. Since software development also requires collaboration and carefully performed reviews, it is common practice for collaborators to work on unrelated tasks in parallel and interrupt one another instead of doing the tasks together, one task at a time.</p>



<p class="wp-block-paragraph">A developer stares at the screen with his head wrapped in his hands, like a chess player hovering over a chess board. Surrounded by noises of irrelevant conversations and other annoyances, he tries to concentrate on the code of Duck Hunt Extreme and ponders, <strong>&#8220;What does this method do? The name is ambiguous.</strong> I will need to read it line by line to figure it out &#8230; &#8220;.</p>



<p class="wp-block-paragraph">Minutes pass; “Oh, I get it, now I need to choose the right name for the method to capture its intent correctly in order to prevent other people from going down the same rabbit hole”; other changes are made, and hours pass. </p>



<p class="wp-block-paragraph">&#8220;Oh my, another pull request,&#8221; another developer sighs as she leaves her work to review what her colleague has done. She wonders <strong>why they aren&#8217;t doing <a href="https://shiftmag.dev/mob-programming-why-do-it-882/" target="_blank" rel="noreferrer noopener">mob programming</a></strong> and reviewing code continuously. </p>



<p class="wp-block-paragraph">She wonders why her colleague renamed the method and tries to determine whether the change is worth checking before going on to look for objections elsewhere. And so, as time passes, the collapse of the solar system slowly approaches, <strong>the customer is still waiting for his chainsaw</strong>, and the ducks are safe and sound.  </p>



<h2 class="wp-block-heading" id="few-larger-functions-and-classes-nothing-too-big-and-some-code-duplications"><strong>&#8220;Few larger functions and classes (nothing too big) and some code duplications&#8221;&nbsp;</strong>&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">Most novice software developers are haunted by imposter syndrome, doubting their knowledge and skills. In contrast, many senior developers suffer from <strong>the curse of knowledge</strong>: the assumption that others share their background knowledge. This problem is sometimes tackled by leaving the novices to do the tasks on their own while seniors work on hard problems, <strong>creating even more background knowledge</strong> and increasing the skill gap.&nbsp; &nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;Hm, chainsaws&#8230; Where should I add this change? It seems to fit inside this big function inside this big class. I guess I could just copy and slightly modify the existing code in another class. <strong>I better not try to refactor this to a simpler solution</strong> since it was written by people far more experienced than me, and my change does not seem to make things much worse,&#8221; a junior developer wonders as she commits her changes, hoping that the senior dev will review them thoroughly and give her feedback. </p>



<p class="wp-block-paragraph"><strong>If a senior developer fails to do what the junior hopes for</strong>, another avalanche will start to form at the top of the hill. I do not feel I need to spell out how this approach will play out in several years. You do not need to work long in the software industry to realize that the slippery slope is not a fallacy.&nbsp;</p>



<h2 class="wp-block-heading" id="misuses-of-exception-here-and-there"><strong>&#8220;Misuses of exception here and there&#8221;&nbsp;</strong>&nbsp;</h2>



<p class="wp-block-paragraph"><strong>SRE engineers are a software company&#8217;s Night&#8217;s Watch on the Wall.</strong> They are software engineers with a wide technological skillset and broad knowledge of the service they are sworn to protect. They monitor the systems and prevent or shorten production outages and service degradations.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">A sleep-deprived SRE engineer groans as he reaches for his pager in the middle of the night, having received a wake-up call from an automated alerting system.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph"><strong>&#8220;How in the hell did the ducks get control of the nuclear weapons?!&#8221;</strong> he screams in disbelief as he reads the error logs filled with misused exceptions.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;Not funny, not funny at all, and highly unprofessional. Who throws such exceptions!?&#8221; he thinks as he returns to bed to catch a few hours of sleep, as an uncomfortable conversation is due on the incident postmortem meeting tomorrow.&nbsp;&nbsp;&nbsp;</p>



<h2 class="wp-block-heading" id="the-lose-lose-situation"><strong>The lose-lose situation</strong>&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">Scenarios like those outlined above are more likely to appear around a code base that settles with good enough code. <strong>Such code can potentially hurt everyone involved:</strong> the customers, the programmers, and the rest of the company.&nbsp;&nbsp;Striving for excellence is not just a matter of professional pride; it is necessary to prevent the endless cycle of suffering.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">I am slightly sleep-deprived as I write these final lines, and I feel like a manic street preacher yelling at the indifferent passers-by: <strong>&#8220;The sins of the past will come to haunt you! Tech debt must be repaid!&#8221;</strong>&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">But here is something I observed and thought long and hard about, and I stand by it: the exceptionally good code can and should be achieved, and it is a goal worth pursuing.&nbsp;&nbsp;&nbsp;</p>



<h2 class="wp-block-heading" id="exceptionally-good-code-is-not-a-myth"><strong>Exceptionally good code is not a myth</strong>&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">There is a set of technical <strong>practices recommended by highly experienced engineers.</strong> When these practices are combined with frequent and honest communication with the customers, they lead to stellar results.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">Some of these are worth your consideration:&nbsp;&nbsp;&nbsp;</p>



<ul class="wp-block-list">
<li>Pair and mob programming to create a shared understanding of the project and to share technical knowledge and skills&nbsp;&nbsp;</li>



<li>Continuous integration to catch and fix errors early&nbsp;&nbsp;</li>



<li>Continuous delivery to ship the features quickly and safely and getting customer feedback as soon as possible&nbsp;&nbsp;</li>



<li>Test-driven development to make progress in safe steps while continuously making the code base more understandable and adaptable to future change&nbsp;</li>
</ul>



<p class="wp-block-paragraph">This is how we achieve code that is better than &#8220;just good enough &#8220;: By listening to our elders, the <a href="https://shiftmag.dev/the-agile-manifesto-jon-kern-1810/" target="_blank" rel="noreferrer noopener">original Agile manifesto signers</a>, and others who have been doing this job for several decades.</p>
<p>The post <a href="https://shiftmag.dev/good-enough-code-is-just-not-good-enough-2844/">&#8220;Good enough&#8221; code is just not good enough</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Why would anyone do something as absurd as mob programming?!</title>
		<link>https://shiftmag.dev/mob-programming-why-do-it-882/</link>
		
		<dc:creator><![CDATA[Hrvoje Rancic]]></dc:creator>
		<pubDate>Mon, 26 Jun 2023 10:14:47 +0000</pubDate>
				<category><![CDATA[Productivity]]></category>
		<category><![CDATA[Ensemble programming]]></category>
		<category><![CDATA[Mob programming]]></category>
		<category><![CDATA[Test-driven development]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=882</guid>

					<description><![CDATA[<p>A few thousand lines of code everyone is satisfied with and understands used to sound like science fiction – now it was reality.</p>
<p>The post <a href="https://shiftmag.dev/mob-programming-why-do-it-882/">Why would anyone do something as absurd as mob programming?!</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="1200" height="630" src="https://shiftmag.dev/wp-content/uploads/2023/05/Mob-Programming.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2023/05/Mob-Programming.png 1200w, https://shiftmag.dev/wp-content/uploads/2023/05/Mob-Programming-300x158.png 300w, https://shiftmag.dev/wp-content/uploads/2023/05/Mob-Programming-1024x538.png 1024w, https://shiftmag.dev/wp-content/uploads/2023/05/Mob-Programming-768x403.png 768w" sizes="auto, (max-width: 1200px) 100vw, 1200px" /></figure>


<p class="wp-block-paragraph">On a warm Spring afternoon, Alice, Bob, Carlos, Dave, and I were quietly enjoying a coffee break when I uttered a question I assumed I would regret: &#8220;Seriously, people, what are your thoughts on mob programming? Why aren&#8217;t we doing it?&#8221;&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">The looks on their faces spoke volumes. Bob was appalled and thought, &#8220;I thought this guy knew a thing or two about parallelism and resource optimization&#8230;&#8221;. Alice, a highly productive senior engineer, was disgusted, thinking about what a waste it was for five engineers to work simultaneously on only one task. Carlos, the youngest of the bunch, had a curious look. &#8220;I wonder if that cute girl from marketing is seeing someone?&#8221; he pondered.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">Dave, our principal engineer, and an omniscient being, in his infinite wisdom, proclaimed that we would try it out. We were already a high-performing team and would deliver everything we committed on time, even if we experimented with this mob programming thing I kept whining about. The biggest benefit would be that they would get me to shut up about it finally<strong>.&nbsp;&nbsp;</strong>&nbsp;</p>



<h2 class="wp-block-heading" id="a-perfect-chance-arises">A perfect chance arises&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">In the words of&nbsp;<a href="https://twitter.com/WoodyZuill" target="_blank" rel="noreferrer noopener">Woody Zuill,&nbsp;</a><a href="https://en.wikipedia.org/wiki/Mob_programming" target="_blank" rel="noreferrer noopener">mob programming</a>&nbsp;or software teaming is:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">A software development approach where the whole team works together on the same thing, at the same time, in the same space, and at the same computer. This is similar to Pair Programming, where two people sit at the same computer and collaborate on the same code at the same time. However, with Software Teaming, we extend the collaboration to everyone on the team while still using a single computer for writing the code and doing other work.&#8221;&nbsp;</p>
</blockquote>



<p class="wp-block-paragraph">The next day, during sprint planning, we found a task that would be a good chance to try that approach out.</p>



<p class="wp-block-paragraph"><strong>We needed to introduce a new authorization mechanism on a database gateway microservice.</strong> This microservice abstracts the massive database infrastructure away from developers of other microservices. It allows them to invoke stored procedures through a simple remote procedure call (RPC) without the hassle of establishing and maintaining database connections, preparing queries, and processing results, performing health checks and failovers when databases are in maintenance, picking a database instance with the fastest response and so on.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">Since it is a high throughput service with many existing clients, backward compatibility, performance, and reliability were top priorities.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">We estimated that <strong>it would take a single person about a week and a half to implement it.</strong> We would try to do it in a mob and see how it goes. We will all go into a room, look at the code on the big screen, discuss it, and code together.&nbsp;&nbsp;&nbsp;</p>



<h2 class="wp-block-heading" id="five-rules-for-mob-programming">Five rules for mob programming&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">On Wednesday, we gathered in a meeting room with a giant TV on the wall. I got up and wrote the rules on the blackboard:&nbsp;&nbsp;&nbsp;</p>



<ol class="wp-block-list" start="1">
<li>No distractions! Ignore Slack, email, and non-critical alerts. We need everyone&#8217;s complete and undivided attention.&nbsp;&nbsp;</li>
</ol>



<ol class="wp-block-list" start="2">
<li>&nbsp;One person at a time writes the code. We take turns at the keyboard, rotating every 20 minutes.&nbsp;&nbsp;</li>
</ol>



<ol class="wp-block-list" start="3">
<li>&nbsp;The main responsibility of the person at the keyboard is to listen and write the code.,&nbsp;&nbsp;</li>
</ol>



<ol class="wp-block-list" start="4">
<li>&nbsp;People away from the keyboard discuss and navigate the person at the keyboard.&nbsp;&nbsp;</li>
</ol>



<ol class="wp-block-list" start="5">
<li>&nbsp;We take regular breaks every two hours.&nbsp;&nbsp;&nbsp;</li>
</ol>



<p class="wp-block-paragraph">We had a brief discussion about how we were to proceed. Alice gathered the requirements in advance and did a little design up front, and Dave, the principal, gave us some pointers and advice before the higher-ups summoned him to join the other Avengers and help save humanity from imminent disaster. &nbsp;</p>



<h2 class="wp-block-heading" id="first-comes-awkwardness-then-comes-progress">First comes awkwardness, then comes progress&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">Bob was the first at the keyboard, and he obviously had stage fright as he constantly missed keystrokes and misspelled words. It was amusing to watch him struggle until it was my turn at the keyboard; I was even worse. Alice and Carlos were also nervous. It is curious how easy it is to reason about the code when looking at it on a big screen as someone else is writing it.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">I tried to think about implementation and design while I was at the keyboard, but it quickly became apparent that <strong>my role was to listen and write</strong>. To obey the rules written on the blackboard.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">The first rotation was stressful for everyone involved, but we pulled through. After a couple of rounds, the mood changed, everyone felt comfortable with the keyboard in hand, and then the magic started to happen.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;I&#8217;m supposed to change this part of the code? I have no idea what it does or how it works?&#8221; Carlos complained. Bob, who was familiar with said code, gave a short lecture and answered follow-up questions. Now Carlos&nbsp;is confidently modifying it.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;How will we ever implement this? It is too much work&#8230;&#8221; two people with little experience with this project asked; we pointed them to the modules that can be reused. </p>



<p class="wp-block-paragraph">&#8220;Wow, how did you do that?!&#8221; &#8211; Alice is impressed with a clever IDE trick; minutes later, at the keyboard, she yawns as she performs the same trick for the second time.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;I have an idea how we can get some data we need without relying on the client to provide it. Let me show you before we go on a break.&#8221; &#8211; Bob, who has been around and seen the world shows us how, and we are all pleased with the proposal. We have routinely done it sub-optimally for years.&nbsp;&nbsp;</p>



<h2 class="wp-block-heading" id="first-impressions">First impressions&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">During our coffee break, we reflect on the experience. Two benefits we&#8217;ve seen so far: <strong>All of us are learning something, and it is a lot of fun.</strong> We are the kind of team that likes to hang out together, and we just found a new way to do it.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">We are a little worried that we are moving too slowly. There are other tasks in the sprint waiting, and Mallory, the product manager, gives us the stink eye as he passes by.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">On Thursday morning, we are back in the office, in a meeting room, ready to continue. We are steadily progressing, but we figured out that 20 minutes at the keyboard is too short, so <strong>we prolonged it to 30. </strong>We are constantly discussing naming conventions and design; <strong>we are continuously refactoring.&nbsp;</strong>&nbsp;</p>



<p class="wp-block-paragraph">It becomes ever more obvious that<strong> the teammates are the customers for the code </strong>and that coding together is like having those&nbsp;<a href="https://martinfowler.com/bliki/OnsiteCustomer.html" target="_blank" rel="noreferrer noopener">customers on-site</a>. You should write the code as if you are trying to get people to pay you for the luxury of reading and maintaining it.&nbsp;&nbsp;&nbsp;</p>



<h2 class="wp-block-heading" id="the-highs-the-lows-the-rollercoaster-ride">The highs, the lows, the rollercoaster ride&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">As we continue to code, someone suggests we approach the part of the code following the&nbsp;<a href="https://www.ibm.com/garage/method/practices/code/practice_test_driven_development/" target="_blank" rel="noreferrer noopener">Test-Driven Development approach</a>.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">We all write tests, some of us after implementation, some before. So, naturally, Alice, who writes them after the implementation, rolls her eyes as she grabs the keyboard and proceeds to write the first failing test. <strong>We hit our first serious obstacles that day.&nbsp;&nbsp;</strong>&nbsp;</p>



<p class="wp-block-paragraph">None of us liked the metrics code around the cache we added, but we had no idea how to write it cleaner, and Dave was still&nbsp;out there fighting Thanos. Another issue was with a part of the code which performs a remote procedure call to our authorization server. It behaved strangely; we did not understand our RPC framework as well as we had thought.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">After a frustrating half an hour, we decided to stop for the day &#8211;<strong> bumping heads against the wall just leads to headaches</strong>. Before our next session, one person will investigate how to do the metrics part better, and another will learn how our RPC framework handles exceptions in asynchronous calls.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">On Friday, we were all working from home; we joined the call and shared the screen. <strong>The obstacles of yesterday were quickly resolved</strong>, and in a couple of hours, we had a working solution &#8211; something that works both locally and in our integration environment. We started polishing it up.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">Bob proposed a simple refactoring, Carlos saw no point in it, but we still did it. When it was done, it was obvious why Bob was keen on doing it. <strong>I wondered how long it would take to explain the same idea through the comments on the pull request.</strong> We started cleaning up the code, following the adage, &#8220;First make it work, then make it right.&#8221;&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">And as often happens with this kind of work, every modest improvement makes way for another unforeseen. As the code was getting cleaner, it was easier to spot mistakes and opportunities for improvement.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">When the new code is a mess and schedules are tight, there is always the temptation to check if the code works say, &#8220;It looks good to me,&#8221; and approve the pull request. In this industry, that is known to be a typical prologue of many <strong>really scary horror stories</strong>. Programming in a mob helped us avoid that pitfall.&nbsp;</p>



<h2 class="wp-block-heading" id="the-verdict">The verdict&nbsp;&nbsp;</h2>



<p class="wp-block-paragraph">On Monday, we decided to call it quits; Bob would do the last 5 percent of the work alone &#8211; add some logging and documentation, write a couple more tests, and deploy it to production.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">We all agreed that <strong>our code was far superior to anything we would have done individually</strong>. A few thousand lines of code everyone is satisfied with and understands used to sound like science fiction – now it was reality.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">When we add up the person-hours it took to deliver the feature &#8211; we were slightly faster as a mob than when using feature branches and PRs.&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">Our engineering manager put it well:&nbsp;</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">In programming, you do 90 percent of the stuff smoothly, then you get stuck on some stupid thing and lose a lot of time figuring it out. When you&#8217;re in a mob, chances are that someone has already bumped into this stupid thing before or can quickly spot a bug you accidentally introduced.&nbsp;&nbsp;&nbsp;</p>
</blockquote>



<p class="wp-block-paragraph">We now have <strong>a complex feature that is thoroughly understood, can be easily improved, and quickly be troubleshot</strong> by everyone on the team. There was no waiting on PR approval; we all learned new concepts and keyboard shortcuts, tried different coding styles, and saw how the interplay of different thought processes plays out.&nbsp;&nbsp;We practiced talking accurately and listening carefully, we delivered the feature, and we had fun.&nbsp;&nbsp;&nbsp;</p>



<h2 class="wp-block-heading" id="the-unexpected-benefits">The unexpected benefits&nbsp;</h2>



<p class="wp-block-paragraph">I believe the main reason our sessions were successful is our team&#8217;s values of&nbsp;<a href="http://www.extremeprogramming.org/values.html" target="_blank" rel="noreferrer noopener">communication and respect.</a>&nbsp;Psychological factors were also at play: we are not afraid to make mistakes and look stupid in front of one another, we cherish the opportunities to share knowledge, and there are no &#8220;my way or the highway &#8220;egomaniacs on the team.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">Company support also plays a part; there is no micromanagement, and if we are steadily delivering value, no one cares how we do it; it is up to us to&nbsp;<a href="https://martinfowler.com/bliki/AgileImposition.html" target="_blank" rel="noreferrer noopener">self-organize</a>.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">The team mindset about mob programming changed dramatically since a little experience can quelch many common-sense prejudices. Common sense is the great deceiver. We have been doing an excellent job so far, but we did not want that to be an obstacle to improvement. We agreed to continue the old way for simple tasks that can be quickly implemented and reviewed; for the more complex stuff, we will consider mob programming.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;You&#8217;re not coughing anymore, and you look kind of cheerful. Is everything okay?&#8221; &#8211; Dinko from human resources asked me.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">&#8220;I finally got the team to try mob programming.&#8221; I smiled as I explained what it was.&nbsp;&nbsp;&nbsp;</p>



<p class="wp-block-paragraph">Dinko was amused, &#8220;It seems to have some health benefits as well; you should write an article about it. It might even end up on&nbsp;<a href="https://shiftmag.dev/" target="_blank" rel="noreferrer noopener">ShiftMag</a>,&#8221; &#8211; he said.&nbsp;&nbsp;&nbsp;</p>
<p>The post <a href="https://shiftmag.dev/mob-programming-why-do-it-882/">Why would anyone do something as absurd as mob programming?!</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>

<!--
Performance optimized by W3 Total Cache. Learn more: https://www.boldgrid.com/w3-total-cache/?utm_source=w3tc&utm_medium=footer_comment&utm_campaign=free_plugin

Page Caching using Disk: Enhanced 

Served from: shiftmag.dev @ 2026-11-16 11:54:17 by W3 Total Cache
-->