<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>ShiftMag</title>
	<atom:link href="https://shiftmag.dev/feed/" rel="self" type="application/rss+xml" />
	<link>https://shiftmag.dev/</link>
	<description>Insightful engineering content &#38; community</description>
	<lastBuildDate>Fri, 13 Nov 2026 14:16:38 +0000</lastBuildDate>
	<language>en-GB</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://shiftmag.dev/wp-content/uploads/2026/09/cropped-Shift-32x32.webp</url>
	<title>ShiftMag</title>
	<link>https://shiftmag.dev/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>How Fin Doubled Its Engineering Capabilities With AI Agents</title>
		<link>https://shiftmag.dev/how-fin-doubled-its-engineering-capabilities-with-ai-agents-13529/</link>
		
		<dc:creator><![CDATA[Ivan Simic]]></dc:creator>
		<pubDate>Fri, 13 Nov 2026 14:16:38 +0000</pubDate>
				<category><![CDATA[Artificial Intelligence]]></category>
		<category><![CDATA[Video]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[artificial intelligence]]></category>
		<category><![CDATA[brian scanlan]]></category>
		<category><![CDATA[Engineer Explains]]></category>
		<category><![CDATA[fin]]></category>
		<category><![CDATA[Infobip Shift 2026]]></category>
		<category><![CDATA[Shift Conference]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=13529</guid>

					<description><![CDATA[<p>Fin's Senior Principal Systems Engineer explains how agents took over low-risk work and how Fin counts the result: merged pull requests.</p>
<p>The post <a href="https://shiftmag.dev/how-fin-doubled-its-engineering-capabilities-with-ai-agents-13529/">How Fin Doubled Its Engineering Capabilities With AI Agents</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph"><strong>Fin (formerly Intercom)</strong> is known for its customer service, chatbots and live chat products and has rebranded to be AI-first and acquired by Salesforce in 2026. <strong>Brian Scanlan</strong>, Senior Principal Systems Engineer at Fin, has been a part of the company for more than 12 years, and he knows all of the important workflows inside and out.</p>



<p class="wp-block-paragraph">In the AI era, Fin managed to double its engineering capabilities through AI agents, which raised the question of how the company should measure success and growth. There needs to be a <strong>way to &#8220;count&#8221; the efficiency of AI agents</strong>, and Brian says the secret (at least for Fin) is in merged pull requests:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">So we look at pull requests on our main applications and just count them. It&#8217;s very simple, it&#8217;s a little crude, it&#8217;s extremely gameable and it&#8217;s not by no means perfect, but I think if you&#8217;re in the right environment, you&#8217;ve got a high trust and people aren&#8217;t trying to game things, but you&#8217;re given guidance as well that, there&#8217;s no compromise around quality, you&#8217;re shipping real features and meaningful work, then it&#8217;s not a bad measurement.</p>
</blockquote>



<h2 class="wp-block-heading" id="in-some-areas-agents-are-doing-all-the-work">In some areas, agents are doing all the work</h2>



<p class="wp-block-paragraph">Just betting on people not abusing pull requests is one thing, but working with many AI agents makes for a rather unique environment. Brian says that <strong>in some areas, much of the work is already delegated to agents</strong>. </p>



<p class="wp-block-paragraph">Brian explained that some fields, such as testing and fixing unreliable tests, are good areas for AI agents. This is because these tasks are predictable, and there is a relatively small amount of patterns that makes them unreliable. There&#8217;s also not a lot of risk:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">And also, if things go wrong, you know, the worst case that&#8217;s gonna happen is you&#8217;re gonna block tests for a while or block deployments for a while, and that&#8217;s not too bad. So this is, like, a relatively low risk area and the patterns are well known. And so at the moment that&#8217;s fully delegated. Our agents just work in the background and fix all the tests.</p>
</blockquote>



<p class="wp-block-paragraph">For major features or incidents, Brian explained that agents are still working alongside humans in a supporting role. He pointed out that agents are being used throughout the process, but the main thinking and &#8220;figuring out what&#8217;s going on&#8221; still includes a lot of human brain power rather than RAM. Still, agent usage depends a lot on the project itself and its maturity, so some projects will be more agent-focused and others will have more human involvement. In general, the company&#8217;s direction is to <strong>go towards more delegation and more independent agents</strong>.</p>



<h2 class="wp-block-heading" id="scaling-ai-agents-made-fins-ci-costs-grow-10-times-faster-than-the-business">Scaling AI agents made Fin&#8217;s CI costs grow 10 times faster than the business</h2>



<p class="wp-block-paragraph">Things are bound to break at this level and scale, so we asked Brian what was the first thing that broke during their efforts to scale AI usage from individuals to a company-wide effort. It turns out that their <strong>continuous integration (CI) system</strong> melted down in a few ways:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">It started getting more unreliable because we were shipping more, we had more flaky tests, but also the costs went through the roof. We were auto scaling our CI environments and the cost was growing, you know, 10 times faster than the business was growing, and that&#8217;s not very comfortable. So we did the work to optimize our CI environment, to make sure our tests were actually valuable tests worth keeping and stuff like that.</p>
</blockquote>



<p class="wp-block-paragraph">The second thing was that when agents became popular and viral in the company, many people started pushing all sorts of things to GitHub, including to public GitHub repos. Brian said that there were &#8220;<strong>definite security problems&#8221; during that time</strong>, but many of these growing pains are now fixed and he is &#8220;worrying less about it now&#8221;.</p>



<h2 class="wp-block-heading" id="react-learn-improve-and-write-down">React, learn, improve and write down</h2>



<p class="wp-block-paragraph">When asked <strong>how to stop agents from just compounding bad decisions</strong> across the code base and projects, Brian admitted that those things are very hard to do, and that these issues remain largely unsolved in their environment:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Early on, we put a lot of work into improving our linting. We have a pretty good testing culture, I think, as well. We rely absolutely on our unit tests. We push to production hundreds of times a day. And so I think doubling down on the fundamentals, make sure that our linting is in good shape, our tests are in good shape, that people are still writing these things, that&#8217;s important. And then, you know, we still have humans in the loop on most of our code reviews.</p>
</blockquote>



<p class="wp-block-paragraph">Fin still has many safeguards in place and Brian says that everyone pays attention and makes sure none of the unreadable code and other problems get accepted. Still, there&#8217;s still &#8220;stuff that gets through&#8221;. In these situations, Brian says that the best advice is to r<strong>eact, learn, improve and then then put that guidance into the context</strong> so that the agent can&#8217;t make the same mistake again.</p>



<h2 class="wp-block-heading" id="organizations-might-be-too-conservative-with-ai">Organizations might be too conservative with AI</h2>



<p class="wp-block-paragraph">Brian was also honest about how people adapt to using AI agents, saying that at the start, it wasn&#8217;t obvious who was going to get benefit out of using AI agents. For him, who&#8217;s been at the company for more than a decade, <strong>AI unlocked many benefits</strong> since he already had a lot of context and knew a lot about the systems in place. On the other side, more junior engineers also benefited from using AI since their onboarding was easier and faster, so they got more done in a shorter time.</p>



<p class="wp-block-paragraph">When it comes to using AI, Brian said that many organizations are too conservative and cautious about being nice to their engineers:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">I think change is hard and organizational change means real kind of thought and care when rolling it out. But that doesn&#8217;t mean you shouldn&#8217;t be ambitious. It doesn&#8217;t mean that you need to not upset anybody. I think to get real growth and real change and real adoption of AI, you have to make it very clear to have expectations that this is the future. </p>



<p class="wp-block-paragraph"></p>



<p class="wp-block-paragraph">And honestly, I think the most important thing for people&#8217;s careers at the moment, where we are right now, is to be adopting these technologies, learning them and figuring out what the future of your own career kind of looks like.</p>
</blockquote>



<p class="wp-block-paragraph">Brian feels like the &#8220;careful approach&#8221;, which might&#8217;ve worked for others in the past, might not be the good strategy for solving the AI adoption issues. </p>



<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<iframe title="AI Agents Scaled: What Breaks First?" width="500" height="281" src="https://www.youtube.com/embed/zzQxSj62abE?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
</div></figure>


<figure class="wp-block-post-featured-image"><img fetchpriority="high" decoding="async" width="1200" height="630" src="https://shiftmag.dev/wp-content/uploads/2026/11/brian-scanlan-shiftmag.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/11/brian-scanlan-shiftmag.png 1200w, https://shiftmag.dev/wp-content/uploads/2026/11/brian-scanlan-shiftmag-300x158.png 300w, https://shiftmag.dev/wp-content/uploads/2026/11/brian-scanlan-shiftmag-1024x538.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/11/brian-scanlan-shiftmag-768x403.png 768w" sizes="(max-width: 1200px) 100vw, 1200px" /></figure><p>The post <a href="https://shiftmag.dev/how-fin-doubled-its-engineering-capabilities-with-ai-agents-13529/">How Fin Doubled Its Engineering Capabilities With AI Agents</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AI Agents Shouldn’t Set Their Own Security Boundaries</title>
		<link>https://shiftmag.dev/ai-agents-shouldnt-set-their-own-security-boundaries-12827/</link>
		
		<dc:creator><![CDATA[Ivan Pelivanovic]]></dc:creator>
		<pubDate>Tue, 10 Nov 2026 14:30:11 +0000</pubDate>
				<category><![CDATA[Artificial Intelligence]]></category>
		<category><![CDATA[Security]]></category>
		<category><![CDATA[AI agents]]></category>
		<category><![CDATA[AI security]]></category>
		<category><![CDATA[NVIDIA]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=12827</guid>

					<description><![CDATA[<p>"They just see friction versus no friction." That is how Ali Golshan, Senior Director of AI Software at NVIDIA, describes AI agents facing security rules.</p>
<p>The post <a href="https://shiftmag.dev/ai-agents-shouldnt-set-their-own-security-boundaries-12827/">AI Agents Shouldn’t Set Their Own Security Boundaries</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[<figure class="wp-block-post-featured-image"><img decoding="async" width="1200" height="720" src="https://shiftmag.dev/wp-content/uploads/2026/10/Ali.jpg?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/Ali.jpg 1200w, https://shiftmag.dev/wp-content/uploads/2026/10/Ali-300x180.jpg 300w, https://shiftmag.dev/wp-content/uploads/2026/10/Ali-1024x614.jpg 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/Ali-768x461.jpg 768w" sizes="(max-width: 1200px) 100vw, 1200px" /></figure>


<p class="wp-block-paragraph">AI agents use tools, reach into internal systems and get work done with barely any human involved. New security problems turn up almost weekly. What if an agent&#8217;s own instructions block it, and it finds a way around them just to finish the job?</p>



<p class="wp-block-paragraph">At <a href="https://www.nvidia.com/en-eu/gtc/" target="_blank" rel="noreferrer noopener">NVIDIA&#8217;s GTC Berlin</a> I was at a session where <strong>Ali Golshan</strong>, Senior Director of AI Software at NVIDIA presented an approach that treats agent security as a problem spanning the entire software stack, from the application and runtime to the underlying infrastructure.</p>



<p class="wp-block-paragraph">His central argument was that<strong> security cannot depend solely on whether a model follows instructions</strong>.</p>



<h2 class="wp-block-heading" id="enforce-rules-where-agents-cant-reach">Enforce rules where agents can&#8217;t reach</h2>



<p class="wp-block-paragraph">One big issue Golshan pointed out is that <strong>agents don’t understand restrictions the way developers meant them to</strong>. If an agent can’t get to a tool or resource it might look for another way to get the job done. In systems this can lead the agent to drift from its original goal or even find a hidden path to sensitive data. As he said:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">They just see friction versus no friction.</p>
</blockquote>



<p class="wp-block-paragraph">The point is that an agent <strong>may treat a security restriction as an obstacle to overcome</strong> rather than a boundary it must respect. If controls exist only in the application layer, where the agent can reason about them, they may not provide a sufficiently reliable enforcement mechanism.</p>



<p class="wp-block-paragraph">Ali argued that security controls should therefore be moved into layers that <strong>operate independently of the agent</strong>. The agent can propose an action, but a separate mechanism should determine whether that action is permitted and enforce the decision. This means separating an agent’s ability to plan from its authority to act.</p>



<h2 class="wp-block-heading" id="every-agent-can-follow-the-rules-while-the-system-as-a-whole-breaks-them">Every agent can follow the rules while the system as a whole breaks them</h2>



<p class="wp-block-paragraph">The problem becomes more complicated when multiple agents collaborate. Golshan described a scenario in which one agent can access GitHub but cannot access the public internet, while another can publish content online but has no access to GitHub.</p>



<p class="wp-block-paragraph">Each agent operates within its individual permissions. Yet if the first agent delegates work to the second, or passes information between them, their combined capabilities may exceed the limits an organisation intended to enforce:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">However, you have three agents that their collective capabilities now exceed your global policy. How do you verify these unintended exfiltration paths?</p>
</blockquote>



<p class="wp-block-paragraph">This illustrates why <strong>checking permissions for individual agents may not be enough</strong>. In multi-agent systems, security controls must also account for delegation and the combined effects of actions across an entire workflow.</p>



<p class="wp-block-paragraph"><strong>Longrunning agents </strong>introduce another challenge. As they work, call tools or delegate tasks, their access requirements can change. Persistent credentials and permissions may accumulate unnecessary risk if they remain active after a task is complete.</p>



<p class="wp-block-paragraph">A more restrictive approach is to grant agents only the permissions they need, for as long as they need them, and revoke access when the relevant work is finished.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="614" src="https://shiftmag.dev/wp-content/uploads/2026/10/AliG-1-1024x614.jpg?x32039" alt="" class="wp-image-12989" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/AliG-1-1024x614.jpg 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/AliG-1-300x180.jpg 300w, https://shiftmag.dev/wp-content/uploads/2026/10/AliG-1-767x460.jpg 767w, https://shiftmag.dev/wp-content/uploads/2026/10/AliG-1.jpg 1200w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /><figcaption class="wp-element-caption">Credit: Ivan Pelivanović</figcaption></figure>



<p class="wp-block-paragraph"><strong>The risks also grow as agents operate for longer periods</strong>. Ali pointed to long-running agents and multi-agent fleets as two developments that make security harder to manage. Over time, an agent may accumulate access, credentials and session state, while collaboration introduces risks that cannot be assessed by looking at each agent in isolation.</p>



<h2 class="wp-block-heading" id="how-to-solve-this">How to solve this?</h2>



<p class="wp-block-paragraph">He advocated a layered approach that combines isolation, narrowly scoped permissions, monitoring and the ability to revoke access quickly.</p>



<p class="wp-block-paragraph">The principle is to <strong>reduce the damage an agent can cause if it makes a mistake</strong>, is manipulated by malicious input or exploits a vulnerability in its environment. Sensitive credentials should remain outside the agent’s direct reach wherever possible, while access to external systems should pass through independently enforced controls.</p>



<p class="wp-block-paragraph">NVIDIA presented its own implementation of this approach through OpenShell, an agent runtime, alongside infrastructure-level controls associated with its BlueField platform and Sentry security framework.</p>



<p class="wp-block-paragraph">The proposed architecture aims to move some enforcement beyond the software environment in which an agent operates. The broader idea is that an agent should not be able to change or bypass the security controls governing its own execution.</p>



<p class="wp-block-paragraph">Ali described this as a <strong>progression from application-level controls to runtime isolation</strong> and, ultimately, hardware-level enforcement. The reasoning is that a runtime is still software and may itself contain vulnerabilities. NVIDIA’s proposed architecture aims to place some security functions in a separate infrastructure layer that the agent cannot modify or bypass directly.</p>



<h2 class="wp-block-heading" id="agent-security-cant-be-solved-one-vendor-at-a-time">Agent security can&#8217;t be solved one vendor at a time</h2>



<p class="wp-block-paragraph">Enterprise environments <strong>combine different models</strong>, cloud platforms, tools and internal systems. If each provider implements its own incompatible approach to identity, permissions and delegation, organisations may struggle to enforce consistent controls across their infrastructure. Shared approaches to agent identity, authorisation, delegation and auditability could let them apply one security policy across different systems. NVIDIA pointed to the Open Secure AI Alliance as work toward open standards for agentic systems.</p>



<p class="wp-block-paragraph">Before deploying agents, ask what each one can access, what it can change, and whether those limits still hold when several agents work together.</p>
<p>The post <a href="https://shiftmag.dev/ai-agents-shouldnt-set-their-own-security-boundaries-12827/">AI Agents Shouldn’t Set Their Own Security Boundaries</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Developers, Your Next Customer Is an AI Agent</title>
		<link>https://shiftmag.dev/developers-your-next-customer-is-an-ai-agent-13455/</link>
		
		<dc:creator><![CDATA[Ivan Brezak Brkan]]></dc:creator>
		<pubDate>Mon, 09 Nov 2026 15:57:05 +0000</pubDate>
				<category><![CDATA[Artificial Intelligence]]></category>
		<category><![CDATA[Business of Software]]></category>
		<category><![CDATA[AI agents]]></category>
		<category><![CDATA[Apify]]></category>
		<category><![CDATA[MCP]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=13455</guid>

					<description><![CDATA[<p>Apify's community of creators earns about $2m a month by building tools for AI agents. Here is what the CEO says it takes to make tools agents will actually use.</p>
<p>The post <a href="https://shiftmag.dev/developers-your-next-customer-is-an-ai-agent-13455/">Developers, Your Next Customer Is an AI Agent</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="1920" height="1080" src="https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-7.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-7.png 1920w, https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-7-300x169.png 300w, https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-7-1024x576.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-7-768x432.png 768w" sizes="auto, (max-width: 1920px) 100vw, 1920px" /></figure>


<p class="wp-block-paragraph">Tomorrow, developers and founders will gather at The Pearl in San Francisco for <a href="https://run.apify.com/" target="_blank" rel="noreferrer noopener">Run by Apify</a>, a one-day <strong>conference about building with AI agents</strong>. Among the speakers giving an opening lecture is Apify CEO <a href="https://www.linkedin.com/in/jancurn/" target="_blank" rel="noreferrer noopener">Jan Čurn</a>, who has been thinking about how to make tools those agents will actually use.<br><br>Last week I had the chance to sit down with him and talk about AI agents. The conversation showed me how solving one problem in the AI space often creates another, more complex but just as interesting. Software, for example, is easier to build than ever, yet finding the right tool is harder because there are so many to choose from.</p>



<p class="wp-block-paragraph">Apify knows this tension well: the company started out in 2016 offering web tools for all sorts of jobs, such as web scrapers and other automations. Since then, it has watched software development change, from tools like Zapier to AI agents. <strong>It&#8217;s those AI agents that are some of Apify&#8217;s customers today</strong>, Jan explained:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">People are no longer selecting the tools themselves, but they have their agents do it on their behalf. The agents are basically like a surogate of the customer.</p>
</blockquote>



<p class="wp-block-paragraph">Here&#8217;s the <strong>uncomfortable part</strong>: if your number one customer is an AI agent, your beautiful UI might not exactly be what&#8217;s needed. Agents read your API and your docs, and they look for tools in specific places. Jan says people have stopped clicking around interfaces too and rather hand the job to an agent.</p>



<h2 class="wp-block-heading" id="if-there-are-50-tools-or-maybe-500-or-5000-which-one-are-you-going-to-pick">&#8220;If there are 50 tools, or maybe 500, or 5,000, which one are you going to pick?&#8221;</h2>



<p class="wp-block-paragraph">Jan and I talked about <strong>how cheap software has become to build</strong>. Good software still costs money, but the cost of creating it has dropped across every part of development. When building something is almost free, we end up with so many tools and products that finding the right one is like walking through fog.</p>



<p class="wp-block-paragraph">With so much choice, it&#8217;s hard to figure out which is good, so we end up doing the same thing, and using the same tools.</p>



<p class="wp-block-paragraph">Jan said that a good example of this is the Apple App Store, where <strong>the number of new apps is growing very fast, but the usage is the same: the supply is massive</strong>, but the demand is pretty much the same as it ever was. For those building software, getting seen was never exactly an easy thing to do, but now it has become exponentially harder:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">If there are 50 tools, or maybe 500, or 5,000, which one are you going to pick? Who can you trust? Trust is becoming really important.</p>
</blockquote>



<p class="wp-block-paragraph"><strong>Apify aims to solve this challenge through their marketplace</strong>, where they host thousands of &#8220;Actors&#8221;, or tools for different tasks their clients and other customers require help with. These tools are created by the community, and Apify pays their creators:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">If those Actors are used by our users and customers, we pay 80% of the revenue to the creators. These creators are making about $2m per month. I think this month will go over $2m, and this is growing maybe four to five times year over year.</p>
</blockquote>



<p class="wp-block-paragraph">Jan added that having this type of spotlight could be a <strong>beneficial thing for developers</strong>, noting that someone could earn several thousand dollars per month by having their tool on the platform.</p>



<h2 class="wp-block-heading" id="you-dont-need-to-be-a-traditional-developer-to-build-for-this-marketplace">You don&#8217;t need to be a traditional developer to build for this marketplace</h2>



<p class="wp-block-paragraph">The marketplace&#8217;s growth is organic in a sense that more users can attract more creators, who will create more tools and Actors for more use cases, and that again attracts more users. Since Actors can be created relatively easily, Jan says that <strong>the barrier of entry is also reasonable</strong>, but there&#8217;s plenty of plans for the future:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Actors are self-contained, well-defined tools that do one job. In a way, these are tools that are pretty good to be completely generated by AI. So over time, we want to bring capabilities that make it easy for creators to create self-healing Actors that will automatically repair themselves. Our goal is for anyone who’s entrepreneurial to be able to do it, not just traditional software developers.</p>
</blockquote>



<p class="wp-block-paragraph">Right now, Apify is also working on <strong>expanding what Actors can do</strong>. In the past, Jan explained, Actors were batch-job oriented, meaning that one Actor would do its task with some inputs and return a result. Now, a lot of customers are looking for use cases similar to APIs, which means quick request and quick response. Apify is thus working on Server Actors, which will bring more features to Apify:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Server Actors will allow us and our creators to bring many new kinds of services to Apify, including APIs to serve data or perform transformations. We also have MCP connectors, so a user can connect external systems like Notion or Slack, and Actors can get access to them. We’re moving away from web data extraction into data as a service, general automation, and agentic tools.</p>
</blockquote>



<p class="wp-block-paragraph">The future looks bright but challenging, and not just for Apify. The same goes for any company building tools for people and AI agents. For developers, getting a tool to work is the easy part. To get an agent to use it, you also need <strong>clear documentation and a good API</strong>, so the agent can understand what the tool does and call it without trouble.</p>



<p class="wp-block-paragraph">Still, building for agents instead of people is something the whole industry is trying to figure out. AI is built on predictions and training, yet it still seems to have a mind of its own. Whether that mind picks your tool out of thousands of others is an open question.</p>
<p>The post <a href="https://shiftmag.dev/developers-your-next-customer-is-an-ai-agent-13455/">Developers, Your Next Customer Is an AI Agent</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Dan Maher from Datadog Says AI Agent Reliability Is an Engineering Problem, Not a Model Choice</title>
		<link>https://shiftmag.dev/dan-maher-from-datadog-says-ai-agent-reliability-is-an-engineering-problem-not-a-model-choice-13430/</link>
		
		<dc:creator><![CDATA[Ivan Simic]]></dc:creator>
		<pubDate>Fri, 06 Nov 2026 12:50:22 +0000</pubDate>
				<category><![CDATA[Artificial Intelligence]]></category>
		<category><![CDATA[Video]]></category>
		<category><![CDATA[agentic AI]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[AI agents]]></category>
		<category><![CDATA[Infobip Shift 2026]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=13430</guid>

					<description><![CDATA[<p>The Senior Technical Advocate and Researcher at Datadog explained what actually determines whether an AI agent is trustworthy.</p>
<p>The post <a href="https://shiftmag.dev/dan-maher-from-datadog-says-ai-agent-reliability-is-an-engineering-problem-not-a-model-choice-13430/">Dan Maher from Datadog Says AI Agent Reliability Is an Engineering Problem, Not a Model Choice</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The use case for milions of AI users today is simple: give the agent a task, and enjoy the result. We might argue that&#8217;s not the ideal way to do things, but it&#8217;s just what it&#8217;s like. So, <strong>how do we know whether an AI agent is trustworthy</strong>?</p>



<p class="wp-block-paragraph">For <strong>Daniel (Dan) Maher</strong>, Senior Technical Advocate and Researcher (STAR) at Datadog, there&#8217;s no &#8220;silver bullet.&#8221; To evaluate whether a model is trustworthy, you need to understand the model and separate it from its harness, and then you just have to &#8220;deal with a certain degree of uncertainty, run your tests, try to get your answers, and loop back over them until you start to see what you believe to be correct.&#8221;</p>



<p class="wp-block-paragraph">Even so, relying on AI can be a double-edged sword. Dan said using it for research today is mostly reliable. AI tools keep getting better, and we&#8217;re seeing many interesting breakthroughs in science and other technical fields. But a human should always stay in the loop. Dan explains:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Currently, state of the art today is yes. Humans have to stay in the loop, even with concepts like LLM as judge, for example. The humans have to stay in the loop because that&#8217;s where we&#8217;re at technologically. Going forward, that may less and less be the case from a practical perspective. However, there may be legal or jurisdictional reasons for which a human has to remain responsible for publishing or taking responsibility for those results.</p>
</blockquote>



<h2 class="wp-block-heading" id="what-many-people-think-of-as-a-models-personality-is-in-fact-a-harnesss-personality">&#8220;What many people think of as a model&#8217;s personality is in fact a harness&#8217;s personality.&#8221;</h2>



<p class="wp-block-paragraph">Simply choosing a &#8220;better&#8221; or &#8220;smarter&#8221; model won&#8217;t make the overall experience of using it more reliable and solid. Dan explained that, while we as users think that models have different personalities, oftentimes that&#8217;s <strong>the personality of a harness</strong>:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">What many people think of as a model&#8217;s personality is in fact a harness&#8217;s personality. And this is an area where a lot of people, engineers alike, I did for a long time not understand this distinction. And so learning to understand the interaction between the model and the harness is one thing.</p>
</blockquote>



<p class="wp-block-paragraph">He added that some models themselves do like to overthink, while others will churn on things forever. In order to deal with the problem of reliability, the best thing we as users can do is to find out what makes each model tick and how it behaves, so that we know how to handle them better.</p>



<p class="wp-block-paragraph">Think of the AI model as the engine of a car, or brain of the operation, but it sits in a jar and can only instruct and think. The harness is everything else; the wheels and tires or arms and legs it needs to actually do things. </p>



<h2 class="wp-block-heading" id="we-have-to-figure-out-if-the-ai-is-correct">We have to figure out if the AI is correct</h2>



<p class="wp-block-paragraph">We often feel like AI is unique in the way that it <strong>confidently answers questions without being 100% sure of what it&#8217;s talking about</strong> but Dan pointed out that we, as humans, also do the same:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">I am capable of giving you an answer to any question you ask. Now, whether that&#8217;s the correct answer is a completely different thing, right? So the idea here, of course, being that we have to figure out, is the AI correct?</p>
</blockquote>



<p class="wp-block-paragraph">What makes a difference between an &#8220;answering machine&#8221; and a model that is actually reliable and helpful is whether the model provides a trail of information. This means that it&#8217;ll tell you where it looked, what it found and what decisions and determinations it&#8217;s doing with that information. Having this information, the model can show it to you as the user and you can decide what&#8217;s valuable and what isn&#8217;t. For Dan, that&#8217;s a much more valuable result than a confident incorrect answer.</p>



<h2 class="wp-block-heading" id="ai-agents-will-fail-similarly-to-how-software-fails">AI agents will fail similarly to how software fails</h2>



<p class="wp-block-paragraph">Echoing that statement, we wanted to know where do AI agents fail first, then? </p>



<p class="wp-block-paragraph">The answer is, of course, both simpler and more complex than one might think. Dan explained that <strong>AI agents will usually fail at the first thing they do</strong>, so users will run into problems with each step of the way. Though, that&#8217;s not something new to AI:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">This is software engineering, right? You write a line of code, you test it, it fails. You fix it, you write the next line of code, you test it, it fails. You fix it. It&#8217;s the same thing with agentic behavior. You have to run it, let it fail, understand why it failed, and then move on to the next thing, be it at the orchestration phase, be it at the inference phase, be it at any of the phases. The phase where you&#8217;re at for the first time is the first place it will fail.</p>
</blockquote>



<p class="wp-block-paragraph">Testing AI agnets and their workflows might be counterintuitive to the end users who expect that the agents will be faultless from the start, but if we view them as software that needs updates (which they, in the end, are), we can go further. Of course, updates here mean that we need to correct them and steer them in the right direction, rather than just letting them do their own thing while blindly believing them.</p>



<h2 class="wp-block-heading" id="benchmarks-are-useful-but-arent-all-knowing">Benchmarks are useful, but aren&#8217;t all-knowing</h2>



<p class="wp-block-paragraph">Users tend to rely on <strong>benchmarks and scores for AI model reliability</strong> or level of capability, and measure it with numbers. This means that model X might be a certain percent better than model Y, which, in theory, makes it superior. Dan agrees that models should have some sort of benchmark of reliability and performance so that users can figure out what to use.</p>



<p class="wp-block-paragraph">On the other hand, the benchmarks today can&#8217;t really tell us whether the model is fit for purpose for a particular goal:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">In that case, that&#8217;s where the practitioner, the organization, the person manipulating the tool needs to come in with their experience, their expertise, their domain knowledge, and be analyzing those outputs, and be analyzing a lot of outputs, and then running those outputs through their own experimentation phases. And this is where something like LLM observability tooling and LLM experimentation tooling becomes increasingly necessary for any organization doing serious work in this space.</p>
</blockquote>



<p class="wp-block-paragraph">The overarching conclusion of our talk with Dan is that AI agent reliability is a very empirical engineering problem and it has much less to do with the model selection than one might thin. The teams treating it as the latter are likely to be very surprised by the former.</p>



<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<iframe loading="lazy" title="Stop Building AI Answer Machines!" width="500" height="281" src="https://www.youtube.com/embed/5qkUgb31mpA?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
</div></figure>


<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="2100" height="1400" src="https://shiftmag.dev/wp-content/uploads/2026/11/DSC00047-scaled.jpg?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/11/DSC00047-scaled.jpg 2100w, https://shiftmag.dev/wp-content/uploads/2026/11/DSC00047-300x200.jpg 300w, https://shiftmag.dev/wp-content/uploads/2026/11/DSC00047-1024x683.jpg 1024w, https://shiftmag.dev/wp-content/uploads/2026/11/DSC00047-768x512.jpg 768w" sizes="auto, (max-width: 2100px) 100vw, 2100px" /></figure><p>The post <a href="https://shiftmag.dev/dan-maher-from-datadog-says-ai-agent-reliability-is-an-engineering-problem-not-a-model-choice-13430/">Dan Maher from Datadog Says AI Agent Reliability Is an Engineering Problem, Not a Model Choice</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>We Fed Years of Support Queries to an AI, Hoping It Becomes Our Best Support Engineer</title>
		<link>https://shiftmag.dev/we-fed-years-of-support-queries-to-an-ai-hoping-it-becomes-our-best-support-engineer-12896/</link>
		
		<dc:creator><![CDATA[Adnan Elezović]]></dc:creator>
		<pubDate>Thu, 05 Nov 2026 15:30:30 +0000</pubDate>
				<category><![CDATA[Artificial Intelligence]]></category>
		<category><![CDATA[AI Support Agent]]></category>
		<category><![CDATA[MCP]]></category>
		<category><![CDATA[RAG]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=12896</guid>

					<description><![CDATA[<p>Our senior support engineers take years to learn which of thousands of records actually matter when a client call drops. We tried to shortcut that with an AI agent, and the key was how we built its memory.</p>
<p>The post <a href="https://shiftmag.dev/we-fed-years-of-support-queries-to-an-ai-hoping-it-becomes-our-best-support-engineer-12896/">We Fed Years of Support Queries to an AI, Hoping It Becomes Our Best Support Engineer</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">While supporting our clients who are using our <a href="https://www.infobip.com/voice" target="_blank" rel="noopener">Voice platform</a>, we often find ourselves <strong>falling into repetitive work</strong>.</p>



<p class="wp-block-paragraph">For example, let&#8217;s observe the example support ticket with the following summary: </p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Our call center agent claims that our customers never hear the first few seconds when the agent greets them, what gives?</p>
</blockquote>



<p class="wp-block-paragraph">Of course, with a description like this, you first have to play an initial round of ping-pong with the client to gather enough information to know where to start, for example asking for any kind of identifiers, such as call IDs, so we can start looking up what went wrong. <strong>When we get the identifier, we start looking into the issue</strong>.</p>



<h2 class="wp-block-heading" id="seniors-know-which-facts-matter-and-that-knowledge-takes-years-to-build">Seniors know which facts matter, and that knowledge takes years to build</h2>



<p class="wp-block-paragraph">On a happy day, we would take a look at <strong>the call records</strong>, and assuming that the issue is caused by some expected behavior, we usually leave enough breadcrumbs here that tell us what happened. This gives us everything we need to provide an answer to our client, and continue with our day.</p>



<p class="wp-block-paragraph">However, <strong>what if the answer isn&#8217;t found here?</strong> That&#8217;s usually when the fun begins and seasoned support engineers make the iconic &#8220;lean forward&#8221;.</p>



<p class="wp-block-paragraph">Depending on the type of issue, you might think you should first <strong>check a few Grafana dashboards and check for anomalities, spikes, weird patterns</strong>, and hopefully they will vaguely match the timestamps in the client complaint and have an a-ha moment. Perhaps the observed packet loss and low bandwidth budget has something to do with the reported missing audio? But considering that this is visible only in one of the five provided examples by the client, you quickly dismiss it and think about what&#8217;s next.</p>



<p class="wp-block-paragraph">Depending on how desperate you are, at this point you might consider looking at <strong>application logs</strong> and try to walk through the process, assuming you know it yourself, and look for the missing piece that will help you understand what happened. At this point, you are already hours into troubleshooting and have talked to several engineers to help you navigate through the forest known as communication platform with many components that are split across many microservices that <em>somehow</em> supposedly work together.</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="700" height="408" src="https://shiftmag.dev/wp-content/uploads/2026/10/lean-forward-meme.jpeg?x32039" alt="" class="wp-image-12899" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/lean-forward-meme.jpeg 700w, https://shiftmag.dev/wp-content/uploads/2026/10/lean-forward-meme-300x175.jpeg 300w" sizes="auto, (max-width: 700px) 100vw, 700px" /></figure>



<p class="wp-block-paragraph">Fast forward across the 10 tabs you have HEPIC, Grafana, backend logs, API docs and various configuration pages. You managed to understand the clients use case, how they consume our APIs to orchestrate their call center and you might as well apply for a job at their startup, but you just can&#8217;t find anything that would cause their pain.</p>



<p class="wp-block-paragraph">At this point<strong> the client is losing patience</strong> and thus the senior customer support engineer gets involved. Takes a look at the ticket, picks up your mouse and checks a few timestamps.</p>



<p class="wp-block-paragraph">They <strong>immediately recognize that the issue</strong> is caused due to the client call orchestrator connecting both parties in the call via a conference only once both call legs are established, causing unpredictable audio clipping on both sides. Barely ten minutes of troubleshooting and the definitive answer was found.</p>



<p class="wp-block-paragraph">If this was your first month on the job, you are probably exhausted at this point and wonder when you will catch up with all these things. It&#8217;s not true to say that these things aren&#8217;t documented, as the API behavior is very well documented in our awesome Voice API docs, but connecting all these dots and knowing the implications simply requires more time&#8230;</p>



<p class="wp-block-paragraph">Okay, so how can we accelerate this?</p>



<h2 class="wp-block-heading" id="just-use-ai">Just use AI?</h2>



<p class="wp-block-paragraph">With AI being all the craze, it was only natural to consider integrating it somehow into this process to accelerate our troubleshooting. And more than anything, <strong>reducing the difficulty of entry for new engineers</strong>, since just knowing where to look is a skill that has to be developed with time and experience.</p>



<p class="wp-block-paragraph">At the time, AI powered chat bots were really popular, especially for providing support, but <strong>citing documentation to support engineers isn&#8217;t really helping</strong>, it&#8217;s actually annoying. No one wants a colleague (AI or not) who just drops links to various docs whenever you ask them to help with something, so this approach was a no-go for us.</p>



<p class="wp-block-paragraph">Take the previous example, our support engineer wouldn&#8217;t even know what exactly to ask the AI, as the issue reported is often just a by-product of something else entirely. We need something that will work with us on figuring out what the issue is, instead of just being an interactive encyclopedia.</p>



<p class="wp-block-paragraph">Thus, the criteria for building an AI tool that is actually useful for us is simple: <strong>if it&#8217;s not doing the same work as me, for me, I don&#8217;t want to use it</strong>.</p>



<p class="wp-block-paragraph">We need something that will receive the same input as our supporters, pull up its artificial sleeves and get to work.</p>



<h2 class="wp-block-heading" id="what-we-built">What we built</h2>



<p class="wp-block-paragraph"><strong>Introducing Lynx</strong> &#8211; an internal support tool that tries to be something rather more than a simple prompt-in, answer-out flow with an LLM behind the curtain. At its&#8217; core, it&#8217;s a tool that goes through a structured investigation loop, leveraging LLM models that make extensive use of tools, both MCP and locally defined ones.</p>



<h2 class="wp-block-heading" id="small-tangent-what-are-tools">Small tangent: what are tools?</h2>



<p class="wp-block-paragraph"><em>While this article aims to cover the overall implementation of Lynx, we will also provide a &#8220;10 sentence or less&#8221; crash course for certain topics.</em></p>



<p class="wp-block-paragraph"><strong>Tools</strong> are essentially just commands that the AI is allowed to invoke. For example, you might want to allow your LLM to do a web search, lookup a phone number, or simply do some algebra.</p>



<p class="wp-block-paragraph">You would do this by defining tools. MCPs are one very popular example of such tools, but when developing an &#8220;LLM wrapper&#8221; on the backend, you can expose certain parts of your application directly to the AI by defining local tools.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<p class="wp-block-paragraph">Now, the aim of Lynx is to<strong> allow our support engineer to initiate an analysis using human language</strong>, providing any info they gathered thus far, or simply forwarding whatever they received themselves. Lynx enters the investigation loop, starting with a system prompt that we curate, giving it an overall goal and instructions how to behave. It then usually starts using the tools at its&#8217; disposal to gather as much data as possible.</p>



<p class="wp-block-paragraph">We provided it tools that connect it to everything any other supporter has access to: HEPIC, Prometheus, application logs, reports, etc.</p>



<p class="wp-block-paragraph">The crucial point to remember here is the fact that the <strong>AI doesn&#8217;t &#8220;know&#8221; voice infrastructure</strong>. It knows how to look things up, correlate them, and attempt to reason about what it finds. And digesting huge amount of textual data is something where LLM really shine.</p>



<p class="wp-block-paragraph">However, since just giving an LLM a bunch of tools won&#8217;t really help it know what to look for, it was important to develop a list of articles how to approach certain problems, what to look, what to use, and how certain things are supposed to function. For this, we came up with a knowledge base for the AI to search for. </p>



<p class="wp-block-paragraph">Also, around this time, <strong>RAG systems</strong> became hugely popular for putting between your AI assistant and your knowledge base, so we obviously had to incorporate that as well.</p>



<h2 class="wp-block-heading" id="small-tangent-what-is-rag">Small tangent: what is RAG?</h2>



<p class="wp-block-paragraph">Retrieval-Augmented Generation (RAG) is an AI framework that allows LLMs to pull relevant text from an knowledge base.</p>



<p class="wp-block-paragraph">There are plenty of amazing articles about this topic, but essentially, <strong>it allows the AI to search for things inside your knowledge base</strong>, and getting results based on some calculated similarity/relevancy score.</p>



<p class="wp-block-paragraph">A simplified overview of how this looks like can be something like this:<br></p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="864" height="744" src="https://shiftmag.dev/wp-content/uploads/2026/10/lynx-demonstration-simple.png?x32039" alt="" class="wp-image-12900" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/lynx-demonstration-simple.png 864w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-demonstration-simple-300x258.png 300w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-demonstration-simple-768x661.png 768w" sizes="auto, (max-width: 864px) 100vw, 864px" /></figure>



<p class="wp-block-paragraph">Looks fine and dandy, right? We just need a few volunteers to write the knowledge base, how wants to go first?</p>


<div class="wp-block-image">
<figure class="aligncenter"><img decoding="async" src="https://miro.medium.com/1*1LPBUmm5gjoDfet6AzMf1A.gif" alt="The Tumbleweed. my favorite email conversation-saving… | by ..."/></figure>
</div>


<p class="wp-block-paragraph">Matter of fact is that <strong>not many people want to write documentation</strong>, especially not for a handbook to be used by an experimental AI companion.</p>



<h2 class="wp-block-heading" id="so-just-use-ai">So&#8230;. Just use AI?</h2>



<p class="wp-block-paragraph">Feels like Deja Vu, but yes, we tackled this problem with AI as well. We realized that <strong>years of real investigation already lives in our internal communication channels</strong>, e.g. our slack channels dedicated to connecting our supporters with our developers.</p>



<p class="wp-block-paragraph">We did the following experiment:</p>



<ul class="wp-block-list">
<li>Dump every historical support conversation, initial support query on one side, human analysis on the other</li>



<li>Have Lynx investigate each reported issue independently, using its tools</li>



<li>Ask Lynx to compare its&#8217; analysis against what the human engineer actually found</li>



<li>Ask Lynx to create or edit existing knowledge base articles based on what it learned</li>
</ul>



<p class="wp-block-paragraph">The result? <strong>Thousands of articles,</strong> based on actual investigations and issues encountered by clients. We suddenly had a manual how to handle all issues encountered in the past few years!</p>



<p class="wp-block-paragraph">We captured exactly the kind of knowledge that we were looking for. The edge cases, the non-obvious correlations, the slack threads that spun for days investigating some obscure issue.</p>



<p class="wp-block-paragraph">For example, if someone uninformed were to ask Lynx about something we refer to as ASTW, they would have gotten the following answer:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="880" height="1024" src="https://shiftmag.dev/wp-content/uploads/2026/10/lynx-graphic-880x1024-1.png?x32039" alt="" class="wp-image-12901" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/lynx-graphic-880x1024-1.png 880w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-graphic-767x893-1.png 767w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-graphic-258x300-1.png 258w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-graphic-.png 1366w" sizes="auto, (max-width: 880px) 100vw, 880px" /></figure>



<p class="wp-block-paragraph">Since the AI model used (Claude Sonnet 4.5, in this specific instance) has only general knowledge and knows nothing about our procedures and abbreviations, all it can do is guesswork, and sadly often with misplaced confidence.</p>



<p class="wp-block-paragraph">However, now after <strong>we constructed our knowledge base</strong>, it&#8217;s a completely different story:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="538" src="https://shiftmag.dev/wp-content/uploads/2026/11/Shift-In-article-IS-2-1024x538.png?x32039" alt="" class="wp-image-13392" srcset="https://shiftmag.dev/wp-content/uploads/2026/11/Shift-In-article-IS-2-1024x538.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/11/Shift-In-article-IS-2-300x158.png 300w, https://shiftmag.dev/wp-content/uploads/2026/11/Shift-In-article-IS-2-768x403.png 768w, https://shiftmag.dev/wp-content/uploads/2026/11/Shift-In-article-IS-2.png 1200w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /></figure>



<p class="wp-block-paragraph">As you can see, we now have an initial version of our knowledge base built from real queries, and their answers are used by the LLM for future questions and investigations.</p>



<p class="wp-block-paragraph">Earlier we mentioned what tools are and that Lynx has a bunch of them, but how did we decide what to plug into Lynx was intentionally left vague. </p>



<p class="wp-block-paragraph">While creating the initial knowledge base, aside from giving Lynx the tools to write articles, we also defined a nifty tool called &#8220;request_tool_integration&#8221;.</p>



<pre class="wp-block-code"><code>@Tool(name = "request_tool_integration", description = """
        
Requests integration of a new tool or data source. USE THIS TOOL when you identify a need for      accessing data that is not currently available. Creates a request ticket for the development team. """)

public String requestToolIntegration(
@ToolParam(description = "Name for the proposed tool") String toolName,
@ToolParam(description = "Description of what the tool should do") String description,
@ToolParam(description = "Justification for why this tool is needed") String justification
) {...}</code></pre>



<p class="wp-block-paragraph">This tool would be used by Lynx to &#8220;file a formal complaint&#8221; whenever it would notice our human supporter was able to check something Lynx wasn&#8217;t able to see for itself. For example, consider the following support thread on our slack channel:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="763" src="https://shiftmag.dev/wp-content/uploads/2026/10/lynx-slack-1024x763-1.png?x32039" alt="" class="wp-image-12903" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/lynx-slack-1024x763-1.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-slack-300x224-1.png 300w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-slack-767x572-1.png 767w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-slack-.png 1453w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /></figure>



<p class="wp-block-paragraph">The prompt was simple, and the human supporter gave a straightforward answer, but when Lynx tried to analyse it itself, its conclusion was wrong. </p>



<p class="wp-block-paragraph">The initial analysis provided by Lynx was vague and mostly guesswork. However, since this was the learning phase, it actually invoked the tool mentioned above with the following description:</p>



<pre class="wp-block-code"><code><strong>Integration</strong> request – Resource: Account Configuration Service

<strong>Identifiers</strong>: account_id: NUMBER, recording_configuration, sftp_server_configured

<strong>Reason</strong>: Cannot determine if an account has SFTP recording delivery configured, which is essential for diagnosing why API recording downloads fail with HTTP 500 errors.
The support team identified this configuration as the root cause, but I have no way to query it independently.,

<strong>Value</strong>: Would enable independent diagnosis of recording access method failures. When a customer reports HTTP 500 on recording downloads,

I could query their account configuration and immediately determine if it's a SFTP-based account (expected behavior) vs CDN-based account (potential bug).

This would reduce support turnaround time and improve first-contact resolution rate. Currently, only account administrators can access this configuration.</code></pre>



<p class="wp-block-paragraph">What happened here is Lynx told us exactly what it needed access to in order to do its&#8217; job, so that&#8217;s actually how we formulated our initial list of tools that need to be integrated into Lynx. Ideally, we wanted to plug Lynx into everything via MCPs so it can access anything a normal supporter would want. This didn&#8217;t turn out to be as straightforward as initially anticipated. Instead, it was big enough of a learning experience that I&#8217;ll write a longer article about it soon.</p>



<h2 class="wp-block-heading" id="back-to-the-original-objective-there-is-a-support-ticket-waiting">Back to the original objective, there is a support ticket waiting</h2>



<p class="wp-block-paragraph">Now, for any given user complaint, <strong>Lynx is able to query for previous troubleshooting experiences</strong> and quickly try to apply that knowledge on the current investigation. Also, it is able to access all relevant information across all our components to quickly digest all records and hopefully find a correlation that will result in an accurate analysis.</p>



<p class="wp-block-paragraph">Once done, it provides a compact report, split into distinct sections, ranging from a quick summary about what it found, to a evidence section containing traces to records from which the conclusions were derived from, allowing the supporter to verify any claim and actually learn from the analysis itself.</p>



<p class="wp-block-paragraph">The result? The difficulty of Voice related troubleshooting significantly dropped, domain specific knowledge is no longer a barrier, and diagnostic data is no longer overwhelming.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="538" src="https://shiftmag.dev/wp-content/uploads/2026/11/Shift-In-article-IS-1-1024x538.png?x32039" alt="" class="wp-image-13393" srcset="https://shiftmag.dev/wp-content/uploads/2026/11/Shift-In-article-IS-1-1024x538.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/11/Shift-In-article-IS-1-768x403.png 768w, https://shiftmag.dev/wp-content/uploads/2026/11/Shift-In-article-IS-1-300x158.png 300w, https://shiftmag.dev/wp-content/uploads/2026/11/Shift-In-article-IS-1.png 1200w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /></figure>



<p class="wp-block-paragraph">Pictured: An example investigation by Lynx of a failed call. The issue was due to a misconfiguration while setting up the client account. Immediately identified by Lynx with provided steps how to remedy it.</p>



<p class="wp-block-paragraph">Aside from having a tool for support engineers, what we ended up with is actually also <strong>useful for anyone needing a quick answer regarding our voice platform</strong>. For example, one day someone from Infobip was wondering what a specific client was up to, and simply asked Lynx the following:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="1017" src="https://shiftmag.dev/wp-content/uploads/2026/10/lynx-demo-1024x1017.png?x32039" alt="" class="wp-image-12905" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/lynx-demo-1024x1017.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-demo-300x298.png 300w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-demo-150x150.png 150w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-demo-768x762.png 768w, https://shiftmag.dev/wp-content/uploads/2026/10/lynx-demo.png 1372w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /></figure>



<p class="wp-block-paragraph">Although the account traffic was already flagged by our other systems internally, it was still a non-technical-user friendly way to request some technical investigation and get an answer rapidly, without having to bother an actual engineer.</p>



<h2 class="wp-block-heading" id="what-the-future-brings-for-lynx">What the future brings for Lynx</h2>



<p class="wp-block-paragraph">All in all, <strong>Lynx is quickly becoming an immense time saver all around</strong>, both for technical employees for who the tool is originally aimed at, but also for non technical employees who would like to get a quick answer from a voice engineer. AI tools often struggle going from &#8220;that&#8217;s cool&#8221; to &#8220;that&#8217;s actually useful&#8221;, and we hope that Lynx will serve as one real world example where we managed to make the cut.</p>



<p class="wp-block-paragraph">The closed learning loop of automated learning of troubleshooting cases where human troubleshooters had to &#8220;jump in&#8221; ensures the knowledge base keeps growing and improving.</p>



<p class="wp-block-paragraph">Regarding next steps, there is still lots to do in terms of improving Lynx, especially regarding improvements in the knowledge base utilization, and overall testing and quality of response improvements. </p>



<p class="wp-block-paragraph">Aside from simply talking about giving it access to more integrations, there is also a lot of discussion to be had how and when an analysis is triggered. </p>



<p class="wp-block-paragraph">Although currently while in testing it&#8217;s always triggered by an human prompt, <strong>we aim to broaden the scope</strong>, for example by integrating it in our monitoring pipeline, so an analysis can be executed as soon as our alerts trigger. No matter the issue, having an Lynx analysis already provided and ready for review is like having an always on duty support engineer, always at your side, and hopefully reduces the amount of times our on-duty supporters have to be woken up.</p>


<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="1280" height="720" src="https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-5.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-5.png 1280w, https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-5-300x169.png 300w, https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-5-1024x576.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-5-768x432.png 768w" sizes="auto, (max-width: 1280px) 100vw, 1280px" /></figure><p>The post <a href="https://shiftmag.dev/we-fed-years-of-support-queries-to-an-ai-hoping-it-becomes-our-best-support-engineer-12896/">We Fed Years of Support Queries to an AI, Hoping It Becomes Our Best Support Engineer</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>How WiredTiger Cache Dirty Pages and Eviction Can Throttle Your MongoDB Writes</title>
		<link>https://shiftmag.dev/how-wiredtiger-cache-dirty-pages-and-eviction-can-throttle-your-mongodb-writes-13357/</link>
		
		<dc:creator><![CDATA[Frane Jelavic]]></dc:creator>
		<pubDate>Wed, 04 Nov 2026 14:10:08 +0000</pubDate>
				<category><![CDATA[Backend]]></category>
		<category><![CDATA[Data]]></category>
		<category><![CDATA[Database Performance]]></category>
		<category><![CDATA[mongodb]]></category>
		<category><![CDATA[wiredtiger]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=13357</guid>

					<description><![CDATA[<p>My MongoDB primary started throttling writes with every secondary healthy, and the cause was hiding in a cache most of us never look at.</p>
<p>The post <a href="https://shiftmag.dev/how-wiredtiger-cache-dirty-pages-and-eviction-can-throttle-your-mongodb-writes-13357/">How WiredTiger Cache Dirty Pages and Eviction Can Throttle Your MongoDB Writes</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="1280" height="720" src="https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-4.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-4.png 1280w, https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-4-300x169.png 300w, https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-4-1024x576.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/11/ShiftMag-article-banner-4-768x432.png 768w" sizes="auto, (max-width: 1280px) 100vw, 1280px" /></figure>


<p class="wp-block-paragraph">While continuing to learn more about MongoDB, specifically the synergy with WiredTiger, I’ve come to notice there is <strong>more than one way client writes can be throttled</strong>.</p>



<p class="wp-block-paragraph">If you haven’t followed my journey while battling with MongoDB, check out my previous post, <a href="https://shiftmag.dev/mongodb-one-write-two-logs-12267/" target="_blank" rel="noreferrer noopener">MongoDB: One write, two logs</a>, where a client write gets persisted and replicated inside a classical replica set. Supporting files for this article are on <a href="https://github.com/FraneJelavic/FraneJelavic.github.io/tree/main/content/writing/wiredtiger-cache-dirty-pages-and-eviction/media" target="_blank" rel="noopener">GitHub</a>.</p>



<p class="wp-block-paragraph">The first and best-known one is<strong> flow control</strong>, the write back pressure most of us have already met. A write counts as majority-committed once most of the replica set members acknowledge it. In a three-node replica set, that means the primary plus at least one secondary. If no secondary acknowledges, the majority commit point stops moving.</p>



<p class="wp-block-paragraph">MongoDB tries to keep that commit point within ten seconds of the primary (with default configuration). It starts throttling at half of that, around five seconds behind. Nothing is rejected, writes just slow down. The primary hands out fewer write tickets until the commit point moves again, but never fewer than a hundred tickets per second.</p>



<p class="wp-block-paragraph">We won&#8217;t talk about flow control here, because in production the throttling came from the primary itself. How is that possible? Let me explain.</p>



<h2 class="wp-block-heading" id="wiredtiger-cache">WiredTiger cache</h2>



<p class="wp-block-paragraph">You’ve read my previous post :), so let’s start assuming some basic understanding of MongoDB -&gt; WiredTiger communication. Every DML statement modifies a page in the WT cache.</p>



<p class="wp-block-paragraph">If we trace a write to a collection, the page holding that document becomes <strong>dirty</strong> inside the cache (if it is in the cache in the first place, otherwise it first gets read in from the data files). The page no longer matches what is on disk, and that mismatch is WT’s signal that the page has to be reconciled before the memory can be reused. Dirty pages leave the cache by one of two routes. <strong>Eviction</strong> runs continuously in the background. <strong>Checkpoints</strong> run every 60s by default.</p>



<p class="wp-block-paragraph">The write itself does not wait for either of them. WT appends every modification to the write-ahead <strong>journal</strong> while the affected page stays dirty in cache, and that append is what makes the write survive a crash (classical write ahead log mechanism). MongoDB flushes the journal every 100ms by default, <code>storage.journal.commitIntervalMs</code>, and immediately for a write that asks for <code>j:true</code>, so the exposure is that interval rather than the 60 seconds between checkpoints. If <em>mongod</em> dies between checkpoints, recovery replays the journal and rebuilds the pages from it. This is also what lets the primary acknowledge a write before its page is rewritten, which is how dirty content piles up between checkpoints. None of it touches <code>collection-*.wt</code> or <code>index-*.wt</code> on disk.</p>



<p class="wp-block-paragraph">Eventually the pages have to move, and that is the <strong>checkpoint’s job</strong>. It reconciles the pages that are still dirty and writes them to the data files as a snapshot consistent up to a single timestamp (MVCC), the same one across every collection and index. Pages that eviction already cleaned are skipped. The previous checkpoint stays valid while the new one is written, and the new one becomes permanent when WT updates its metadata table in one atomic step to point at it. Only then can the journal records it covers be discarded, which is what bounds journal growth.</p>



<h2 class="wp-block-heading" id="eviction-threads">Eviction threads</h2>



<p class="wp-block-paragraph">The cache has a fixed size, so something has to keep it from filling up. One eviction server walks the cache and picks candidates. Worker threads, by default four, do the heavy lifting.</p>



<p class="wp-block-paragraph">When the threads start, their progress is governed by <strong>four thresholds</strong>. They are two pairs measuring fullness and dirty percentage of the cache.</p>



<p class="wp-block-paragraph">The first pair watches how full the cache is. If we cross 80 percent, <code><strong>eviction_target</strong></code>, the workers start freeing space in the background. If we cross 95 percent, <code><strong>eviction_trigger</strong></code>, the application threads are summoned to help the workers.</p>



<p class="wp-block-paragraph">The second pair watches how much of that cache is dirty, using the same scale, with much lower numbers. Cross 5 percent, <code>eviction_dirty_target</code>, the workers start to evict dirty pages. If we cross 20 percent, <code>eviction_dirty_trigger</code>, the application threads join in.</p>



<p class="wp-block-paragraph">In a write-heavy workload the dirty pair is the one you meet first. A cache that is 30 percent full can be 30 percent dirty if every page in it was written, which is past the 20 percent trigger and nowhere near the 80 percent line.</p>



<p class="wp-block-paragraph">The logic is simple. The target starts the workers, and the trigger is the point where workers have “lost the battle” and need help from application threads. This is also the point where <strong>client writes start getting throttled</strong>.</p>



<p class="wp-block-paragraph">An excellent analogy to the upper mechanism is a restaurant. Waiters are application threads. They take orders and carry plates, which is your reads and writes. The kitchen shelf is the cache and holds a fixed number of plates. A clean plate is a page that matches the data files and can be reused. A dirty plate is a page somebody changed.</p>



<p class="wp-block-paragraph">The dishwashers are the eviction workers. While they keep up, nobody in the dining room notices. If we pass a point where there are too many dirty plates, the dishwashers cannot keep up, so we pull in the waiters. Since the waiters are now washing dishes, taking orders slows down, new dirty plates arrive more slowly, and the kitchen gets its chance to catch up. And never mind the review the customer leaves after waiting too long :D.</p>



<p class="wp-block-paragraph">Two details keep the analogy honest. A plate somebody is still eating from cannot be collected. That is a page pinned by an active reader, and it is a real reason eviction stalls while the cache keeps growing. And the trigger is not a request. An application thread that finds the cache past a trigger does the eviction itself, before it can read or write any more data. That is the latency you feel.</p>



<h2 class="wp-block-heading" id="mitigation">Mitigation</h2>



<p class="wp-block-paragraph">In my scenario this meant I needed more dishwashers, more time for them to work, or both. </p>



<p class="wp-block-paragraph">The workload was bulk writes that also triggered extra reads. That needed fixing at the source, but first I needed to <strong>reduce the backpressure</strong>. This cluster ran MongoDB 8.3.8. The version matters because the available configuration parameters differ between releases. The first change was to increase the eviction thread count from 4 to 20. On 8.3.8, dedicated server parameters allow this at startup or runtime:</p>



<pre class="wp-block-code"><code>db.adminCommand({
  setParameter: 1,
  wiredTigerEvictionThreadsMax: 20,
  wiredTigerEvictionThreadsMin: 20
})
</code></pre>



<p class="wp-block-paragraph"><strong>More threads alone were not enough</strong>. I also wanted dirty-page eviction to start earlier, giving the workers more time before application threads had to join in. That meant lowering <code>eviction_dirty_target</code> from 5 percent to 1 percent.</p>



<p class="wp-block-paragraph">There is a dedicated <code>wiredTigerEvictionDirtyTargetGB</code> parameter, but it only takes effect at startup and accepts an absolute size. For a 15.5 GiB cache, 0.155 would approximate a 1 percent target. Changing the percentage on the running instance required the raw <strong>WiredTiger configuration</strong>. The dirty-target part of that change was:</p>



<pre class="wp-block-code"><code>db.adminCommand({
  setParameter: 1,
  wiredTigerEngineRuntimeConfig: "eviction_dirty_target=1"
})
</code></pre>



<p class="wp-block-paragraph">MongoDB’s <a href="https://www.mongodb.com/docs/manual/reference/parameters/#mongodb-parameter-param.wiredTigerEngineRuntimeConfig" target="_blank" rel="noopener">manual</a> advises against changing this parameter without direction from MongoDB engineers because it has implications across both WiredTiger and MongoDB.</p>



<p class="wp-block-paragraph">Neither runtime command persists across restarts. The thread parameters can be stored under <code>setParameter</code> in <em>mongod.conf</em>.</p>



<p class="wp-block-paragraph"><strong>The result was 20 eviction threads available instead of 4</strong>, with background dirty-page eviction starting at 1 percent instead of 5 percent of the cache. That target is about 159 MiB on a 15.5 GiB cache. It is not a cap: dirty content can rise above it, and the dirty trigger remains at 20 percent.</p>



<p class="wp-block-paragraph">There is a cost to starting earlier and allowing more workers to participate. <strong>Eviction can consume more CPU and disk I/O</strong>. Setting both thread limits to 20 also prevents the thread count from scaling down, although those threads can wait when there is no work. For this workload, the aim was to absorb more of the eviction work in the background before it reached the application threads.</p>



<h2 class="wp-block-heading" id="my-takeaway">My takeaway</h2>



<p class="wp-block-paragraph">Even a two-parameter change can change how the cluster behaves, from incidents and slowdowns during write inflow to steady work across all workloads. </p>



<p class="wp-block-paragraph">The important thing is to <strong>understand the underlying technology and the mechanisms underneath</strong>. Understanding the why’s and the how’s. AI is incredible at helping us gain more knowledge and understanding, but we are the ones making the decision. </p>



<p class="wp-block-paragraph">This is the next step in software engineering I’m comfortable with.</p>



<figure class="wp-block-video"><video height="900" style="aspect-ratio: 1600 / 900;" width="1600" controls src="https://shiftmag.dev/wp-content/uploads/2026/11/wiredtiger-cache-glossary-v001.mp4"></video></figure>
<p>The post <a href="https://shiftmag.dev/how-wiredtiger-cache-dirty-pages-and-eviction-can-throttle-your-mongodb-writes-13357/">How WiredTiger Cache Dirty Pages and Eviction Can Throttle Your MongoDB Writes</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		<enclosure url="https://shiftmag.dev/wp-content/uploads/2026/11/wiredtiger-cache-glossary-v001.mp4" length="1000632" type="video/mp4" />

			</item>
		<item>
		<title>Jensen Huang&#8217;s Advice to Young Developers: &#8220;Become the World&#8217;s Best User of AI&#8221;</title>
		<link>https://shiftmag.dev/jensen-huangs-advice-to-young-developers-become-the-worlds-best-user-of-ai-13274/</link>
		
		<dc:creator><![CDATA[Ivan Pelivanovic]]></dc:creator>
		<pubDate>Tue, 03 Nov 2026 14:10:23 +0000</pubDate>
				<category><![CDATA[Artificial Intelligence]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[Jensen Huang]]></category>
		<category><![CDATA[NVIDIA]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=13274</guid>

					<description><![CDATA[<p>Nvidia's CEO says young people shouldn't fear AI taking their jobs. They should learn to use it to do things they couldn't do before.</p>
<p>The post <a href="https://shiftmag.dev/jensen-huangs-advice-to-young-developers-become-the-worlds-best-user-of-ai-13274/">Jensen Huang&#8217;s Advice to Young Developers: &#8220;Become the World&#8217;s Best User of AI&#8221;</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="1200" height="720" src="https://shiftmag.dev/wp-content/uploads/2026/11/gtc-berlin.jpg?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/11/gtc-berlin.jpg 1200w, https://shiftmag.dev/wp-content/uploads/2026/11/gtc-berlin-300x180.jpg 300w, https://shiftmag.dev/wp-content/uploads/2026/11/gtc-berlin-1024x614.jpg 1024w, https://shiftmag.dev/wp-content/uploads/2026/11/gtc-berlin-768x461.jpg 768w" sizes="auto, (max-width: 1200px) 100vw, 1200px" /></figure>


<p class="wp-block-paragraph"></p>



<p class="wp-block-paragraph">At Berlin’s packed Tempodrom, NVIDIA founder and CEO <strong>Jensen Huang</strong> was welcomed like a rock star at the company’s <a href="https://www.nvidia.com/en-eu/gtc/" target="_blank" rel="noreferrer noopener">GTC conference</a> where he shared the <a href="https://www.nvidia.com/en-eu/gtc/keynote/" target="_blank" rel="noreferrer noopener">big announcements about the future of AI</a> and NVIDIA&#8217;s products.</p>



<p class="wp-block-paragraph">The following day, Huang sat down with journalists, analysts and content creators for a dedicated Q&amp;A session. Alongside around 70 other attendees, we had the opportunity to hear what the head of <a href="https://companiesmarketcap.com/nvidia/marketcap/" target="_blank" rel="noreferrer noopener">the world’s most valuable company</a> thinks about whether AI will replace junior roles and  whether it is actually making us more productive.</p>



<p class="wp-block-paragraph">And no, that’s not AI photo.<strong> I really did get a photo with Jensen</strong>!</p>



<h2 class="wp-block-heading" id="with-ai-we-can-do-things-we-couldnt-before">With AI, we can do things we couldn&#8217;t before</h2>



<p class="wp-block-paragraph">The conversation about artificial intelligence at work often comes down to how quickly it can generate code, summarise a document or write a piece of content. Huang believes <strong>we should be thinking about productivity in broader terms</strong>:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">There are two ways to think about productivity. One is, of course, that whatever you used to do, you can now do more efficiently. But the better way of thinking about AI is that you can now do things you couldn&#8217;t do before.</p>
</blockquote>



<p class="wp-block-paragraph">As an example, he pointed to engineering and design. Rather than manually working through a limited number of possible solutions, engineers can use AI to explore a much wider design space while accounting for constraints such as cost, energy efficiency and manufacturability. The value of AI, in this case is that <strong>it opens up possibilities that would have been difficult to explore otherwise</strong>:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">AI can give you thousands of suggestions. That is real productivity, and this principle can be applied to almost any industry.</p>
</blockquote>



<p class="wp-block-paragraph">Huang believes <strong>we should stop measuring productivity solely by how much faster we can complete existing tasks</strong>:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">That is just not very ambitious. We should raise our ambitions, not just give ourselves more free time. I think we have always thought about productivity in precisely the wrong way. Technological progress works in the opposite way to what people often expect. Every time we invented a new technology, the economy got bigger and, frankly, we got busier.</p>
</blockquote>



<h2 class="wp-block-heading" id="what-about-juniors">What about juniors?</h2>



<p class="wp-block-paragraph">Huang was also asked about the global jobs market and the<strong> lack of entry-level opportunities</strong> for people graduating from university or just starting their careers:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Of course, some jobs will become less necessary than before because of AI, or perhaps less interesting. But new jobs, new companies and new businesses will emerge. That is the nature of the world. These changes are inevitable, and it is a good thing that the world keeps changing.</p>
</blockquote>



<p class="wp-block-paragraph">So how should young people prepare for those changes? We asked Huang what he would do if he were starting his career today:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">If I were a young person today, I would, without a doubt, aim to become the world&#8217;s best user of AI. I would be completely obsessed with how to use it, how to use different tools, open and closed models, everything. I would use all the different types of harnesses. I would learn how to manage its memory. How to create knowledge graphs, whether to use graphs. I would figure all of that out. I would become an expert at it. I would be constantly running new AI agents.</p>
</blockquote>



<p class="wp-block-paragraph">So, Huang believes young people should <strong>focus on mastering AI tools</strong>, dismissing predictions that more than half of young graduates will struggle to find jobs as ridiculous. </p>



<p class="wp-block-paragraph">And this advice is not limited to people working in tech. A plumber, for example, could use AI to help a customer design a new kitchen, while a furniture salesperson could offer a more detailed interior design proposal:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">By adopting technology, you can decide to do more with your own career, whatever your job may be.</p>
</blockquote>



<p class="wp-block-paragraph"><strong>That does not mean knowing how to use AI tools automatically replaces professional expertise</strong>. On the contrary, the more capable these tools become, the more important it is to evaluate their output, recognise mistakes and understand the problem they are supposed to solve. Someone who knows their field and understands how to apply AI within it has a tangible advantage over someone who relies solely on model outputs.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="614" src="https://shiftmag.dev/wp-content/uploads/2026/11/Jensen-Huang-1024x614.jpg?x32039" alt="" class="wp-image-13278" srcset="https://shiftmag.dev/wp-content/uploads/2026/11/Jensen-Huang-1024x614.jpg 1024w, https://shiftmag.dev/wp-content/uploads/2026/11/Jensen-Huang-300x180.jpg 300w, https://shiftmag.dev/wp-content/uploads/2026/11/Jensen-Huang-767x460.jpg 767w, https://shiftmag.dev/wp-content/uploads/2026/11/Jensen-Huang.jpg 1200w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /><figcaption class="wp-element-caption">Photo: Ivan Pelivanović</figcaption></figure>



<h2 class="wp-block-heading" id="ai-agents-from-different-companies-will-work-together">AI agents from different companies will work together</h2>



<p class="wp-block-paragraph">The conversation also turned to a future in which <strong>AI agents will play an increasingly important role in the workplace</strong>. Huang expects specialised agents to become more advanced and capable of handling increasingly complex tasks. One agent could research data, another analyse the findings, and a third prepare a report. A coordinating system would oversee their work and direct the individual steps.</p>



<p class="wp-block-paragraph">The NVIDIA CEO also <strong>expects agents from different systems and companies to start working together</strong>. This could connect, for example, industrial design tools with manufacturing platforms and business systems:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">An NVIDIA agent and a Mercedes agent, or an NVIDIA agent and a Siemens agent, could start working together once we connect them, whenever they determine that it is necessary.</p>
</blockquote>



<p class="wp-block-paragraph">But as agents gain more autonomy, <strong>the question of oversight</strong> becomes increasingly important. Huang highlighted security and access control as two of the key challenges in the development of AI agents. Systems should be given only the permissions necessary to complete a specific task, with clearly defined access rules and continuous monitoring of their activity.</p>



<p class="wp-block-paragraph">For companies, this means establishing data access policies, clear boundaries for agent autonomy and appropriate oversight mechanisms. An agent capable of carrying out tasks independently can improve efficiency, but mistakes or unintended actions can also have more serious consequences.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="614" src="https://shiftmag.dev/wp-content/uploads/2026/11/Huang-2-1024x614.jpg?x32039" alt="" class="wp-image-13279" srcset="https://shiftmag.dev/wp-content/uploads/2026/11/Huang-2-1024x614.jpg 1024w, https://shiftmag.dev/wp-content/uploads/2026/11/Huang-2-300x180.jpg 300w, https://shiftmag.dev/wp-content/uploads/2026/11/Huang-2-767x460.jpg 767w, https://shiftmag.dev/wp-content/uploads/2026/11/Huang-2.jpg 1200w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /><figcaption class="wp-element-caption">Photo: Ivan Pelivanović</figcaption></figure>



<h2 class="wp-block-heading" id="europe%e2%80%99s-ai-future"><strong>Europe’s AI future</strong></h2>



<p class="wp-block-paragraph">In Berlin, Huang also addressed Europe’s position in the global AI race. Europe does not need to develop every part of AI technology to become a relevant player, he argued. What matters more is identifying areas where it already has a competitive advantage, directing investment towards them and enabling new companies to emerge and apply AI in ways that were previously unimaginable:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Of course, you should aspire to do more. But, as with everything in life, you start by focusing on the areas where you are particularly good. You need to prioritise them because those are the areas where you are most likely to succeed first.</p>
</blockquote>



<p class="wp-block-paragraph">In Huang&#8217;s vision, Europe&#8217;s success over the next five years will be measured by <strong>how quickly existing companies integrate AI into their businesses</strong>:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Success means that every single company in Europe is an AI company.</p>
</blockquote>



<p class="wp-block-paragraph">That does not mean every company should develop its own AI model but <strong>integrating AI into products, services and business processes</strong>, as well as in the emergence of new companies that do not yet exist. <a href="https://www.konsulteer.com/article/nvidia-plans-investments-in-four-french-startups-as-france-s-sovereign-ai-push-widens" target="_blank" rel="noreferrer noopener">NVIDIA has already invested close to</a> $10 billion in Europe&#8217;s technology ecosystem, including startups, universities and research institutions:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">I believe in Europe&#8217;s AI startups. I believe they are Europe&#8217;s scouts. Not all of them have to succeed as independent companies. Some will find a sustainable business model, while others may be acquired by larger players. What matters is that they all have the opportunity to try.</p>
</blockquote>



<p class="wp-block-paragraph">Making that happen will also require a stronger venture capital ecosystem &#8211; Europe needs more investors willing to fund startups, as well as the capacity to support those investments.</p>



<h2 class="wp-block-heading" id="too-much-optimism"><strong>Too much optimism?</strong></h2>



<p class="wp-block-paragraph">Huang sees AI as an <strong>opportunity to expand human capabilities</strong>, develop new industries and create jobs that may not even exist today. At the same time, he acknowledges that this development comes with risks and that companies must take responsibility for the safety of their systems.</p>



<p class="wp-block-paragraph">His optimism, however, comes from the perspective of someone leading a company whose chips, systems and platforms power a large part of today&#8217;s AI infrastructure. NVIDIA benefits from a world in which more companies develop AI, invest in data centres and automate their operations. That does not necessarily make Huang&#8217;s predictions wrong, but it is worth considering the business interests behind the vision he presents.</p>



<p class="wp-block-paragraph">His vision describes a world in which AI can expand what individuals and companies are capable of achieving. Whether that opportunity translates into a genuine competitive advantage will also depend on who has the knowledge, resources and conditions needed to make the most of it.</p>



<p class="wp-block-paragraph"></p>
<p>The post <a href="https://shiftmag.dev/jensen-huangs-advice-to-young-developers-become-the-worlds-best-user-of-ai-13274/">Jensen Huang&#8217;s Advice to Young Developers: &#8220;Become the World&#8217;s Best User of AI&#8221;</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>MotherDuck CEO Says Single-Node Analytics Can Outperform Distributed Systems</title>
		<link>https://shiftmag.dev/motherduck-ceo-says-single-node-analytics-can-outperform-distributed-systems-12404/</link>
		
		<dc:creator><![CDATA[Marko Crnjanski]]></dc:creator>
		<pubDate>Mon, 02 Nov 2026 11:19:54 +0000</pubDate>
				<category><![CDATA[Data]]></category>
		<category><![CDATA[Software Engineering]]></category>
		<category><![CDATA[AI agents]]></category>
		<category><![CDATA[Data Analytics]]></category>
		<category><![CDATA[DuckDB]]></category>
		<category><![CDATA[Jordan Tigani]]></category>
		<category><![CDATA[MCP]]></category>
		<category><![CDATA[MotherDuck]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=12404</guid>

					<description><![CDATA[<p>Jordan Tigani, MotherDuck CEO, argues that many analytics teams are paying a complexity tax for workloads that never needed it.</p>
<p>The post <a href="https://shiftmag.dev/motherduck-ceo-says-single-node-analytics-can-outperform-distributed-systems-12404/">MotherDuck CEO Says Single-Node Analytics Can Outperform Distributed Systems</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="1200" height="630" src="https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-4.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-4.png 1200w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-4-300x158.png 300w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-4-768x403.png 768w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-4-1024x538.png 1024w" sizes="auto, (max-width: 1200px) 100vw, 1200px" /></figure>


<p class="wp-block-paragraph">I spoke with <a href="https://www.linkedin.com/in/jordantigani/" target="_blank" rel="noreferrer noopener">Jordan Tigani</a>, CEO of MotherDuck, about a question analytics teams should ask before building a cluster: <strong>does this workload actually need one</strong>? </p>



<p class="wp-block-paragraph">MotherDuck is a cloud analytics service built on DuckDB, an open-source SQL database for analytics; and before co-founding it, Tigani helped build Google BigQuery as an early engineer and led product at SingleStore. </p>



<p class="wp-block-paragraph">We discussed <strong>when distributed systems are worth the complexity</strong> and how analytics teams can build and govern simpler alternatives.</p>



<h2 class="wp-block-heading" id="choose-architecture-based-on-what-queries-actually-process-not-just-total-data-size">Choose architecture based on what queries actually process, not just total data size</h2>



<p class="wp-block-paragraph">First I wanted to satisfy my curiosity about how BigQuery was built, so I asked Tigani what the starting point had been. </p>



<p class="wp-block-paragraph">He said it was <strong>the need for distributed systems, which divide work across multiple machines</strong>. I then asked what had convinced him that many modern analytics workloads might no longer require that approach and he said:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">When I was working on BigQuery I used to run a lot of analytics to understand our customers and what they were doing. I noticed that most of our customers didn’t have a lot of data. And even the ones who did have a lot of data, many of their queries only processed a small amount of data at once. We had some of the largest customers in the world, Walmart, HSBC, Equifax. They were all using a tool called &#8220;Big Query&#8221; but the queries they were running were not big at all.</p>
</blockquote>



<p class="wp-block-paragraph">Tigani also noted that he mostly just filed that information away as a curiosity. Later, while he was working at SingleStore to help build its SaaS database product, he began exploring the machines available on AWS. &#8220;I was shocked; you could get a machine with hundreds of cores and terabytes of RAM, and it wasn’t even expensive,&#8221; Jordan explained.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">The third piece that tied it all together was when I saw how fast you could implement things in a single-node system. I had been following DuckDB for a little bit, and they added a join optimization that was similar to something we’d done in BigQuery years earlier.</p>
</blockquote>



<p class="wp-block-paragraph"><strong>That optimization took months to build in BigQuery because of all its moving parts</strong>. DuckDB built it in days and shipped it alongside other improvements. &#8220;That was shocking, just how fast you could move if you didn’t have to deal with all of the distributed systems machinery.&#8221;</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">So machines are getting bigger. But workloads are not. And distributed systems add a tax, both in the performance you can possibly expect but also the time it takes to build and improve things. I put those together and thought, &#8220;wow, if we were going to create BigQuery today, we would have done it very differently.&#8221;</p>
</blockquote>



<h2 class="wp-block-heading" id="isolate-workloads-to-avoid-slowdowns-and-complex-resource-management">Isolate workloads to avoid slowdowns and complex resource management</h2>



<p class="wp-block-paragraph">From there, I turned to query execution in MotherDuck:<strong> how it decides whether queries run locally, in the cloud (or both)</strong> and how developers can check.</p>



<p class="wp-block-paragraph">&#8220;For context, MotherDuck has a feature we call “dual execution” which means we can run queries either locally or in the cloud,&#8221; Jordan explained.</p>



<p class="wp-block-paragraph">He also explained that this lets MotherDuck integrate seamlessly with applications already using DuckDB, and also lets customers build highly performant applications where results get cached locally and follow-up queries can run out of that local cache:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">However, we never run anything locally unless the user makes it clear that is what they want to do. For example, if they’re using a local duckdb database and then they join it against a MotherDuck database, we’ll run part of the query locally and part in the cloud.</p>
</blockquote>



<p class="wp-block-paragraph">He said there are also cases where <strong>users want to run entirely in the cloud</strong> and don’t want the dual-execution machinery, which includes a copy of DuckDB. For those cases, they can use MotherDuck’s PostgreSQL endpoint, which makes MotherDuck look just like a PostgreSQL server.</p>



<p class="wp-block-paragraph">Speaking about the advantages and limitations of MotherDuck’s hypertenancy model compared with a traditional shared data warehouse, where multiple users or workloads draw on shared computing resources, Jordan noted that hypertenancy is a fancy way of saying &#8220;everybody gets a duck.&#8221; In other words, if you have 100 users, every one of those users gets <strong>their own individual DuckDB instance running in the cloud</strong>.</p>



<p class="wp-block-paragraph">&#8220;We combine that with serverless execution, so we can start virtually instantaneously and shut down as soon as queries complete,&#8221; Tigani said and explained: </p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">The rationale from this was after seeing how complicated workload management was in other databases. If you have more than one user or workload accessing the same hardware, you have to figure out how to share resources. If you go from one user to 10 users querying concurrently, either that first user’s queries are going to slow down, the new users are going to run slowly, or you had vastly over-provisioned.</p>
</blockquote>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="538" src="https://shiftmag.dev/wp-content/uploads/2026/10/motherduck_1-1-1024x538.jpg?x32039" alt="" class="wp-image-13196" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/motherduck_1-1-1024x538.jpg 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/motherduck_1-1-768x403.jpg 768w, https://shiftmag.dev/wp-content/uploads/2026/10/motherduck_1-1-300x158.jpg 300w, https://shiftmag.dev/wp-content/uploads/2026/10/motherduck_1-1.jpg 1200w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /><figcaption class="wp-element-caption">MotherDuck’s architecture combines shared storage with isolated DuckDB compute instances, called Ducklings, and interfaces for developer tools and AI agents. Compute resources can scale independently for different users and workloads.</figcaption></figure>



<p class="wp-block-paragraph">What I also learned from Jordan is that <strong>workload management tends to get really complicated</strong>, both for users and for people building the database. For example, Tigani said, configuring workload management in Teradata is so complicated that one of their most anticipated features was a mechanism for machine learning to be able to help you tune it correctly. In other words, it was too hard for humans to do it well:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">For MotherDuck we thought, what if we made it so every workload was a different completely isolated database? Then workload management is simple. You don’t need to worry about one user slowing down another one, or complex priority schemes to decide who got what resources. You also don’t have to worry about over or under provisioning.</p>
</blockquote>



<p class="wp-block-paragraph">The other advantage of this, he added, is that <strong>it scales really nicely with how people actually use an analytical database</strong>. There are often a couple of users who have really heavy workloads and a lot of users who have much lighter workloads. So the heavy users scale up to big instances, the lighter users get tiny ones. And none of them slow each other down.</p>



<p class="wp-block-paragraph">&#8220;If you’re building a SaaS application this ends up being even more useful, because you can give each one of your end users a different database. In other words, let’s say you’re building a stock portfolio app and you want to give your users the ability to analyze their holdings,&#8221; Jordan explained.</p>



<p class="wp-block-paragraph">According to Tigani, each of those users gets a different DuckDB instance; these instances turn off when not being used, so it doesn’t matter how many users you have: <strong>you only pay for what’s actually running</strong>. And all of your users are nicely isolated from each other.</p>



<h2 class="wp-block-heading" id="storage-formats-and-postgresql-compatibility">Storage formats and PostgreSQL compatibility</h2>



<p class="wp-block-paragraph">Our conversation also moved to <strong>DuckLake, Apache Iceberg and Delta Lake</strong>, technologies for organizing tables and managing their metadata in a data lake. </p>



<p class="wp-block-paragraph">I asked Tigani what problem DuckLake was designed to solve that Iceberg and Delta Lake did not solve well enough. His detailed answer was:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Do you remember when everyone was putting things on the blockchain, and a handful of sane people said &#8220;wait, this is basically just an incredibly complicated database that can only do a couple of updates per second.&#8221; That’s kind of what Iceberg and Delta are; they’re a super complicated way of storing data on S3 and can only do one or two updates per second.</p>
</blockquote>



<p class="wp-block-paragraph">DuckLake he expleined, is a <strong>cleaner way of envisioning how a data lake can work </strong>by storing the metadata about how to find the active bits of the table in a database instead of on S3. This means it is a lot faster to find what you’re looking for as well as makes it much more reliable to make updates.</p>



<p class="wp-block-paragraph">Jordan also mentioned that people say that a lot of the weaknesses of Iceberg are being solved in subsequent versions, but it is very hard to get around architectural limitations by adding complexity on top:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Of course, a lot of people use Iceberg, and there is a lot of momentum behind it. So while we believe that DuckLake is a better way to store your data, Iceberg isn’t likely going anywhere. So we have first class support for iceberg, including the various Iceberg catalogs, in MotherDuck.</p>
</blockquote>



<p class="wp-block-paragraph">MotherDuck’s support for the <a href="https://www.postgresql.org/docs/current/protocol.html" target="_blank" rel="noreferrer noopener">PostgreSQL wire protocol</a>, the communication protocol used by PostgreSQL clients and servers, raised another practical topic. I asked where compatibility with Postgres ends and what differences developers should understand before using it in production.</p>



<p class="wp-block-paragraph">Explaining this, Tigani said the primary reason for Postgres wire protocol support is that it creates virtually universal connectivity. A lot of people can connect to DuckDB, but virtually everyone knows how to connect to Postgres.</p>



<p class="wp-block-paragraph">He also noted that the secondary reason is that <strong>a lot of people use Postgres for things that are really analytics and then start to hit the limits of what Postgres can do</strong>. “By switching to the postgres wire protocol, we can make it super easy to switch to MotherDuck; you don’t have to change your queries, your connector, or your code; it should just work,” Jordan explained and added:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">That said, I should add all the usual caveats. While DuckDB SQL is mostly a superset of Postgres SQL, it is not a strict superset. That is, the vast majority of Postgres functions work in DuckDB, but there are some corner cases where things might be missing. There are also cases where some things may work subtly differently in DuckDB than in postgres. We’re hard at work on ensuring compatibility, but when you take a large system as complex as Postgres, there are going to be some differences.</p>
</blockquote>



<h2 class="wp-block-heading" id="treat-ai-agents-like-any-production-tool-grant-only-the-access-they-need">Treat AI agents like any production tool: grant only the access they need</h2>



<p class="wp-block-paragraph">An important part of our conversation focused on AI agents querying production data through the <a href="https://shiftmag.dev/tag/mcp/" data-type="post_tag" data-id="573" target="_blank" rel="noreferrer noopener">Model Context Protocol</a>, or MCP, a standard for connecting AI applications to external tools and data sources. I asked Jordan which security and governance controls teams should put in place before enabling that access. He explained:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">There isn’t anything special about MCP that isn’t true with other AI tools, or with other tools in general. It is easy, and perhaps natural, for people to see tools that can produce magical results and to jump in and want to start using them in production. However, if set up poorly, these magical tools can cause embarrassing security holes. The principle of least privilege is a good one to follow.</p>
</blockquote>



<p class="wp-block-paragraph">He also urged teams to be careful about access permissions:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Be careful who can add new tools. Trim security groups. Watch out for confused deputies (things that run as privileged users that provide services to other users can accidentally give those other users access to things they shouldn’t be able to see or do).</p>
</blockquote>



<p class="wp-block-paragraph">Building on this theme, Tigani said that <strong>it is also easy to have a knee-jerk reaction to security and risk</strong>. However, it is important to think through the actual threat models. What is the actual harm that can be done? What things would happen in order for that harm to be achieved? And then what can you do to mitigate those situations?</p>



<h2 class="wp-block-heading" id="match-review-effort-to-the-risk">Match review effort to the risk</h2>



<p class="wp-block-paragraph">Before the interview ended, I asked Jordan <strong>how engineering teams should test, review and monitor work produced by AI agents</strong> as they generate more SQL queries, data pipelines and analytics applications.</p>



<p class="wp-block-paragraph">He answered that at MotherDuck, virtually everyone in the company creates Dives, which are their vibe-coded dashboards. According to Tigani, most of these people wouldn’t be comfortable using SQL, and yet they use these tools to help them in their jobs every day:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">For most cases, the advantage of having a quick and easy answer is well worth the possibility that it will be wrong. And the benefit of having the line-of-business users actually generate the visualizations is that they are the ones who can detect when something goes wrong; they are going to be better than an analyst on a centralized data team to know that the revenue from a particular customer looks off.</p>
</blockquote>



<p class="wp-block-paragraph">Tigani also mentioned that the amount of scrutiny varies depending on how important it is that the answer is right. If you’re reporting numbers to a regulator, you probably need them to be right. But the vast majority of use cases are not actually like that.</p>



<p class="wp-block-paragraph">&#8220;If you’re building dashboards to understand customers in your sales book, or trying to gauge the success of your marketing campaigns, <strong>rapid iteration is going to be more important than perfection</strong>,&#8221; Jordan said, adding:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Additionally, in my experience, the quality of AI written SQL is higher than human written SQL. You really don’t have to review it to figure if it is going to work. You don’t need to check to see if there is an off-by-one error in the bounds of the window function, or if it passed the arguments correctly. Important visualizations, important data pipelines, and context should be all treated like product code. Stored in source control. Changed via pull-requests with commit messages showing what was actually changed as well.</p>
</blockquote>



<p class="wp-block-paragraph">&#8220;Even if it is AI writing the code, <strong>you should make sure it is auditable by a human</strong>. But as AI gets better, humans are going to be doing less code review to check it to see if it is right and more auditing to see what happened when something goes wrong,&#8221; Jordan concluded.</p>
<p>The post <a href="https://shiftmag.dev/motherduck-ceo-says-single-node-analytics-can-outperform-distributed-systems-12404/">MotherDuck CEO Says Single-Node Analytics Can Outperform Distributed Systems</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>You Shouldn&#8217;t Trust AI Agents in Production</title>
		<link>https://shiftmag.dev/you-shouldnt-trust-ai-agents-in-production-13232/</link>
		
		<dc:creator><![CDATA[Marko Crnjanski]]></dc:creator>
		<pubDate>Fri, 30 Oct 2026 12:35:53 +0000</pubDate>
				<category><![CDATA[Artificial Intelligence]]></category>
		<category><![CDATA[Video]]></category>
		<category><![CDATA[AI agents]]></category>
		<category><![CDATA[AI security]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Darko Mesaros]]></category>
		<category><![CDATA[LLMs]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=13232</guid>

					<description><![CDATA[<p>Because AI agents are probabilistic and can't be trusted on their own, developers need to build layered safeguards around them, and the cost of running them in production deserves as much attention as safety.</p>
<p>The post <a href="https://shiftmag.dev/you-shouldnt-trust-ai-agents-in-production-13232/">You Shouldn&#8217;t Trust AI Agents in Production</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Darko Mesaros, a Distinguished Developer Advocate at AWS, <strong>doesn&#8217;t trust his AI agents</strong>. We spoke with him after his talk at WeAreDevelopers San Jose about where to set limits for them and why production costs deserve more attention.</p>



<p class="wp-block-paragraph">He thinks that distrust is a sensible place to start. Agents can be wrong, so if you let them touch business operations, you need safeguards around their decisions.</p>



<p class="wp-block-paragraph">Let&#8217;s start with why he doesn&#8217;t trust them.</p>



<h2 class="wp-block-heading" id="your-talk-at-wad-san-jose-starts-with-a-strong-statement-i-don%e2%80%99t-trust-ai-agents-what-exactly-shouldn%e2%80%99t-developers-trust-about-them">Your talk at WAD San Jose starts with a strong statement: &#8220;I don’t trust AI agents.&#8221; What exactly shouldn’t developers trust about them?</h2>



<p class="wp-block-paragraph"><strong>Darko</strong>: The reason I don’t trust my AI agents is that they’re ultimately probabilistic systems. They’re not deterministic. When you give agents serious responsibilities, such as handling refunds, selling things or dealing with sensitive customer data, relying on a probabilistic system is a problem. <strong>Agents by themselves, without guardrails around them, are problematic</strong>.</p>



<p class="wp-block-paragraph">I’ve spent a lot of time teaching developers how to approach this. Building agents is more than having an agent and access to an LLM. It’s about building the system around it, with guardrails and everything you need to make the agent a little more secure.</p>



<h2 class="wp-block-heading" id="what%e2%80%99s-the-biggest-difference-between-an-ai-agent-that-works-in-a-demo-and-one-that-is-actually-ready-for-production">What’s the biggest difference between an AI agent that works in a demo and one that is actually ready for production?</h2>



<p class="wp-block-paragraph"><strong>Darko:</strong> Believe it or not, cost. When we build an agent for a demo, we might pick the biggest model because we want to make sure the demo works.</p>



<p class="wp-block-paragraph">The challenge comes when you start scaling that out. When an agent handles hundreds of thousands of customer requests in different ways, cost becomes an issue. <strong>Optimization isn’t something you worry about in a demo</strong>. You don’t worry about prompt caching, RAG or those things.</p>



<p class="wp-block-paragraph">In an actual production system, you start to consider cost. Beyond safety and guardrails, that’s the biggest difference between a demo agent and a real-life production agent.</p>



<h2 class="wp-block-heading" id="when-building-an-agent-what-safety-layer-should-developers-add-first-guardrails-permissions-observability-or-something-else">When building an agent, what safety layer should developers add first: guardrails, permissions, observability or something else?</h2>



<p class="wp-block-paragraph"><strong>Darko:</strong> The answer is yes to all of those things, because proper defense is built in layers. A single layer, a single guardrail, may not prevent everything. You may want guardrails, MCP input validation and deterministic policy checks around certain tool calls. All of those things are important.</p>



<p class="wp-block-paragraph">As I mentioned in my talk, observability becomes incredibly important <strong>because you really don’t know what your agents are doing</strong>, especially at that scale. Being able to observe them and run evaluations periodically or in real time gives you a lot more peace of mind. So, to answer the question of which one to add, the answer is yes.</p>



<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<iframe loading="lazy" title="AWS Expert on 3 Must-Haves Before Shipping AI Agents" width="500" height="281" src="https://www.youtube.com/embed/Nv0uRM1W4V0?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
</div></figure>



<h2 class="wp-block-heading" id="how-do-you-know-when-an-ai-agent-has-made-a-mistake-if-its-output-still-looks-perfectly-reasonable">How do you know when an AI agent has made a mistake if its output still looks perfectly reasonable?</h2>



<p class="wp-block-paragraph"><strong>Darko:</strong> You have to combine a few things, and we go back to observability. Implementing automated tracing is critical so you can trace what’s going on in real time. But tracing doesn’t stop at the agent. You also need to trace the backend system the agent works with.</p>



<p class="wp-block-paragraph">If you start detecting anomalies, such as an increasing number of small or large orders, or something unusual, <strong>you need to look at both the agent and the backend</strong>. Combining those signals into a proper anomaly detection system is how you could potentially catch certain things happening in real time.</p>



<p class="wp-block-paragraph">It’s never just an agent. It’s an agent plus everything else behind it.</p>



<h2 class="wp-block-heading" id="how-much-autonomy-should-we-actually-give-ai-agents-in-production-where-should-developers-draw-the-line">How much autonomy should we actually give AI agents in production? Where should developers draw the line?</h2>



<p class="wp-block-paragraph"><strong>Darko:</strong> Ultimately, responsibility for what happens, for a system or code that goes into production, rests with the developer.</p>



<p class="wp-block-paragraph">How much autonomy should we give agents? <strong>It depends on the blast radius of a change or of a system going down</strong>. Is this a simple chatbot that tells customers the opening times of your business? Sure. If something goes wrong, it’s not horrible.</p>



<p class="wp-block-paragraph">But if the system handles banking information or sensitive data, or processes purchases, you may want to reduce its autonomy. It’s not only about the code or change being submitted; it’s also about the system you’re submitting it to. Unfortunately, the answer here is: it depends.</p>



<h2 class="wp-block-heading" id="if-a-developer-is-shipping-their-first-production-ai-agent-tomorrow-what-are-the-three-things-they-absolutely-need-to-have-in-place">If a developer is shipping their first production AI agent tomorrow, what are the three things they absolutely need to have in place?</h2>



<p class="wp-block-paragraph"><strong>Darko:</strong> Number one, proper observability. They need to be able to observe and monitor what their agent is doing, run evaluations against the models and their decisions and choices, <strong>and constantly monitor those things for changes</strong>.</p>



<p class="wp-block-paragraph">Number two, policies. A policy language such as Cedar lets you put policies in front of tool execution so you can prevent certain tools from executing.</p>



<p class="wp-block-paragraph">Number three, consider an established harness. It’s tempting to build your own because we’re all developers, but I encourage developers to consider something like the Strands SDK, LangGraph or LangChain. <strong>It makes things easier and lets you focus on your business problem</strong> and spend less time figuring out how to wrangle the agent.</p>


<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="1200" height="620" src="https://shiftmag.dev/wp-content/uploads/2026/10/darko_main.jpg?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/darko_main.jpg 1200w, https://shiftmag.dev/wp-content/uploads/2026/10/darko_main-766x396.jpg 766w, https://shiftmag.dev/wp-content/uploads/2026/10/darko_main-300x155.jpg 300w, https://shiftmag.dev/wp-content/uploads/2026/10/darko_main-1024x529.jpg 1024w" sizes="auto, (max-width: 1200px) 100vw, 1200px" /></figure><p>The post <a href="https://shiftmag.dev/you-shouldnt-trust-ai-agents-in-production-13232/">You Shouldn&#8217;t Trust AI Agents in Production</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>How to Review a 1,000+ File PR Without Losing Your Mind</title>
		<link>https://shiftmag.dev/how-to-review-a-1000-file-pr-without-losing-your-mind-13054/</link>
		
		<dc:creator><![CDATA[Mario Brcina]]></dc:creator>
		<pubDate>Thu, 29 Oct 2026 13:33:06 +0000</pubDate>
				<category><![CDATA[Artificial Intelligence]]></category>
		<category><![CDATA[Software Engineering]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[AI agents]]></category>
		<category><![CDATA[code review]]></category>
		<category><![CDATA[PR review]]></category>
		<guid isPermaLink="false">https://shiftmag.dev/?p=13054</guid>

					<description><![CDATA[<p>The scariest part of a 1,186-file PR is how quickly 15% coverage can start to feel like a thorough review.</p>
<p>The post <a href="https://shiftmag.dev/how-to-review-a-1000-file-pr-without-losing-your-mind-13054/">How to Review a 1,000+ File PR Without Losing Your Mind</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Let me present you the PR in question: ticket <strong>DAED-190, &#8220;Switch gw edit to new UI stack&#8221;</strong>. </p>



<p class="wp-block-paragraph">In one PR, a colleague replaced the legacy JSP gateway-edit page with a React single-page app. Because that meant changing the UI stack, the backend had to move with it: new REST controllers, a new application service, new tests, a Maven frontend build, and an early commit that deleted everything nobody referenced anymore.</p>



<p class="wp-block-paragraph">The diff, against master:</p>



<pre class="wp-block-code"><code>1186 files changed, 19809 insertions(+), 473971 deletions(-)
</code></pre>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="393" src="https://shiftmag.dev/wp-content/uploads/2026/10/image-6-1024x393.png?x32039" alt="" class="wp-image-13224" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/image-6-1024x393.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/image-6-300x115.png 300w, https://shiftmag.dev/wp-content/uploads/2026/10/image-6-767x294.png 767w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /><figcaption class="wp-element-caption">Numbers are inflated because bundled third-party frontend was committed before this PR. The right panel excludes it.</figcaption></figure>



<p class="wp-block-paragraph">Bitbucket helpfully informed me that <strong>only the first 1000 files are shown</strong>. That number was the whole reason this turned into a project of its own.</p>


<div class="wp-block-image">
<figure class="aligncenter size-full"><img loading="lazy" decoding="async" width="567" height="103" src="https://shiftmag.dev/wp-content/uploads/2026/10/image-5.png?x32039" alt="" class="wp-image-13222" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/image-5.png 567w, https://shiftmag.dev/wp-content/uploads/2026/10/image-5-300x54.png 300w" sizes="auto, (max-width: 567px) 100vw, 567px" /></figure>
</div>


<p class="wp-block-paragraph">So I sat down and asked myself: <em>how do I actually do this well?</em></p>



<h2 class="wp-block-heading" id="review-fails-through-lack-of-coverage-not-lack-of-skill"><strong>Review fails through lack of coverage, not lack of skill</strong></h2>



<p class="wp-block-paragraph"><strong>Doing it manually was not realistic</strong>. I&#8217;d read for days, hold a shrinking amount of the change in my head, get tired, and still miss things, and the things I&#8217;d miss would be the boring, important ones (a field that quietly lost its validation or an action that no longer exists).</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">The failure mode of manual review at this size is not &#8220;I review it wrong&#8221;, it&#8217;s &#8220;I review 15% of it and quietly convince myself that was enough&#8221;.</p>
</blockquote>



<p class="wp-block-paragraph">The obvious alternative, throwing the diff at an LLM and asking &#8220;review this&#8221;, is worse. The diff alone doesn&#8217;t fit in the context window. And even if it did, the agent has no baseline: &#8220;is this line correct?&#8221; is a much weaker question than &#8220;does this behave like the thing it replaces?&#8221; When you&#8217;re replacing a UI, the interesting question is never <em>does the new code look right</em>, it&#8217;s <em>does the new code still do what the old code did</em>. You can&#8217;t answer that from the diff.</p>



<p class="wp-block-paragraph">I needed an <strong>approach that kept the context small enough for the agent to be reliable, while still covering the whole change</strong>.</p>



<h2 class="wp-block-heading" id="what-i-had-going-for-me-and-what-i-didnt">What I had going for me (and what I didn&#8217;t)</h2>



<p class="wp-block-paragraph">Before choosing an approach, it&#8217;s worth being honest about the shape of the thing you&#8217;re reviewing.</p>



<p class="wp-block-paragraph"><strong>In my favor</strong>: this is a relatively small CRUD service with only a handful of flows. A list page, a copy flow, an edit form, not a sprawling distributed system with a dozen integration points. That matters, because it means the surface a rewrite can break is finite and enumerable. Users had already been testing the new UI on IO (Integrating Environment), so the obvious, everyday path (open a gateway, change a field, save) had human eyes on it before the PR ever reached me.</p>



<p class="wp-block-paragraph"><strong>Which is exactly why I didn&#8217;t trust it</strong>. People test the happy path. They click through the flow that works, confirm the screen looks right, and move on. Nobody spends their afternoon feeding a 4000-character string into a 100-column field, or pulling the session out from under a save.</p>



<p class="wp-block-paragraph">So I deliberately weighted the review toward the unhappy paths: missing validations, error handling, what happens when a write fails halfway through. A green light from testers on IO told me the good path worked but told me almost nothing about the bad ones.</p>



<p class="wp-block-paragraph"><strong>And one thing working against me</strong>: I already knew, from weeks of troubleshooting this service, that it had holes: plenty of its fields were not properly validated or handled to begin with. So &#8220;the new code matches the old code&#8221; was never going to be enough on its own. If the baseline is leaky, faithfully reproducing the baseline reproduces the leaks.</p>



<p class="wp-block-paragraph">That&#8217;s part of why the review ended up covering DB constraints and failure modes rather than stopping at parity, and a big reason so many of the findings turned out to be inherited rather than new.</p>



<h2 class="wp-block-heading" id="divide-and-conquer">Divide and conquer</h2>



<p class="wp-block-paragraph">Two things made the difference.</p>



<p class="wp-block-paragraph"><strong>First, clone both sides side by side.</strong> I checked out master and the PR branch into two separate directories:</p>



<pre class="wp-block-code"><code>./master/infobip-gateway-edit      ← the old JSP page, the baseline
./pr_branch/infobip-gateway-edit   ← the React/Bepo SPA</code></pre>



<p class="wp-block-paragraph">That sounds trivial but it changes the whole exercise. Now &#8220;what did master do here?&#8221; is a file read, not a guess. Every claim the review makes can be traced back to a line in the old code. The agent is comparing two concrete states, and not reasoning a diff. </p>



<p class="wp-block-paragraph">(I also cloned the two SQL repositories that hold the table definitions, because part of the review ended up being &#8220;does the new code respect the actual database constraints?&#8221;. More on that later.)</p>



<p class="wp-block-paragraph"><strong>Second, break the review into small steps and do one per session.</strong> This is the part that made the agent reliable. A single &#8220;review this PR&#8221; prompt would have produced a confident, shallow wall of text. Sixteen focused prompts produced something I could actually trust, because each one had a narrow scope, a written-down baseline, and a bounded amount of context to hold.</p>



<p class="wp-block-paragraph">The prompt I gave the agent did not say &#8220;review the changes&#8221;. It asked a narrower, more provable question, and it did so in very concrete terms.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="512" src="https://shiftmag.dev/wp-content/uploads/2026/10/additions_by_language-1024x512.png?x32039" alt="" class="wp-image-13208" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/additions_by_language-1024x512.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/additions_by_language-300x150.png 300w, https://shiftmag.dev/wp-content/uploads/2026/10/additions_by_language-768x384.png 768w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /></figure>



<h3 class="wp-block-heading" id="the-instruction-that-started-it">The instruction that started it</h3>



<p class="wp-block-paragraph">Here is the first prompt in full: it names what we&#8217;re checking, the order, the output files, and how we&#8217;ll work together.</p>



<p class="wp-block-paragraph"></p>



<div class="wp-block-group"><div class="wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained">
<details class="wp-block-details has-background is-layout-flow wp-block-details-is-layout-flow" style="background-color:#f5f5f5"><summary> <strong>&#8211; First-phase review prompt (<em>click to expand)</em></strong></summary>
<div class="wp-block-group has-background" style="background-color:#f5f5f5"><div class="wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained">
<h3 class="wp-block-heading" id="code-review-strategy">Code review strategy</h3>



<p class="wp-block-paragraph">Since PR is so large, DO NOT try to review it by listing changes and checking implementation correctness.</p>



<p class="wp-block-paragraph">Instead we will focus on:</p>



<ul class="wp-block-list">
<li>does branch version has all the fields and functionalities that were present on master</li>



<li>does branch version has all the validations that were present on master</li>
</ul>



<p class="wp-block-paragraph">This will be the only things we will focus in first part of the review.</p>



<p class="wp-block-paragraph">So we need to make a plan and do it in steps so we manage context efficiently.</p>



<p class="wp-block-paragraph">Here is my proposed order:</p>



<ol start="1" class="wp-block-list">
<li>Document all fields present on master (by checking source code of master branch) and store documentation in new md file (call it <code>./MASTER_FIELDS_DOCS.md</code>)</li>



<li>Document all functionalities/actions on master (that can be done by user) and store documentation in new md file (call it <code>./MASTER_ACTIONS_DOCS.md</code>)</li>



<li>Document all validations on master and store documentation in new md file (call it <code>./MASTER_VALIDATIONS_DOCS.md</code>)</li>



<li>At this step you should have all the docs needed (that will serve as reference), and you can start reviewing are all fields that were present on master (documented in step 1) present in PR branch</li>



<li>Same thing like previous point, in this step you will check are features/actions present on master (documented in step 2) also present in PR branch</li>



<li>And finally, you will check are validations present on master (documented in step 3) also present in PR branch</li>
</ol>



<p class="wp-block-paragraph">Before any of the above (as step 0) create review plan with ordered items (based on the above instructions) that will serve you and me to track the progress. Idea is that we have plan written in md file and then go over it step by step. After finishing one step you stop, update plan file and report back. I check result, compact session and give green light to continue to next session.</p>
</div></div>
</details>
</div></div>



<p class="wp-block-paragraph">Two things are doing most of the work here. The prompt leads with a <strong>negative</strong> instruction (<em>don&#8217;t</em> review the diff), which steers the agent away from the obvious-but-wrong approach. And it hands over a concrete first output (three named files), a stopping rule, and a coordination protocol (<em>stop, update the plan, report back</em>). None of that is clever; it&#8217;s just specific. That specificity is what kept sixteen sessions coherent.</p>



<h2 class="wp-block-heading" id="why-the-plan-lives-in-a-file">Why the plan lives in a file</h2>



<p class="wp-block-paragraph">The move that mattered most wasn&#8217;t clever. It was one of the earliest practices in agentic coding: a plan file (<code>REVIEW_PLAN.md</code>) with a status table at the top:</p>



<figure class="wp-block-table"><table class="has-fixed-layout"><tbody><tr><td>#</td><td>Step</td><td>Output file</td><td>Status</td></tr><tr><td>0</td><td>Create review plan</td><td>REVIEW_PLAN.md</td><td>DONE</td></tr><tr><td>1</td><td>Document all master fields</td><td>MASTER_FIELDS_DOCS.md</td><td>DONE</td></tr><tr><td>2</td><td>Document all master actions</td><td>MASTER_ACTIONS_DOCS.md</td><td>DONE</td></tr><tr><td>3</td><td>Document all master validations</td><td>MASTER_VALIDATIONS_DOCS.md</td><td>DONE</td></tr></tbody></table></figure>



<p class="wp-block-paragraph">The workflow was:</p>



<ol start="1" class="wp-block-list">
<li>Agent does <strong>one</strong> step.</li>



<li>Agent <strong>stops</strong>, updates the status table, and reports back.</li>



<li>I read the result and decide if it&#8217;s good enough.</li>



<li>I compact the session (context reset), give the green light, and the agent starts the next step.</li>
</ol>



<p class="wp-block-paragraph">Two things come out of this. The agent never has to hold more than one step in mind, which keeps the context (and therefore the hallucination risk) small. And <strong>I</strong> never have to hold the whole PR in mind either, because the plan file does that for me. If I come back a week later, <code>REVIEW_PLAN.md</code> tells me exactly where we are. (This turned out to matter: I later re-checked comments after the developer pushed new commits, and the plan file meant I didn&#8217;t have to reconstruct anything.)</p>



<p class="wp-block-paragraph">The other side of the workflow: the intermediate documents are <em>baselines</em>. <code>MASTER_FIELDS_DOCS.md</code> lists every field on the old page with its type, options, and visibility rules. <code>MASTER_ACTIONS_DOCS.md</code> lists 17 groups of user actions.</p>



<p class="wp-block-paragraph"><code>MASTER_VALIDATIONS_DOCS.md</code> lists <strong>47 numbered validations</strong> (V1–V47). Each entry names its source file and line, so the parity check in a later step is mechanical rather than interpretive: you can diff a document against the new code, which is <em>much</em> more reliable than asking an agent to &#8220;look for missing validations&#8221;.</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="374" src="https://shiftmag.dev/wp-content/uploads/2026/10/the_loop-1024x374.png?x32039" alt="" class="wp-image-13209" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/the_loop-1024x374.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/the_loop-300x110.png 300w, https://shiftmag.dev/wp-content/uploads/2026/10/the_loop-766x280.png 766w" sizes="auto, (max-width: 1024px) 100vw, 1024px" /><figcaption class="wp-element-caption">An overview of the loop.</figcaption></figure>



<h2 class="wp-block-heading" id="building-on-the-baseline-phase-2">Building on the baseline (phase 2)</h2>



<p class="wp-block-paragraph">Once the three parity axes were done (fields, actions, validations), I had a solid, evidence-backed answer to &#8220;did the rewrite lose anything?&#8221; That&#8217;s the question a UI rewrite can actually get wrong in a way users notice.</p>



<p class="wp-block-paragraph">But parity isn&#8217;t correctness. A rewrite can keep every field and every rule and still be broken in ways the old page wasn&#8217;t. So I extended the plan with a second phase, still small steps, still one per session, this time walking the application like an engineer building it from scratch and asking <em>which parts have to work for this to be safe to ship?</em></p>



<p class="wp-block-paragraph">I kicked off that phase with a second prompt, the same shape as the first: a concrete goal, the scope I was deliberately excluding, and an ordered list of steps.</p>



<div class="wp-block-group"><div class="wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained">
<details class="wp-block-details has-background is-layout-flow wp-block-details-is-layout-flow" style="background-color:#f5f5f5"><summary><strong>&#8211; Second-phase review prompt (<em>click to expand</em>)</strong></summary>
<div class="wp-block-group has-background" style="background-color:#f5f5f5"><div class="wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained">
<h3 class="wp-block-heading" id="goal"><strong>Goal</strong></h3>



<p class="wp-block-paragraph">This is the second part of the review. In the first part we did NOT check the implementation details, we only checked that the branch version carries over everything that was on master (all fields, all functionalities/actions, all validations). That baseline is confirmed and documented now, and the comments that came out of it are already posted on the PR.</p>



<p class="wp-block-paragraph">Now we go one level deeper and review the PR implementation for correctness. Same rule as before: the PR is huge, so we will NOT do it line by line. We treat the app as a set of standalone parts and check each part in isolation, one step per session, so we manage the agent&#8217;s context efficiently.</p>



<p class="wp-block-paragraph">Code quality and code organisation are NOT the focus. Behaviour is. We will focus on correctness and checxk does the app work correctly and safely for the user.</p>



<h3 class="wp-block-heading" id="code-review-strategy-2"><strong>Code review strategy</strong></h3>



<p class="wp-block-paragraph">Instead of diffing against master, we walk the app the way an engineer building it from scratch would, and ask which parts have to work for it to be safe to ship. So we will check these, each as a standalone unit:</p>



<ul class="wp-block-list">
<li>are the DB constraints that back the form actually enforced (length, nullability, numeric width, uniqueness, CHECK), or do they only blow up later in SQL Server</li>



<li>are the writes atomic, and what happens when two operators change the same gateway at the same time</li>



<li>what does the user actually see when something fails (validation error, missing record, unexpected error)</li>



<li>does the new stack respect the same permissions the old tool required</li>



<li>is the new REST contract consistent with what the frontend sends and expects</li>



<li>does the SPA work at runtime (routing/deep links, races, i18n, keyboard/a11y)</li>



<li>what is cached (HTTP, Spring, browser) and can any of it serve stale content</li>



<li>keystore file upload: limits, validation, error path</li>



<li>what we can observe in production (logging, audit trail, metrics/health)</li>



<li>does the build produce the right assets, reproducibly</li>
</ul>



<h3 class="wp-block-heading" id="scope-notes"><strong>Scope notes</strong></h3>



<ul class="wp-block-list">
<li>Security: verify only that the authorization the old tool required is carried over. Do NOT review the SSO internals.</li>



<li>API contract: treat the old routes as internal (no external callers confirmed), so focus on internal consistency only.</li>



<li>The write flow that matters is COPY. Users only copy a template gateway; add/update and the auxiliary actions are not used today and are being removed.</li>
</ul>



<h3 class="wp-block-heading" id="here-is-my-proposed-order"><strong>Here is my proposed order</strong></h3>



<p class="wp-block-paragraph">Continue the numbering from the plan file you already have (steps 0-6 are done). For each step, record traceable findings plus a verdict, and write the output md file.</p>



<ol start="7" class="wp-block-list">
<li>Verify branch validations against DB constraints (length, nullability, numeric width, uniqueness, CHECK)</li>



<li>Data integrity, transactions &amp; concurrency</li>



<li>Error handling &amp; failure modes</li>



<li>Security: authorization transfer only</li>



<li>API contract &amp; integration (no external callers)</li>



<li>Frontend runtime correctness</li>



<li>Caching (HTTP + Spring + frontend)</li>



<li>File upload (keystore)</li>



<li>Observability &amp; operability</li>



<li>Build, packaging &amp; assets</li>
</ol>



<p class="wp-block-paragraph">Add these steps to the plan file and keep the same way of working: after finishing one step you stop, update the plan file and report back. I check the result, compact the session and give the green light to continue to the next step.</p>



<h3 class="wp-block-heading" id="files-you-should-read"><strong>Files you should read</strong></h3>



<p class="wp-block-paragraph">Read following files:</p>



<ol start="1" class="wp-block-list">
<li><code>./REVIEW_PLAN.md</code></li>



<li><code>./INSTRUCTIONS_FOR_AGENT.md</code></li>



<li><code>./MASTER_FIELDS_DOCS.md</code></li>



<li><code>./MASTER_ACTIONS_DOCS.md</code></li>



<li><code>./MASTER_VALIDATIONS_DOCS.md</code></li>



<li><code>./PR_COMMENT.md</code></li>



<li><code>./JIRA_TASK.md</code></li>
</ol>
</div></div>
</details>
</div></div>



<figure class="wp-block-table"><table class="has-fixed-layout"><tbody><tr><td>#</td><td></td><td></td></tr><tr><td>7</td><td>Validations against <strong>DB constraints</strong> (length, nullability, numeric width, uniqueness, CHECK)</td><td>REVIEW_VALIDATIONS_AGAINST_DB.md</td></tr><tr><td>8</td><td><strong>Data integrity, transactions &amp; concurrency</strong></td><td>REVIEW_FINDINGS_DATA_INTEGRITY.md</td></tr><tr><td>9</td><td><strong>Error handling &amp; failure modes</strong></td><td>REVIEW_FINDINGS_ERROR_HANDLING.md</td></tr><tr><td>10</td><td><strong>Security</strong>: authorization transfer only</td><td>REVIEW_FINDINGS_AUTHORIZATION.md</td></tr><tr><td>11</td><td><strong>API contract &amp; integration</strong></td><td>REVIEW_FINDINGS_API_CONTRACT.md</td></tr><tr><td>12</td><td><strong>Frontend runtime correctness</strong></td><td>REVIEW_FINDINGS_FRONTEND_RUNTIME.md</td></tr><tr><td>13</td><td><strong>Caching</strong> (HTTP + Spring + frontend)</td><td>REVIEW_FINDINGS_CACHING.md</td></tr><tr><td>14</td><td><strong>File upload (keystore)</strong></td><td>REVIEW_FINDINGS_KEYSTORE_UPLOAD.md</td></tr><tr><td>15</td><td><strong>Observability &amp; operability</strong></td><td>REVIEW_FINDINGS_OBSERVABILITY.md</td></tr><tr><td>16</td><td><strong>Build, packaging &amp; assets</strong></td><td>REVIEW_FINDINGS_BUILD_ASSETS.md</td></tr></tbody></table></figure>



<p class="wp-block-paragraph">I wrote these steps the way I&#8217;d audit a service: transactions, then failure paths, then who&#8217;s allowed in, then the contract, then the runtime, then the edges. Each one is a standalone unit of &#8220;does this part work?&#8221;, so the agent could check it in isolation without needing the other fifteen steps in context.</p>



<p class="wp-block-paragraph">Two of the steps earned their place in ways I didn&#8217;t predict:</p>



<ul class="wp-block-list">
<li><strong>DB constraints (step 7)</strong> turned into the single most valuable step. Comparing the form metadata against the actual DDL found that 43 text columns had no length enforcement anywhere, client <em>or</em> server, so an over-long value would travel all the way to SQL Server and come back as <em>&#8220;String or binary data would be truncated&#8221;</em>, with the save handler flattening every database error into one generic message. There&#8217;s no way to find that by reading a diff; you find it by resolving each form field to its column and checking the constraint is enforced somewhere.</li>



<li><strong>Caching (step 13)</strong> found that the new SPA loads its bundles from fixed, unversioned URLs, so a deploy could leave browsers running last week&#8217;s JavaScript against this week&#8217;s backend. That one was real, and it got fixed. (Which is its own small lesson: a review that produces at least one concrete, fixed bug has paid for itself.)</li>
</ul>



<h2 class="wp-block-heading" id="what-i-didnt-review-on-purpose">What I didn&#8217;t review <em>on purpose</em></h2>



<p class="wp-block-paragraph">I ignored code quality and organization. Not because it doesn&#8217;t matter, but because I had to choose, and at 1186 files I couldn&#8217;t do everything (see &#8220;the line&#8221; below). What I prioritised was <strong>app correctness and the things a user would notice</strong>. These include a missing field, a missing validation, a save that half-commits, an error page instead of an error message or a bundle that won&#8217;t refresh.</p>



<p class="wp-block-paragraph">I&#8217;m confident there is dead code left in that PR. There is certainly some code that could be written more clearly, or factored differently. I flagged the small amount of dead code I tripped over while looking for correctness issues (unused imports, unreferenced configuration properties), but I didn&#8217;t go looking for it.</p>



<p class="wp-block-paragraph">That was a conscious trade, not an oversight.</p>



<h2 class="wp-block-heading" id="tools">Tools</h2>



<p class="wp-block-paragraph">I used <strong>Pi</strong> with an AI model served from our own infrastructure.</p>



<p class="wp-block-paragraph">I keep Pi deliberately bare (no MCPs and no skills) because I also use it with models running locally on my Mac, where RAM limits how much context I can hold, so I keep the harness lean to leave as much of the window as possible for the actual work. This review ran against the company-hosted model, so that constraint didn&#8217;t really bite here, but the bare setup is what I&#8217;m used to.</p>



<p class="wp-block-paragraph">When a task genuinely needs tooling, I reach for other agents (Codex, Claude Code) that have it wired up. For this review what I needed wasn&#8217;t tools; it was a patient reader with one small, well-defined job per session, and that&#8217;s what the step-by-step structure provided.</p>



<h2 class="wp-block-heading" id="what-the-review-produced">What the review produced</h2>



<p class="wp-block-paragraph">All told, the process produced 16 review steps, 13 findings documents, and a set of PR-ready comments. Not every step produced a comment: several produced an explicit <em>&#8220;no regression, no comment needed&#8221;</em> verdict, which is just as valuable, because it&#8217;s a documented answer instead of a silent omission. A few of the things it turned up:</p>



<ul class="wp-block-list">
<li><strong>A real caching bug</strong> in the new SPA bundle URLs (found, commented, <strong>fixed by the developer</strong>).</li>



<li><strong>A dropped keystore</strong> on copy-without-reupload (the copy path read the source gateway and then discarded it, so the row ended up with the old filename and password but a <code>NULL</code> keystore blob).</li>



<li><strong>A blank page</strong> when the SPA is opened at a URL that isn&#8217;t a copy URL: the URL parser throws <em>before</em> the promise <code>.catch</code> that renders the &#8220;cannot open the gateway&#8221; message, and there&#8217;s no error boundary, so the operator gets nothing.</li>



<li><strong>The dead</strong> <code>action</code> <strong>envelope field</strong>, and assorted smaller items.</li>



<li>And on the other side, a genuine <strong>improvement</strong> worth calling out: the branch is <em>stricter</em> than master on server-side bounds: 18 <code>@Range</code> checks and the <code>GatewaySpeed</code> <code>@DecimalMin/Max</code> rules now run on save where master had them dormant. I asked the developer to confirm the stricter rejections were intentional, but it&#8217;s a good change.</li>
</ul>



<p class="wp-block-paragraph">A handful of the findings resolved in the developer&#8217;s favour, too: two &#8220;missing&#8221; grids turned out to be non-functional on master in the first place (their buttons were disabled and the permission helper always returned <code>false</code>), and one authorization concern from the plan turned out not to be a real gate at all. </p>



<p class="wp-block-paragraph">Part of doing this well is being willing to be wrong in writing, and the baseline documents made that checkable rather than a matter of opinion.</p>



<h2 class="wp-block-heading" id="the-line-you-have-to-draw">The line you have to draw</h2>



<p class="wp-block-paragraph">Somewhere in a PR this size you have to decide you&#8217;re done. You will not read everything. You will miss nuances. There is dead code you didn&#8217;t flag and duplication you didn&#8217;t mention.</p>



<p class="wp-block-paragraph">That&#8217;s not failure. It&#8217;s the job. The alternative is holding a 1186-file change to an impossible standard and never shipping the review. So I drew the line at app correctness, accepted that code-quality nits would survive, and moved on, because other things were waiting for my attention.</p>



<p class="wp-block-paragraph">If you take one thing from this: <strong>scope the review to a small number of axis you can actually prove</strong>, write the baseline down, and let the structure (not willpower) keep you honest.</p>



<h2 class="wp-block-heading" id="credit-where-its-due">Credit where it&#8217;s due</h2>



<p class="wp-block-paragraph">A list of findings can give the wrong impression, so I want to be explicit about something: <strong>most of what this review flagged was not introduced by this PR.</strong> It was copied, faithfully, from the old master implementation.</p>



<p class="wp-block-paragraph">The non-atomic copy (step 8), the generic &#8220;database error&#8221; that names no field (step 7), the audit that is best-effort and swallows its own failures (step 15), the permission helper that always returned <code>false</code>, the grids whose buttons did nothing (steps 5/10), the lengthless text columns: the new code reproduces the old behaviour, warts and all. </p>



<p class="wp-block-paragraph">That is exactly what a rewrite is <em>supposed</em> to do, and finding inherited behaviour is not the same as finding a defect. I said as much in the write-ups: several findings are marked &#8220;parity&#8221; or &#8220;no regression&#8221; rather than &#8220;bug&#8221; for precisely that reason.</p>



<p class="wp-block-paragraph">Which reframes the numbers. In a PR this size (a full UI stack replacement, plus the backend changes to support it), the count of genuinely <em>new</em> defects was small. Most of the review&#8217;s output is a map of what the old system already did, not a list of things the rewrite broke.</p>



<p class="wp-block-paragraph">Large part of that is the developer&#8217;s work. He kept parity deliberately, wrote tests that pin behaviour, and fixed the real issues (the caching bug, the keystore drop, the case-sensitivity bug in the new field check) as they came up. But I also think it&#8217;s a genuine sign of how far LLM-assisted development has come. </p>



<p class="wp-block-paragraph">Producing a large, coherent, mostly-correct rewrite, where the majority of what a careful reviewer finds is the <em>old</em> code&#8217;s pre-existing behaviour rather than new mistakes, is not something I would have expected a couple of years ago. The human still matters: someone had to guide the change, land it, and decide what was worth reviewing. </p>



<p class="wp-block-paragraph">But the baseline quality of the large machine-written change was high, and the remaining edge cases were the exceptions, not the rule.</p>



<h2 class="wp-block-heading" id="caveats">Caveats</h2>



<p class="wp-block-paragraph">This is <strong>one developer&#8217;s approach to one specific PR</strong>. It is not a general methodology and it isn&#8217;t a claim that everyone should review big PRs this way.</p>



<p class="wp-block-paragraph">It worked well <em>here</em> for a particular reason: this PR was a <strong>rewrite of an existing behaviour</strong>, which gives you a natural baseline (the old code) and a natural set of questions (does the new code still do all of it?). A 1000-file PR that <em>adds</em> a new feature has no such baseline, and the same divide-and-conquer axes wouldn&#8217;t transfer directly. </p>



<p class="wp-block-paragraph">A PR that&#8217;s mostly dependency bumps or generated code would want something else again.</p>



<p class="wp-block-paragraph">Treat it as a worked example you can borrow from, not a rulebook. The transferable part isn&#8217;t the sixteen steps. It&#8217;s the habits: clone both sides, pick a small number of provable axes, write the plan down, keep each session narrow, and stop when you&#8217;ve covered what you decided mattered.</p>


<figure class="wp-block-post-featured-image"><img loading="lazy" decoding="async" width="1280" height="720" src="https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-4-1.png?x32039" class="attachment-post-thumbnail size-post-thumbnail wp-post-image" alt="" style="object-fit:cover;" srcset="https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-4-1.png 1280w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-4-1-300x169.png 300w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-4-1-1024x576.png 1024w, https://shiftmag.dev/wp-content/uploads/2026/10/ShiftMag-article-banner-4-1-768x432.png 768w" sizes="auto, (max-width: 1280px) 100vw, 1280px" /></figure><p>The post <a href="https://shiftmag.dev/how-to-review-a-1000-file-pr-without-losing-your-mind-13054/">How to Review a 1,000+ File PR Without Losing Your Mind</a> appeared first on <a href="https://shiftmag.dev">ShiftMag</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>

<!--
Performance optimized by W3 Total Cache. Learn more: https://www.boldgrid.com/w3-total-cache/?utm_source=w3tc&utm_medium=footer_comment&utm_campaign=free_plugin

Page Caching using Disk: Enhanced 

Served from: shiftmag.dev @ 2026-11-16 04:33:53 by W3 Total Cache
-->