When businesses create messaging campaigns on Infobip, they must go through an approval process. They share brand details and campaign information, such as the purpose, target age group, and campaign type, which is checked for consistency and compliance with platform safety rules like WhatsApp or Viber terms of use.

Once a campaign goes live, it’s our job as developers to ensure the messages being sent continue to reflect what was approved.

There’s a whole spectrum of ways a campaign can drift, though. Some changes are obvious safety problems, like phishing or illegal content. Others are more subtle, like using the wrong brand voice, adding marketing language to transactional messages, or changing personalized content outside the approved template. The hardest cases are the ones that look compliant at first but still blur the line.

…and that’s precisely what makes this a tough engineering problem.

Keyword matching isn’t enough to catch message drift

Legitimate traffic and drifting traffic can look very similar, even when they mean different things. Rule-based systems and keyword matching can’t always tell the difference. Detecting drift requires understanding meaning, context, and who is sending the message.

If that’s a little hard to picture in the abstract, here’s an example. It’s rarely this overt in practice, but it makes the shape of the problem obvious.

Imagine you’re a long-time client of an eye clinic. Every year, like clockwork, you receive the same familiar birthday message from the same phone number:

Hello Sam, this is Clermont Family Eyecare, we want to wish you a happy birthday!

Generic, but consistent – a small customer-care touch from a business you trust.

This year, the message arrives slightly altered, from that same number:

Hello Sammy, this is For Your Eyes Only Eye Care, we want to wish you a very happy birthday!

The tone is friendly and the structure looks the same. But the brand name has changed, and the link is new. You tap it, and suddenly you’re on a page with explicit content, not eyecare. In one click, trust is broken – even though the message came from the same number and passed every superficial check.

On average, this system, when completed, is meant to process just short of 300 million SMS and RCS messages daily (crazy, right?). So, the engineering question becomes: how do you detect drift semantically, in production, at that scale, without reasoning about every message individually? Let me walk you through how we got to an answer.

The system needs to be designed from first principles

A few observations about the traffic led to the architecture.

The first is the most obvious: messaging campaigns are repetitive by nature. Messages are templated, sometimes identical within a campaign. At our scale, this immediately points toward grouping messages and reasoning about drift at the group level rather than per message.

There’s an important operational detail here you should know: when businesses create messaging campaigns on our platform, they go through an approval process. They provide brand information and campaign metadata like intent, recipient age or campaign type categorization, which is reviewed for internal consistency and against platform safety guidelines (think WhatsApp or Viber Terms of Use). Once a campaign goes live, it’s our job to ensure the messages being sent continue to reflect what was approved.

You can see that the problem isn’t really about tracking a semantic distribution of campaign messages and watching it shift over time, even though the word drift might suggest that. Historical traffic isn’t what defines drift here. The problem is closer to continuous content validation: preserving the intent stated in the original campaign metadata and looking for new patterns that warrant separate interpretation and potential action.

Even though this framing focuses on novel data, historical traffic still matters. If incoming messages reliably match traffic that has already been evaluated, we can decide in real time whether they’re legitimate or drifting. Messages that match approved intent should be handled immediately, and only unexplained content needs to be isolated for further analysis.

A message entering our system takes one of two paths:

• A real-time classification path, which assigns it to a previously observed group with a known verdict, or
• A deferred discovery path, where it’s held aside to help form a new group.

And there you have it: the gist of the design. The rest of the post is about how we turn that into a production system.

The architecture

In practice, this is a multi-stage microservice pipeline. A message flows through it as follows:

1. Ingestion and reassembly. The orchestrator service ingests incoming traffic and reassembles multi-part messages into their original textual form.
2. Vectorization. To compare messages by intent rather than wording, we map them into a shared semantic space. A dedicated vectorization service takes a batch of messages and returns their embedding vectors. Messages expressing similar intent end up close together in this space, even when the wording differs. That way, grouping by proximity in the embedding space gives us grouping by semantics.
3. Search. The orchestrator queries an Elasticsearch index of known cluster representatives using the message’s embedding. If a match is found, the message is assigned to that cluster, and the cluster signature is promoted to a local cache, so subsequent traffic from the same sender resolves faster. This is the classification path in action.
4. Noise accumulation. If no match is found, the message takes the discovery path: it’s written to a noise index. The name comes from clustering terminology: technically, it’s an “unclassified” index, holding candidate data for future clusters.
5. Clustering. A scheduled job continuously pulls accumulated noise and runs clustering to discover new semantic groups. Clustering is performed on sender level. In practice, we’ve observed that for a sender, message volume is a great primary signal for parameter configuration, and after an obscene amount of per-sender experiments, we did manage to settle on a base of volume-keyed parameter sets. Although not as ubiquitously successful as maybe a best-of-n configs approach would have been, at our scale the compute savings make it the right call.
6. LLM-as-a-judge. Finally, cluster representatives are relayed to an LLM, which produces a verdict for the cluster, not for individual messages. The model also outputs per-message reasoning, which is what makes the verdict actionable downstream.

Walking the eyecare message through the pipeline

Let’s replay the message path from the start with this architecture in mind.

The message arrives at ingestion alongside other traffic from the same eyecare clinic sender. The orchestrator reassembles it and the vectorization service returns its embedding. The orchestrator queries Elasticsearch for known clusters associated with this sender, and let’s imagine that there are several: the standard birthday message, an annual check-up reminder, a glasses-ready-for-pickup notification.

The incoming embedding sits near the birthday cluster, but not near enough to match. The brand name swap and the shift from “Sam” to “Sammy” are subtle on paper, but they move the vector just far enough away.

No match. The message goes to the noise index.

Over the next clustering interval, more messages like it accumulate (you can be pretty sure they would, this reads like a classic sender takeover). The clustering job runs, a new cluster forms around them, and a handful of representatives are stored and passed to the LLM judge.

The judge compares the cluster against the sender’s approved campaign metadata and flags it for brand impersonation. (The changed recipient personalization remains unflagged: on its own, it’s a minor surface variation that doesn’t shift the underlying intent.) Reasoning is attached. A human reviewer picks it up, and the sender is paused before the next batch of spam goes out.

The message arrived through the same number, used the same structure, and would have passed any rule-based check. The only reason the system caught it was because the meaning had shifted.

Clustering is the leverage point

At 300 million messages a day, per-message LLM inference is a non-starter on cost and latency alone. It would also be wasteful, because most incoming traffic is heavily templated or duplicated. Grouping by intent first means we make one judgment instead of a million.

Embeddings group messages together, the cluster amortizes the cost of judging them, and the LLM is reserved for the moment judgment is actually necessary: when something new appears.

Two (in my opinion) interesting design decisions follow from this:

• First, we deliberately don’t let clusters evolve. A cluster represents a stable behavior snapshot, a fixed meaning at a point in time. Letting clusters expand would let them quietly absorb new variations in intent and language, which is exactly what we’re trying to detect. Keeping them fixed forces any notable change in semantics to surface as a new cluster, and makes downstream judgment more coherent because the examples within a cluster genuinely belong together.
• Second, this tightening of group intent allows us to represent each cluster by a few select examples rather than its full membership. This keeps the classification index lean for fast matching, and it gives the next stage, cluster judgment, a focused, reviewable input.

Tradeoffs we accepted

As no architecture is without its flaws, the most interesting one is overclustering.

The system inherits the limitations of its embedding model: embeddings capture intent well, but not every kind of shift. We expected greedy cluster expansion, but in practice the embedding space often treats benign surface variation (like a swapped greeting or different recipient name) as meaningfully different. Legitimate traffic from one campaign gets fragmented across many clusters, which inflates downstream cost and dilutes the signal we care about.

The lesson is that embeddings alone won’t always get us to clean intent groups.

The other two costs are more predictable:

• Discovery latency. The classification path is real-time, but the discovery path isn’t: messages that don’t match an existing cluster sit in the noise index until the next clustering job runs. For genuinely novel drift, this means a delay between first appearance and first judgment. We tune the noise accumulation period to keep this window short for senders above a certain message volume, assuming that low-volume senders will continue to grow with time and graduate to a size above the threshold.
• Cluster proliferation from non-evolving clusters. Forcing semantic shifts to surface as new clusters means a sender with legitimate variation across message templates ends up represented by many clusters. We manage this with cluster aging policies explicitly, but the lean cluster representations also help in that regard, so the memory cost remains bounded.

Model provides a recommendation, not a decision

A verdict on a cluster may end the pipeline, but the case is only closed when a human says it is.

When the LLM judge flags a cluster, the verdict and reasoning go to a human review queue. Reviewers see the cluster representatives, the approved campaign metadata, and the model’s reasoning side by side. They can confirm the verdict, override it, or adjust its severity, and their decision propagates downstream.

From there, action depends on severity: mild drift leads to client notification and guidance, more serious cases can block specific message types and trigger a root cause analysis, and the most severe, spam-grade drift can block senders entirely.

The important thing to note is that the model provides a recommendation, not a decision.

The scariest drift is the one that still looks trusted

A few things became clearer as we built this out than they were going in:

• The most interesting drift is quiet: a message that looks familiar, comes from the same number, but means something slightly different and slowly erodes trust before anyone notices.
• Judging clusters instead of individual messages changes the question. Is this message okay? is narrow and easy to answer wrong. What is this sender doing now? is the question that matters, and a group of similar intent is the right unit to ask it about.

What’s immediately next is concrete, on two fronts.

1. The first is resolving the overclustering issue described earlier: collapsing benign surface variation back into single clusters so the LLM judge isn’t doing redundant work.  Lightweight named entity recognition and a smidge of the old regex to capture emails or phone numbers should do the trick, we hope.
2. The second is scaling: the system currently runs in a subset of regions and handles on the order of tens of millions of messages a week, and the immediate work is extending that toward the 300-million-a-day ceiling it’s designed for.

The eyecare message at the start of this post is just a small example of a bigger problem. What we’re really trying to defend is the meaning of a trusted channel.

And for now, please don’t click weird links in your texts. Sam learned the hard way, and we’re not quite there yet on link-checking.

The post How We Catch Messaging Campaign Drift Before It Spreads appeared first on ShiftMag.