We were entering unfamiliar territory, different from our usual experience. This caused some initial uncertainty, but we recognized that adjustments were essential to move forward effectively.

Here’s the story of that project.

This is how Infobip Signals was born 

The product in question was our new product – Signals.  

Its purpose was to detect and block artificially generated traffic, specifically SMS traffic sent through our platform but initiated by our clients’ end users. This primarily refers to 2FA messages. To give you some context, these are the messages containing verification codes that you, the end-user, request and receive via SMS.  

We had some skeleton service as a baseline and a couple of initial ideas of where to start.  

Our first task was to verify existing and develop new ways to detect such traffic. This quickly consumed our development time, resulting in hours of analyzing traffic graphs, querying databases, and reviewing in-house reports of suspicious traffic increases.  

The success was moderate, but we felt like we were chasing our tails. With inconclusive results, it was hard for us as developers to implement something we weren’t sure would work. 

We faced many open questions that were impossible to ignore. 

Consider what you’d look for if someone were artificially inflating your SMS traffic and increasing your bill: 

• Who is responsible? 

• How are they doing it? 

• Are they using multiple phones or virtual numbers? 

• Is it automated or human-operated? 

• What are the volumetric patterns in the traffic to watch for?  

Let’s bring in the Data Scientists 

It became clear that we needed a new approach to tackle this problem effectively. 

We decided to bring in a Data Science team to handle identifying traffic patterns. This allowed us to focus more on implementation, even though it still involved many uncertainties that we needed to address. 

Data scientists excelled at deciphering requirements, automating data processing, analyzing statistics, and detecting anomalies—tasks beyond traditional developer roles. 

Meanwhile, our operations team managed daily tasks and client interactions. This setup is crucial as our product expands, inevitably introducing greater complexity and unknowns. 

Have the procedures in place

So, returning to when we, the developers needed to decipher traffic patterns on our own – it was chaotic. We had unstructured blocks of text written from individual perspectives, yielding no actionable conclusions for implementation. 

Thus, we devised a structured procedure that resembles the following: 

This procedure involves rechecking results with the team at various stages, systematically reducing the uncertainty that plagued this stage of development. It also provides the flexibility to halt implementation at multiple checkpoints if needed. 

Most importantly, this approach compelled us to move forward and built in the capability to revert pattern detection to testing phases if issues arose. 

Getting our hands dirty 

In our case, we conducted our investigation by engaging with our clients, who were highly cooperative and motivated to resolve the issue. To provide context, some reported annual losses they estimated at 60 million USD.  

This collaboration yielded valuable insights and fresh perspectives on the problem. For example, we learned that attackers were using VPNs to shuffle IP addresses, thwarting attempts to correlate IPs with phone numbers and to be able to identify bad actors more accurately. Such information was instrumental and beyond what we could have gathered independently. 

It’s always reassuring when your suspicions are confirmed, isn’t it? Even better when clients guide your development in the correct direction. 

On the research front, our focus was predominantly technical – determining which methods, algorithms, or technologies would effectively detect fraudulent traffic. 

Consider that we’re dealing with tens of thousands of messages per second, requiring processing that incorporates historical data analysis. This is a significant challenge. Therefore, it was essential for us to allocate sufficient time to validate the technical aspects of our solution. 

Don’t forget to backlog!

If we lacked sufficient information to implement suspicious traffic pattern detection, we simply deferred it to the backlog. It may sound like basic Agile development, but it’s crucial to regularly revisit these tasks and actively break them down. Often, as we advanced, new insights shed light on old problems, allowing us to resume work on them. 

Tasks like these are fundamental to the product and require heightened attention and frequent review. 

And always keep your backlog lean! 

Measure twice and cut once

Considering complexity versus benefits, some features simply aren’t worth pursuing. There’s nothing wrong with discarding such ideas—it’s far better than implementing something incorrectly and enduring its consequences later. 

For instance, we encountered cases where intricate traffic analysis would only impact a minuscule fraction of mobile numbers, clearly falling into the “not worth it” category. However, most decisions aren’t as straightforward. 

Always consider key metrics (request volume, current loads, user numbers, etc.). This enables you to swiftly reject unnecessary or resource-intensive features before they integrate into your system.  

Fast cycle PoCs

Don’t forget to create a proper Proof of Concept (PoC)—it may not necessarily make it to production but is crucial for testing ideas. Good component isolation ensures easy disabling or removal of these procedures if needed. 

Think of PoCs as experimental branches of a tree – some may thrive, others may not. Develop them with the Pareto principle in mind: focus on the 20% of features critical for the PoC to function 80% of the time.. This approach verifies whether the concept is viable for production. You might use it in production briefly to gather and analyze data or find it needs rewriting for scalability. 

Valuable concepts earn dedicated development time. Lack of allocated time can signal a feature’s lower priority or potential for removal. 

However, PoCs shouldn’t replace thorough planning; they’re ideal for situations where uncertainty requires gathering data before making a decision. 

Remember, it doesn’t have to be perfect

Embracing probabilities was a shift for me, moving away from exact specifications to a world where uncertainties could still be valuable. 

Not everything needs to be perfectly accurate to be beneficial. 

While this advice might not directly apply to your project, the key is to remain open-minded. Recognize when you’re stuck in a mindset where every problem seems like a nail to be hammered. By stepping back, reflecting, and adjusting as necessary, you can navigate challenges more effectively. 

Reaping rewards

And what were the results? Speaking from my own experience, I can attest that this approach worked well for us. However, as we understand, the success of any project is often influenced by various factors, some of which are beyond our control. 

Ultimately, we achieved a profitable and moderately complex product with satisfied paying customers. 

In summary, this article can be condensed into two key steps: 

1. Let the team grow

Understanding and solving a problem takes time. I recently came across an article that emphasized the importance of this – hence the 268% higher failure rates for Agile software projects (and no, I’m not advocating for waterfall methodology!). 

Often, the rush from receiving a feature request to implementation is too brief for thorough execution. 

The most successful methods allocate adequate time and effort to understanding and breaking down problems. This principle should be embraced by everyone, from project managers to developers. 

This approach not only results in effective solutions but also fosters team growth and diverse experiences. It allows for self-organization, learning opportunities, and extensive PoC development and builds confidence as the product evolves. 

2. Profit

Our efforts resulted in a reliable solution trusted by dozens of clients, including leading global companies. They depend on us to block suspicious traffic effectively and reduce costs—an immense responsibility given the direct impact on their traffic volume. 

This trust has significantly increased our traffic, benefiting everyone involved. 

The challenge isn’t the unknowns themselves – they’re always present. It’s about how you approach and systematically address them, uncovering unexpected benefits along the journey. 

The post How to handle any mystery project that falls in your lap appeared first on ShiftMag.

Houston, we have a problem!

What the hell is going on?! Alerts started popping up everywhere, multiple services crashed and mass confusion arose between colleagues trying to pinpoint the center of the problem.

This does not happen very often, and when it does, it usually involves a bigger problem that is out of our reach (Kafka problems, datacenter issues, etc.) and has nothing to do with our code, but this was not the case.

Let me present you the feature

Our feature allowed our customers to import a large number of mobile numbers and whitelist them, ensuring that the SMS that is sent to those numbers is not blocked at any moment, even if our analysis detected them as fraudulent. This ensured that traffic sent to those mobile numbers was always sent. We call these “Trusted MSISDNs” 

The problem was how we distributed those numbers to other services, which needed to be aware of this information. 

Our architecture wasn’t ready

Since we are working with microservice architecture, we are using a configuration service to send configuration updates to other services using HTTP(RMI). The configuration service also received information about Trusted MSISDNs, stored them in the database, and then sent them as part of the configuration file to other services. This is where the problem started. 

We did not expect that a huge amount of Trusted MSISDNs would become a burden to our configuration file: 

1. Configuration files became much larger – resulting in HTTP timeouts   

2. Serialization and De-serialization took more time and used much more RAM 

3. Configurations were stored in memory and updated often, resulting in being too big for our virtual machine 

As a result, when services tried to fetch configuration, they either had timeouts all the time, making them work with non-updated configuration, or they fetched configuration and crashed because not enough RAM was added to the service to handle de-serialization.  

You cannot think of everything

This functionality was developed first as a proof of concept, and eventually, it was improved to the point where we used it in production. We followed the architecture we already had, and it worked well without apparent problems. 

Millions of numbers were imported, the UI tested, and the feature was in production for many weeks. Sometimes, you can’t think of everything, and in our case, the estimation of imported numbers was not big enough—and we thought big.

We recognized that as our product evolves, our architecture must also adapt to accommodate our expanding customer base. This functionality received more attention than we initially thought, and we needed to change our approach.

Solving HTTP issues and optimizing future features

We have moved from RMI and started using Kafka to resolve our HTTP(RMI) issues, and the in-memory configuration was switched to Redis as a fast-responding database. 

The problem that occurred opened our eyes and made us take the first step in changing how we handled configurations. It also changed how we think about future features, where we already took different approaches and where a huge amount of data does not represent a problem.

This alert showed us a potentially big problem over time that could be much harder to handle if it came all at once. It was time for a change.

Crisis – > Opportunity

Problems sometimes lead us to resolve more than one issue and tackle core problems. However, due to our busy schedules, these issues never get the spotlight. 

Sometimes, unforeseen challenges can drive significant improvements, transforming crises into opportunities. You never know. Maybe the problem you have today will make your life easier tomorrow.  

Embrace problems and good luck!

Find out more about how to combat fake OTP traffic.

The post From a feature to a problem – from a problem to an opportunity appeared first on ShiftMag.