We were entering unfamiliar territory, different from our usual experience. This caused some initial uncertainty, but we recognized that adjustments were essential to move forward effectively.
Here’s the story of that project.
The product in question was our new product – Signals.
Its purpose was to detect and block artificially generated traffic, specifically SMS traffic sent through our platform but initiated by our clients’ end users. This primarily refers to 2FA messages. To give you some context, these are the messages containing verification codes that you, the end-user, request and receive via SMS.
We had some skeleton service as a baseline and a couple of initial ideas of where to start.
Our first task was to verify existing and develop new ways to detect such traffic. This quickly consumed our development time, resulting in hours of analyzing traffic graphs, querying databases, and reviewing in-house reports of suspicious traffic increases.
The success was moderate, but we felt like we were chasing our tails. With inconclusive results, it was hard for us as developers to implement something we weren’t sure would work.
We faced many open questions that were impossible to ignore.
Consider what you’d look for if someone were artificially inflating your SMS traffic and increasing your bill:
It became clear that we needed a new approach to tackle this problem effectively.
We decided to bring in a Data Science team to handle identifying traffic patterns. This allowed us to focus more on implementation, even though it still involved many uncertainties that we needed to address.
Data scientists excelled at deciphering requirements, automating data processing, analyzing statistics, and detecting anomalies—tasks beyond traditional developer roles.
Meanwhile, our operations team managed daily tasks and client interactions. This setup is crucial as our product expands, inevitably introducing greater complexity and unknowns.
So, returning to when we, the developers needed to decipher traffic patterns on our own – it was chaotic. We had unstructured blocks of text written from individual perspectives, yielding no actionable conclusions for implementation.
Thus, we devised a structured procedure that resembles the following:
This procedure involves rechecking results with the team at various stages, systematically reducing the uncertainty that plagued this stage of development. It also provides the flexibility to halt implementation at multiple checkpoints if needed.
Most importantly, this approach compelled us to move forward and built in the capability to revert pattern detection to testing phases if issues arose.
In our case, we conducted our investigation by engaging with our clients, who were highly cooperative and motivated to resolve the issue. To provide context, some reported annual losses they estimated at 60 million USD.
This collaboration yielded valuable insights and fresh perspectives on the problem. For example, we learned that attackers were using VPNs to shuffle IP addresses, thwarting attempts to correlate IPs with phone numbers and to be able to identify bad actors more accurately. Such information was instrumental and beyond what we could have gathered independently.
It’s always reassuring when your suspicions are confirmed, isn’t it? Even better when clients guide your development in the correct direction.
On the research front, our focus was predominantly technical – determining which methods, algorithms, or technologies would effectively detect fraudulent traffic.
Consider that we’re dealing with tens of thousands of messages per second, requiring processing that incorporates historical data analysis. This is a significant challenge. Therefore, it was essential for us to allocate sufficient time to validate the technical aspects of our solution.
If we lacked sufficient information to implement suspicious traffic pattern detection, we simply deferred it to the backlog. It may sound like basic Agile development, but it’s crucial to regularly revisit these tasks and actively break them down. Often, as we advanced, new insights shed light on old problems, allowing us to resume work on them.
Tasks like these are fundamental to the product and require heightened attention and frequent review.
And always keep your backlog lean!
Considering complexity versus benefits, some features simply aren’t worth pursuing. There’s nothing wrong with discarding such ideas—it’s far better than implementing something incorrectly and enduring its consequences later.
For instance, we encountered cases where intricate traffic analysis would only impact a minuscule fraction of mobile numbers, clearly falling into the “not worth it” category. However, most decisions aren’t as straightforward.
Always consider key metrics (request volume, current loads, user numbers, etc.). This enables you to swiftly reject unnecessary or resource-intensive features before they integrate into your system.
Don’t forget to create a proper Proof of Concept (PoC)—it may not necessarily make it to production but is crucial for testing ideas. Good component isolation ensures easy disabling or removal of these procedures if needed.
Think of PoCs as experimental branches of a tree – some may thrive, others may not. Develop them with the Pareto principle in mind: focus on the 20% of features critical for the PoC to function 80% of the time.. This approach verifies whether the concept is viable for production. You might use it in production briefly to gather and analyze data or find it needs rewriting for scalability.
Valuable concepts earn dedicated development time. Lack of allocated time can signal a feature’s lower priority or potential for removal.
However, PoCs shouldn’t replace thorough planning; they’re ideal for situations where uncertainty requires gathering data before making a decision.
Embracing probabilities was a shift for me, moving away from exact specifications to a world where uncertainties could still be valuable.
Not everything needs to be perfectly accurate to be beneficial.
While this advice might not directly apply to your project, the key is to remain open-minded. Recognize when you’re stuck in a mindset where every problem seems like a nail to be hammered. By stepping back, reflecting, and adjusting as necessary, you can navigate challenges more effectively.
And what were the results? Speaking from my own experience, I can attest that this approach worked well for us. However, as we understand, the success of any project is often influenced by various factors, some of which are beyond our control.
Ultimately, we achieved a profitable and moderately complex product with satisfied paying customers.
In summary, this article can be condensed into two key steps:
Understanding and solving a problem takes time. I recently came across an article that emphasized the importance of this – hence the 268% higher failure rates for Agile software projects (and no, I’m not advocating for waterfall methodology!).
Often, the rush from receiving a feature request to implementation is too brief for thorough execution.
The most successful methods allocate adequate time and effort to understanding and breaking down problems. This principle should be embraced by everyone, from project managers to developers.
This approach not only results in effective solutions but also fosters team growth and diverse experiences. It allows for self-organization, learning opportunities, and extensive PoC development and builds confidence as the product evolves.
2. Profit
Our efforts resulted in a reliable solution trusted by dozens of clients, including leading global companies. They depend on us to block suspicious traffic effectively and reduce costs—an immense responsibility given the direct impact on their traffic volume.
This trust has significantly increased our traffic, benefiting everyone involved.
The challenge isn’t the unknowns themselves – they’re always present. It’s about how you approach and systematically address them, uncovering unexpected benefits along the journey.
