“Developers should own security. And test in production.”

Not long ago application and API security relied solely on manual methods like knights guarding a castle.

Now, with Agile and DevOps leading the charge and orchestrated ephemeral infrastructure as our new arsenal, we’ve upgraded to a tech-savvy defense, ensuring efficiency and effectiveness like never before.

In this interview, we talk about it with Larry Maccherone, Dev[Sec]Ops Transformation Architect at Contrast Security.

There are almost no dedicated QA departments anymore

“Back in the early days of Agile, I used to advocate for cross-functional teams owning software products, including quality. After these talks, QA professionals would often express concern about their siloed roles, fearing changes if development teams took over quality responsibility, often commenting, “What you said sounds idealistic. It’ll never work at my organization. The developers will just put out crap.”

A decade later, there are almost no dedicated QA departments, and the ones that are left are more of a body shop with all of their people on semi-permanent ‘loan’ to specific dev teams.

Also, we have to remember that back then, developers didn’t do QA the way QA people did QA. “They tried to eliminate all manual testing and replace it with automated testing.”

The DevOps and cloud-native movements were similarly disruptive to traditional IT Ops and were also dominated by automation. Security is now well into a similar tradition which is also marked by more tools deeply integrated into the process.

Shifting the responsibility for security to dev teams

We also asked Larry about the challenges that the industry faces when considering application and API security as a siloed function.

“Siloed functions tend to result in local optimization rather than holistic systems thinking. In systems thinking (referred to as “flow” as the first of three “ways” of DevOps), you try to optimize for value delivery of the entire software development system”, Larry claims.

To get more concrete, Larry says that in a silo, security folks find the vulnerabilities, but they can’t fix them themselves. That leaves them in the position of essentially begging the developers to fix them.

“In this same siloed example, the development teams are measured on how many features they ship. Security is responsible for security. So, the local optimization is to ignore the requests from security to fix the vulnerabilities they found.”

On the other hand, if you shift responsibility for security to the development team, they can optimize for the best overall outcome. Sometimes that means, they can make tradeoffs. At times they might legitimately decide that the risk of not fixing a vulnerability is outweighed by the risk of missing a market opportunity. At other times, they will decide that the security vulnerability fix is the most valuable way to spend their limited resources.

Also, according to Maccherone, developers are prone to automate things, which makes them more efficient overall.

Developers just want their code to be merged

A particular practice that is now universally used by development teams is the pull/merge request. It is the very center of how development teams coordinate their work.

It’s the process of a developer’s morning work reaching the shared codebase by afternoon and ultimately going into production. Almost all automated quality and security checks can (and should) be run prior to the decision to merge that code.

The person clicking on the “merge” button is generally a teammate of the developer for that pull/merge request.

This pull/merge request reviewer, will look at the changes but will also be informed by the automated quality and security checks that were run prior to their review.
Developers are passionate about the code they wrote and want nothing more than for it to be merged. They will see any negative feedback from a security check and respond to it immediately.

Done this way, the vulnerability is often resolved on the same day that it was injected. At this point, the code is very familiar to the developer, so the cost of fixing is an order of magnitude lower, especially when you consider that there was no vulnerability inventory carrying costs.

The situation Larry described above is only possible if each pull/merge request has its own isolated infrastructure for the automated quality and security checks to run.

With traditional methods, having five developers on one team would require maintaining five separate test environments, costly and often idle. Yet, with container-based ephemeral infrastructure, a single developer’s request can spawn multiple workers to conduct quality and security checks simultaneously, completing tasks rapidly and freeing up resources for others.

Technology, practices, and culture

And what about a future Larry sees emerging for application and API security testing?

He emphasized that significant industry transformations, such as the DevOps movement, hinge on three key components: technology, practices, and culture.

In DevOps, ephemeral build and test infrastructure alongside cloud-native technologies represent the technological shifts, while adopting the pull/merge request signifies a change in practices. Culturally, the pivotal shift involves transferring operational responsibilities to the development team.

So, what is the coming revolutionary change for Application and API Security?

It follows the revolution pattern for DevOps, namely those three things – technology, practices, and culture, but it also builds upon the changes that DevOps has brought.

However, before we jump into it, we are going to take a little detour to one discipline that fundamentally changed along with the DevOps movement: performance testing.

Performance testing in production: cheaper, faster, and safer

“Performance testing was long considered a necessary evil. It was expensive and inaccurate, but without it, you ran the risk of having an outage or at least annoyed customers when your application appeared non-responsive.”

Creating a test environment similar to production was necessary, but the closer it mirrored reality, the higher the costs, potentially doubling production expenses.

Then, running extensive automated test scripts in the performance environment incurred high costs, but the real issue was their reliance on artificial traffic, which would lead to inaccurate predictions of real user performance.

“Because of these difficulties, many didn’t even bother with performance testing and took the risk of releasing without it,” Larry says.

Then came technology, practice, and cultural transformations that disrupted the performance testing landscape and avoided all the previous problems. Developers started doing performance testing in production.

The enabling technological change came in the form of application performance monitoring agents that were efficient enough to run in production. The practice change was canary deployments. The cultural shift was realizing that doing performance testing in production was cheaper, faster, and safer than doing it in pre-production when we previously believed you must trade these factors off against each other.

Test your API security in production too

At this point, you might be thinking, “What does this performance testing diversion have to do with the future of application and API security testing?”

Well, what if we can safely accomplish app and API security testing in production?

Doing so avoids the downsides of the current methods:

• SAST’s inaccuracy and long scan times
• DAST and IAST’s poor coverage
• SCA’s lack of context
• WAF’s ineffectiveness

“Like the performance testing of yesteryear, these downsides, along with the cost of licensing, rolling out, and maintaining all of these disparate tools, cause many to skip some or all of them and risk getting hacked. However, to achieve a revolution like this, we’ll need all three elements of the other revolutions: technology, practices, and culture,” Larry emphasizes.

The technological innovation we need is the same as the one we need for the performance testing revolution – agents that are safe and efficient enough to run in production. We’ve been making steady improvements to our IAST agent for years such that it’s an order of magnitude more efficient than it was a few years ago, but we recently beta-released new agents that take a fundamentally different approach when running in production environments, which have improved the efficiency by another order of magnitude.

The practice change is that we need those agents installed by engineering teams in production environments. The cultural transformation is similar to the performance testing one—the conviction that doing application and API security testing in production is cheaper, faster, and more secure than doing it in pre-production.

So, Larry believes we are on the cusp of a revolution in app and API security testing like the one that occurred for performance testing.