You have kicked-off your annual application security assessment, but by the time the final report comes in, so have a bunch of new features from your developers. Since your pen test report can’t keep-up with your modern development cycles, it is now (and always) obsolete. You can check-off your compliance checkbox, but you’re not anymore secure than you were before. If this sounds familiar, it is clearly time for an update.

After a few hundred pen tests, I have come across my fair share of weird and wonderful bugs. If you’re interested in some proof of that, take a look at the following write-ups (shameless self-plug for previous blogs here): Account takeover vulnerability in Azure’s API Management Developer Portal and Using HTTP request smuggling to hijack a user’s session – exploit walkthrough.

However, no matter how much time I spend crafting an impactful exploit, and detailed recreation flow for these vulnerabilities, I see the same patterns and uncertainties around fixing these bugs. Over time, I’ve come to realize that this irritation is a pen testing process problem.

In the following post, I will share my perspective on the most common issues with both vulnerability scanning, and the traditional pen testing model, by comparing them with an alternative (more modern) process – Pen Testing as a Service (PTaaS).

What is PTaaS?

The name alone implies PTaaS is cloud-native and entails an element of automation – both desirable traits for an agile AppSec program. PTaaS is the delivery of on-demand manual pen testing and continuous security monitoring to keep your agile applications secure – no matter how often your production code changes.

Vulnerability scanning vs PTaaS

Let’s start with how vulnerability scanning compares against PTaaS. While vulnerability scanners can be useful for some use-cases, they are limited in their ability to detect novel attack vectors. Interestingly in 2023, more than 20% of all reported vulnerabilities from Outpost24’s PTaaS platform were classified as high or critical severity. In 2022, that number was around 14%. These findings are produced manually through our in-house team of pen testers and represent the blind spots that automated scanners have routinely missed. They also highlight the increase in high-severity vulnerabilities that come alongside the added complexity of modern web applications.

In a recent engagement, a customer who previously solely relied on automated testing was particularly surprised when our PTaaS team reported a large number of critical findings related to privilege escalation and even remote code execution. For more information about the limitations of vulnerability scanning, check out this post from a fellow pen tester: Broken access control and why vulnerability scanners can’t detect them.

Traditional pen testing vs PTaaS

In other engagements with customers who have made the switch from traditional pen tests to PTaaS, the following key benefits are often highlighted to me.

Benefit 1 – No more waiting around for the final report

The timeline for a traditional pen test has long been a barrier to produce valuable results. By the time the onboarding process is completed, the testing is bound to take place on soon-to-be outdated code. This is a frustrating experience for developers who are likely to have made changes that are relevant to the findings and creates uncertainty on the validity of any given vulnerability in the report.

PTaaS takes a much faster approach by allowing testers to report vulnerabilities in real-time to customers in an easy-to-use interface. To further complement this, Outpost24 takes a hybrid approach to PTaaS with manual and automated (manually verified) results. This means that customers receive their first findings on day one of any given assessment. Our interface also enables developers by notifying them the moment a new vulnerability report is created, helping them to properly prioritize code changes during their development cycles, before it is too late.

Benefit 2 – Remediations are validated immediately

Traditional pen testing will usually result in a static PDF report at the end of the test cycle. Even if a report is received relatively quickly, and the developers are able to make changes to address the reported issues, it may be months before a new test and report can be created to verify the results of these changes.

Given the advantages of PTaaS’ rapid reporting methods, developers are able to integrate their remediation process with the testers. They can request additional tests to confirm if an implemented fix is working, and to determine if the fix is robust enough to protect against any possible bypasses. At Outpost24, this is all possible in real-time commonly resulting in fixes being applied and verified within days, all while further testing is conducted by the PTaaS teams.

Benefit 3 – Visibility and updates during the testing process

Another issue present in the traditional pen testing process is the lack of visibility during a test. While customers will be aware of when a test has started, they often have no visibility of their findings, or any issues that may have been raised during the test.

In stark contrast, another advantage of PTaaS’ real-time reporting is the ability to instantly and continuously have visibility of the entire testing process. Once an application is onboarded, customers will know instantly when testing will begin/end. They can also view and interact with all of the reported findings, as well as any issues that have been spotted from testing, such as any broken functionality, or testing credentials that are not working.

Benefit 4 – Direct communication with the pen testers

A common theme in traditional pen testing is a rather boxed-in approach, where the testers themselves are not available for questions regarding findings and deployed fixes. This can be frustrating for customers.

One of the numerous advantages of PTaaS is its ability to allow customers to open a direct communication channel between their developers and the pen testers. This allows the developers and testers to actively discuss vulnerabilities and deployed fixes during the engagement, often resulting in a back-and-forth style approach to remediation where the developer’s initial fixes are re-tested to ensure robustness, and then improved upon with the help of the tester’s expertise.

Benefit 5 – Encouraged creative testing

An unavoidable aspect of all pen testing is checklist driven testing. This is required to ensure full coverage of the application’s features, as well the possible vulnerabilities that may affect each feature. However, traditional pen testers can often focus too much on hitting every item on the checklist, rather than digging into an aspect of the list that looks to be critically significant to the application. Outpost24’s testers are taught early on in their training process to manage their time carefully to allow for creative and in-depth analysis of critical context-dependant features to ensure that vulnerabilities with highest potential impact to the customer’s application are identified every time. This is still combined inevitably with the checklist driven testing, to produce wide reaching coverage of the application.

Benefit 6 – Large pool of testers

If a customer has been using a single vendor for some time, they will often quickly discover that the same testers are being assigned to their applications as they are familiar with its functionality from previous tests. While this can be a good methodology, customers often want a fresh perspective on their applications, and will end up requesting new testers, or in some cases, bringing in a second vendor to ensure a variety of testers can review their critical applications. This can be extremely costly.

At Outpost24, our PTaaS solution exposes a large pool of testers to all applications over the course of their engagement. This means that applications will regularly be rotated through new testers, each of which has a unique skillset that allows them to take on a new perspective of the application. This in turn leads to more creative and impactful vulnerabilities and ensures the application’s security posture is evaluated by multiple experienced testers.

Getting started with PTaaS

Outpost24’s PTaaS solution, SWAT, gives you the most accurate view of your AppSec vulnerabilities. By combining the depth and precision of manual pen testing with continuous vulnerability scanning, we help business secure web applications at scale. For more information, you can check out our the SWAT product page, or speak with an expert.