[Editor’s note: Here is an article I wrote for PenTest Magazine on how penetration testers can structure their work and its results to provide a lot more value to target organizations. We’ve used these principles on penetration tests and ethical hacking engagements in companies where I’ve worked with really positive impact, and I hope you find the concepts useful, or at least thought provoking. The whole goal of these recommendations is to avoid the death spiral of the Really Crappy Penetration Test, a plague on our industry described in more detail in the article. Have fun! –Ed.]
Maximizing Value in Penetration Testing and Ethical Hacking
By Ed Skoudis
SANS Institute Penetration Testing Curriculum Lead
Founder, Counter Hack Challenges
The penetration testing business faces a great danger as more and more people jump into the field offering very low-value penetration tests that are little better than an automated vulnerability scan. In this article, we’ll discuss how to conduct your tests and write up results so that they can provide significant business value to the target organization. If you are an in-house penetration tester in an enterprise, providing more business value through your work can help improve your job security in a tumultuous economy, and, better yet, may help you land that fat raise you’ve been hoping for. If you are a third-party penetration tester, providing more business value can lead your career to the point where you will command a higher bill rate. What’s not to like?
I read a lot of penetration testing reports. In my work as an expert witness analyzing large-scale breaches, I’m regularly called upon to look at the previous five years of penetration testing and vulnerability assessment reports of a large number of companies. Also, in my own pen testing work with my team, I review many of my team’s reports, as well as take a critical eye to my own reporting output, always with the goal of making our results better and more meaningful. In any given week, I read between two and five pen testing reports, and I spend a lot of time thinking about their effectiveness.
And, I’ve got to tell you, a lot of penetration testers generate absolutely horrible reports. Some of them are little more than regurgitated vulnerability scanning results, all packaged up and labeled as “Penetration Test Results.” Admittedly, some organizations desire what I like to call the “RCPT”, the Really Crappy Penetration Test. That is, they want to procure a test so that they can check off a compliance box saying that they got a penetration test, but the last thing they want is test results that make an effective argument for changing things in their environment.
Although there is, sadly, a distinct market segment of enterprises that desire the RCPT, other organizations demand more business value for the penetration testing expenditures, as they should. As a penetration tester, yes, you could take the easy way and deliver low-quality results from low-quality tests, catering to the RCPT market. But, I’m hoping you’ll strive to do better. I strongly believe that it’s in all of our best interest to do so. If the RCPT comes to dominate and tarnish the definition of a penetration test, we’ll all be worse off. Fewer organizations will want to employ us for the high-quality work we all love to do.
The folks working on the Penetration Testing Execution Standard (PTES) have done some fantastic work in defining procedures for conducting thorough, high-value penetration tests, and I celebrate their work. What I’d like to focus on in this article, however, is tips for helping to maximize the business value of your penetration testing results, especially in the report itself. Look, most penetration testers can scan and exploit a target environment. But what really differentiates the best of the best from the merely good is the ability to provide value and drive change that helps an organization improve its security stance. That has to be our relentless focus, as we strive to avoid the pit of the RCPT.
Tip #1: Keeping the Main Thing the Main Thing: It’s Not All About Shell or Even Domain Admin… It’s Really About Business Risk
As penetration testers, our hearts dance when we pop a target box, getting that much-coveted shell access to the machine. You know it and I know it. But please realize that merely compromising machines actually isn’t the ultimate goal of your work. It’s a means to an end. The end is to determine the business risk the organization faces in association with the vulnerabilities you’ve discovered. As you conduct a test, and especially as you prepare the report, make sure you always keep the main goal in mind: to determine, demonstrate, and explain the risk to the business, as well as methods for mitigating that risk.
One item in which some penetration testers fall short in determining business risk involves a view of a target environment as just a group of individual machines. Once they’ve gotten shell on one of them, such pen testers figure that they have a “high-risk” finding, and they call it a day. The real bad guys don’t do it that way. That initial compromise is the toe in the door, and they view the entire group of machines and the network itself as their target. The real bad guys, whose work we need to mimic to understand business risk properly, pivot mercilessly, bouncing from that initial compromised machine to other machines in the target environment.
Pivoting through a target, some penetration testers set their sites on seemingly very juicy prey: Domain Admin rights in a Windows environment. But, honestly, even that very important level of access is still a means to the end of demonstrating business risk. Decision makers in management of the target organization likely will not understand the risks they face if their penetration testers tell them that an attacker can conquer shell on a machine or even Domain Admin rights on their Windows environment. The penetration tester who can show the implications of this access, such as the ability to access millions of sensitive healthcare records or control systems that contain vital trade secrets, will provide so much more value.
Joshua “Jabra” Abraham has written convincingly about “goal-oriented” penetration testing, in which a penetration tester focuses on achieving certain goals beyond discovering vulnerabilities in a target environment. Abraham cites goals such as remotely gaining internal system access, gaining Domain Admin access, and gaining access to credit card information. I strongly support the idea of goal-oriented testing, and urge penetration testers to work with target system personnel to define their goals in terms of business issues (not just technical achievements) that are important to the target organization.
When initially scoping a penetration test, make sure you ask target system personnel what their most important information and processing assets are, and what their nightmare scenarios for computer attacks might be. Sometimes, you may have to stretch their minds a little bit about what an attacker could actually do. Have an open and honest discussion about the possibility of economic loss (due to down time, stolen money, diminished competitive advantage through stolen trade secrets, etc.), regulatory and compliance oversight (if a breach were to occur and government investigators were to come a-calling), lawsuit possibilities from customers or business partners, brand/reputation tarnishment, and physical threat to life and limb. In a frank discussion about these points, I often ask target system personnel, “What keeps you awake at nights in terms of computer attacks?” This isn’t about spreading Fear, Uncertainty, and Doubt, the lame FUD used to scare people into better security practices. Instead, this is about an honest view of security risks and how a penetration test can help determine how realistic those risks are.
For example, I was once discussing with a manufacturing company their biggest worries about what an attacker could do in compromising their computing infrastructure. They were focused on whether a bad guy could deface their website. I asked them whether they thought about an attacker who got access to their internal environment and stole their sales contacts, swiped their future product plans, or gained control of their manufacturing equipment controls causing it to malfunction or shut down. “Are those things possible?” they asked. “Let’s structure a penetration test so we can carefully see if they are,” I responded, as we set more business-centric goals for the test.
Figure 1: Pen Tester C Has Defined Business-Centric Goals that Go Beyond Shell and Domain Admin
Consider the three penetration tests illustrated in Figure 1. In the first test (indicated by Pen Tester A with green text and arrows), the penetration tester gets shell access on a target machine and reports that a critical exploitable vulnerability was discovered, but stops there. In the second test, Pen Tester B (whose work is illustrated by text and arrow B in blue) has gone deeper than the first tester, pivoting after exploiting the initial flaw, by dumping hashes, conducting a pass-the-hash attack, and gaining access to a machine with a Domain Administrator token on it. This tester then seizes the Domain Admin token, and writes up the results in a report, claiming victory due to Domain Admin compromise. Pen Tester B has certainly demonstrated some risks associated with the original flaw better than the first pen tester. But, it isn’t until we get to the third penetration tester, Pen Tester C, shown with red arrows and text, who continues pivoting even after gaining Domain Admin privileges, getting access to a machine with highly sensitive trade secrets. This third penetration tester will be able to best express the risk the organization faces due to the collective flaws in its environment, and make the best argument to management for action. Note that not only does Pen Tester C pivot more than A or B, but Pen Tester C is also focused on showing business risk by gaining access to sensitive trade secrets instead of just technical dominance of the target environment.
Tip #2: Remember Who Your Primary Audience Is… Not Other Pen Testers
Many really skilled penetration testers write their reports so that they will impress people like themselves, that is, other penetration testers. I am often tempted to do this myself, as I get into a mindset of “I want to knock the socks off of other penetration testers with the amazing work I did here, so I’m going to describe it all in terms that pen testers will understand and get excited about.” While the temptation is understandable, it should be avoided. Impressing other penetration testers shouldn’t be the real goal of our reports, as they aren’t the audience that will allow us to provide the most business value in our reports.
Who is? For your executive summary, decision makers are. These people can allocate resources to help alleviate the issues you’ve discovered if you can make a convincing business-centered point to them. The remainder of your deliverable, however, should be written with an eye toward providing maximum value to the enterprise security professional and the operations team. Phrase your discussion and recommendations so that they help security people and system administrators implement your recommended fixes. How? That’s what Tips number 3 and 4 are all about.
Tip #3: Provide the “How-To” In Your Recommendations
In your recommendations for remediation, don’t just describe at a high-level the changes that need to be made, but instead, include a practical step-by-step description of how to implement your recommended change. Provide command-line or GUI screenshot examples that show how to make your recommended changes.
Consider a straight-forward sample finding that many network penetration testers encounter, illustrated in Figure 2. Suppose the target organization has internal Windows machines that support NTLMv1, an older, weaker form of Windows authentication. A fairly low-value finding would involve copying and pasting the result from a vulnerability scanner recommending that the target organization move to NTLMv2. But how?
Figure 2: Different Styles of Recommendation Carry Different Levels of Business Value (and Risk of Something Going Horribly Wrong)
A bit higher-value finding would tell the target organization to set the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\LSA\LMCompatibilityLevel registry key value to 5 for servers. Such a recommendation is certainly better than just saying to move to NTLMv2, but it still leaves open some questions of how target environment personnel can do this.
A higher-value finding might include information about running a command such as the following on the discovered impacted servers, which will alter the given setting to the recommended value:
C:\> reg add hklm\system\currentcontrolset\control \lsa /v lmcompatibilitylevel /t REG_DWORD /d 0x5
To provide even more value, you can include a walk-through of how to implement this finding using Group Policy and then apply it to an entire Windows environment. The bottom line here is to always look at your recommendations, and see how well you’ve answered the question, “But how?”
I know what you are thinking. At this point, you are likely concerned that the more detailed you get with your recommendation, the more risk that target system personnel will blindly follow it, potentially wreaking havoc in a production environment. This concern is quite valid, and must be managed in the report itself. That’s why I like to include language with every single finding that says:
“These changes are based on their applicability to numerous environments, but could have unknown consequences in this particular environment. For that reason, any recommended changes should be evaluated in a test environment first, and then rolled out through proper, formal change control processes. If you do not test these configurations in an experimental environment, they could result in downtime or other damage to a production environment.”
I like to put this text in bold face font and italicize it to emphasize its importance. I include it with every finding that requires a change of configuration. Why every finding? In many enterprises, a penetration test report is split up among multiple groups or individuals, with each group assigned tasks to fix a subset of findings and receiving only the pages corresponding to their actions. If you include that text with only one finding, another group may get a separate part of the report, and not see the vital caveat you included in another section of the report. Does this get redundant? Yes, but that redundancy is the price of reducing risk.
Tip #4: Provide the “How-To-Check-Remediation” In Your Recommendations
Now, here’s a real gem that can help differentiate your penetration testing results and significantly increase their value. In your recommendations for remediation, include not only the “how-to” for implementing the fix, but also include a description of how the organization can verify that the remediation is in place. That way, they can have some level of assurance that the fix is working. The “how-to-check” description may be prose, but I like to go further, providing one or more command lines, GUI screenshots, or tool configurations that do the job of verifying the efficacy of the remediation. You should write such recommendations so that they can be carried out by a skilled security professional or a very knowledgeable system administrator, but don’t write them in a manner that only other penetration testers would be able to perform your recommended actions.
For some findings, including a checking step is trivially easy. Consider the NTLMv2 recommendation discussed earlier. You could add the following to that recommendation, significantly improving its value:
“You can check this setting by running the following command:
C:\> reg query hklm\system\currentcontrolset \control\lsa /v lmcompatibilitylevel
You should verify that its output is 5, an indication that the system is configured to use NTLMv2.”
For small tweaks to configuration or the application of various patches, Windows commands such as reg, “wmic qfe”, and “wmic product” are especially helpful. On Linux, you’ll often rely on cat, grep, rpm, and running a program with the –version option as a check.
For more complex recommendations, crafting a checking step that is suitable for non-penetration testers can be much more of a challenge. For example, writing a procedure to test whether Cross-Site Scripting (XSS) defenses have been implemented at first seems very difficult. If you suggest that they try to enter certain specific test XSS strings to evaluate their newly implemented filters, it is quite possible that the filters remove only the specific test strings you’ve provided! The organization would then have a false sense of security, as other XSS strings would still work against the target application. That’s why I’ll sometimes craft my verification process around the running of a given tool with a specific configuration. So, for XSS, I’ll suggest that the organization run a particular free XSS scanning tool that I know will put the application through its paces and give a reasonably good read on whether they have defended against XSS more comprehensively than by just filtering a few test strings.
When I first proposed adding these checking recommendations to our reports, some folks at the penetration testing company where I worked protested, saying that this will lengthen the report writing time and drive up our costs. But, I’ve found that adding this extra information really only requires a few minutes for each recommendation, and lends itself to templatization. It may mean that your reports take ten percent longer to write, but their value to target system personnel will be significantly greater.
At first blush, third-party penetration testers who do assessment projects for other enterprises may think that this recommendation will cost them future remediation verification work. That is, if you tell your customers how to check their own remediation in your report, they’ll be less likely to come back to you for a retest to verify their fixes. While that is certainly true, quite honestly, retest work focused on verifying fixes tends to not be terribly interesting, nor financially lucrative. I’d rather provide as much value up front as I can, with the knowledge that I’m helping to cement the customer relationship for their next real penetration test.
Tip #5: Prioritize Your Findings Carefully According to Impact and Probability of Exploitation
The vast majority of penetration testing reports that I read prioritize finding based solely on whether the issue is high, medium, or low risk. While such rankings do provide a broad signal to decision makers and technical personnel about where they should focus their remediation activities first (high-risk items), the so-called “HML” (High-Medium-Low) mechanism often lacks the granularity many organizations need for prioritization with the high-risk category itself. That’s why I recommend categorizing risks according to both their potential impact as well as their probability of being successfully exploited. That way, organizations can get a better feel for the risk factors and focus their efforts on items that are simultaneously high impact and rather likely to be exploited.
Of course, there are far more complex methods for assigning risk levels to discovered flaws, such as the Common Vulnerability Scoring System (CVSS) developed by FIRST. While CVSS is an excellent method for detailed analysis of flaws, some penetration testers find that its complexity and precision make it difficult or costly to use in routine penetration testing. I’ve found that categorizing issues according to impact and probability to be a happy medium between the too-simple HML approach and the more complex CVSS scheme.
In your executive summary at the start of a penetration testing report, it can be useful to provide a graphical summary of discovered issues according to their relative importance to the organization. For HML-style findings, many penetration testers just cut and paste a bar chart showing the relative count of high, medium, and low-risk issues discovered, which doesn’t really convey that much information or value, as shown in Figure 3.
Figure 3: A Traditional Bar Chart Used with the HML Model Doesn’t Convey Very Much Information or Business Value
Going beyond the simple bar chart, our team has had a lot of success in showing a graphical summary of discovered issues based on impact and probability of successful exploitation using a multi-dimensional graph, such as that shown in Figure 4. Here, we have a matrix with the probability of successful exploitation running along the X-axis, and the potential impact going up the Y-axis, with a relative ranking of 1 to 5. Note that we indicate the relative number of issues discovered at each intersection by including a circle whose area corresponds to the number of findings there. A bigger circle indicates that the pen test team identified more instances of this kind of finding. We have had several customers tell us that this kind of chart provides a more meaningful summary of our results, and allows decision makers to more quickly understand results and assign resources necessary for remediation.
Figure 4: A Matrix Showing Impact and Probability, with Circle Size Indicating the Number of Each Type of Issue
It is important to note that all of the recommendations I’ve described here presume that you perform excellent technical work. You must continuously strive for that. Then, to add that final polish to your results, apply one or more of these tips to maximize the business value of your work.
We’ve discussed several different approaches for providing significantly more value in your penetration tests. Now, I’m not expecting that every reader will follow every single tip here. But, I do hope that you’ll incorporate at least one or two of these practices, helping to drive up the business value of the work you do. Working together to help define and provide high-value penetration testing will help our industry avoid the valueless death spiral of the Really Crappy Penetration Test.
SANS Fellow and Pen Test Curriculum Lead
Author of SANS 504 and 560 Courses
Founder, Counter Hack Challenges