Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼


Beyond the Badness-ometer

Gary, the author of Software Security (Addison-Wesley, 2006), Exploiting Software (Addison-Wesley, 2004), and Building Secure Software (Addison-Wesley, 2001), among other books, is the CTO of the software-quality management firm Cigital.

Penetration testing is a technique commonly practiced by computer security specialists, but its power is limited—especially when it comes to software.

Since the days of SATAN (the first network security port scanner), computer security specialists have honed penetration testing into a powerful art. Ironically, SATAN's authors Dan Farmer and Wietse Venema's approach—improving the security of your site by breaking into it using automated tools—got Farmer summarily fired from Silicon Graphics in 1996; 10 years later, any sysadmin not regularly using a network port-scanning tool (think Nessus) is ripe for dismissal. We've come a long way in a decade.

Penetration testing certainly has demonstrated its value as a diagnostic exercise for network security. The reason penetration testing works so well for network security is that it focuses on configurations of fairly well-understood systems. That is, networks should be configured certain ways, firewalls should block certain ports, common applications such as sendmail need to be properly patched up, and operating systems on servers should be up to date. All of these essential security practices can be probed using network penetration testing.

Enter the Badness-ometer

The problem with diagnostic exercises is that they're diagnostic exercises. That is, a penetration test can tell you whether your system can be compromised in a straightforward manner (often using canned, black-box probes), but it can't tell you that you're completely secure. Both SATAN (www.fish2 .com/satan/) and Nessus (www.nessus .org) use very straightforward probes. However, penetration testing like this simply demonstrates a negative. Whether a problem found through penetration testing is easily corrected depends entirely on the nature of the problem. It's straightforward enough to tweak firewall rules and bring operating systems up to date, but if a penetration test finds a problem deep inside your software, what are you to do?

When software branches off into nonstandard applications, penetration testing begins to lose its potency. Think about it in terms of testing any arbitrary program for a negative condition. Many "application security testing" tools rely on canned black-box tests to do their business. That is, they try to simulate a "hacker in a box" for testing software security.

The problem with this approach is twofold. First off, if black-box canned tests do find security problems, then you know your software is in really bad shape. However, if you pass all of the black-box tests with flying colors, you really don't know anything about your security posture at all (other than that you passed a bunch of tests); hence, the term "badness-ometer." Badness-ometers register values in the range from "Deep Trouble" to "Who Knows." Note that a badness-ometer is not at all the same thing as a security-ometer. Second, penetration testing is often outsourced to consultants, many of who claim to be "reformed attackers"—well, they say they're reformed. How can you know whether you're getting all of the results? Is knowing two of five ways to hack into your system good or bad? What makes admitted criminals (reformed attackers) worthy of trust?

Diagnosis of a problem with black-box testing often does not result in a clear mitigation plan. So far, that hasn't been a problem with the use of network scanning tools (SATAN derivatives), but it's a big problem with software security.

Embrace the Badness

Then again, it's good to know if you're in deep trouble. If you know, maybe you'll do something about it. For that reason, I hope everyone buys a badness-ometer for their software and applies it today. We're all very likely to be shocked at the state that we're in—a state of untenable software risk. Then maybe we'll turn to building security into our systems from day one.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.