Fuzzers can be used in different phases of the development process, but mainly they apply to:
- Developers that can detect bugs and work to fix them as they code, for example to test the network interfaces and input processing algorithms
- QA teams use fuzzers as automated tools for performing Security QA, as a part of their QA process
- External software auditors, or security testers, use fuzzers to get a testing environment with which products can be audited to see how secure they are without needing access to sensitive information such as the source code
Fuzzing technology provides the necessary assurance that security vulnerabilities that would be found after a product release will be found while it is still in development. This is important both from the costs of potential technical support (to handle recall/patching) to the costs incurred by losing credibility with customers or from receiving a reputation for bad security practices.
Now that fuzzing has reached a stage of maturity and is being used as a reliable tool for organizations to fight against 0-day vulnerabilities (unpublished vulnerabilities used by hackers to attack while evading detection by signature-based defense tools), it is also likely that hackers will utilize this technology to attack products more than ever before. The difference is that now the tools that hackers use are available to developers to integrate into testing suites that are implemented by organizations and applied to their testing methodology.
Although fuzzers are exhaustive in their testing, and are likely to find many security vulnerabilities, they can only prove the existence of bugs -- not the absence. A fuzzer can not guarantee that every vulnerability was discovered and fixed.
In addition, fuzzers find bugs in programs with a certain likelihood of false positives. The fact that a certain request has triggered an internal program error, raised an exception, or caused the application to hang or crash, does not necessarily mean that the problem is a real security vulnerability. Test cases found by automated tools should be investigated by a developer that knows exactly how serious a threat it really is.
Despite those shortcomings, black box testing remains a popular testing mechanism and has demonstrated a successful record of finding security holes. In fact, this is the "secret sauce" used for many years by both hackers and security researchers to uncover many of the thousands of security vulnerabilities published in recent years.
On the practical side, while black box testing can stand on its own merits, it can also complement other testing techniques such as source code auditing of different forms. It is a good idea to develop a testing methodology that includes black box testing, in addition to the rest of your current QA testing process.
