Limitations
No single tool or practice can possibly discover or fix all security vulnerabilities. Multiple tools, training and best practices must be employed instead to ensure that software is as secure as possible. Some of the limitations of static analysis tools include false positives, vulnerabilities that only show up in the environment and a false sense of security.
False positives can be very difficult to weed through and often require significant security experience to discern which warning should be fixed. Fixing every warning reported by a static analysis tool can cause unnecessary code churn, which may lead to the introduction of more functional and security flaws. False positives that are not easily interpretable can overwhelm developers to the point that they become disenchanted with the tool. This plays into the psychological acceptability of the tool; if it is too difficult or causes significantly more work for the developer or tester to use, it may be discarded, despite the potential of increased security in the end product.
Many vulnerabilities occur only in specific environments and will only be discoverable on certain install beds. These vulnerabilities are often caused by low system resources, library and API versioning problems, and insecure settings. These types of vulnerabilities are impossible to detect before runtime. Static analysis tools attempt to simulate resource contention and some versioning problems, but the vulnerability can only be detected at runtime with dynamic analysis tools such as a debugger or fault injection tool. Library and API versioning problems are difficult to detect and test for before runtime; however, with strong naming conventions, a developer can take precautions to ensure a certain library has been installed on the system. This approach, while secure, is sometimes unused because testing on every available system is difficult, costly and limits the number of compatible systems the application can be installed on.
Static analysis tools can not foresee possible insecure deployment settings and thereby miss this significant attack surface. Ensuring the product ships securely and is installed securely by default is challenging and must be outlined in the requirements from early on. It is essential to understand:
- The environment your software will be deployed in,
- How your users will use your software
- How knowledgeable the users are, and
- How much configuration is likely to occur after the product has been installed on the system.
Once these questions have been answered, secure requirements can be written to help keep your users safe in many deployment situations.
