Debunking Myths: Application Security Checklists Suck

There is a pervasive sentiment amongst the security community about checklists: they suck. We’ve all seen inflexible audit checklists that seem to be highly irrelevant to the specific system being audited. Moreover, we are all too aware of organizations doing the bare minimum to meet a checklist item on an audit report, even at the expense of achieving *real* security. It’s no wonder that so many IT professionals react with disdain when they see yet another checklist. They are often generic because auditors want an extensible assessment process that doesn’t become out of date when technologies change. Other times they are rigid or outdated because they are too low level and don’t allow for compensating controls. The problem is, not all checklists suck. Dr. Atul Gawande and his team have shown that a simple 6-step checklist can reduce deaths due to surgical defects by over 40%. It’s not only surgery; Dr. Gawande found that pilots, engineers and investment fund managers have all recorded measurable gains from using checklists. If checklists are so successful in other industries, why do people abhor them in software security? The reasons are simple:

1. Software is dynamic. Each application is unique, and a “one size fits all” checklist just won’t apply to different types of software with different features. For example, a security audit checklist for an online banking web application won’t reflect the risks of a mobile healthcare application. Effective software security checklists themselves need to be dynamic and tailored to the threats of a specific application.
2. Technology changes rapidly. A static technical checklist can fall out-of-date quickly. Development teams may think checklists that reference old technology are outdated and not applicable to their projects. Effective software security checklists need to be fluid and updated with changes to technology.
3. Generic isn’t good enough. A simple generic checklist just doesn’t cut it. For example, many organizations refer to the OWASP Top 10 as a simple software security checklist. Digging beneath the surface, the OWASP Top 10 simply enumerates major classes of application security risks. Digging deeper, a risk like “A3: Broken Authentication and Session Management” can mean over a dozen different requirements for a single application. Only experienced assessors understand the lower-level, technical risks contained within “A3: Broken Authentication and Session Management” and there may even be discrepancies between assessors. Effective software security checklists need to be specific.
4. Process overhead. Developers working in large companies may already be flooded with process checklists. They often see a new checklist as another distraction from actually building software. At the same time, most developers don’t see functional requirements in the same light as checklists. That’s because requirements lists are made specifically for an application and communicate needs from stakeholders. No software security checklist is as effective as a tailored set of security requirements.

Dynamic, low-level, tailored software security requirements are more effective than any static, generic software security checklist. In fact, effectively designed software security requirements can predict 97% of common software security flaws.