Measurement and Metrics in Secure Software Development
Security metrics are measurements that can be applied to software development as a way to improve the security characteristics of the software being developed. Guidance on software measurement and analysis can be found in the International Organization for Standardization and International Electrotechnical Commission (ISO/IEC) 15939 (Software Measurement Process standard), the Capability Maturity Model2 Integration (CMMI) Measurement and Analysis process area, and the guidance provided by the Practical Software and Systems Measurement (PSM) project. The commonly used CMMI model outlines practices that are organized around two major goals as follows, aligning measurement and analysis activities with organizational and project goals, and performing the measurement and analysis activities.
Aligning measures outlined in CMMI are follows:
- Establish measurement objectives.
- Specify measures.
- Specify data collection and storage.
- Specify analysis procedures.
- Practices for performing measurement are:
- Collect measurement data.
- Analyze measurement data.
- Store data and results.
- Communicate results.
Result of implementing these practices;
- Goal driven measurement which is an important first step in ensuring managements support.
- The project is forced to target and measure items necessary to meet objectives.
- Provides a framework for decision and process improvement that results in a better process and product.
It is important that security concerns be clearly identified and addressed in all steps of the measurement and analysis process. The organization should assess risks and address probable threats, then translate these into specific requirements that address security in the design. The end goal is developing the ability to implement a development process that will ensure the “building in” of security requirements.
After security requirements are specified, measurement objectives are formulated that will provide insight into achieving security requirements. Threat modeling and attempts to identify likely sources of attacks can be performed. Methods such as analytical questions can help determine measurement objectives. A few example questions include:
- What vulnerabilities have been detected? Are current development practices adequate to prevent recurrence?
- What process points are most vulnerable to the introduction of security-related risks?
- What proportion of defects relate to security concerns and requirements?
- Do practitioners comply with security-related processes and procedures?
- Have measures associated with security requirements and implementation been defined and planned?
- What are the critical and vulnerable modules? Have vulnerabilities been identified and addressed?
Measurements do not have to be complicated. Example, in the requirements phase it is useful to know whether security related concerns have been included in defining system requirements. This could be measured initially as yes or no. Another example could be measuring the percentage of sources of input that have validation checks. The target for this measure would likely be 100% depending on any associated effects. As a guideline, measures should be as simple as possible while still meeting information needs.