Here is the Security Metrics Foundations presentation that yesterday's blog entry was largely based upon. I look forward to comment on the presentation, since it is still a work in progress. After spending a few more minutes on the English version, will spend some time on the Japanese version and do a write-up on the Japanese blog later today or early tomorrow. Stay tuned!!!
For Starters...
Over the past couple months I have worked to pull a lot of information together on security metrics and creating an implementation guide and implementation kit. Andrew Jaquith's book, Security Metrics, Replacing Fear, Uncertainty, and Doubt, covers the subject quite well from the metrics definition phase through to an attempt to dashboard with balanced scorecard, but I hope to focus more in this series on the implementation with tools and cover the reporting phase in more depth. The other endeavor focuses on getting the book published in Japanese, where I can use the kit and tools for implementation in my current home market.
Over the past couple years I have been using one of the original Core Duo MacBook Pro computers, and yesterday my kind and beautiful wife let me purchase a new 2.6 Ghz MacBook Pro - with LED backlight. I must say that using this system is a computing experience unto itself. Graphics are crispy and the ride is extremely smooth.
The 'Defining' Moment
Let us approach the definition of a security metric for purposes of this guide and toolkit from a bottom-up perspective. First by explaining what a measurement is, then completing the metric definition with just that. Later in the guideline, we will cover the topic from a top-down approach for implementation guidance.
In establishing a standard definition for a security metric, we must outline the elements of measurements that will be used to develop metrics. This is defined in NIST publication, Performance Measurement Guide for Information Security (SP800-55), and provides a solid foundation for our metrics.
Measurement ID - A unique identifier for the measurement for database and organization reasons.
Measurement Goal and Objective - This is applied to the measurement to outline the security goal and objective that this measurement is helping to achieve.
Unit of Measure - A numeric statement that begins with the word "number", "servers", or "incidents". For example, a virus measurement can be in terms of two different measurement units - number of computers infected, or number of virus incidents reported.
Type - Identify whether the measure is implementation, effectiveness and efficiency, impact, or multiples of these types.
Formula - For measurements that require a calculation, the formula for such calculation needs to be documented.
Target - The satisfactory rating for the measure expressed in measurement units, or a directly related unit to the unit of measure for the measurement.
Implementation Evidence - Our definition of evidence required to support the execution of measurement and derived values, will slightly depart from the NIST definition to include more environments besides those found in government agencies.
Frequency - How often the data will be collected and analyzed, and how often the measurement results will be reported.
Responsible Parties - Identify the following owners:
Data Source - Where the data will come from. The application, system set, environment from where data will be pulled.
Reporting Format - A description of the measurement reporting format, or provide a sample of the reporting format.
A metric name, extended description, and unit of measure tells us what the metric represents, while a measurement procedure and established measurement frequency ensures consistency in measurement (more about consistency soon). Thresholds and target value all combine to allow the metric to be applied toward objectives and goals in the security program. All of the elements listed above from NIST guidance can be applied to a metric. But a metric may combine multiple measurements to come up with a metric. For example, a metric could be "dollar cost per virus quarantined every year". The bottom right-hand of the mind map below is a set of metric elements that I have found useful for security metric implementation.
Now that we know what characteristics a security metric will possess, we can define measurement standards by which security metrics will be determined. As such, measurements used to determine security metrics will follow the following rules:
An objective of security metrics implementation is to report based on concrete, quantitative facts that are pulled from systems or processes, therefore, some kind of unit of measurement needs to be used to express values that are reported as a part of metrics. Some would say that percentages would also suffice in reporting measurements, however, we recommend to not allow for measurements to be expressed as percentages. On the other hand, metrics can be reported as a percentage of the previous measurement period, so as to allow for comparison of metrics based upon a baseline or an established benchmark.
Measuring at regular intervals, such as monthly or weekly, will drive consistency and foster development of benchmarks and target values. Benchmarks and target values should not be determined without data provided by at least two past measurements taken at two past consistent intervals. Taking measurements at regular intervals is also related to the following rule, which is to apply measurement methodology in a consistent manner. Consistency in measurement methods will ensure that any conclusions drawn from the data will be accurate, and that target and baseline values will be the clearest representation of the measurement value. And finally, to ensure consistent measurement, measurement methods should be documented so that any person that performs measurement will apply the same consistent method as past measurements.
Now that we have outlined the requirements for our security metrics and underlying measurement requirements, the Wikipedia definition is offered:
"Metrics are a system of parameters or ways of quantitative and periodic assessment of a process that is to be measured, along with the procedures to carry out such measurement and the procedures for the interpretation of the assessment in the light of previous or comparable assessments. Metrics are usually specialized by the subject area, in which case they are valid only within a certain domain and cannot be directly benchmarked or interpreted outside it."
