How to measure execution capability.
Execution capability is measured and represented as an un-biased score (rather than an opinion based assessment), as with similar execution capability metrics identified earlier in this position paper. Initiatives are scored independently. A scoring of overall execution capability of an institution is insufficient, as conditions vary from initiative to initiative. An overall assessment of an agency’s execution capability will not identify accurately the specific vulnerabilities that exist for a specific initiative.
The following factors are recommended for measuring execution capability for technology resource projects:
Applying Domains of Measurement
An execution capability metric uses domains of measurement to identify classes and sub-classes of factors that affect projects. Each domain addresses a specific area of measurement (such as: alignment to strategy, operational capabilities, understanding of business rules & processes, clarity of vision, technical capabilities, management capabilities, etc.)
Data is collected through indirect survey methodologies to remove gaming and other forms of participant response manipulation. Each question must be weighted through the entire survey, and multiple domains are addressed indirectly.
As domain data is collected through indirect survey data, weighting and scoring of each domain is represented. Domain scores collectively contribute to a single execution capability score, which is weighted from method based criteria.
As scoring identifies the execution capability with each domain, vulnerabilities are listed from established criteria that relate to the specific domain and type of initiative being measured.
Compiled through structured methods of data collection and scoring (i.e. SAT, personality profiles, bond ratings and credit scores), execution capability for each specific initiative reflects a moment in time for the specific project and the specific conditions that will impact its execution.
The Origins of Execution Capability
Execution capability is a sub-category of decision bias, which is a principal factor measured in behavioral economic theory. Behavioral economics, along with the related sub-field behavioral finance, analyzes the effects of psychological, social, cognitive, and emotional factors on the economic decisions of individuals and institutions, and the consequences for market prices, returns, and resource allocation.
Behavioral economics is primarily concerned with the bounds of rationality of economic agents and functions as a method of economic analysis that applies psychological insights into human behavior to explain economic decision-making and the impacts of decision bias.
There are three prevalent themes in behavioral finance:
- Heuristics: Humans make 95% of their decisions using mental shortcuts or rules of thumb.
- Framing: The collection of anecdotes and stereotypes that make up the mental emotional filters individuals rely on to understand and respond to events.
- Market inefficiencies: These include mispricings and non-rational decision making.
Decision bias – also known as cognitive bias – are tendencies to think in certain ways that can lead to systematic deviations from a standard of rationality or good judgment, and are often studied in behavioral economics.
Although the reality of these biases is confirmed by replicable research, there are often controversies about how to classify these biases. Some are effects of information-processing rules (i.e., mental shortcuts), called heuristics, that the brain uses to produce decisions or judgments.
Biases have a variety of forms and appear as cognitive (“cold”) bias, such as mental noise, or motivational (“hot”) bias, such as when beliefs are distorted by wishful thinking. Both effects can be present at the same time.