Metrics That Matter: Improving Project Progress and Performance Assessment

Publication No

RS322-1

Type

Research & Development Product

Publication Date

Nov 01, 2016

Pages

Research Team

RT-322

Order Now

DOCUMENT DETAILS

Abstract

Key Findings

Filters & Tags

Abstract

Assessment of project progress and performance is critically important to the successful delivery of capital facility projects. Project managers are often misinformed in their perceptions of project performance until the project nears its end. Major challenges are related to the lack of consistent, objective, and reliable metrics and indicators, and the lack of appropriate interpretation of these data for establishing suitable corrective action plans. Research conducted by CII Research Team 322 (RT-322) identified several key Forecasting and Diagnostic metrics that can provide insights into a project’s actual progress, performance, and forecast at completion. It also provides information on ways to improve the reliability of these metrics.

The RT-322 research methodology included a broad literature review of more than 120 sources to identify progress, performance assessment, and performance forecasting metrics. The review also identified factors to improve metric reliability in several areas, including project scope definition, execution planning, and risk management. Next, the researchers distributed a survey to collect data on the metrics and reliability techniques used on completed projects. In total, 44 survey responses were collected, representing mostly large, industrial projects. Subsequently, 10 case studies provided more in-depth analysis of some of these projects and their use of metrics. This was followed by a Delphi session with 16 subject matter experts with more than 350 years of combined experience, who evaluated and validated the findings from the survey and the case studies. The Delphi session further refined a list of in-flight project control metrics divided into the following categories: Core (“must have”), Validation (metrics that confirm the validity of the Core metrics), Innovative (metrics that are not currently in wide use, but are considered potentially beneficial), and Other Significant (other metrics that fall outside the previous categories, but are perceived to have value).

Core metrics were initially selected based on the following project characteristics: large, industrial, reimbursable cost, balanced cost and schedule goals, moderate complexity, and contractor perspective. However, when considering Core metrics for other project characteristics, it was discovered that these metrics tended to be the same across projects—the only differences were the frequency of data collection and the level of effort involved in collecting and analyzing these data. Further statistical analysis of survey responses revealed, that compared to projects using fewer Core metrics, projects that used more Core metrics for project controls experienced higher rates of success at meeting their original budgets. A correlation between the use of more Core metrics and better project cost outcomes was observed at the 95% confidence level using the Spearman’s rank correlation method.

The RT-322 research identified in-flight project control metrics for determining project progress measurement, performance assessment, and performance forecasting. Critical factors were identified that individually and collectively influence the reliability of project progress and performance assessment metrics. A useful dictionary of Core metrics, including their definitions, uses, and interpretation information, as well as “metric maps” (demonstrating how each metric is calculated), was produced. Furthermore, the Project Controls Improvement (PCI) Tool was created to provide a more standardized or systematic way of assessing a project’s progression and performance. Using the PCI Tool, project stakeholders can identify the gaps in their project control systems or learn more about Core metrics and steps they can take to improve metric reliability within a dynamic and interactive software environment.

Key Findings

Identified a list of 111 potential project control metrics based on the data gathered. Classified in-flight project control metrics based on their roles, importance, and functionality:

Roles: Forecasting (predicting project cost and schedule outcomes based on the current understanding of project progress and performance) vs. Diagnostic (signaling progress and performance issues to inform corrective actions).
Importance: Core (“must have” metrics that provide the greatest insight into project controls) vs. Significant (supplement or complement Core metrics as needed).
Functionality: Data (Data Collection), Information (Progress Measurement), Knowledge (Performance Assessment), Insight (Performance Forecasting).

(RS322-1, p. 9)

Use and interpretation of Core metrics enable project stakeholders to better understand and implement them to manage and control project performance. The use and interpretation details for each Core metric includes: background information and explanations, requirements and timing for use, application scenarios, and relationships among the different metrics.

(RS322-1, p. 12)

RT-322 found that the Core metrics would remain the same for projects with different characteristics. However, it is anticipated that the main differences would be in the level of effort and the frequency of data collection and reporting. The study found the following project characteristics to have the greatest impact for creating this difference in the use of Core metrics: Perspective (Owner versus Contractor); Project Size; Contract Type; Project Complexity; and Schedule or Cost Priority.

(RS322-1, p. 13)

Statistical analysis of data revealed that projects that used more Core metrics for project controls experienced higher rates of success at meeting their original budgets. A correlation between the use of more Core metrics and better project cost outcomes was observed at the 95% confidence level using the Spearman’s rank correlation method.

(RS322-1, p. 14)

a. The team identified 15 Critical Reliability Factors (CRFs) that influence the reliability of project control metrics:

Project Scope Definition
Project Execution Planning
Project Control Planning
Progress Measurement
Schedule and Cost Development and Tracking
Change Management
Risk Management
Progress Audits
Metric Trend Analysis
Schedule Forecasting
Cost Forecasting
Communication
Teamwork
Accountability
Project Control Audits

b.   CRFs are defined as factors that individually and collectively influence the reliability of project progress and performance assessment metrics.
c.   83 reliability indicators associated with these CRFs are also provided.
d.   Each indicator has a specific assigned timeframe, defined by five timing milestones (i.e., prior to, at the beginning, during, at the end, post) for different project phases.
e.   These specific timing milestones should be carefully considered for effective implementation of CRFs and indicators.
f.   There may be additional CRFs and indicators depending on project characteristics, requirements, and project team perspective; project teams can identify these additional CRFs and expand on the provided list.

(RS322-1, p. 17)

Filters & Tags

Knowledge Area

Performance Assessment, Project & Program Management, Project Controls

Project Phase

Detailed Scope, Detailed Design (Engineering), Procurement, Construction, Commissioning & Startup

Project Function

Project Management, Project Controls, Construction, Commissioning & Startup

Industry Group

Light Industrial, Heavy Industrial, Buildings, Infrastructure

Research Topic

Improving Project Progress and Performance Assessment

Keywords

Project Controls, Progress Measurement, Performance Assessment, In-flight Project Control Metrics, Core Project Control Metrics, Forecasting and Diagnostics Metrics, Metrics Dictionary, Metric and Measurement Reliability, Reliability Improvement Factors, rt322