What Gets Measured, Gets Done

In's an old cliche, to be sure: "What gets measured gets done" but the message is clear: measuring something gives you the information needed to make sure you actually achieve what you set out to do.

Last Friday, we briefly discussed the Section 809 Panel's recommendations concerning the Defense Contract Audit Agency (DCAA). See "Section 809 Panel Issues First Report". Today we want to go into more detail on the Panel's second recommendation (Recommendation #6 Overall) concerning DCAA: Revise the elements of DCAA's annual report to Congress to incorporate multiple key metrics. Most of the following is taken verbatim from the Panel's report.

Congress’s reporting requirement for DCAA lacks critical metrics to adequately measure DCAA’s performance. To alter the conduct of the DoD’s financial and business system oversight functions, success must be defined to be consistent with improving mission focus, valuing time, and simplifying compliance. 

Congress currently emphasizes the number of audit reports completed, and recovering or sustaining questioned costs, over measurements on the other advisory services DCAA provides—which are not mentioned in the report requirements today. DCAA’s annual report to Congress requires some measurements of delayed audits; however, the current report emphasizes the number of audits and questioned costs.  

If DCAA is operating effectively, its success cannot be measured only in questioned and sustained costs. As DoD and contractor internal controls improve, there may be fewer costs to question and sustain. In contrast, worsening DoD and contractor internal controls may increase costs questioned and sustained. Similarly, DCAA’s success as an organization cannot be measured by the quantity of audits at the expense of quality. Congress’s current emphasis on questioned costs and DCAA’s emphasis on return on investment alone do not adequately demonstrate performance. DCAA is not, and should not, be considered a profit center. Most importantly, the current DCAA report has no measure of DCAA’s primary customers’ (contracting officer or acquisition team) satisfaction with the quality and timeliness of DCAA’s work.

Congress must measure DCAA’s success in a manner that helps DoD meet contract objectives. Although detecting contractor noncompliance is important, preventing noncompliance through education and training of contractors and contracting officers is DCAA’s original mission and the best use of its resources.   Questioned costs and sustained costs should remain part of DCAA’s report to Congress; however, these metrics alone are misleading and should not be viewed in isolation of the other key parts of DCAA’s mission regarding service to the contracting offer and the acquisition team. The current report makes no mention of contracting officer or acquisition team satisfaction with the quality and timeliness of DCAA’s work, yet they are DCAA’s primary customer. Congress should measure DCAA using a balanced scorecard consisting of multiple key metrics to include the following:

• A description of the regulatory requirements that create compliance difficulties for contractors, including an analysis of how those regulatory requirements affect contractors of different sizes and industries.
• The total number of new audit or advisory engagements, by type (preaward, incurred cost, other postaward, and business system), with time limits expiring during the fiscal year that were completed or were awaiting completion, as compared to total audit and advisory engagements completed or awaiting completion during the year.
• On‐time performance relative to time limits for each type of audit or advisory engagement (shown separately for the DCAA and qualified private auditors retained by the agency).
• The time limit (expressed in days) for each type of audit or advisory engagement, along with the shortest period, longest period, and average period of actual performance (shown separately for the DCAA and qualified private auditors retained by the agency). 
• For preaward audits and advisory engagements of contractor costs, sustained costs as a total number and as a percentage of total questioned costs, where questioned costs are expressed as the impact on negotiable contract costs (shown separately for the DCAA and qualified private auditors retained by the agency).
• For postaward audits and advisory engagements of contractor costs, the questioned costs accepted by the contracting officers and contractors as a total number and as a percentage of total questioned costs, where questioned costs are expressed as the impact on reimbursable contract (shown separately for the DCAA and qualified private auditors retained by the agency).
• The aggregate cost of performing audits, set forth separately by type of audit.
• The ratio of sustained questioned costs to the aggregate costs of performing audits, set forth separately by type of audit.
• The total number and dollar value of postaward audits that are pending for a period longer than 1 year as of the end of the fiscal year covered by the report, and the fiscal year in which the qualified proposal was received, set forth separately by type of audit.
• A summary of the reasons for the difference between questioned and sustained costs shown in the statistical tables.
• A description of outreach actions towards industry to promote contract compliance and professional development of the DCAA workforce (shown separately for collaborative outreach actions and other outreach actions).
• A statistically representative survey of contracting officers from DoD buying commands and the DCMA and representatives of small and large businesses to measure the timeliness and effectiveness of audit and advisory services provided by the DCAA (shown separately for the DCAA and qualified private auditors retained by the agency).

You will no doubt have noted that the focus of this recommendation is designed to move DCAA back to supporting its primary customers; contracting officers and the acquisition community.