FY 2020 Annual Performance Plan and Report - Data Sources and Validation

Tools supporting this measure are publicly available at https://cds.ahrq.gov/cdsconnect/topic/opioids-and-pain-management.

• Category 1—FFS with no link of payment to quality;
• Category 2—FFS with a link of payment to quality;
• Category 3—alternative payment models built on FFS architecture; and
• Category 4—population-based payment.

CMS considers CMMI models in categories 3 and 4 that directly test how providers are paid and that the agency can use to measure these payments for this measure.  For example, CMS considers Accountable Care Organizations (ACOs), which include Pioneer ACO and the Medicare Shared Savings Program, for this measure because the model allows CMS to understand how providers are receiving Fee For Service payments.  CMS also considers advanced primary care medical homes, which include Comprehensive Primary Care and Multi-Payer Advanced Primary Care Practice, in this total.  In contrast, CMS does not include models such as the Health Care Innovation Awards that provide funding to organizations to support providers in moving toward this goal because the model is not necessarily explicitly testing how providers are paid.

Numerator: Model payment actuals based on model specific data, such as the number of aligned beneficiaries and annual per beneficiary spending.  Denominator: The CMS OACT actual or estimated annual Part A and B expenditure.

CMS staff and contractors provide beneficiary alignment and expenditure data to CMMI. Model teams and contractors use quality assurance measures and data cleaning, including an audit and validation process of the programs that calculate the results to ensure the reliability of the results.

The second step applies “forward cleaning logic” to the respondent-level data.  In certain cases, a beneficiary may answer “No” to a screener question but subsequently answer the dependent questions (this is against protocol).  In these cases, CMS programmatically sets responses for the dependent questions as “missing.”  

After forward cleaning, the vendor runs the respondent-level dataset through the CAHPS scoring macro.  Harvard developed and maintains the CAHPS scoring macro, which has been used for over 10 years to correctly calculate case-mix adjusted (CMA) CAHPS scores.  The macro outputs 0-100 CMA scores and reliabilities for each CAHPS item and Summary Survey Measures (SSM).  CAHPS produces scores at both the ACO and national level.  These scores then undergo a final programmatic check to ensure 1) the CMA item and SSM scores are produced for every ACO, 2) the scores and reliabilities are within scale, and 3) reliability flags are correctly assigned to items and SSMs.

The CERT Program selects a random sample of Medicare Fee-for Service (FFS) claims from a population of claims submitted for Medicare FFS payment.  CMS performs complex medical review on the sample of Medicare FFS claims to determine if the claims were properly paid under Medicare coverage, coding, and billing rules.

As part of a national contracting strategy, CMS gathers adjudicated claims data and medical policies from the States for purposes of conducting medical and data processing reviews on a sample of the claims paid in each state.

As part of a national contracting strategy, CMS gathers adjudicated claims data and medical policies from the states for purposes of conducting medical and data processing reviews on a sample of the claims paid in each state.

Data are validated using parallel coding, reasonableness checks on each file, version-to-version changes by variable and service types, and year-over-year comparisons.

CMS reports the prevalence of antipsychotic use in the last three months of the fiscal year.  The numerator consists of long‑stay residents receiving an antipsychotic medication on the most recent assessment.  The denominator is all long-stay nursing home residents, excluding residents with schizophrenia, Tourette’s syndrome, or Huntington’s disease.  If they have resided in the nursing home for 101 or more days, residents are long-stay residents.  The baseline number reflects the prevalence of use in the last quarter of CY 2011.  CMS selected this quarter because it was the last quarter in the pre‑intervention period.

In addition, the AHRQ National Scorecard draws on two other measurement systems to capture additional adverse event types not captured by the MPSMS: select NHSN and AHRQ Patient Safety Indicator data, which are included in order to generate a more comprehensive set of preventable patient harms.  Over 90 percent of this dataset is not dependent on coding or coding practices, making it a highly reliable account of patient safety harms occurring on a national scale.  MPSMS collects preliminary data and provides it to AHRQ for validation and analysis before the delivery of finalized compiled results. 

CMS uses earlier data to fill-in gaps for preliminary estimates.  As all the data for a given year become available, CMS produces a final number.  More details are available at: https://www.ahrq.gov/sites/default/files/wysiwyg/professionals/quality-patient-safety/pfp/hacreport-2019.pdf.

AHRQ and CMS are in the process of updating the MPSMS data source, to which new clinical measures were last added in 2005.  For example, MPSMS does not currently monitor opioid-related adverse drug events.  An updated baseline will accommodate major updates to the measurement system during the performance period.

The Sentinel Data Quality Review and Characterization Programs are used by the Sentinel Operations Center (SOC) for data quality review and characterization of the SDD. To create the SDD, each data partner transformed local source data into the Sentinel Common Data Model format. The SOC created a set of data quality review and characterization programs to ensure that the SDD meets reasonable standards for data transformation consistency and quality and that the SDD data meets expectations needed for a distributed health data network.

The Sentinel Coordinating Center will track the number of medical product analyses conducted by ARIA by using the Sentinel query tracking database. FDA updates these data on a continuous basis at the end of every calendar quarter and makes them publicly available online.

This web-based data collection method communicates errors and warnings in the built-in validation process.  To ensure data quality the program conducts data verification for all RSR submissions.  Recipients receive reports detailing items in need of correction and instructions for submitting revised data.  The web system has an array of reports available through which the grantees and their funded providers can identify data issues that need to be resolved.  In addition, the program provides technical assistance and training during and after the submission period to address quality issues.

TEDS compiles client-level data for substance abuse treatment admissions from state agency data systems. State data systems collect data from facilities about their admissions for treatment and discharges from treatment. TEDS is an admission-based system, but it does not include all admissions.

Many facilities that report TEDS data receive state funds or federal block grant funds to provide alcohol and/or drug treatment services.  State laws require substance abuse treatment programs to report publicly funded admissions.  Some states only collect information on publicly funded admissions.  Other states are able to collect privately funded admissions from facilities that receive public funding.  States report these data from their state administrative systems to SAMHSA.