Outlier policy and the National Diabetes Audit Programme

The National Clinical Audit Advisory Group (NCAAG) has produce statistical guidance on how potentially outlying performance of healthcare providers can be identified. This page summarises the outlier policy and its impact on the National Diabetes Audit programme.

Choice of performance indicator

Performance indicators must provide a valid measure of a provider’s quality of care in that there is a clear relationship between the indicator and quality of care, and relate to frequently occurring events to provide sufficient the statistical power

Choice of target (expected performance)

The expected performance may be based either on external sources (research evidence, clinical judgment, audit data from elsewhere) or on internal sources (such as average performance of all providers, though maybe excluding the provider in question or outliers).

Data quality

Three aspects of data quality must be considered and reported:

• case ascertainment: number of patients included compared to number eligible, derived from external data sources; impact on the generalisability (representativeness) of the results
• data completeness: in particular performance indicator data and data on patient characteristics required for case-mix adjustment
• data accuracy: tested using consistency and range checks, and if possible external sources

Case-mix (risk) adjustment

Comparison of providers must take account of differences in the mix of patients between providers by adjusting for known, measurable factors that are associated with the performance indicator. These are likely to include age, sex, disease severity and co-morbidity. Other possibilities include socio- economic status and ethnicity.

Adjustment should be carried out using an up-to-date statistical model. The model should have been rigorously tested as regards its power of discrimination (such as the area under the ROC) and its calibration (such as, goodness-of-fit) and, together with details of the model, both attributes should be publicly reported. Judgment as to the adequacy of a model will depend on the performance indicator selected and the clinical Provider lead clinician: clinician contact for NCA in provider organisation context so universal, absolute values cannot be provided.

Detection of a potential outlier

Statistically derived limits around the target (expected) performance should be used to define if a provider is a potential outlier: more than 2 standard deviations from the target is deemed an ‘alert’; more than 3 standard deviations is deemed an ‘alarm’.

Note that these are definitions of statistically significant differences from expected performance, differences that may not be clinically significant if based on large numbers of patients.

Management of a potential outlier

Management of a potential outlier involves several people: 

• NCA supplier: the team responsible for managing and running the audit nationally 
• NCA supplier lead: person responsible for the audit, often chair of the Board of Management of the audit 
• Provider lead clinician: clinician contact for NCA in provider organization
• Provider medical director and chief executive will need to be involved.

The following table indicates the stages that may be needed in managing a potential outlier, the actions that need to be taken, the people involved and the time scale. It aims to be both feasible for those involved, fair to providers identified as outliers and sufficiently rapid so as not to unduly delay the disclosure of comparative information to the public.

For CCGs the Lead Clinician will be the Clinical Leader or the Chief Clinical Officer; The Accountable Officer will be the Chief Officer or the Chief Clinical Officer. In Wales the lead clinician will be the LHB Primary Care AMD, where appointed, or the Medical Director; Accountable Officer will be the Chief Operating Officer.

Involvement of the regulator

The CQC are included in the guidance so as to provide them with assurance that organisations are engaging appropriately in the process. The CQC recognises that alert level outliers may be identified as a result of chance alone. For alarm level outliers the CQC expects to see evidence of appropriate initial and substantive action plans. The CQC will consider the data as part of its monitoring process. The CQC will not usually take regulatory action if organisations are responding appropriately to each stage of the outlier management process at alert and alarm level.

The NDA Core Audit includes NICE Care Process completion rates, NICE treatment targets and disease outcomes (diabetic complications). Presently, it comprises data from more than 90% of people with diabetes managed in both General Practice and Specialist Centres.

Choice of performance indicator

Care Processes and Treatment targets are recommended. Achieving the NICE treatment targets is the foundation of delivering evidence based long-term complications prevention. And Care Process completion rates have recently been linked to long-term outcomes within the longitudinal analysis. Care Processes and Treatment Targets are recorded in more than 90% of people with diabetes every year. This high data quality and the large numbers even at individual provider level make statistical comparisons robust.

Care process completion rates and Treatment target achievement rates vary substantially between General Practices, between specialist services and between CCGs.

Type 1 patients managed exclusively in primary care are generally older and have proven self-management capabilities. Type 2 patients referred for specialist care input usually have diabetes management, diabetes complications or co-morbidities so do not represent ‘usual care’. Accordingly it is recommended that specialist services be assessed on their management of the Type 1 diabetes patients attending their services and General Practices/CCGs on the management of the Type 2 diabetes patients registered with them.

Choice of target (expected performance)

The expected performance would be less than 2SD from mean care process or treatment target achievement (HbA1c less than or equal to 58mmmol/mol, more than 86mmol/mol, SBP less than or equal to 140, Total serum Cholesterol less than 5mmol/l) rate of all providers. More than 2SD but less than 3SD below mean would raise an alert. More than 3SD below mean would be an alarm

Data quality

Three aspects of data quality must be considered and reported:

• case ascertainment: data is extracted for approximately 100% of the patients (subject to patient dissent) registered with participating practices and specialist services (specialist services enter their own data so could do incomplete submissions)
• data completeness: nearly 100% for age/sex/IMD (small percentage missing from some GP records); around 80% for ethnicity (improving year on year)
• data accuracy: good - electronic record extract of clinical procedure, lab and physiological data

Case-mix (risk) adjustment

In multivariate statistical modeling for care processes a significant part of the variation in completion is accounted for by factors such as age, sex, ethnicity and social deprivation (c statistic more than 0.8) whereas for treatment targets a significant part of the variation is accounted for by age but sex, ethnicity and social deprivation are not explanatory (c statistic 0.6).

The models will be checked annually.

Detection of a potential outlier

Statistically derived limits around the target (expected) performance will be used to define if a provider is a potential outlier: more than 2 standard deviations from the target will be deemed an ‘alert’; more than 3 standard deviations deemed an ‘alarm’. 

CCGs will be assessed on their Type 2 Diabetes data. Specialist services will be assessed on their Type 1 diabetes data

Management of a potential outlier

The NDA clinical lead and the team responsible for managing and running the audit in partnership with HQIP will follow the 8step process specified in the national outlier policy.

For any outlying CCG or Trust the NDA team will provide limited guidance on data quality and data interpretation on request (one meeting with analyst and Project Manager) on how to interpret individual practice data and how to help services assess their DQ. 

The usual lines of accountability (clinical team, clinical director, executive clinical director, CEO/Trust Board, CCG) will apply in respect of specialist teams. For CCGs the Lead Clinician will be the Clinical Leader or the Chief Clinical Officer; The Accountable Officer will be the Chief Officer or the Chief Clinical Officer. In Wales the lead LHB clinician will be the LHB Primary Care, the AMD, where appointed, or the Medical Director; the Accountable Officer will be the Chief Operating Officer.

This report includes the long-term outcomes of diabetes care. The prevalence and severity of cardiovascular, kidney and eye complications are determined primarily by the characteristics and effectiveness of care provided 5 to 20-plus years ago. They are not considered appropriate to use for outlier management.

The prevalence of diabetic foot disease is also duration of diabetes dependent but its consequences, ulcers and amputations, are modifiable by current care (discussed below in the NDFA section).

The acute hyperglycaemic complications DKA/HHS might be candidates for 3 year cumulated measurement but for severe hypoglycaemia mostly treated at home, by paramedics or in A&E there is presently no reliable measure.

NPiD includes: process and treatment target measures of preparation for pregnancy, and antenatal care; maternal complications and delivery measures; foetal and infant complications measures. Pregnancy in women with diabetes is high risk but uncommon (3,500 to 4000 per year in England & Wales).

Approximately equal numbers of women have Type 1 and Type 2 diabetes. The principal location of usual, non-pregnancy diabetes care is specialist for Type 1 and General Practice for Type 2. Typically a specialist service will have around 10 women with Type 1 diabetes becoming pregnant each year while an individual general practice might have one pregnant diabetic woman every 2 to 3 years.

Specialist diabetes services in collaboration with high risk obstetric services manage the antenatal care and delivery of women with diabetes and lead the design and implementation of pre-pregnancy pathways and services in their locality.

Choice of performance indicator

All the scientific evidence reflected in NICE Guidance and corroborated by NPiD analyses is that foetal and maternal outcomes are most influenced by preparation for pregnancy and very early antenatal management.

A composite indicator comprising percent achieving pre-pregnancy NICE glucose control, antenatal folic acid supplementation and early first specialist antenatal review is recommended.

This applies to every woman and even though it is a ‘whole pathway’ measure (specialist and primary care, doctors, midwives nurses) it is advocated by the advisory group as the best single performance measure. Because of small numbers it would have to be cumulated over 3 years for each specialist joint diabetes-antenatal team to be statistically valid.

Secondary measures that would be Diabetes-Antenatal team specific could be glucose control in the third trimester and rate of admission to Neonatal Intensive Care by Type of Diabetes again cumulated over 3 years.

Choice of target (expected performance)

The expected performance would be less than 2SD from mean indicator achievement rate of all providers. More than 2SD but less than 3SD below the mean would raise an alert. More than 3SD below the mean would be an alarm.

Data quality

Three aspects of data quality must be considered and reported:

• case ascertainment: very few women decline to participate so close to 100% case ascertainment for each participating unit, and from 2018 no need to consent women for England due to direction
• data completeness: patient characteristics are linked from the Core NDA so completeness is close to 100%; the three measures that form the ‘bundle’ have a more than 95% data completeness in NPiD
• data accuracy: tested using consistency and range checks, and if possible external sources

Case-mix (risk) adjustment

Modeling includes age, ethnicity, IMD and type and duration of diabetes.

Detection of a potential outlier

Statistically derived limits around the case-mix adjusted mean performance will be used to define if a provider is a potential outlier: more than 2 standard deviations from the target will be deemed an ‘alert’; more than 3 standard deviations deemed an ‘alarm’.

It will then be determined whether there are similar or radically different patterns for women with Type 1 Diabetes (for whom pre-pregnancy diabetes management responsibility usually rests with specialist services) and women with Type 2 Diabetes (for whom pre-pregnancy management responsibility usually rests with General Practice).

In the event of divergence Specialist services will be assessed on their Type 1 diabetes data.

In the event of the problem applying to both Type 1 and Type 2 or just to Type 2 then a ‘whole system’ approach will be applied as noted below.

Management of a potential outlier

The NDA and NPiD clinical lead and the team responsible for managing and running the audit in partnership with HQIP will follow the 8step process specified in the national outlier (appendix).

If the issue is restricted to Type 1 diabetes the usual lines of accountability (clinical team, clinical director, executive clinical director, CEO/Trust Board, CCG) will apply.

If it is restricted to or also includes Type 2 Diabetes because of the ‘whole system’ footprint of responsibility the first action will be to establish in the relevant locality who will assume lead responsibility and who will be the group members of the local investigating team (drawn from clinical team, clinical director, executive clinical director, CEO/Trust Board, CCG). For CCGs the line of responsibility would probably be (diabetes clinical lead, medical director, Chair/Director of Quality, CEO/Board, NHSE) (appendix).

Major amputations are often regarded as the key apex measure of diabetic foot care effectiveness. However because they are infrequent they need to be cumulated over several years to become statistically comparable. Also, because most vascular services now work on a ‘hub and spoke’ model, amputation rates at Hub Trusts are inflated and at Spoke Trusts deflated with respect to the activity of their Trust Diabetic Foot services.

It is possible to make valid comparisons of cumulated 3 year amputation rates at population (CCG) level where the numerator is provider independent and the denominator is the whole population of people with diabetes.

It is the responsibility of CCGs to ensure that their population has rapid access to a specialist Multidisciplinary Foot Diabetic Foot Team (MDFT) and in NDFA thus far (it is just approaching its third annual report) delays in reaching the team after ulcer onset seem to correlate well with ulcer severity, delay in recovery and mortality.

Time to first assessment is a central part of NICE guidance. About a quarter to a third of localities do not, at present, seem to have organised diabetic foot-care services. There are variations at provider level in rates of achieving the treatment goal – ‘alive and ulcer free’.

Choice of performance indicator

• the percentage of patients taking longer than 2 weeks to reach MDFT (good measure of ‘whole pathway’ performance) measured at CCG level
• 3 year cumulated case-mix adjusted major amputation, population based rates measured at CCG level (based on report 2 results)
• the percentage of patients with less severe & severe ulcers at presentation that are alive and ulcer free 12 week later measured at service provider level

Choice of target (expected performance)

The expected performance would be less than 2SD from mean treatment target achievement rate of all providers. More than 2SD but less than 3SD would raise an alert. More than 3SD would be an alarm.

Data quality

Three aspects of data quality must be considered and reported:

• case ascertainment:
  • amputations comprehensive –HES
  • time and treatment rates: participating services endeavor to include all patients but prior to August 2017 the need for individual consent was probably a variably modifying factor
• data completeness:
  • HES and time to first assessment should be 100%
  • patient characteristics are linked from Core so more than 90%
  • some patients get lost to FU so resolution rates around 90%
• data accuracy: no data that is fundamentally unstable; data submitted by teams is checked by them before being submitted for analysis

Case-mix (risk) adjustment

Amputation rates are standardized against local and whole national diabetic populations and, where appropriate, case-mix adjusted for age, ethnicity, and social deprivation.

For time to resolution (alive and ulcer free) the current model includes age, sex, ethnicity, IMD, Type and Duration of Diabetes, smoking habit.

Detection of a potential outlier

Statistically derived limits around the case-mix adjusted mean performance will be used to define if a provider is a potential outlier: more than 2 standard deviations from the target will be deemed an ‘alert’; more than 3 standard deviations deemed an ‘alarm’.

CCGs will be assessed on their amputation and time to first assessment data. Specialist services will be assessed on their 12 week resolution for severe ulcers data.

Management of a potential outlier

The NDA clinical lead and the team responsible for managing and running the audit in partnership with HQIP will follow the 8step process specified in the national outlier policy (appendix).

The usual lines of accountability (clinical team, clinical director, executive clinical director, CEO/Trust Board, CCG) will apply in respect of specialist teams; for CCGs it would probably be (diabetes clinical lead, medical director, Chair/Director of Quality, CEO/Board, NHSE) (appendix).

NaDIA is an annual snapshot bedside questionnaire and service structure survey audit. It can therefore track national trends robustly but cannot provide statistically valid between trust comparisons.

NaDIA has identified three key hospital harms (severe hypoglycaemia, inpatient DKA/HHS and inpatient onset foot ulceration) that are appropriate apex indicators of the underlying whole hospital service effectiveness. From 2018 forwards they will be collected continuously thereby providing measures of overall effectiveness and harm.

Choice of performance indicator

• the percentage of patients having severe hypoglycaemia (requiring injectable therapy)
• the percentage of patients having inpatient DKA/HHS
• the percentage of patients having new inpatient foot ulcer

Choice of target (expected performance)

The expected performance would be less than 2SD from mean rate of all providers. More than 2SD but less than 3SD would raise an alert. More than 3SD would be an alarm.

Data quality

Three aspects of data quality must be considered and reported: 

• case ascertainment: will depend on comprehensive reporting by Trusts; it will be possible to detect under-reporting accurately because NDA Core identifies the number of patients admitted to each Trust by type of diabetes, so it will be clear if unexpectedly low rates are being recorded
• data completeness: patient characteristics will be linked from Core so more than 90%; denominator data on characteristics and volumes of patients from HES
• data accuracy: all data required will be subject to review by submitting organisations

Case-mix (risk) adjustment

Models will be created to take account of patient characteristics and hospital size/casemix characteristics. The necessary volumes of data will probably not be generated until about 2019.

Detection of a potential outlier

Statistically derived limits around the mean case-mix adjusted performance will be used to define if a provider is a potential outlier: more than 2 standard deviations from the target will be deemed an ‘alert’; more than 3 standard deviations deemed an ‘alarm’.

Management of a potential outlier

The NDA clinical lead and the team responsible for managing and running the audit in partnership with HQIP will follow the 8step process specified in the national outlier policy (appendix).

The usual specialist team lines of accountability (clinical team, clinical director, executive clinical director, CEO/Trust Board, CCG) will apply.