Summary Hospital-level Mortality Indicator (SHMI) review update January 2019

This update provide an overview of the current SHMI review which will include:

• short-term improvements, some of which are presentational, some of which are methodological
• more significant long-term methodological changes that are being evaluate or considered
• issues that may also impact upon the SHMI, that must also be accounted for

Take the time to digest the following and provide feedback to NHS Digital by 4 February 2019. User feedback is invaluable to this process, many of the short-term tactical improvements that have been proposed and are currently being implemented resulted from the user feedback obtained from the User Survey undertaken by NHS Digital in 2018.

The SHMI has been published by NHS Digital since September 2011. It is a complex indicator and there are a range of academic views on various aspects of the methodology. The SHMI is subject to continuous evaluation and has undergone several methodology reviews. Part of the review process requires the indicator to be assured through the Indicator Methodology and Assurance Service (IMAS), hosted by NHS Digital on behalf of the wider health and social care system. As part of this process the SHMI series as a whole is now due for re-assurance.

There are known changes to the clinical terminology on the horizon which will impact the SHMI alongside which, technology is continuously improving which has facilitated the implementation of more advanced statistical techniques. In addition, it is good practice to periodically consider whether any new developments should be adopted.

The purpose of the review is to ensure that the SHMI is fit for purpose, to consider alternative methodologies for mortality indicators and recommend future work, as well as to assess the impact of changes to clinical terminologies on the SHMI.

A SHMI Engagement group consisting of stakeholders and users of the indicator has been established and a technical sub-group consisting of academic experts has been convened to lead this work. They will report to the National Medical Director at NHS England, Professor Stephen Powis.

The following groups have been established to ensure any proposals are considered robust by experts in the field.

Engagement group

Aim: To provide an overall steer for the review

Chair: Celia Ingham-Clark (NHS England)

Membership: Representation from NHS England, NHS Digital, NHS Improvement, Public Health England (PHE), Department of Health and Social Care (DHSC), Care Quality Commission (CQC), academics from University College London (UCL), Imperial College London, NHS Hospital Trust, NHS Foundation Trust

Technical sub group

Aim: To provide technical input on statistical modelling and technical aspects of the SHMI

Chair: Daniel Ray (NHS Digital)

Membership: Representation from NHS Digital, academics from University College London (UCL), Imperial College London, University Hospitals Birmingham (UHB), University of Cambridge, University of Sheffield, Nuffield Trust, Bradford Institute of Health Research, Leeds Institute of Data Analytics

Consultation group

Aim: To provide a broad level of consultation / input from users of the solution

Membership: Users from NHS, the research community, NHS organisations, general public

The review is currently progressing based on three main workstreams:

Short-term tactical improvements:

• presentational improvements or additional requirements
• refinements to the existing methodology

Overall solution review.

Changes to currencies (ICD or SNOMED).

Delivered

The SHMI Engagement Group has been established and has met three times. They have agreed the changes made so far on the publications, and the approach for the short-term tactical improvements.

The SHMI Review Technical Sub-Group has been established and met once. They have agreed the terms of reference, reviewed the short-term tactical improvements and agreed the key areas of focus for the overall solution review.

The first set of presentational improvements were delivered in the September SHMI publication, these included:

• Improvements to the layout of the SHMI homepage and the publication page making it easier for users to find information 
• Integration of the interactive funnel plot into Power BI, centralising visualisations in a single tool
• the addition of a shortened text descriptor label for diagnosis groups

Improvements to timeliness. The SHMI publication released on 22 November 2018 was the regular quarterly publication, which was delivered a month earlier than previously scheduled.

To be delivered

The elements listed below have been agreed by the SHMI Engagement Group and the SHMI Technical Sub-Group. The recommendation was that it would be preferable to users if changes were implemented at the same time rather than via a phased approach.

The second set of presentational improvements are scheduled to be published in the May 2019 publication:

• revised data disclosure rules to be consistent with the updated Hospital Episode Statistics (HES) suppression methodology
• in addition to publishing at the trust level, the SHMI will be published by site of treatment
• the inclusion of additional fields in the record level extract service data for provider organisations 
• Inclusion of a one-page summary aimed specifically at a clinical audience o Publishing control limits and SHMI bandings for a subset of the SHMI diagnosis groups (larger diagnosis groups) 
• the aspiration is to publish the SHMI on a monthly basis from May 2019

Refinements to the existing methodology are also scheduled to feature in the May 2019 publication:

Addition of seasonality within the statistical models employed to calculate the expected number of deaths. 

This has been evaluated from a governance and change in SHMI value perspective. The percentage change in SHMI values was less than 1 per cent for all trusts and no trusts changed banding. The benefit of this methodological change will be on the Variable Life-Adjusted Display (VLAD) charts which visualise trends in outcomes over time. 

Including an adjustment for birthweight in the perinatal diagnosis groups. 

The inclusion of birthweight improves the accuracy of the perinatal diagnosis group models and therefore the overall SHMI.

Changes to two diagnosis groups (separating live births from maternity related conditions and non-Hodgkin’s lymphoma from Hodgkin’s disease).

This additional level of granularity adds further credibility to the SHMI by creating more clinically appropriate groupings and has minimal impact upon the overall result. 

The cumulative effects of all the methodological changes will be assessed and reported as part of the implementation process, with “before” and “after” SHMI values and bandings being calculated.

The following work items are at differing stages of development, it is intended these will be considered by the SHMI Review Technical sub-group:

The definition of the outcome, ie whether to predict deaths from the date of discharge or the date of admission:

• at present the SHMI covers patients admitted to hospitals in England who died either while in hospital or within 30 days of being discharged 
• It has been proposed that the definition of the event be reviewed, one possibility is “Deaths within 30 days of admission, whether in hospital or post discharge”
• this would be a fundamental change to the SHMI, which would break the historic time series
• not all hospital deaths would be considered, as is currently the case (the current methodology does not limit the duration of the stay in hospital for inclusion in the SHMI)
• a change could however bring the SHMI more into line with other indicators that are employed nationally and internationally

Adjusting for co-morbidities, whether to employ Charlson, Elixhauser or individual diagnosis group indicators

• the objective of this change would be to make the predictive models underlying the SHMI more accurate
• at present the Charlson co-morbidity index underpins the SHMI, however, studies have shown that the Elixshauser index or even bespoke comorbidity indicators are more predictive
• the alternative approaches do however add complexity to the models and a balance must be achieved between complexity and explanatory power

A review of the characteristics to be included within the statistical models and the consistency of those characteristics across model. 

• the objective of this change would be to make the predictive models underlying the SHMI more accurate
• at present the SHMI employs the same characteristics in the models for all 140 diagnosis groups
• it has already been shown via the short-term improvements that this is not the most appropriate approach
• due to improvements in data quality, it is now possible to broaden the scope of the data included within the SHMI
• The predictions would therefore be based on more relevant data and be more accurate as a result

The adjustment to be employed for over-dispersion

• the approach employed to create the control limits and therefore classify a trust as “higher than expected”, “as expected”, “lower than expected” has been debated since the initial development 
• a series of new approaches will be considered once the underlying solution has been developed
• an approach must be adopted that acts as a “Smoke Alarm”, while accounting for the inherent variability in the data

Consider the use of different modelling techniques

• the objective of this change would be to make the predictive models underlying the SHMI more accurate
• since the development of the SHMI more data is now available, in addition more advanced modelling techniques can be deployed due to improvements in computing power
• the process would consider which techniques would provide the same if not further benefits to users of the SHMI, while maintaining transparency of the approach and improving the accuracy of the prediction

Inclusion of emergency admissions with a zero length of stay

• there are variations between trusts in the recording of zero length of stay emergency admissions
• some trusts include this activity in the outpatient dataset rather than the Admitted Patient Care (APC) dataset, and so these records are not included in SHMI for all trusts
• when the SHMI was first established, one of the principles was that it should include activity for all admitted patients (except day cases, regular attenders and stillbirths). However, excluding this activity could improve the consistency of the indicator between trusts. This could impact the time series

Further work on the suitability of diagnosis groups, including consideration of suspicion of sepsis

• the objective of this change would be to make the SHMI more relevant to users
• changes may be required to the diagnosis groups employed within the SHMI, of which there are 140 at present
• this will require the involvement of clinical expertise
• certain diagnosis groups may be better reported upon as part of a review of the contextual indicators that accompany the SHMI

We strongly encourage interested parties to participate in this review. If you have any questions or comments related to the Presentational or Methodological changes proposed within this document, please send them to [email protected]. The deadline for the submission of comments or feedback is 4 February 2019.