Skip to main content

Indicator Governance Board - Chair's report 2017/18

Annual report of the Indicator Governance Board

Background and context

The Francis Enquiry into Mid Staffordshire Hospital described the need for mechanisms to bring transparency to health information. Following the introduction of the Health and Social Care Act (2012) and its statutory regulations, the Indicator Governance Board (IGB or the Board) was created to provide the approval function required for items to be included in the National Library of Quality Assured Indicators (the Library).

These regulations not only set out the requirements for the Library but also make it clear that any person or organisation can put forward indicators for inclusion.

In order to encourage the cross-system ownership of indicator assurance implicit in the regulations and promote the benefits of assurance, IGB has for the last two years been operating with the involvement of partner organisations. They include NHS Digital, Public Health England, NICE, NHS Foundation Trusts, NHS England, Health Education England, NHS Scotland, NHS Improvement, NHS Providers, Medicalchain, Hathersage Centre, Sustainability Development Unit, Local Government Association, CQC and Healthwatch England.

To ensure the Board continues to work in such a way that NHS Digital’s statutory obligations relating to indicator assurance are supported and to maintain awareness of its principles and objectives, the Chair of the IGB has committed to reporting on the Board’s activity on an annual basis.

2017 overview

The central remit of the IGB is to provide the approval function required to allow indicator methodologies to be included in the Library. It has established criteria for the robust development of indicators, overseeing a process by which to assure their quality. Since its inception, just under 500 indicators have been submitted for review. IGB secretariat also holds information on a pool of over 8600 additional indicators developed locally which have not yet been forward for assurance.

Expanding coverage from those clinically-based, the first social care indicator was approved by IGB in 2017 as well as the first “green” indicator.

The Board continues to address challenges resulting from cross-organisational resource restrictions, reducing the number of indicators which have been brought through for assurance. As a result, focus has been on indicators within the Library whose approval period had lapsed as well as determining potential quick wins from the pool which would require minimum effort to get ready for approval.

The project to deliver a fully functional Library compliant with statutory regulation continues and is now in Beta testing.

Alongside additional members to cover a broader spectrum of subject matter experts, an independent Chair has been recruited from outside the current member organisations, taking up his post in April 2018.

Message from the Interim Chair

John Varlow image

I took up the mantle of Interim Chair during the search for a new independent Chair; this annual report marks my formal handover to Richard Popplewell who will take the reins in April 2018. I am sure he will continue to take the work of the Board forward, building on its achievements to date. My handover of chairing responsibilities also marks my own stepping down from the Board as I move on in my career. Not only do I wish Richard every success in chairing going forwards, but I would like to say a personal thank you to the IGB members, MRG attendees and support staff behind the scenes who have helped and inspired me. I am sure their ongoing commitment and professionalism will help IGB continue to grow and develop over the coming year.

The IGB works to strengthen the cross-system partnerships required to embed assurance, giving the Board the ability to provide impartial challenge to ensure that only the most robust of indicators are approved for inclusion in the Library. With this in mind, over the past year we have strengthened these partnerships by adding more clinical and Allied Health Professional representation with the added benefit of new members promoting development and use of indicators in their own areas of practice.

Ensuring indicators are well-defined and based on transparent methodologies and robust data provides benefits to the health and social care system as a whole. A recognised and trusted process for quality assuring indicators increases confidence amongst users and reassures organisations that what is being producing is of value. Discouraging the production of poor, inconsistent or unnecessary indicators helps reduce burden and delivers cost savings to organisations and the system.

The operation of the IGB provides a great example of what can be achieved when organisations work together to improve health and care information. The Board continues to take an active role in supporting national reporting frameworks and provides advice to embed common understanding by enabling coherent and system wide understanding in indicator definitions. This helps avoid duplication and reduces the potential for confusing or misleading information.

This report presents a summary of the achievements of the Board over the last 12 months in championing the principles of transparency and quality through the assurance of health and care indicators.

John Varlow signature

John Varlow
Interim Chair – Indicator Governance Board

What are indicators and why are they so important?

Each indicator is an evidence-based measure; its importance is derived from its linkage to whether government policy is carried out, a quality or care standard is being met or a critical business question answered.

Benefits of assuring an indicator

Being able to trust the quality of information used by both commissioners and providers of health and care services to benefit patients is important.

The principle benefit of assurance is that it instils transparency. In being able to access indicators based on transparent methodologies, users can be clear on what is being measured, how it is measured and identify any limitations associated with the approach. Transparency allows scrutiny of methodologies which strengthens associated metadata and, in turn, drives improvements in quality. This means decisions can be made based on the best information, avoiding costly and unnecessary investigation due to a lack of understanding of poor information.

Indicators which have been assessed and approved for their robustness by the IGB are contained within the Library, which directs users to indicators that are based on good data and transparent methodologies. Assurance provides users with confidence in the indicator and benefits producers by allowing them to demonstrate that they can be considered a trusted source of information.

There are other benefits arising from quality assuring indicator methodologies. Assurance challenges those developing indicators to consider why it is needed, whether there is anything already out there and whether commonly understood definitions and methods are being used. It also informs a consistent application of the methodology so that there is common understanding of the results, allowing comparison and benchmarking against like organisations, teams or activities. In advocating the avoidance of duplication or multiplicity of indicators, the IGB plays an important role in reducing burden.

When is a metric an indicator?

Not an indicator:

  1. Dissemination of raw or basic aggregated data or descriptive statistics (such as from a data collection or survey).
  2. Basic descriptive statistics only (such as counts or percentages).

Indicator:

  1. Rates, precision measures (confidence intervals) and/or ratios.
  2. Standardisation techniques or casemix adjustment are used
  3. Statistics referred to/described as being “indicators” such as Provisional Accident and Emergency Quality Indicators for England.

Benefits of the IGB

There are thousands of health and care related indicators with numerous examples of duplication or instances of similar indicators reporting on the same subject (often constructed in different ways). Additionally, new indicators are being added by organisations all the time. This can result in confusion amongst users and creates unnecessary cost burdens to producers.

IGB illustrates what can be achieved when organisations are willing to work together to promote quality in health and care information. A key success is the recognition amongst partner organisations in the value of shared ownership of assurance.

The IGB discusses all governance issues of those indicators that come to them for approval, such as duplication, general strategic value of new indicators, impacts across organisations plus issues relating to information governance and interpretation.

Members of the IGB play a key critical role in its validation that indicators are developed in a robust and transparent manner for the benefit of the entire system. To this end, IGB plays an instrumental role in helping to shape the culture around the development of good quality indicators within health and adult social care.

Indicators are succinct measures that aim to describe as much about a system as possible in as few points as possible. They help us understand a system, compare it and improve it. Indicators are extremely important forms of measurement but they can also be controversial – like all powerful tools, they can easily do as much harm as good.

David Pencheon
The Good Indicator Guide

National Health Service Institute for Innovation and Improvement, 2008

Criteria used in the assessment of quality

Clarity - Is it clear what the indicator will measure?

Rationale - Are the reasons and evidence for measuring this clear?

Data - Are the data in the measure fit to support the purpose?

Construction - Will the methods used support the stated purpose? Is it clear what methods are used and how they have been justified?

Interpretation - Is the presentation of the indicator suitable and are all potential users able to interpret the values?

Risks and usefulness - Are any limitations, risks or perverse incentives associated with the indicator explicitly stated? Can the indicator be used for quality improvements?

Meeting statutory regulation

The Health and Social Care Act 2012 regulations called for the establishment of a database of quality indicators to hold indicator specifications, subject to an assessment of the appropriateness of describing it as a “quality” indicator.

This National Library of Quality Assured Indicators contains assured specifications along with a marker showing the “level of confidence” that the indicator is quality plus an assessment review date. The Library helps users of indicators by providing a trustworthy, centralised point for finding quality assured indicator metadata, enabling them to recreate the indicator should they wish.

NHS Digital was assigned responsibility for developing and maintaining the Library. IGB provides the approval function for the Library, ensuring impartiality and a cross-system perspective of indicators.

The usefulness, purpose or source data of indicators may change over time. A component of IGB’s activity is to set regular review dates so assured indicators can be re-assessed. Over the last year the IGB has reviewed a number of indicators with lapsed certification, either agreeing a renewal or retiring from the Library due to being superseded by newer or revised indicators.

Figure 1: Indicator process flow

IGB and statutory regulation

Part 3 of The National Institute for Health and Care Excellence (Constitution and Functions) and the Health and Social Care Information Centre (Functions) Regulations 2013, number 259, sections 24-30 detail the requirements of an indicator and the library of quality indicators. The Health and Social Care Information Centre (now NHS Digital) has responsibility for developing and maintaining the Library, however the regulations also make clear that for indicators to be included in the library, approval is required. IGB fulfils this approval function, ensuring impartiality in any decision making.

Regulations also provide the framework for assessing indicators. The IGB have approved a set of criteria and considerations which are used when assessing the quality of indicators. These build on best practice from across the health and care system and reference guidance such as the “Good Indicator Guide”, “Evaluating the Quality of Clinical Health Indicators” used in the Compendium of Public Health indicators and learning taken from the development of Indicators for Quality Improvement (IQI).

Relationship with other groups

IGB sits within a wider network of bodies involved with information standards and the assurance of health and care information. It undertakes work in collaboration with other National Boards and stakeholders in order to support the development of common standards for indicators (including methodology, metadata and assurance) in a manner compatible with statute.

IGB is accountable and reports to the National Quality Board which provides a forum where key NHS oversight organisations can come together to share intelligence, agree action and monitor quality. It liases with the Data Standards Assurance Services (DSAS) where cross cutting issues relating to information standards are identified and providing recommendations on indicators which relate to new or existing standards.

Representatives attending the IGB accept the responsibility to keep their organisations informed of the decisions of the Board.

IGB relationship with other bodies

Membership

To encourage cross-system ownership of indicator assurance implicit in the regulations, the IGB operates on the basis of shared sovereignty between its members. The IGB has sought to maintain continuity in organisational representation as well as reaching out to parts of the system not previously involved. In the past year, representation has been added from NHS Scotland and NHS Providers as well as subject matter expertise provided by clinicians and allied health professionals.

Following a recruitment campaign, an independent Chair was appointed as of 1st April 2018. Secretariat to the Board is provided by the NHS Digital Indicator and Methodology Assurance Service (IMAS), reflecting that NHS Digital carries a statutory obligation to maintain the Library.

IGB membership

Pragmatic partnership

One of the projects being undertaken by NHS England looks at improving access to general practice in order to assist in better management and provision of appointments. There is a substantial demand to understand capacity and waiting times for appointments.

The Third Next Available Appointment is intended to provide an indication of the number of days a patient would theoretically need to wait for an appointment at the time a booking was made. It does not take account of patient preferences such as if the patient chooses to make a later appointment with a particular clinician rather than one who was available earlier.

IGB initially reviewed the indicator in October 2017 noting that in this development phase, standard deviation values were only available from one GP system supplier, accounting for 36% of practices.

Although the application itself was of good quality, the opinion was that additional work was required to ensure all the methodology is transparent.
Recognising that data quality is an area of concern, IGB approved the indicator, cautioning that it should not be used for comparative purposes without an agreed confidence interval methodology.

Alignment with the Code of Practice for Official Statistics

The IGB ensures impartiality; integrity and transparency are at the heart of delivering indicator assurance. Upholding these principles align assurance with the UK Statistics Authority’s Code of Practice for Official Statistics. In particular Principle Four of the Code of Practice focuses on the need for statistics to be produced to sound methods and assured quality.

The “criteria and considerations” endorsed by the IGB allow the assurance process to document how practices of the Principle are met, including:

  • ensuring official statistics are produced according to scientific principles.
  • details are published of the methods adopted, including explanations of why particular choices were made.
  • ensuring that official statistics are produced to a level of quality that meets users’ needs, and that users are informed about the quality of statistical outputs.
  • adopting quality assurance procedures, including their coherence with other statistical products.
  • seeking to achieve continuous improvement in statistical processes by, for example, undertaking regular reviews.

Common reasons applications are not assured

  1. No sponsor
  2. Lack of clear description
  3. Purpose not clearly stated
  4. Exclusions missed or not justified
  5. Incorrect methodology in indicator or underlying statistical model
  6. Appropriate statistical adjustments either missing or incorrect
  7. Data source not justified
  8. Lack of clarity as to how it will be interpreted or acted on
  9. Indicator contradicts methods used in similar indicators
  10. Lack of transparency – others can’t recreate the indicator 

Objectives for 2017

The IGB Chair’s report for 2016 set out a number of key opportunities and challenges facing the Board in the coming year. These include:

  • promoting indicator assurance within organisations represented on the Board, and make links within the wider health and care system
  • ensuring that indicators being assured are prioritised in line with stakeholder requirements
  • ensuring links are made with major reviews relating to health and care information, such as the Carter Review and Kings Fund review of NHS Outcomes Framework
  • demonstrating the work already done by the Board and what services are available, such as the Library providing a central resource for assured methodologies
  • continuing to provide governance in the assurance, encouraging a lean, customer focused process
  • engaging with the analytical community to embed the principles of quality indicators into the development process
  • broadening membership of the Board
  • re-launching IGB with a clear mission statement
  • ensuring that the governance function is seen by the wider system to add value rather than burden
  • delivering education and training to those in the business of indicator development with the goal of driving up the quality of applications coming to IGB, thereby reducing the timescale for approval
  • working with health and social care organisations to imbed governance of indicators developed locally
  • addressing indicators whose period of assurance has lapsed

Work has been undertaken during 2017 to deliver against the above and to embed the Board’s position as the central point for indicator assurance in the system. This has been done in line with the principles set out in the statutory obligations relating to assurance

Meeting objectives

Considerable promotional activity was carried out by IMAS on behalf of IGB in fiscal year 2017/2018. This includes discussion with 44 different stakeholders to introduce indicators and the benefits of IGB assurance. A further 35 are currently in the pipeline.

A full Creating a Good Quality Indicator training programme complete with slides and trainer guidance notes have been created and are available for organisations wishing to increase the quality of applications.

With IGB’s support, three organisations have imbedded their own indicator methodology reviews based on the forms and process used by IGB. The follow-up objectives throughout 2018 are to both ensure consistency across all organisations as well as upgrade at least 10% of local reviews of appropriate indicators to inclusion into the Library.

Representation from a total of five new organisations have been added to IGB since the last Chair’s Report: The Hathersage Centre, Medicalchain, Mhabitat, NHS Providers and the Sustainable Development Unit.

Indicator library, repository and directory

Indicators are held in three different categories:

  1. Library: has undergone rigorous quality assurance and has been approved by IGB
  2. Repository: submitted for assurance, either in process or has been halted by applicant
  3. Directory: local indicators from the delegated methodology review submitted, potential to undergo assurance

The usefulness and purpose of indicators may change over time. To account for this IGB members agree regular review dates so assured indicators can be re-assessed. As of 31st March 2018, 32 lapsed indicators have commenced with reviews of the methodology as a preparation for submission to the IGB.

Indicators that have been or are going through assurance

As of 31st March 2018, in the past year since the last Chair’s report:

  • 30 indicators / metrics have been reviewed by IGB
  • 11 have been rejected by IGB with support being given to revise the applications
  • 2 lapsed indicators have been reviewed and approved
  • 32 previously assured indicators are in the process of being reviewed
  • 70 indicators have been placed on hold by their submitter

In total:

  • 486 indicators have been received for assessment
  • 126 indicators which meet the quality criteria are currently in the Library
  • 5,338 indicators are in the Directory

Independent methodology review

The IGB seeks to take advice on the quality of indicator specifications from those with expertise in indicator construction in order to help inform the decision to approve indicators for the Library. At present this advice is taken from the Methodology Review Group (MRG) which is hosted by NHS Digital on behalf of the health and care system. Public Health England is also piloting a delegated review group which will further increase capacity once fully established (see above).

Since the beginning of 2017 those organisations represented at IGB who did not previously have organisational representation on the MRG were encouraged to put forward candidates. In tandem with this activity, work has been undertaken to expand the “independent” membership of the group (i.e. members not attending on behalf of an organisation). This has strengthened the diversity of experience within the group and re-enforces its impartiality. Along with NHS Digital, who chairs the group, representation is drawn from NHS England, Public Health England and NHS Improvement. In addition independent representatives have joined the group including those with backgrounds in academia, public health consultancy and the charity sector.

The independence of the process is further enhanced through the use of Peer review. Over the past year the IGB has continued to develop peer review networks to engage reviews from clinicians, GPs, Allied Health Professionals, statistical methodology experts, epidemiologists, academia, independent health journal reviewers and those with prior experience of indicator development.

Delegated methodology review

There are thousands of health and care related indicators with numerous examples of duplication or instances of similar indicators reporting on the same subject. Additionally new indicators are being added by organisations all the time. The IGB has agreed that the capacity for assuring indicators can be increased by allowing other bodies to share the outputs of their methodology review with the IGB. This “delegated methodological review” supports the collaborative way of working between the partner organisations within the IGB.

The IGB needs to be satisfied that the Delegated Methodology Review arrangements are consistent with the established indicator and methodology assurance process (i.e. consistent quality criteria, transparency, inclusion of independent reviewers) and abide with the spirit of statutory regulations.

In the last year Public Health England has been trialling methodology review with outputs anticipated to be brought to the IGB during 2018.

Indicator breakdown

Indicators are often not presented as stand-alone measures, but as part of a wider set or “framework” of indicators. These can be indicators that together provide an overview of the performance of a wider system, such as the NHS Outcomes Framework (NHSOF) and Adult Social Care Outcomes Framework (ASCOF). They can be to provide information, e.g. the CCG Outcomes Indicator Set (CCGOIS), encourage and monitor improvement, e.g. the CCG Improvement and Assessment Framework (CCGIAF) and A&E Quality or to provide incentives, e.g. Payment by Results (PbR). In assessing the quality of these indicators the IGB looks not only at the methodology of the indicator itself but how it fits with the purpose of the framework. Although the IGB seeks to encourage consistency in definition, differences in how an indicator is intended to be used can mean using the same methodology is not appropriate (i.e. a slight change in the question being asked may affect what should be included and excluded in the indicator).

There are however examples of indicators assured by the IGB that are not part of frameworks, such as the Summary Hospital level Mortality Indicator (SHMI), Patient Reported Outcomes Measures (PROMS), GP Workload Tool and academic research. Consideration is also given as to any wider purpose or context surrounding these indicators.

Indicators in the Library, by area

ASCOF Adult Social Care Outcomes Framework Other Third next available appointment
CCGIAF Clinical Commissioning Group Improvement and Assessment Framework PROMS Patient Reported Outcome Measures
CCGOIS Clinical Commissioning Group Outcomes Indicator Set SDU Sustainable Development Unit
IAPT Improving Access to Psychological Therapies SHMI Summary Hospital-level Mortality Indicator
NHSOF NHS Outcomes Framework 7DS Seven day services

Within the current year, 61.48% or 5,338 of the 8,682 available indicators have been uploaded into the Directory. Items originating from NICE, NHS England, Health Education England and NHS Digital are still to be addressed.

Summary of uploads into the Directory

Composition of library, repository and directory

Library indicators by organisation

At 96%, by far the most assured indicators within the Library come from NHS England with 2% from DH; roughly 1% each originate from NHS Improvement plus independent researchers and academia.

Repository indicators by organisation

Just under 75% of all indicators within the Repository originated from NHS England with the next highest from the Department of Health. Slightly under 1% were from NHS Improvement with about 2% from independent sources.

Directory indocators

Of the over 8,600 indicators eligible for uploading into the directory, more than half were initiated by Public Health England with NHS England responsible for just over a fifth.

Case Study: Core carbon footprint of NHS trusts and foundation trusts

The Climate Change Act (2008) was introduced to ensure that the UK cuts its absolute carbon emissions by 80% by 2050 and 34% by 2020, calculated on the baseline set in 1990.

The Act enables the UK to become a low carbon economy, setting in place national, legally binding reduction targets which are driving carbon reduction as well as climate change adaptation.

At over a third of all public sector emissions, the NHS is its largest producer and has a duty to respond to these targets. The health and social care system has the most to gain from reducing greenhouse gas emissions from its own activity due to the severity and diversity of the impact on human health therefore NHS providers must be part of the solution to reduce the burden on health.

Rick Lomax, Sustainability Analyst in the Sustainable Development Unit, first approached the IMAS in May 2017 to discuss the process and what would be required. It was explained that an Information Analyst would be assigned to support the application and a number of discussions between the two ensued as the content was addressed. IMAS explained that it was possible to request that the Methodology Review Group (MRG) provide advice on how to strengthen the application before its formal submission.

At its review in November 2017, MRG requested that the paperwork deliver greater explanation around the estimation associated with the indicator and the potential degree for error with some sensitivity analysis provided. With a number of potential indicators proposed, it suggested that the submission focus on the absolute emissions with a floor area denominator as primary with others detailed as supplemental. MRG was also not clear on how the indicator would be used to compare a Trust’s data over time as well as benchmark against other Trusts’ performance.

Returning to MRG the following month after addressing all of the queries captured in the Appraisal Log, data quality was the only area of concern; MRG’s view was that a caveat should be included that data should be used with caution but recommended that the indicator proceed to the IGB.

IGB forms the final stage of the indicator assurance process and the indicator was duly presented at its next meeting on 18th January 2018. Board members agreed that the application had much merit and it was approved for inclusion with the caveat into the National Library of Quality Assured Indicators for a period of one year.

The SDU are very happy to have an approved indicator as it sets a clear standard for calculating core carbon emissions in NHS trusts and wider whilst helping raise the profile of environmental and sustainability management in the health sector.
The process was just as useful and beneficial as the outcome as it provided useful challenge, drove justification of the indicator and has now developed a more robust indicator because of the engagement from MRG and IGB. The IMAS team and MRG were clear and timely in their support and communication which made the process far more seamless than I had anticipated from the outset.

Rick Lomax
Sustainable Development Unit

Feedback

Any feedback should be made to the Indicator Governance Board. Likewise, if you are unclear regarding any of the content in this report or have any queries about the assurance process in general, please contact IMAS:

Indicator and Methodology Assurance Service
NHS Digital
1 Trevelyan Square
Boar Lane
LEEDS
LS1 6AE

Email: igb@nhs.net
Website: https://digital.nhs.uk/services/indicator-methodology-and-assurance-service
Follow us at LinkedIn: https://www.linkedin.com/showcase/nhs-digital-igb/

Last edited: 11 February 2019 9:46 am