Skip to main content

Publication, Part of

NICE Technology Appraisals in the NHS in England (Innovation Scorecard) to December 2021

Official statistics

Frequently Asked Questions

Background

What is the Innovation Scorecard?

The Innovation Scorecard is an action taken from The Department of Health’s (DH) paper ‘Innovation, Health and Wealth’ (Dec 2011).

Further information and report can be found at: https://www.gov.uk/government/news/accelerating-adoption-of-innovation-in-the-nhs

More recently, the Accelerated Access Review (AAR) makes a number of references to the development of the Scorecard in particular its role in measuring the uptake of medicines and potential to measure other technologies and in future, the Innovation Scorecard should be the single source of information on the use of innovation in the NHS.

Further details and final report can be found at: https://www.gov.uk/government/organisations/accelerated-access-review

The Innovation Scorecard aims to support and drive NHS compliance with NICE Technology Appraisals (TAs) and was designed to help the NHS identify variation which can then be justified, challenged or acted upon.

It is published by the NHS Digital on behalf of the Office for Life Sciences. The work is informed by collaborative working with colleagues from: the Association of the British Healthcare Industries (ABHI), Cabinet Office, DH, NHS Digital, the NHS, NHS England, NICE, Office for Life Sciences, and the pharmaceutical industry. Recent developments in 2021 include:

  • Introduction of Secondary Care Medicines Data (SCMD) as supplied by Rx-Info and published by the NHS BSA.
  • All medicines are now reported at organisation, regional and national level.
  • Addition of 3 new groupings and revision of existing medicine groupings to align with the 5 High Health Gain policy areas for NHS England and NHS Improvement.
What are NICE TAs?

Technology appraisals  are recommendations on the use of new and existing medicines and treatments within the NHS, including medicines, medical devices, diagnostic techniques, surgical procedures and health promotion activities.

Some medicines and treatments may be covered by more than one technology appraisal. Each technology appraisal may contain more than one recommendation. NICE classify their recommendations into four categories:

  • Recommended - the medicine or treatment is recommended for use:
    • In line with the marketing authorisation from the European Medicines Agency (EMA) or from the Medicines and Healthcare Products Regulatory Agency (MHRA) or
    • In line with how it is used in clinical practice in the NHS
    • or both
  • Optimised - the recommendations have a material effect on the use of a medicine or treatment, and it is recommended for a smaller subset of patients than originally stated by the marketing authorisation. This test of materiality takes into account advice from clinical experts on the anticipated use of the technology in routine clinical practice. In some instances, an optimised recommendation is made because the committee considers that a medicine or technology is only a cost-effective treatment option for a specific group of people; for example in people who are resistant to or cannot tolerate other medicines.
  • Only in research - The medicine or treatment is recommended for use only in the context of a research study, for example, a clinical trial. Often, particularly in the case of promising new technologies, sufficient clinical evidence has not been collected at the time of the appraisal and so the Appraisal Committee is unable to recommend the technology for use in the NHS until further evidence on its effectiveness is available for re-appraisal.
  • Not recommended - the medicine or treatment is not recommended. In most instances, a technology will not be recommended if there is a lack of evidence for its clinical effectiveness or if the technology is not considered to be a cost-effective use of NHS resources, compared with current NHS practice.

The technologies included in an appraisal may not be the only treatment for the condition recommended in NICE guidance, or otherwise available in the NHS. Therefore, if a NICE technology appraisal recommends use of a technology, it is as an option for the treatment of a disease or condition. This means that the technology should be available for a patient who meets the clinical criteria set out in the guidance, subject to the clinical judgement of the treating clinician. The NHS must provide funding and resources when the clinician concludes and the patient agrees that the recommended technology is the most appropriate to use, based on a discussion of all available treatments.

Some medicines and medical technologies previously covered by TAs are now included in clinical guidelines or in best practice guidelines.

NICE Technology Appraisals for Medicines

Further guidance on NICE TAs and those technologies that are considered by NICE guidance other than the TA process is available at: http://www.nice.org.uk/About/What-we-do/Our-Programmes/NICE-guidance/NICE-technology-appraisal-guidance

Where does the data used in the Scorecard come from?

The Scorecard includes a range of data from a number of sources which includes:

  • English Prescribing Dataset (EPD) published by the NHS Business Services Authority
  • Secondary Care Medicines Data (SCMD) as supplied by Rx-Info and published by the NHS Business Services Authority
  • Hospital Episode Statistics (HES) and Quality Outcomes Framework (QOF) data from NHS Digital
  • Mid-year population estimates from Office for National Statistics
  • Defined Daily Doses (DDD) from World Health Organisation (WHO) Collaborating Centre for Drug Statistics Methodology
  • Actual Daily Doses (ADD) from NHS Digital

 

 


Publication of the Innovation Scorecard

When and where is the Innovation scorecard published?

The Innovation Scorecard is published by NHS Digital with the first release in January 2013. Initially published on a quarterly basis the publication has been bi-annual since October 2019.

All releases of the Innovation Scorecard are available on the NHS Digital website and can be accessed at: https://digital.nhs.uk/data-and-information/publications/statistical/nice-technology-appraisals-in-the-nhs-in-england-innovation-scorecard

What Technology Appraisals does the Innovation Scorecard cover and why aren’t all Technology Appraisals included?

A process for selecting the medicines and medical technologies for inclusion was developed to ensure a systematic approach for inclusion within the scorecard. The Strategic Metrics Group and Technical Working Groups agreed a series of inclusion and exclusion criteria, and a rationale for any medicines excluded.

Inclusion criteria for medicines

  • The medicine has a TA published from 2013 onwards, which is positively appraised by NICE. The date will move forward a year as part of an annual cycle starting with the October publication.
  • TAs published prior to 2013, which are deemed to be innovative will be considered for inclusion.
  • The medicine has a TA prior to 2013 but is of interest to the NHS. This may include those medicines which do not have widespread use across the whole service, for example, where there is unwarranted variation or where there is known significant therapeutic value.

Exclusion criteria for medicines

  • Where a significant proportion of data is not currently available, for example, those medicines which can be purchased over the counter, or dispensed via routes other than hospital or community pharmacies, etc.
  • Where the medicine is not new and therefore, not considered to be innovative as the treatment would be well established already.
  • The patient population is small and therefore there is a risk of identification when presenting the data
  • Where inclusion of the data available is misleading or of limited value to the NHS

Inclusion criteria for medical devices and diagnostics

  • Any product receiving a positive NICE Technology Appraisal, or medtech guidance assessment or diagnostic products assessment (and meeting criterion ii) should automatically be considered for inclusion on the Innovation Scorecard.
  • Products subject to guidance less than 10 years old is the indicative timescale for including medtech and diagnostic products on the Innovation Scorecard, but this is a guide not a rule.
  • Heavier weighting should be given to products that are aligned with key priorities e.g. the NHS Innovation Accelerator, the Innovation Tariff or the AAR. Clinical or system priorities (e.g. as stated in the 5 Year Forward View) should be similarly weighted.
  • Any product being included on the Innovation Scorecard should have a clear usage/uptake data source(s).

Inclusion criteria for the estimates report

The inclusion of estimates to show expected uptake versus actual uptake were first included in the scorecard in the September 2014 publication. The estimate approach requires its own criteria to determine which medicines should be selected for that methodology. The “Use of NICE appraised medicines in the NHS in England” report provides historical selection criteria for those medicines included in the estimate approach.

Are there any data quality issues associated with the underlying data that I should be aware of?

Yes, data volumes can vary between publications due to an issue known as backtracking in the secondary care medicines dataset. Backtracking is a process where pharmacy stock control systems alter historical volumes based on the stock issued. On this basis data should be treated as provisional. Users should always use the figures in the latest publication to ensure they are the most up to date figures available. Further information is provided in the Background Quality Report.

Where do I access Trust formularies?

The Innovation, Health and Wealth Strategy included a recommendation that all NHS organisations should publish information which sets out which NICE technology appraisals are included in their local formularies by 1 April 2013.

Formularies are available on individual Trust websites. Please contact the hospitals directly with any queries relating to the formularies.

What is meant by compliance?

Compliance is subject to interpretation, and the measurement of compliance is not simple. An important aspect of this is consideration of the circumstances under which a medicine is administered or a medical technology used, and whether this aligns with the precise recommendations within the NICE guidance. The ”Innovation, Health and Wealth” report sets out actions for the NICE compliance regime, so readers are advised to view this report.

The data presented here is insufficient to determine if an organisation is or is not compliant with NICE guidance.

 

 


Interpretation of the Innovation Scorecard

What is the Innovation Scorecard showing?

The Innovation Scorecard should not be used for performance management. It is intended to identify where variation in the adoption of TAs may exist between healthcare organisations and for these organisations to understand, be challenged and explain any variation. This is based on the assumption that reduced variation will result in improved quality of care.

It is important to be aware that observed use of a medicine or technology may differ for a range of reasons and should not be assumed to definitely indicate either ‘under’ or ‘over’ prescribing or implementation. It is recommended that any exploration of the variation uses a structure such as the pyramid of investigation for special cause variation[1] (see diagram below).

Diagram showing a pyramid with annotations which read from bottom (the widest part of the pyramid) to top - Data; Patient case-mix; structure or resource; process of care; and individual

More likely explanations are listed towards the bottom of the pyramid, and so it is suggested that these are investigated first. The data presented in this publication requires careful interpretation and should be used in conjunction with other indicators and information from other sources that together form a holistic view of use of medicines and medical technologies. For example, a technology may not be the only treatment for a particular condition recommended in NICE guidance, or otherwise available in the NHS. Medicines are generally recommended as options for treatments. Other options may include non-appraised medicines or other appraised medicines. Therefore variation in the use of individual medicines would be expected.

Assessment of compliance cannot be made using currently available data.

Indicators of variation may be more appropriate for CCGs than for Trusts, as medicine use in primary care is more likely to be for the management of common long term conditions. Some appraised medicines are used wholly or mainly in tertiary care hospitals; others are more likely to be used in less specialist hospitals. A medicine which is used for a very rare condition (tens of doses per CCGs or Trust per quarter) would normally be expected to show wider variation between organisations.

Variation in the use of medicines or medical technologies between NHS organisations may be due to a number of valid factors including:

  • Natural variation in populations, both in demographic profile and disease prevalence
  • Variation in presentation to the NHS by the relevant populations
  • Variation in choice of preferred treatment option at the local level
  • Variation in the use of alternative products or procedures
  • Differences in the extent to which local utilisation information is available
  • Differences in services provided between organisations, for example differences in the extent to which a service is provided by primary or secondary care
  • Difference in levels of informed patient dissent to intervention

A detailed examination of the reasons for variation for individual technologies is beyond the scope of this report.

 

[1] Lilford R., Mohammed M. A., Spiegelhalter D., Thomson R. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet 2004; 363: 1147-54.

Is it appropriate to compare the data over the different time periods?

The publications show use of medicines over time. Data on the Innovation Scorecard is presented by calendar quarter which covers up to 5 years of data at the National (England) level and up to 2 years of data at lower NHS Organisation levels. 

It is not appropriate to draw conclusions from direct comparisons of the data in each of the scorecards, as many factors may be responsible for any apparent variation. These include: 

Clinical and demographic issues:

  • differences in populations served, local prevalence of disease, services provided and changes in clinical practice.

Data issues:

  • The NHS reorganisation has led to changes in the individual hospitals included in some Hospital Trusts, and there have been changes in the general practices included in some CCGs.
  • Data which has been standardised by Finished Consultant Episode (FCE) days of hospital care (introduced in March 2014 publication) cannot be compared with data standardised by FCE bed days in previous publications.
At what organisational level is the information available, and why?

Data in the Scorecard are presented in several ways – based on available data at specific NHS organisation levels.

Data for medicine utilisation is reported at National (England), NHS Region level, STP level, CCG and Trust organisation level where appropriate. 

 

How do I compare one organisation with other similar organisations?

The web platform tool allows comparison of medicine use at CCG level and medicine purchased at NHS Trust level. Additionally, maps presented on the tool shows use based on quintile proportions. Data for all organisations are provided in the CSV files.

Why can’t I find my local hospital in this data?

Data is provided at NHS Hospital Trust level. The majority of trusts are made up of a number of local hospitals.

Can individual patients be identified from this publication?

No, we are legally obliged to avoid disclosure of potential identification of individual patients by implementing suppression handling where required.

How and why are medicines standardised?

NHS organisations differ widely in the populations they serve and so data which does not take this into account can be misleading.

For CCG, STP, Region and national data the number of resident patients has been used to take account of the size of population served. The smallest CCG has a population less than 65,000. Larger CCGs have populations of over 850,000.

For hospital Trusts data the number of Finished Consultant Episode (FCE) days of hospital care for the time period under consideration (taken from the Hospital Episode Statistics data) has been used to standardise the data. The values vary significantly, with more specialist hospitals, for example, the Royal National Hospital for Rheumatic Diseases NHS Foundation Trust reporting fewer than 10,000 days of hospital care per year, whereas larger Trusts such as Bart’s Health NHS Trust report over 800,000 days of hospital care per year.

Trust level data should not be compared with the national, Region or CCG data.

This scorecard covers some highly specialised medicines and technologies so differences in use across organisations are to be expected.

Medicines presented as grouped medicines are also standardised either by resident populations, FCE days of hosiptal care or by national registers i.e. QOF. 

How are eligible patient populations estimated?

Data on the number of patients diagnosed with a particular condition, eligible for a specific intervention, or receiving an intervention may be recorded locally but are not reported centrally by the NHS; therefore the eligible population for any TA can be problematic to estimate. To develop estimates of the eligible population, information is required to refine population numbers to the particular circumstances where the medicine is recommended by NICE; from overall disease prevalence to the proportion of patients within a particular stage of a disease and then to the particular indication recommended by NICE. In some cases further details are also required, for example, the proportion of patients likely to discontinue treatment or choose alternative treatments.

What are the limitations with the different data sources?

The NHS currently does not have any robust data reporting or collection process which gathers data which can be used to measure compliance with NICE TAs.

The data sets used in this publication were developed for other specific purposes and are shown here to provide an indication of the implementation of NICE appraised technologies. Therefore users of the Scorecard should be familiar with the limitations of the data when used for this purpose.

Patients can receive medicines from the NHS by a variety of routes. The most common is to receive a prescription from their general practitioner or other community based prescriber, which is then dispensed by a community pharmacy. Hospitals also administer and supply medicines to patients.

However there are a number of other ways patients can receive medicines which may limit the coverage of centrally available data, and result in under-reporting of utilisation. Some of the main areas are described below:

  • Medicines are supplied directly to a patient without a prescription being issued or the supply being recorded in a system which supports central collection of data. For example:
    • Some medicines (e.g. for smoking cessation or contraception) are supplied directly to patients from specific clinics, and this use does not appear in any central data collection.
    • Some mental health services or clinics purchase medicines and supply or administer them directly to patients without using a prescription or order in such a way that relevant information is not routinely captured in national data sets.
    • Some urgent supplies of medicines are supplied directly to patients and this use does not appear in any central data collection, for example, supplies provided by ‘out of hours’ services.
  • There are some known gaps in the central data collections, including:
    • Homecare: Supplies of medicines are delivered directly to the patient’s home, and may not be routinely recorded on hospital pharmacy systems. This service is generally contracted to a commercial supplier by hospital Trusts. The homecare market is a significant proportion of total medicines use in secondary care.
    • Some medicines are provided by an aseptic unit where the medicine has been individually prepared for administration to a particular patient. This data may not routinely appear in the hospital pharmacy systems.
    • Outsourced dispensing: Hospitals are increasingly using local commercial suppliers to dispense medicines to patients attending out-patient clinics and these supplies may not be routinely included in hospital pharmacy systems.
    • Hospitals can contract with the private sector to provide specific care or administer specific medicines, which are not recorded in the data from the commissioning hospital. This is most commonly seen with particular medicines, such as ranibizumab, where specific monitoring is not required.

Some medicines are included in multiple TAs; and some medicines are licensed for other indications than those covered by TAs. The available datasets do not allow identification of utilisation by indication, therefore it is not possible to report use for a particular condition.

What are Medicine Groupings?

In January 2016, medicine groupings were introduced to the Innovation Scorecard. The medicine grouping methodologies have been developed by analysts and pharmacists at NHS Digital, ABPI, OHE, NICE, OLS and NHS England. All methods have been reviewed and agreed by the technical working group for the Innovation Scorecard. Comments and feedback on this new section of the publication would be appreciated. Please email any feedback to enquiries@nhsdigital.nhs.uk.

These groupings have been developed to show combined use of medicines that are options for treatment of a specific condition, and where a TA covers more than one medicine for the same indication or two or more TAs cover the same specific condition. It is more informative to compare uptake of combined options for treatment than only showing uptake of the individual medicines. These groups only contain those medicines with a TA. Other options for treatment may be available but are not reported in the Innovation Scorecard.

Data for the medicine groupings is not comparable to the data published in previous Innovation Scorecard publications.

Actual Daily Dose (ADD) is a new prescribing measure developed for and introduced to the Innovation Scorecard in January 2016. They have been developed where current prescribing measures, such as DDDs, are not available or representative of prescribing practices in England.

  • Unlike DDDs that use a value for a drug (chemical substance), ADDs use a unique value for each presentation based on the actual dose likely to be taken.
  • ADDs assign a unique value for each presentation of a drug based on units (tablets, capsules, patches etc.) and the recommended frequency of daily use (e.g. one a day, three times a day).
  • The general assumption being that each dose is equal to a unit of each preparation.
  • The advantage of ADD is that a measure can be created that matches the prescribing patterns when a DDD has not been calculated; was created for a different indication (e.g. a DDD for DOACs is based on utilisation in secondary care, but different doses of medicines are used in primary care that are not covered by the DDD); the dose is variable depending on age or a dose and hence presentation is specific to an indication.
  • Therefore, different presentations of the same chemical are likely to have different ADD values whereas a single DDD value for a chemical substance is usually applied to all presentations of the substance.
  • For this publication, the ADDs have been created using each presentation strength of the medicines linked with licenced dose regimens and NICE guidance.
What is the Estimates report?

This report includes medicines where it has been possible to estimate the number of patients predicted to be treated with the medicines and comparing that to the observed use. The report is produced and written by NICE. This work is complex due to difficulties in deriving estimates of the number of patients expected to be treated and obtaining appropriate utilisation data for specific medicines and indications. Medicines may have multiple indications (some NICE appraised and some not) and there may be a number of medicines with the same indication. A TA generally recommends a medicine as a treatment option for consideration alongside other options. There are known gaps in medicine utilisation data (such as supplies made directly to patients via the homecare route or by outsourced dispensing).

For January 2016, the estimate report was reformatted to separate the estimates and measures of uptake from the more detailed methodology used to calculate the estimates. The methodology and reference data lists are now provided in a separate annex in the second half of the estimates report. Feedback on these changes made to the structure of the report and the changes to the methodology and the introduction of ranges is welcome. Any comments can be sent to enquiries@nhsdigital.nhs.uk

The Government Statistical Service (GSS) Methodological Committee recommended that estimates should include a range to provide an indication of the level of uncertainty in the data. For this publication all estimates now include a range:

Full details of the calculations are available on Appendix 1: Estimate Calculations within the estimate report.

This report was originally based on the ‘Use of NICE Appraised Medicines in the NHS in England 2012’ report published in January 2014. This publication is a result of collaborative work between NHS Digital and NICE, with contributions from the pharmaceutical industry, DH and the NHS. The report was commissioned by DH, to meet their commitments in the Pharmaceutical Price Regulation Scheme agreements (PPRS 2009 and PPRS 2014). The report can be found at: https://webarchive.nationalarchives.gov.uk/20180328130852tf_/http:/content.digital.nhs.uk/catalogue/PUB13413/

 

 


General Queries

If I have questions specifically regarding the policy and strategy behind the Innovation Scorecard where do I direct these?

Questions regarding policy and strategy behind the Innovation Scorecard should be sent to: officeforlifesciences@bis.gsi.gov.uk

Please quote ‘Innovation Scorecard’ in correspondence.

How can I input into the development of the Innovation Scorecard?

We welcome comments and suggestions to improve this publication. Please use the associated feedback form.

Alternatively, please send them to our contact centre at enquiries@nhsdigital.nhs.uk, quoting ‘Innovation Scorecard’ in the correspondence.

How can I appeal or suggest corrections to the Innovation Scorecard?

Please send them to our contact centre at enquiries@nhsdigital.nhs.uk, quoting ‘Innovation Scorecard’ in the correspondence.

 

 


Glossary

Acronym Definition

AAR

Accelerated Access Review

The AAR aims to speed up access to innovative drugs, devices and diagnostics for NHS patients.

ABHI

Association of British Healthcare Industries

ABPI

The Association of the British Pharmaceutical Industry

ADD

Actual Daily Doses

ATC

Anatomical Therapeutic Chemical

A classification system, where the active substances are divided into different groups according to the organ or system on which they act and their therapeutic, pharmacological and chemical properties.

Drugs are classified in groups at five different levels.  The drugs are divided into fourteen main groups (1st level), with pharmacological/therapeutic subgroups (2nd level).  The 3rd and 4th levels are chemical/pharmacological/therapeutic subgroups and the 5th level is the chemical substance. 

The 2nd, 3rd and 4th levels are often used to identify pharmacological subgroups when that is considered more appropriate than therapeutic or chemical subgroups.

CCG

Clinical Commissioning Group

CG

Clinical Guideline

CMU

Commercial Medicines Unit (Department of Health)

DDD

Defined Daily Dose.

Defined daily doses (DDDs) are a World Health Organisation (WHO) statistical measure of medicine consumption. DDDs are used to standardise the comparative usage of various medicines between themselves or between different health care environments.

DG

Diagnostics guidance

Evaluates single or multiple technologies, particularly in complex care pathways, and uses cost-effectiveness analysis to assess technologies including those which offer additional patient benefit for additional investment.

DH

Department of Health

DQ

Data Quality

ePACT data

Primary Care Prescribing Data. Data on prescribing by primary care collected by the NHSBSA.

FCE Finished Consultant Episode - 

FP10

Prescriptions written in primary care but dispensed in the community

FP10HP

Prescriptions written in hospital but dispensed in the community

HES

Hospital Episode Statistics

HPAI

Hospital Pharmacy Audit Index – a data set owned by QuintilesIMS

Homecare medicines

The supply of hospital prescribed medicines direct to patients in their own homes.

HSCIC

Health and Social Care Information Centre

IH&W Board

Innovation, Health and Wealth Board

Industry data

Sales data provided by the Pharmaceutical Industry

IPG

Interventional procedures guidance

Median

The middle number of a group of numbers;

that is, half the numbers have values that are greater than the median, and half the numbers have values that are less than the median

MHRA

Medicines and Healthcare Products Regulatory Agency

MIB

Medtech innovation briefings

NICE advice designed to support NHS and social care commissioners and staff who are considering using new medical devices and other medical or diagnostic technologies.

Modal

The most frequently occurring number in a group of numbers

MTG

Medical technology guidance

Evaluates new, innovative medical devices and diagnostics that:

  • deliver treatment - like those implanted during surgical procedures
  • give greater independence to patients

detect or monitor medical conditions

NHS

National Health Service

NICE

National Institute for Health and Care Excellence

NOACs

Novel Oral Anticoagulants

NPSG

National Pharmaceutical Supply Group

OHE

The Office of Health Economics

OLS

Office for Life Sciences

ONS

Office of National Statistics

Outsourced Supply

The supply of hospital prescribed medicines direct to outpatients from a contracted dispenser.

PCA

Prescription Cost Analysis

Prescription Services

A department within the NHS Business Services Authority

Primary Care

Primary care is the term for the health care services provided within the local community, acting as the first point of consultation.

Quintile

A quintile is a statistical term describing a division of organisations into five defined intervals based upon the values for the organisations and how they compare to the entire set of organisations. Each quintile contains 20% of the total organisations. The organisations have been ordered from smallest to largest use. The first quintile represents the lowest fifth of the data (1-20%); the second quintile represents the second fifth (21% - 40%) etc.

Secondary Care

Secondary care is known as acute healthcare and can be either elective care or emergency care. Also referred to as hospital Trusts.

SCMD Secondary Care Medicines Dataset supplied by Rx-Info

STP

Sustainability and Transformation Partnerships

The NHS and local councils have formed partnerships in 44 areas covering all of England, to improve health and care. Each area has developed proposals built around the needs of the whole population in the area, not just those of individual organisations.

TA

NICE Technology Appraisal

Trust

An NHS hospital Trust, also known as an acute Trust provides secondary health services. A hospital Trust may include a number of local hospitals, under a single overall management arrangement.

UKSA

UK Statistics Authority

WHO

World Health Organisation

 



Last edited: 28 April 2022 9:31 am