Skip to main content

How we assess health apps and digital tools

All products published on the NHS Apps Library must meet a set of standards.

How the assessment works

Developers will need to answer a range of digital assessment questions (DAQ), to make sure that only safe and secure apps and digital tools are published on the NHS Apps Library.

The questions in the assessment have been designed by experts from technical and policy backgrounds. They:

  • cover national standards, regulations and industry best practice
  • help us to see how a product performs against important criteria

The number of questions depends on the product's:

  • complexity
  • potential clinical effectiveness
  • data protection responsibilities

The more complicated the app or digital tool, the more questions that'll need to be answered.

The questions examine evidence of how:

  • a developer's product performs
  • users could see benefits to their health and wellbeing
  • the product passes our tests

Step 1: check eligibility

Any product must meet the following criteria to be considered for publication on the NHS Apps Library.

It's essential that:

  • the product is already available to the public in the App Store, Google Play, or by other means
  • the developer can be contacted directly by product users
  • the product doesn’t use any form of NHS branding, unless permitted to do so
  • the developer must have evidence that a proper interoperability review has been done, if the product connects to any NHS services
  • the product must be registered and have the relevant certification, if it's a medical device
  • the product must be registered with the General Pharmaceutical Council, if it provides a pharmacy service
  • the developer must provide the healthcare professional's registration status and names, if the product requires registered healthcare professionals to operate it
  • developers are registered with the Care Quality Commission, if required
  • developers can provide a guest login for use by those assessing their product

The developer's organisation needs to be registered as one of the following eligible types:

  • Public Limited Company (PLC)
  • Private Company Limited By Shares (Ltd)
  • Company limited by guarantee
  • Unlimited Company (Unltd)
  • Limited Liability Partnership (LLP)
  • Community Interest Company
  • Industrial and Provident Society (IPS)
  • Royal charter
  • Public body
  • Charitable organisation

Step 2: register details

Developers will be asked for information about their organisation and the product they're submitting for assessment.

We'll need to know:

  • the registered address of the developer's organisation
  • contact details for the person managing the assessment
  • details of the developer's Care Quality Commission registration, if the developer is required to have one

We'll then ask for information about the product, like:

  • what health theme it addresses
  • who the intended users are
  • how much it costs

This section also covers the developer's business model, device registration and use of NHS branding.

This information provided will be reviewed by NHS Digital and any extra information required will be requested.

Developers will be assigned an assessor to help them complete their product's technical assessment.

Step 3: technical assessment and standards

The image below shows the stages involved, followed by detail to explain it.

Process flow of the technical assessment of apps

Available evidence on outcomes

We must make sure that all products do what they say they do.

We will ask developers to show us how their product improves health and wellbeing.

For example, if an app is designed to help patients with their mental health, developers must give examples of how it could help people or has already done so.

We also ask if there is any evidence of the clinical, economic or behavioural benefits of using a developer's product.

This could be how it has helped to improve symptom control, clinical outcomes or patient reported outcomes.

Clinical safety

We must make sure that developers have taken all appropriate action to keep patients safe using their product.

For example, if an app reminds patients to take their medication, developers must give evidence to show that any risk of reminders being incorrect has been completely removed or made as low as possible.

Developers of any product that could put a user at risk must meet the clinical risk management standards required by the Health and Social Care Act 2012.

This would mean producing Hazard Logs and Safety Case Reports, which would be reviewed by experts at NHS Digital.

Data protection

We must make sure that any personal information collected or shared, by an app or digital tool, is handled in a safe, fair and lawful way.

This would include health information recorded by the user, like diabetes readings, or available from a person’s health record.

The UK Data Protection Act 2018:

  • gives people rights and control over their information
  • places greater responsibilities on organisations to use people’s information appropriately and securely

The developer must:

Security

Here we:

  • assess the security assurance of an app or digital tool
  • make sure that a user’s data has been correctly categorised, taking account of data protection regulations and clinical impact

We also ask for confirmation that a security assessment has been carried out against applicable Open Web Application Security Project standards.

Usability and accessibility

We need to make sure that a person can understand and use an app or digital tool effectively.

Text must be clear and easy to read.

Action buttons must be big enough, easy to press and marked with commands that make sense to users.

The product's functions must do what the user expects and not perform any extra actions that are not made clear.

All products are assessed against Web Content Accessibility Guidelines 2.1, which are the agreed international standards for digital accessibility that all web content must satisfy.

This:

  • is to make sure that products provide access to as many people as possible, including older users, younger users and those with disabilities
  • might involve being able to increase text size where needed and work with voice software to help people with visual impairment

The usability of an app or digital tool must satisfy the International Organization for Standardization’s requirements and recommendations for human-centred design principles and activities throughout its life cycle.

Interoperability

We need to test how well a product exchanges data with other systems.

For example, how it connects with a patient’s medical record, or collects readings from another device, like a smart watch or a blood pressure monitor.

This process helps developers to use data within their products, to build new functions that benefit users.

To do this, developers use Application Programming Interfaces (APIs), which allow third parties to view a product’s data in a more digestible format.

Not all apps exchange data, but those that do must adhere to NHS England’s Open API policy.

These rules make the sharing process simple, while also keeping it safe and secure.

Technical stability

These questions are used to understand how an app or digital tool has been tested and how testing will continue over time.

Developers must show how patients can report any problems with a product and how the developer will work to correct them.

These questions also cover what happens to any patient information a product has collected if the patient stops using it, or it's shut down by the developer.

Step 4: get published on the library

We will publish details about the product on the NHS Apps Library, after the assessor tells the developer that the product has successfully completed the technical assessment.

Once the product is published:

  • developers must tell us when they make any significant updates to their product, as it might need to be reassessed
  • we will tell the developer about any changes to the DAQ that mean the product needs to be reassessed for it to remain on the library

The DAQ roadmap for developers highlights upcoming changes in regulations that we are working to include in future versions of the DAQ.

Preview the questions

We know that it helps developers to see the digital assessment questions before they start the assessment process.

The questions are included here:

  • as a heads-up for developers to know what to expect
  • for information purposes only and can't be submitted for assessment

How the questions are updated

Technology improves, so standards and regulations have to keep up.

It is very important that the digital assessment questions (DAQ) are frequently reviewed and updated, to make sure that products on the NHS Apps Library maintain expected high standards.

We work closely with our experts and partner organisations to review and update the DAQ.

The DAQ roadmap for developers shows planned changes based on relevant national standards, policy and regulations for apps and digital tools in healthcare.

The product will need to be reassessed if there are any major updates to it.

Last edited: 30 August 2019 12:24 pm