Skip to main content

Using logic models to assess digital health products

How do you evaluate the success of a digital health product? Ian Roddis and Vicki Litherland explain how digital delivery teams from NHS Digital and Public Health England worked together to assess whether the NHS website is fit for purpose.

By Vicki Litherland and Ian Roddis. 9 July 2019

Staff from the NHS website project team take part in a logic model workshop

Our combined team wanted to find out how successful the NHS website is in achieving its intended health and health-related outcomes. We used a logic model to help us.

What is a logic model?

A logic model is a tool to help you visually represent how things like inputs, outputs and activities relate to each other and identify how a product will produce short, medium and long-term outcomes.

We found that creating a logic model for the NHS website helped bring everyone together to discuss the health outcomes relevant to the service, as well as debate different ways the service could achieve them. Health outcomes are the direct and specific results of a product, targeted at a specific group of users. This included both high level ‘mega mission’ outcomes as well as ‘vertical journeys’, for example on a specific condition.

Visual of a generic logic model, showing hypothesis and assumptions and vertical columns, with blank post it notes at the top, with other columns

Creating a logic model for the NHS website

The NHS website is a mature product (more than 10 years old) but is in a period of improve it.  We held a workshop with the website delivery team and other people involved in the project to develop our logic model. Creating it collaboratively as one big NHS Digital/ Public Health England team was an effective and fun way of getting everyone to agree on what’s important, consider all perspectives and ensure we had included all available information. This built on recent work to confirm our mission for the NHS website and we used the outcomes from both sessions as the frame of reference for evaluating success.

When people look for ‘care and wellbeing information’ the content, tools and services they come across should be easy to use, clinically safe and trigger the appropriate action relevant to them.”

Although we designed our logic model for digital health products, you can use a logic model for any product or service that you’re working on. Ideally, you should start creating your logic model in the early stages of developing your new product or service, so everyone is on the same page and return to it regularly. Logic models should inform the design of your evaluation, data collection and product.

How do you create a logic model?

The steps for creating a logic model are:

1: Hypothesis and assumptions

Explain the problem you’re trying to solve or the theory you’re testing. We formed this hypothesis for the NHS website.

“When people look for ‘care and wellbeing information’, the content, tools and services they come across should be easy to use, clinically safe and trigger the appropriate action relevant to them.”

 

Consider any assumptions you have about what your product is going to do and the changes you expect it to achieve. One assumption we have is that people want to self-diagnose and have evidence for their own prognosis, so they’ll follow the advice on the NHS website and know when it is appropriate to contact a health professional. Another is that people will use the website to quickly find the most appropriate service local to them.

2: Outcomes

Think about how your product can benefit its users. List the short, medium and long-term outcomes that your product is trying to achieve.

An example of a short-term outcome is that patients feel able to take appropriate action ‘now’. They would then feel more in control of their health, which is a medium-term outcome, potentially reducing anxiety and leading to greater empowerment. A long-term outcome is increased life expectancy. Many short and medium-term outcomes contribute to a long-term outcome. Remember this is a hypothesis, a logic model builds on assumptions which may be proven over times as evidence is revealed.

3: Inputs and activities

List the resources you have available for your project, including:

  • budget
  • equipment
  • previous research and evidence
  • the people and organisations working on it

Decide on and record the processes, events, technology and actions you’ll use to build your product. For example, the NHS website programme has a large multidisciplinary team with relevant skills available to maintain and update the website using an agile approach.

4: Outputs

Consider how your product and the interactions with your users can contribute to achieving the selected outcomes. An output for our team is that we must ensure the content on the NHS website is safe and accessible to consume wherever it’s used – whether on our website or via content syndicated elsewhere – so that users receive the help they need when they access the service.

Think about how your product may also negatively impact these intended outcomes. It’s important to always consider unintended consequences and risks associated with your product.

5: Context and barriers

Describe the users of your product and the policies the product is designed to support. For example, users of the NHS website are at varying stages of their lives and have a wide range of socio-economic needs.

Discuss any constraints that may stop you from building your product or prevent it from being successful. Make sure you consider the context in which your product sits – whether that be political, environmental or societal factors – or simply competition.

NHS website project team staff participate in a logic model workshop

Downloads

If you want to set up your own logic model in a workshop setting, the documents below explain how you can develop it.

To find out more contact the evaluation team at Public Health England, which includes Kassandra Karpathakis (product manager), Ros-Mari Mitova (delivery manager), Vicki Litherland and Iain Cooper (content designers), Claire Rackstraw (user researcher), Felix Greaves (service owner), and Charlotte Fountaine (service designer). Without their work this project would not have reached the levels of success it has achieved.

Last edited: 5 August 2019 8:59 am