Skip to main content
Blog
Finding the vulnerable before COVID does
When the University of Oxford developed a model to predict those at higher risk of dying from COVID-19, we had a challenge: to securely integrate this risk assessment tool into our data systems to identify exactly who these people were. Lee Gathercole, Technical Architect, explains how we did it.

At the beginning of the pandemic, the Chief Medical Officer for England asked leading academics, clinicians and scientists to create a way of predicting who may be at high risk of serious illness if they catch coronavirus (COVID-19).

Young woman looking out a window

From this, the COVID-19 Risk Stratification programme was commissioned by the New and Emerging Respiratory Virus Threats Advisory Group to produce a data-driven mathematical model.

The University of Oxford led the work and used anonymised GP data to build a model, QCovid®, that identified combinations of factors such as sex, age, BMI, ethnicity and underlying medical conditions to determine an individual's risk of hospitalisation and death from COVID-19.

Historically, risk tools such as QCovid® were given to GP system suppliers to integrate into the GP system and used locally to identify those at high risk. However, the Chief Medical Officer for England wanted this risk stratification tool to be executed centrally. The research needed turning into reality and this is where NHS Digital came into the picture.


The challenge

The University of Oxford’s research paper was submitted for publication and the decision to run it centrally had been made, but the actual technical implementation of the model was not something we had control over. The researchers commissioned Oxford Computer Consultants (OCC) to produce a .Net DLL which implemented the findings of the paper.

This gave us our challenge: to take a risk assessment model that had been baked into an external component developed by a third party and implement it sustainably within existing NHS Digital architecture, to be safely used with identifiable population data to assess people’s risk of dying from COVID-19.

We had a significant architectural constraint in the design. Put simply, we had been thinking it was a case of writing some big SQL statements to produce a risk score.  Now, it was about writing some big SQL statements to organise the data so it could be passed to an external component, which we have to host and hope is quick enough that it can process 20 million records in a reasonable time. 

Not only was this a significant architectural challenge, but also a significant assurance headache. How do we assure that a component we've been given by a third party is behaving appropriately and able to cope with the non-functional demands we are about to impose upon it?

It didn't fail, it was just blisteringly fast.

Which platform?

The Data Access Environment (DAE) is built and managed by our Data Processing Service (DPS) team and is part of our strategic data platform. It is hosted in Amazon Web Services (AWS) and uses Apache Spark and Databricks as the technology stack. There were some usability issues with the DAE (which are being addressed) however its ability to process large quantities of data is second to none.

DAE didn't initially have all the datasets we required, and the Oxford component would not run natively on the platform. The choices were to put time and effort into making it work on DAE or do something tactical which solved an immediate problem. The tactical solution would not address a desire to see this problem as a more strategic one around population health management. Could what we deliver with the population risk assessment be used for future stratification work? Are there some patterns that could be reused for other preventative models?

We opted to adopt a Strategic Data Platform approach and, with great support from our teams, we managed to deliver our solution on the DAE platform.

When we first started out, we had no real sense of how long it would take us to run the full datasets in production. In our test environment, we had 700,000 records and the performance was ok, but our prediction was it would take 14 hours for our first run of all 20 million records we needed to process. This is really where DAE came into its own. It actually took under an hour. This exceeded all our expectations, even to the point where we had thought the job had failed. It didn't fail, it was just blisteringly fast. 


Managing the third-party Black Box

We had to build our architecture around the complexity of the OCC-provided model. Ideally, we would have deployed the model directly into DAE, but the .Net runtime is not supported. So we opted to leverage AWS Lambdas to parallelise the processing and mitigate any performance issues. We did a series of performance tests and identified the optimal balance of number of concurrent processes to execute. This meant that we were able to process 15 million records through the model in 10 minutes.  What is great about this approach is that it is purely an on-demand service. No cost is incurred when we are not processing data through it.

The other consideration with the third-party component was assuring it to run on our platform. Our first decision was that any data we sent towards the model would have to be through an anonymised process. If something unexpected happened, we needed to make sure personal data would not be exposed. We also put it in its own network and prevented any outbound traffic, again reducing the risk of any data leakage. The last step was the PEN (penetration) testing as the final piece of assurance that the model and our implementation of calling the model did not expose any vulnerabilities.

What we could not assure was that the model had been implemented correctly. We validate that the given inputs match the given outputs through our Continuous Integration Pipeline, but we could not validate that a given input produces the correct output, just that it matches OCCs expected output.


Real-life impact

Prior to Christmas, it was unclear how this stratification could be usefully implemented. That was until the vaccination prioritisation came into play. At the start of January 2021, people assessed as being high risk of dying from COVID-19 were prioritised for vaccination in line with guidance from the Joint Committee on Vaccination and Immunisation. The simplest way of achieving this was by leveraging some of the existing data flows used to add people to the Shielded Patient List (SPL). On 15 February we added 820,000 high risk patients to the SPL, all of these have been prioritised for vaccination and are now actively being vaccinated. A further 700,000 people were added on 22 February, taking the total to 1.5 million high-risk patients identified by the risk stratification process.


What next?

This was quite a challenging programme of work. It changed suddenly at the start of January, with a fixed end date based on ministerial commitments. What we have implemented here is a solution that could be extended for future programmes of work specifically around population health management. We have been able to leverage the Strategic Data Platform and implement an approach that makes it possible to 'bring your own algorithm’.

The next phase of work, due to be implemented shortly, is to provide a central view of the processed data for a local clinician to review. We have an opportunity here to provide a much richer level of data than previously available in the Summary Care Record or the GP record, which will help clinicians make informed decisions about the patients they care for. 


Related subjects

We've used the University of Oxford’s QCovid® risk prediction model to identify additional people to be added to the Shielded Patient List (SPL).
Mark Reynolds, Interim Chief Technology Officer at NHS Digital, explains the architecture behind the NHS Shielded Patient List, which is helping to protect the most vulnerable members of society during the coronavirus pandemic.
NHS Digital has published the Shielded Patient List (SPL), which is enabling partner organisations across government to support and protect those who need shielding at this time.

Author

Last edited: 18 June 2021 11:26 am