This is the largest vaccination programme the NHS has ever delivered. We’re now approaching the milestone of 61 million vaccinations and with each jab in the arm we’re a little closer to returning to a more normal way of life.

This kind of success does not come easily. There have been some difficult moments for our team along the way as we developed an architecture to glue together the three different delivery models which evolved on the ground very quickly. These are:

1. Hospital hubs – Includes hospitals administering vaccines to their own staff and certain patients within the hospital setting.
2. Vaccination centres – Mass vaccination of the public, booked via the National Booking System.
3. Local vaccination services – This includes GPs, community pharmacies, roving vaccinations in residential care (care homes) and for housebound people, and services for detained estates (including prisons).  

The Data Processing Service (DPS)

Pivotal to bringing these different models together was the development of a single data processing architecture with NHS Digital’s Data Processing Service at its heart.

The DPS  is an Amazon Web Services (AWS) service running Databricks and making use of many cloud-native AWS managed components.

It is capable of receiving large amounts of vaccine event data from the point-of-care applications and processing this using a set of data quality and other business rules.

The Data Processing Service can also forward the relevant data to authorised 3^rd parties such as the patient’s GP record and the Medical and Healthcare Products Regulatory Agency (MHRA) to allow them to monitor adverse reactions.

One of the key functions of DPS is to undertake data quality checks to drive up the quality of data captured, received and processed downstream. We are currently on target to drive down the error rate to 0.11% which, given the scale of the programme and the diversity of delivery models, is a massive achievement. This has been very important to the recently launched NHS COVID Pass service, which uses this data.

The National Booking Service 

Our NHS.UK team developed the National Booking Service, which allows the public to book into the mass vaccination and community pharmacy sites. There is a lot of tech behind the public-facing Book a Coronavirus Vaccination service to identify the individual booking and to ensure they are eligible to book.

It also includes a staff-facing system for contact centre teams, vaccination site administrators and a check-in app for when the patient arrives at the vaccination site. My colleague James Higgott published a really interesting blog post on the National Booking Service, which focuses on the importance of service design.

Given this is an open and publicly available service, we needed to manage volumes and protect the user experience and overall service during spikes. For instance, we see large spikes in volumes when media outlets, such as the BBC, run news stories. 

Behind the scenes we have implemented “waiting room” technology, which people are more familiar seeing when booking tickets for popular events such as stadium concerts. In addition, the overall NHS.UK service supporting National Booking Service utilises the Akamai content delivery network to ensure static content such as web page text and images is delivered to end user’s devices more efficiently.

The booking service has been designed to scale up (and down) through use of Azure cloud-based services, with the waiting room technology kicking in when the extreme spikes occur. This allows users to be held in a queue and then go on to book their appointment after a short wait.

NBS has exemplified our API-first strategy, adopting our latest set of FHIR based APIs for accessing patient demographic data alongside other supplier-based APIs.

Tried and trusted….

Alongside the new services in other parts of the architecture, we have reused and repurposed tried and tested services such as MESH (Message Exchange for Social Care and Health) to flow data between services such as the patient’s GP. MESH has dealt with extreme volumes as it processes vaccination data, as well as Covid-19 test results alongside the existing traffic across the NHS.

We made use of standardised mechanisms of data exchange, such as the FHIR Digital Medicines APIs which we had originally developed to allow pharmacy systems to update a patient's GP record following a flu vaccination.

The combination of developing new services alongside the reuse and repurposing of existing services has been fundamental to the success of the programme combined with blood, sweat and tears across the teams and suppliers involved.

Evolving local and national models

From the first day I joined the programme back in November, one of the big unknowns was the availability of the vaccine and how quickly doses would be administered. That made assessing the volume and performance requirements of the various IT components very difficult. 

Earlier I talked about what we did with the National Booking Service, introducing waiting room technology, but with other frontline services we did not have that luxury and had to make assumptions around the expected volumes.

One of the areas where we hit problems was the point-of-care applications as we hadn’t fully anticipated the ingenuity and efficiency of our frontline NHS colleagues.

We had assumed the model would simply be ‘patient arrives, record is opened and vaccination record is written’. Instead, many vaccination sites organised themselves as ‘human snakes’, whereby the data collection elements were broken down and individual stations were created – for example, individual stations for patient check-ins, for finding the patient in the Personal Demographic Service, for asking eligibility questions, and so on. This resulted in over a 5-fold increase in database read and writes because the process was split up across multiple admin staff as patients flowed from station to station.

Some of the systems involved were not cloud-based so we didn’t have the luxury of ‘auto scaling’. In order to cope with the volumes, multiple instances of the infrastructure were ordered and installed over a couple of weeks. Thankfully, we got the capacity we needed due to the commitment from the teams and suppliers involved.

The other element causing challenges is the edge cases. All this is classic NHS stuff, dealing with cross-border patients (from the devolved administrations), patients not registered with a GP practice and people with no NHS number.

The technical challenges revolved around how the data processing pipeline should manage these edge cases whilst ensuring a record of the vaccination event, for example:

• The patient has no NHS number (thus doesn’t exist in our systems),  so how do we record the vaccination event? 
• Welsh, Scottish and Northern Irish patients aren’t in all our systems either but can be vaccinated in England (for example, they work in an English hospital). We therefore need to send the vaccination data to the relevant country’s vaccination programme (this goes the other way for English patients as well)

The actual percentage of patients impacted is relatively small. However, the impact on the individual can be high and can create unwanted stress. We have been working with colleagues across the devolved authorities to agree mechanisms to manage these patients and to ensure they are not disadvantaged.

We introduced a change to the Summary Care Record Application to allow other care settings such as emergency departments access to a patient’s COVID-19 vaccination history.

Other solutions involved introducing new processes and working with the National Back Office to ensure data could be corrected to allow it to flow to the correct patient’s record.

When designing systems, delivery teams can sometimes casually describe thorny problems as “only edge cases”. We’ve tried to take a different approach: behind every edge case is a person and they matter.

We are now starting to turn some attention to how we address some of the technical issues we have accrued (given the pace we have delivered) and move towards a more strategic architecture fit for whatever COVID-19 throws at us in the coming months and years. One of the big tech themes is looking at a more event-based architecture and developing an API-first type service. My colleague Brian Diggle talked about this in an earlier blog post.

We are also looking at how the service needs to evolve to handle seasonal flu and the possibility of other vaccination types in the future.

The last eight months or so have been some of the hardest and most challenging times for everyone involved in the vaccination programme, I don’t think anyone has worked on a project which is so important for the health of every person in the country. When you talk to the team, you hear about the long hours and moments of stress, but mostly you hear how proud they are of what’s been achieved.

The service is made up of many different solutions, data flows and integration points across many organisations and suppliers and is a lot more complex than you would think. Much of this is driven by the clinical necessity to track each vaccine given, ensuring that a patient’s GP record is updated and that adverse reactions can be monitored and tracked for every citizen across the different vaccine types and different clinical settings.

The end-to-end process is made up of 8 core steps, from selecting those groups to invite, sending those invites out and allowing those patients to book an appointment of their choosing, right through to data collection at the point of vaccination and then to how that data flows to other authorised services. More about the initial steps in the process can be found on the SCW CSU website.

In addition, there is a 9th step to support the management of the supply of the vaccine, which I won’t cover within this blog for security reasons. 

The diagram below details the high-level architecture of the service and how these solutions are glued together. The description below the diagram explains in a bit more detail what some of the key services are and how they operate.