Normally, people went to their GPs for sick notes. We wanted to reduce this pressure. Additionally, minimising the number of people attending GP surgeries, whilst displaying COVID-19 symptoms would reduce the spread and ensure GP surgeries remained open. Working with commissioners in NHSX and DWP, our aim was to quickly deliver a robust service, which would help in our combined efforts to prevent the spread of the disease.
We had 2 main objectives:
Firstly, to deliver an effective digital channel for this service as quickly as possible.
Secondly, because of the profile of the service and its likely usage, we had to ensure it was secure, resilient, performant and maintainable.
Cloud by default
We used Microsoft Azure as our chosen cloud provider. Cloud providers remove a lot of the build, maintenance and operational aspect of software delivery, leaving you to get on with building the application. Our default stance was to leverage as much commodity cloud software services as possible. Not only do cloud services expedite the delivery of the service, it intrinsically supports many of our other principles too.
Automate all the things
One of the first things we did, before building out the functional service, was to build a deployment pipeline. A deployment pipeline is something that takes an application code a developer writes and ultimately deploys that into a production environment for our users to use. We used Azure DevOps in combination with Terraform to do this.
Establishing a deployment pipeline early on, with Terraform and DevOps, meant that we could start iterating the functional changes at a much higher pace, in the confidence that every iteration could be safely deployed into a production-like environment. This paid dividends throughout the project, especially when responding to urgent updates or patches in a timely fashion.
Scalability
A major benefit of using cloud services is that they are designed to scale, capable of serving millions of customers globally. What we did not know was if our application scaled to support the expected load on the service.
In parallel to building our automated deployment pipeline, we also looked at building out a performance testing rig that could simulate 50,000 users hitting our service at the same time. Using Azure Container Instances Service we could emulate two million isolation notes being generated within one hour.
Design for failure
Cloud services will fail. Unless the application architecture accounts for this, it can have some undesirable consequences. We accounted for this in our design in a number of ways.
We deployed the service into multiple geographical regions, to protect us from a regional outage. This was a straightforward task because we had already automated the environment build.
The database has an in-built automatic failover capability. If the database had an issue in one region, it would automatically recover itself in the secondary region, without impacting the service’s availability
We used Storage Queues and Functions to add resiliency into our email notification functionality. Rather than overload the Gov.Notify service we could throttle our requests through to Gov.Notify and retry in the event of a failure, meaning we always sent an email to our user.
These three things were the big factors in being able to offer a service that is available 99.999% of the time.
What did we learn?
The timelines and pressure were challenging, but hand on heart, it was one of the most interesting, exciting and rewarding experiences I have gone through. As a team we have created bonds that I’m sure will carry on for many years and much of the positive experiences have been introduced back into the NHS App team.
- Not all cloud services are created equal – We had to rectify a mistake before launch by changing our final solution to the App Service and App Service Environment (ASE). Although the services are similar, ASE did not really lend itself to cope with fluctuating surges in demand, which was clearly one of our main objectives. Had this not being such a condensed timeline, we would have likely performed some technical spike prior to making such a fundamental technical decision.
- We don’t all have to be in the same office – This project kicked off right at the start of the COVID-19 lockdown. We had no choice but to spend most of our time on a Zoom call watching eight other tired souls. However, it changed our general software development behaviour. We leveraged screensharing tools and collaboration so that multiple people could work on a problem at the same time. It made the experience much more interactive and we had a true cross-functional team which meant we could make decisions and bring them to fruition very quickly.
- Design systems work – Being able to use the NHS.uk design system saved a huge amount of time and allowed us to deliver a working prototype within a couple of days. For organisations out there trying to build uniform services, getting this right is an absolute must.
- Solid engineering pays dividends – It is all too easy to be focused on delivering features or functionality a user can see. We tried to do both the functional and non-functional side of the engineering, in equal measure, during the isolation note project. We believe it allowed us to deliver a much more robust and stable product, than had we just focused on just shipping some website functionality. What it also did was lay the foundation for similar COVID related services to accelerate their delivery. Both the Symptom Checking service and Covid Testing service used the foundational work laid by isolation note.
Our conclusion
Investing in engineering pays for itself in the long run.
Interested in working at NHS Digital? Search our latest job opportunities.
Author
Latest blogs
Last edited: 22 December 2021 4:24 pm