Our data services have come a long way in the past 14 weeks. The pandemic has provided the opportunity to build our capabilities in data and analytics, business intelligence and data dashboards. It’s shown how fast we can move when we really have to.

Our data dashboards provide visualisations of our open data sets. We currently have the following on our data dashboards hub:

Data can be filtered by factors such as location, age, gender or medical condition, depending on which dashboard you’re using. This makes the data much more usable.

We’re also making private versions of these data sets that we share with a small number of approved users. This data may not be appropriate to put in the public domain because of the need to protect patient confidentiality but gives more detail to policy makers and allows them to make informed decisions about their local areas.

There’s a whole ecosystem of users putting this data to work. It is very exciting and has been a privilege to be a part of.

Before coronavirus, we focused our dashboards and open data on monthly or yearly data sets offering insights into key topics such as immunisations or maternity. We were working in dedicated publication teams in our Data Services directorate.

The virus changed this. 111 calls where COVID-19 was suspected was our first use case. We figured that, if we could share this data and make it easily accessible, it would support planning and policy across England.

A team was rapidly pulled together from inside and outside NHS Digital.  Goldman Sachs provided invaluable support in developing dashboards. Our internal data science team built the data pipelines, the NHS Digital Bedrock platform team deployed and configured the necessary infrastructure and the data visualisation provider Tableau provided vital help at the front end.

Our service teams picked up user onboarding, designing a process on the fly to authenticate users. Our statistical, Information Governance and Legal teams made sure we were producing robust data and insights and doing everything we needed to ensure we fulfilled our legal and ethical obligations. The key to all this cross-team collaboration is that our experts in each field took responsibility for guiding us through the process and supported us in navigating the complexity. We could not have done this without that support.

In producing our first dashboard, for 111 triages, we stood up a 32-core Tableau instance, the web team integrated this into a public-facing webpage via a new API and uplifted our web meta-data to include schema.org, to tune internet searches and link to our key COVID-19 pages. That all happened in less than three weeks. I have never worked on a data programme that has moved that fast!

Very quickly, we were asked to work on data about shielded patients, e-Referrals and COVID-19 testing. These all posed new challenges. For example, the work on testing data involved building a secure Tableau environment. Initially, this was done using standard Tableau functionality but, as demand for more granular data increased, we upgraded this to 2FA and hardened the cyber wrap for Tableau and the underlying infrastructure.

Our Dashboard support team developed a new user onboarding and authentication process that enabled us to onboard hundreds of users in a matter of days and our service management team worked tirelessly to get an end to end support wrap in place.

It is exhilarating to reflect that what started as a small band of analysts, developers and other techies building some rapid prototypes on desktops has become a solution serving thousands of users. This hasn’t been achieved without significant growing pains. We accumulated tech debt quickly and we are still paying this off. In order to get data out quickly, pipelines were not fully automated. We take turns to work evenings and weekends to run and quality assure the data. We had to make snap decisions on what to prioritise for users and we had to be constantly be ready to flex and change as we discovered more about the ever-changing data and user landscape.

Initially, everyone in the team had to wear multiple hats.  I was, at various points, a day project manager, tester, analyst and chief beggar for more resources. I cannot acknowledge enough the experience and guidance provided by Goldman Sachs, which proved invaluable on many occasions.

Currently, it feels like we are entering a new phase of maturity. We have developed an amazing technical and organisational capability that has the potential to support the health and social care system long after the threat from COVID-19 has subsided.

NHS Digital colleagues who have deep experience in running major patient facing services such as the NHS App and 111 Online, are hardening our software development life cycle (SDLC) and improving our ability to operate at scale. This is an important phase, to building capability to ensure we can offer a fully supported and scalable service.

I hope our (ever growing) team will look back at the experience of the last few months as the first, frantic baby steps on a long journey!

I believe the systems we are building can enable people to improve lives and society at large. What we have done to support the coronavirus response sets us up to make more use of data analytics and visualisation in a post-virus world. It will add value in other areas, such as cancer, maternity, research, and more.

I hope that, by making as much of the data open and accessible on our website as possible, we are contributing to the public record of  the coronavirus pandemic. The epidemic has not ended and its story has yet to be written and I am sure the data we are putting into the public domain will play a significant role in public discourse about what we are going through for many years to come.