The need for accurate, robust and timely health and care data has never been clearer. Public health directors and policy makers need easily accessible and useable data to support decision making in the face of the coronavirus (COVID-19) pandemic.

Our data services have come a long way in the past 14 weeks. The pandemic has provided the opportunity to build our capabilities in data and analytics, business intelligence and data dashboards. It’s shown how fast we can move when we really have to.

Our data dashboards provide visualisations of our open data sets. We currently have the following on our data dashboards hub:

Data can be filtered by factors such as location, age, gender or medical condition, depending on which dashboard you’re using. This makes the data much more usable.

We’re also making private versions of these data sets that we share with a small number of approved users. This data may not be appropriate to put in the public domain because of the need to protect patient confidentiality but gives more detail to policy makers and allows them to make informed decisions about their local areas.

There’s a whole ecosystem of users putting this data to work. It is very exciting and has been a privilege to be a part of.

Our service teams picked up user onboarding, designing a process on the fly to authenticate users. Our statistical, Information Governance and Legal teams made sure we were producing robust data and insights and doing everything we needed to ensure we fulfilled our legal and ethical obligations. The key to all this cross-team collaboration is that our experts in each field took responsibility for guiding us through the process and supported us in navigating the complexity. We could not have done this without that support.

In producing our first dashboard, for 111 triages, we stood up a 32-core Tableau instance, the web team integrated this into a public-facing webpage via a new API and uplifted our web meta-data to include schema.org, to tune internet searches and link to our key COVID-19 pages. That all happened in less than three weeks. I have never worked on a data programme that has moved that fast!

Very quickly, we were asked to work on data about shielded patients, e-Referrals and COVID-19 testing. These all posed new challenges. For example, the work on testing data involved building a secure Tableau environment. Initially, this was done using standard Tableau functionality but, as demand for more granular data increased, we upgraded this to 2FA and hardened the cyber wrap for Tableau and the underlying infrastructure.

Our Dashboard support team developed a new user onboarding and authentication process that enabled us to onboard hundreds of users in a matter of days and our service management team worked tirelessly to get an end to end support wrap in place.

It is exhilarating to reflect that what started as a small band of analysts, developers and other techies building some rapid prototypes on desktops has become a solution serving thousands of users. This hasn’t been achieved without significant growing pains. We accumulated tech debt quickly and we are still paying this off. In order to get data out quickly, pipelines were not fully automated. We take turns to work evenings and weekends to run and quality assure the data. We had to make snap decisions on what to prioritise for users and we had to be constantly be ready to flex and change as we discovered more about the ever-changing data and user landscape.

Initially, everyone in the team had to wear multiple hats.  I was, at various points, a day project manager, tester, analyst and chief beggar for more resources. I cannot acknowledge enough the experience and guidance provided by Goldman Sachs, which proved invaluable on many occasions.

Currently, it feels like we are entering a new phase of maturity. We have developed an amazing technical and organisational capability that has the potential to support the health and social care system long after the threat from COVID-19 has subsided.

NHS Digital colleagues who have deep experience in running major patient facing services such as the NHS App and 111 Online, are hardening our software development life cycle (SDLC) and improving our ability to operate at scale. This is an important phase, to building capability to ensure we can offer a fully supported and scalable service.