Skip to main content

Data Processing Services release notes

We are developing our data processing services to transform how we collect, process and use data to improve health and care. To support this work we will provide regular programme updates. 

DPS Core Announcements and planned outages

Date Time Reason
Thursday 15 April 2021 18:00 to 23:59

DPS Core maintenance

Thursday 22 April 2021 18:00 to 23:59 DPS Core maintenance
Thursday 29 April 2021 18:00 to 23:59 DPS Core maintenance

For any service issues outside of these dates and times, please contact us via the National Service desk.

Email: ssd.nationalservicedesk@nhs.net 

Release notes February 2021

DPS Core 24/02/2021

Added or Changed:

  1. Landing SDCS Pillar 2 Data in DPS
  2. Oximetry at Home
  3. HES Sensitive APC ETL
  4. DPS Central Metastore - MVP
  5. POS disseminations
  6. Coverage & Dashboard DQ Reports
  7. Support production of MPS V HES ID methodological change paper
  8. POS legacy data migration
  9. Data Manager/Production Assistance
  10. Early Disease Data Flows into DPS
  11. Onboard Ambulance Data Set
  12. MHSDS TOS v4.1.12 Updates
  13. Cancer Dataset for PHE
  14. MHSDS TOS v4.1.10 Updates
  15. ECDS/AE and APC linkage
  16. Ingest and process Key-worker test results from NPEx for COVID-19
  17. Ingest and Process Pillar 2 Data (NPEx)
  18. DPS Core Run & Maintain 20/21

Fixed:

  1. DPS Core Run and Maintain Improvements 2020/21
  2. Moonshot - Scaling DPS

DPS Core 17/02/2021

Added or Changed:

  1. DPS Core Platform Activities 2020/21
  2. Service Management Data Volume Reporting Extract
  3. Moonshot - Scaling DPS
  4. Portal ODS data from DSS_CORPORATE
  5. Cancer Dataset for PHE
  6. GPDRP Patient and Appointment Report Processing
  7. COVID-19 SGSS processing
  8. Coverage & Dashboard DQ Reports
  9. Landing SDCS Pillar 2 Data in DPS
  10. DPS - Permission to Contact - Day 3
  11. CSDS: DARS output
  12. Ingest and Process Pillar 2 Data (NPEx)
  13. DPS Core Run and Maintain Improvements 2020/21
  14. AHAS V HES output comparison
  15. (Hue & RStudio) Phase 1 Tech Debt

Fixed:

  1. DSP Core Platform Activities 2019/20
  2. ECDS/AE and APC linkage
  3. GPES v3 Data
  4. POS sending files for processing

DPS Core 10/02/2021

Added or Changed:

  1. DPS Core Run and Maintain Improvements 2020/21
  2. IAPT: Commissioning Extract
  3. Coverage & Dashboards; Data Quality (DQ) solution for AHAS
  4. ECDS/AE and APC linkage
  5. DPS Core Platform Activities 2020/21
  6. DPS Central Metastore - MVP
  7. Moonshot - Scaling DPS
  8. Ingest and Process Pillar 2 Data (NPEx)
  9. Landing SDCS Pillar 2 Data in DPS
  10. (Hue & RStudio) Phase 1 Tech Debt
  11. ePMA - WellskyPhase2
  12. PLICS Data Onboarding - DPS Core activity
  13. Covid-19 Vaccination Data
  14. Coverage & Dashboard DQ Reports

Fixed:

  1. DPS - Permission to Contact - Day 3
  2. Core Pull Data From Portal
  3. COVID-19 SGSS processing
  4. DPS Core Triage 2020/21
  5. DPS Core Run & Maintain 20/21
  6. Portal ODS data from DSS_CORPORATE

DPS Core 04/02/2021

Added or Changed:

  1. IAPT: Commissioning Extract
  2. CSDS: DARS output
  3. Ingest and Process Pillar 2 Data (NPEx)
  4. Coverage & Dashboard DQ Reports
  5. ePMA - WellskyPhase2
  6. Portal ODS data from DSS_CORPORATE
  7. PCRM Data
  8. Data Manager/Production Assistance
  9. DPS Core Platform Activities 2020/21
  10. GPDC Migration to Cloud
  11. Data Managers Enablement
  12. Moonshot - Scaling DPS
  13. DPS Core Run & Maintain 20/21
  14. Coverage & Dashboards; Data Quality (DQ) solution for AHAS
  15. Pseudonymised data
  16. Deliver replacement solution for HES_ONS asset within AHAS/DPS
  17. DPS Core Triage 2020/21
  18. COVID-19 SGSS processing
  19. DPS - Permission to Contact - Day 3
  20. DPS Core Run and Maintain Improvements 2020/21
  21. Ingest and process Key-worker test results from NPEx for COVID-19
  22. Cancer Dataset for PHE
  23. Code Promotion - Onboarding Mental Health

Fixed:

  1. IAPT: Remaining Derivations - End Of Month
  2. General work relating to COVID-19
  3. Core Pull Data From Portal

Release notes January 2021

DPS Core 27/01/2021

Added or changed

  1. Deloitte Datamine
  2. Core Pull Data From Portal
  3. Coverage & Dashboard DQ Reports
  4. Ingest and Process Pillar 2 Data (NPEx)
  5. DPS - Permission to Contact - Day 3
  6. Moonshot - Scaling DPS
  7. DPS Core Run & Maintain 20/21
  8. GPSS Use of Privitar
  9. Pseudonymised data
  10. AMS001: Create Data Pump between DPS & SQL Clones
  11. ECDS/AE and APC linkage
  12. POS disseminations
  13. Data Managers Enablement
  14. Risk Stratification - Population Health Management use case
  15. Coverage & Dashboards; Data Quality (DQ) solution for AHAS
  16. Deliver replacement solution for HES_ONS asset within AHAS/DPS
  17. Land Datasets via GDP
  18. Covid-19 Vaccination Data
  19. Onboard Ambulance Data Set
  20. Databricks Cluster Restructuring

Fixed

  1. AHAS V HES output comparison
  2. CSDS: Views and Combined View
  3. DPS Core Run and Maintain Improvements 2020/21
  4. IAPT: Remaining Derivations - End Of Month

Release notes December 2020

DPS Core 16/12/2020

Added or changed

  1. GP Data RFO Testing, Alerting and Monitoring
  2. Pseudonymised data
  3. GP Data Post Go-Live Enhancements
  4. Sentinel Stroke National Audit Programme (SSNAP)
  5. Covid-19 Vaccination Data
  6. Core Pull Data From Portal
  7. General work relating to COVID-19
  8. Moonshot - Scaling DPS
  9. Ingest and Process Pillar 2 Data (NPEx)
  10. DPS Core Run and Maintain Improvements 2020/21
  11. HESID transition to MPSID in SQL Server
  12. AMS001: Create Data Pump between DPS & SQL Clones
  13. CSDS: DARS output
  14. Data Managers Enablement
  15. DAE Run and Maintain Improvements
  16. CSDS: De-ID Integration into Pipeline
  17. IAPT: Provider Post-Deadline Extracts
  18. Early Disease Data Flows into DPS
  19. POS disseminations
  20. COVID-19 CHESS processing
  21. POS local pos sends files for processing
  22. GPDC DDC Operational Support and Maintenance 2020-21
  23. AHAS V HES output comparison
  24. Coverage & Dashboard DQ Reports

Fixed

  1. CSDS: Commissioning Extract
  2. DSP Core Tech Debt
  3. ECDS/AE and APC linkage
  4. GPES v3 Data

Release notes November 2020

DPS Core 25/11/2020

Added or changed

  1. ePMA - WellskyPhase2
  2. COVID-19 SGSS processing
  3. DAE Run and Maintain Improvements
  4. GP Data EMIS GPSS Testing
  5. SUS+ Cloud CI pipeline
  6. GPES v3 Data
  7. Moonshot - Scaling DPS
  8. POS technical tasks
  9. Ingest and process Key-worker test results from NPEx for COVID-19

Fixed

  1. Monthly cancer data extract into DPS
  2. POS sending files for processing
  3. HESID transition to MPSID in SQL Server
  4. AMS001: Create Data Pump between DPS & SQL Clones

Release notes October 2020

DPS Core 28/10/2020

Added or changed

  1. DPS Core Triage 2020/21
  2. GDP/CP/TK Phase 1 - Preparation
  3. GP Data Run and Maintain Improvements
  4. Risk Stratification - Population Health Management use case
  5. DPS Core Run and Maintain Improvements 2020/21
  6. Data deduplication in AHAS
  7. IAPT: Remaining Derivations - End Of Month
  8. DPS Central Metastore - MVP
  9. Ingest and process Key-worker test results from NPEx for COVID-19
  10. IAPT: De-ID Integration into Pipeline
  11. Onboard Ambulance Data Set
  12. Generic Data Pipeline
  13. GP Data RFO Testing, Alerting and Monitoring
  14. HES_ID and MPS_ID analysis on Historic Episodic Data
  15. DPS Core Run & Maintain 20/21
  16. IAPT: Output for Casemix Cloud
  17. GP Data Post Go-Live Enhancements
  18. Data Manager/Production Assistance
  19. HESID transition to MPSID in SQL Server

Release notes September 2020

DPS Core 30/09/2020

Added or changed

  1. GP Data retokenisation as a separate Databricks job
  2. DPS Core Platform Activities 2020/21
  3. Mental Health Multiple Submission Window Model Re-submission report
  4. Enhanced Splunk Logging and Monitoring
  5. Onboard Ambulance Data Set
  6. GP Data Post Go-Live Enhancements
  7. DPS Core Run & Maintain 20/21
  8. Ingest and process Key-worker test results from NPEx for COVID-19
  9. PCareMeds Phase 2
  10. Data Manager/Production Assistance
  11. Further Code Promotion Enhancements
  12. IAPT TOS uplift V24
  13. COVID-19 SGSS processing
  14. Virtual Wards
  15. DAE Run and Maintain Improvements
  16. DPS - Permission to Contact - Day 3
  17. DPS Core Run and Maintain Improvements 2020/21
  18. GP Data  De-ID Tokenisation
  19. GP Data V&P Testing
  20. Data deduplication in AHAS
  21. GP Data Run and Maintain Improvements

Fixed

  1. GPES-DPS to Deliver GP Clear PID data into DPS - COVID-19 Response
  2. IAPT TOS v2.0.23 upgrade work
  3. CSDS: Historical data and Index uplift process
  4. APC AHAS_FLAT - Readiness for live parallel testing by 7/10
  5. CSDS: Provider Post-Deadline Extract
  6. AMS001: Create Data Pump between DPS & SQL Clones
  7. GP Data Ingestion Parse and Store with XSD Validation
  8. POS sending files for processing

Release notes August 2020

DPS Core 27/08/2020

Added or changed

  1. CSDS: Validations for pilot
  2. Ingest and process Key-worker test results from NPEx for COVID-19
  3. DPS Core Platform Activities 2020/21
  4. DPS - Permission to Contact - Day 2
  5. DAE Run and Maintain Improvements
  6. Mental Health Multiple Submission Window Model Core Processing
  7. AHAS V HES output comparison
  8. DSP Core Tech Debt
  9. GPES-DPS to Deliver GP Clear PID data into DPS - COVID-19 Response
  10. Privitar token vault priming
  11. IAPT TOS v2.0.23 upgrade work
  12. DPS Core Run and Maintain Improvements 2020/21
  13. GP Data RFO Testing, Alerting and Monitoring
  14. GP Data Ingestion Monitoring for New Submissions
  15. POS sending files for processing
  16. Data Production - Permission to Contact - Day 2

Fixed

  1. CSDS: Pre-D Assurance
  2. De-ID (utilising Privitar) Tech Debt / Bugs / Tasks
  3. CSDS: Pipeline Development
  4. Portal: In Hours Releases
  5. GP Data Ingestion Parse and Store with XSD Validation
  6. IAPT: End of month derivations for pilot
  7. CSDS: Remaining Validations (non-pilot)
  8. Data Production - Permission to Contact - Day 3

Release notes July 2020

DPS Core 29/07/2020

Added or changed

  1. ICNARC Pipeline via GDP
  2. Data Production - Permission to Contact - Day 2
  3. COVID-19 CHESS processing
  4. GPES-DPS to Deliver GP Clear PID data into DPS - COVID-19 Response
  5. DAE Run and Maintain Improvements
  6. COVID-19: eRS Processing into DPS (tactical)
  7. HES_ID and MPS_ID analysis on Historic Episodic Data
  8. CDS daily data in ahas_curr_clear - beta phase differences/defects
  9. Ingest and process Key-worker test results from NPEx for COVID-19
  10. DPS assets for COVID-19
  11. COVID-19: CPAS Model Predictions
  12. AMS001: Create Data Pump between DPS & SQL Clones
  13. DSP Core Tech Debt
  14. GP Data V&P Testing
  15. DPS - Permission to Contact - Day 2
  16. Deloitte Datamine
  17. RStudio Shared workspaces
  18. COVID-19: SPL on DPS
  19. DPS Core Platform Activities 2020/21
  20. Generic Data Pipeline
  21. IAPT: Provider Pre-Deadline Extracts

Fixed

  1. Automate the application of Databricks Groups
  2. GP Data MESH Submission Testing
  3. Robust Code Promotion Enhancements
  4. CSDS: Provider Pre-Deadline Extract

Release notes June 2020

DPS Core 24/06/2020

Added or changed

  1. DPS Core Run and Maintain Improvements 2020/21
  2. NICOR: Ingestion of datasets

Fixed

  1. MHSDS v4.1.9 TOS Updates

Release notes May 2020

DPS Core 29/05/2020

Added or changed

  1. IAPT: Remaining Derivations (non-pilot - not End of Month)
  2. Mental Health Multiple Submission Window Model Core Processing
  3. Ingest and process Key-worker test results from NPEx for COVID-19
  4. COVID-19 SGSS processing
  5. DAE Run and Maintain Improvements
  6. Generic Data Pipeline
  7. MHSDS v4.1.9 TOS Updates
  8. GP Data RFO Testing, Alerting and Monitoring
  9. CSDS: Views and Combined View
  10. CSDS: Remaining Derivations (non-pilot)

Release notes April 2020

DPS Core 29/04/2020

Added or changed

  1. GP Data RFO Testing, Alerting and Monitoring
  2. Managing Databricks Permissions for DAE Users
  3. DAE Run and Maintain Improvements
  4. General work relating to COVID-19
  5. DPS Core Platform Activities 2020/21
  6. GP Data - COVID-19 GO-LIVE - Clear Data Release
  7. GP Data  De-ID Tokenisation

Fixed

  1. Ingest and process Key-worker test results from NPEx for COVID-19
  2. PCareMeds Phase 1
  3. DPS Core Run and Maintain Improvements 2020/21

DPS Core 01/04/2020

  1. Completed Analysts Linkage epic MVP.
  2. We completed the assurance for 'Multi Submission Window' so that organisations can submit Mental Health data many times.
  3. Created a perinatal view of data
  4. Completed the local cross check work on historic Hospital Episode Statistic in preparation for the data to pass through the Master Person Service
  5. Implemented changes defined in Mental Health Technical Output Spec
  6. Volume and performance testing of the MESH connection
  7. Fixed a date problem for Primary Care Medicines data

Release notes March 2020

DPS Core 18/03/2020

  1. Code Promotion Ephemeral logs – implementation for Data Managers completed.
  2. Maternity DSCRO extracts (includes UPID) produced.
  3. Null MAT derivations populated.
  4. Spike of MH 4.1 TOS updated.
  5. Group Permissions model for DM Toolkit implemented.

DPS Core 11/03/2020

  1. HES MPS ID scope defined

DPS Core 04/03/2020

  1. MH month end automation completion.
  2. MH YTD Extract work completed.
  3. MPS ID on HES – MPS request for historic data created.

Release notes February 2020

DPS Core 26/02/2020

  1. Implement group permissions model for DM Toolkit.
  2. Produce Maternity DSCRO extracts and historic linkage file (including UPID) and disseminate.

DPS Core 19/02/2020

  1. Start investigation into XML ingestion into DM Toolkit

DPS Core 12/02/2020

  1. Spike re: agree technical approach to code repositories.
  2. Remove blocking issues and complete DM Toolkit user onboarding
  3. Code Promotion – agree approach on how best to make available to DMs and implement
  4. Multiple Submission Window model YTD extract and cross-check functionality – verify and action requirements
  5. Commence work on generating MPS ID on HES historic data (test data on ref is required – SB) to enable analysis by the business

DPS Core 05/02/2020

This release has improved our internal capabilities in two key areas:

  1. A 'minimum viable product' has been developed to provide better self-sufficiency for our Data Managers to promote the code they have generated to operate in our live service.
  2. We have made good progress on the capability to allow Mental Health data providers to submit data multiple times in the year with a path forward working with our internal business.

Release notes January 2020

DPS Core 29/01/2020

In this release we have progressed the following:

  1. We have improved the way we process larger file sizes as received for Primary Care Medicines data
  2. Engagement has begun with our Data Managers for wider usage of their toolkit for onboarding data
  3. Elaborated requirements for improvements to the generation of the Unique Pregnancy Identifier.

DPS Core 22/01/2020

We are continuing to work on the best way to manage the processing of the large primary care medicines file. 

We have started preparatory work to understand requirements to link Data Processing Services to the Data Access Request Service

We have improved the way our data analysts can access and manipulate data on the platform.

We have made updates to the Data Manager Toolkit enabling extracts to new locations via the SEFT data transfer system. 

The first phase of the Data Manager Toolkit has been released to data managers.

The internal data services teams are now developing on the platform, promoting mental health data quality and maternity data analysis to the live system. 

We have completed work on perinatal linked data to support data services analysts. We also now derive a 'Universal Pregnancy ID'. 

We are progressing submissions and data validations to enable multiple submissions of mental health data.

Last edited: 8 April 2021 10:17 am