Data Processing Services release notes

For any service issues outside of these dates and times, please contact us via the National Service desk.

Email: [email protected] 

For specific user impact release notes relating to associated services please see:

Data Access Environment release history.

Strategic Data Collection Service in the cloud - release history.

DPS Core 24/02/2021

Added or Changed:

1. Landing SDCS Pillar 2 Data in DPS
2. Oximetry at Home
3. HES Sensitive APC ETL
4. DPS Central Metastore - MVP
5. POS disseminations
6. Coverage & Dashboard DQ Reports
7. Support production of MPS V HES ID methodological change paper
8. POS legacy data migration
9. Data Manager/Production Assistance
10. Early Disease Data Flows into DPS
11. Onboard Ambulance Data Set
12. MHSDS TOS v4.1.12 Updates
13. Cancer Dataset for PHE
14. MHSDS TOS v4.1.10 Updates
15. ECDS/AE and APC linkage
16. Ingest and process Key-worker test results from NPEx for COVID-19
17. Ingest and Process Pillar 2 Data (NPEx)
18. DPS Core Run & Maintain 20/21

Fixed:

1. DPS Core Run and Maintain Improvements 2020/21
2. Moonshot - Scaling DPS

DPS Core 17/02/2021

Added or Changed:

1. DPS Core Platform Activities 2020/21
2. Service Management Data Volume Reporting Extract
3. Moonshot - Scaling DPS
4. Portal ODS data from DSS_CORPORATE
5. Cancer Dataset for PHE
6. GPDPR Patient and Appointment Report Processing
7. COVID-19 SGSS processing
8. Coverage & Dashboard DQ Reports
9. Landing SDCS Pillar 2 Data in DPS
10. DPS - Permission to Contact - Day 3
11. CSDS: DARS output
12. Ingest and Process Pillar 2 Data (NPEx)
13. DPS Core Run and Maintain Improvements 2020/21
14. AHAS V HES output comparison
15. (Hue & RStudio) Phase 1 Tech Debt

Fixed:

1. DSP Core Platform Activities 2019/20
2. ECDS/AE and APC linkage
3. GPES v3 Data
4. POS sending files for processing

DPS Core 10/02/2021

Added or Changed:

1. DPS Core Run and Maintain Improvements 2020/21
2. IAPT: Commissioning Extract
3. Coverage & Dashboards; Data Quality (DQ) solution for AHAS
4. ECDS/AE and APC linkage
5. DPS Core Platform Activities 2020/21
6. DPS Central Metastore - MVP
7. Moonshot - Scaling DPS
8. Ingest and Process Pillar 2 Data (NPEx)
9. Landing SDCS Pillar 2 Data in DPS
10. (Hue & RStudio) Phase 1 Tech Debt
11. ePMA - WellskyPhase2
12. PLICS Data Onboarding - DPS Core activity
13. Covid-19 Vaccination Data
14. Coverage & Dashboard DQ Reports

Fixed:

1. DPS - Permission to Contact - Day 3
2. Core Pull Data From Portal
3. COVID-19 SGSS processing
4. DPS Core Triage 2020/21
5. DPS Core Run & Maintain 20/21
6. Portal ODS data from DSS_CORPORATE

DPS Core 04/02/2021

Added or Changed:

1. IAPT: Commissioning Extract
2. CSDS: DARS output
3. Ingest and Process Pillar 2 Data (NPEx)
4. Coverage & Dashboard DQ Reports
5. ePMA - WellskyPhase2
6. Portal ODS data from DSS_CORPORATE
7. PCRM Data
8. Data Manager/Production Assistance
9. DPS Core Platform Activities 2020/21
10. GPDC Migration to Cloud
11. Data Managers Enablement
12. Moonshot - Scaling DPS
13. DPS Core Run & Maintain 20/21
14. Coverage & Dashboards; Data Quality (DQ) solution for AHAS
15. Pseudonymised data
16. Deliver replacement solution for HES_ONS asset within AHAS/DPS
17. DPS Core Triage 2020/21
18. COVID-19 SGSS processing
19. DPS - Permission to Contact - Day 3
20. DPS Core Run and Maintain Improvements 2020/21
21. Ingest and process Key-worker test results from NPEx for COVID-19
22. Cancer Dataset for PHE
23. Code Promotion - Onboarding Mental Health

Fixed:

1. IAPT: Remaining Derivations - End Of Month
2. General work relating to COVID-19
3. Core Pull Data From Portal

DPS Core 27/01/2021

Added or changed

1. Deloitte Datamine
2. Core Pull Data From Portal
3. Coverage & Dashboard DQ Reports
4. Ingest and Process Pillar 2 Data (NPEx)
5. DPS - Permission to Contact - Day 3
6. Moonshot - Scaling DPS
7. DPS Core Run & Maintain 20/21
8. GPSS Use of Privitar
9. Pseudonymised data
10. AMS001: Create Data Pump between DPS & SQL Clones
11. ECDS/AE and APC linkage
12. POS disseminations
13. Data Managers Enablement
14. Risk Stratification - Population Health Management use case
15. Coverage & Dashboards; Data Quality (DQ) solution for AHAS
16. Deliver replacement solution for HES_ONS asset within AHAS/DPS
17. Land Datasets via GDP
18. Covid-19 Vaccination Data
19. Onboard Ambulance Data Set
20. Databricks Cluster Restructuring

Fixed

1. AHAS V HES output comparison
2. CSDS: Views and Combined View
3. DPS Core Run and Maintain Improvements 2020/21
4. IAPT: Remaining Derivations - End Of Month

DPS Core 16/12/2020

Added or changed

1. GP Data RFO Testing, Alerting and Monitoring
2. Pseudonymised data
3. GP Data Post Go-Live Enhancements
4. Sentinel Stroke National Audit Programme (SSNAP)
5. Covid-19 Vaccination Data
6. Core Pull Data From Portal
7. General work relating to COVID-19
8. Moonshot - Scaling DPS
9. Ingest and Process Pillar 2 Data (NPEx)
10. DPS Core Run and Maintain Improvements 2020/21
11. HESID transition to MPSID in SQL Server
12. AMS001: Create Data Pump between DPS & SQL Clones
13. CSDS: DARS output
14. Data Managers Enablement
15. DAE Run and Maintain Improvements
16. CSDS: De-ID Integration into Pipeline
17. IAPT: Provider Post-Deadline Extracts
18. Early Disease Data Flows into DPS
19. POS disseminations
20. COVID-19 CHESS processing
21. POS local pos sends files for processing
22. GPDC DDC Operational Support and Maintenance 2020-21
23. AHAS V HES output comparison
24. Coverage & Dashboard DQ Reports

Fixed

1. CSDS: Commissioning Extract
2. DSP Core Tech Debt
3. ECDS/AE and APC linkage
4. GPES v3 Data

DPS Core 25/11/2020

Added or changed

1. ePMA - WellskyPhase2
2. COVID-19 SGSS processing
3. DAE Run and Maintain Improvements
4. GP Data EMIS GPSS Testing
5. SUS+ Cloud CI pipeline
6. GPES v3 Data
7. Moonshot - Scaling DPS
8. POS technical tasks
9. Ingest and process Key-worker test results from NPEx for COVID-19

Fixed

1. Monthly cancer data extract into DPS
2. POS sending files for processing
3. HESID transition to MPSID in SQL Server
4. AMS001: Create Data Pump between DPS & SQL Clones

DPS Core 28/10/2020

Added or changed

1. DPS Core Triage 2020/21
2. GDP/CP/TK Phase 1 - Preparation
3. GP Data Run and Maintain Improvements
4. Risk Stratification - Population Health Management use case
5. DPS Core Run and Maintain Improvements 2020/21
6. Data deduplication in AHAS
7. IAPT: Remaining Derivations - End Of Month
8. DPS Central Metastore - MVP
9. Ingest and process Key-worker test results from NPEx for COVID-19
10. IAPT: De-ID Integration into Pipeline
11. Onboard Ambulance Data Set
12. Generic Data Pipeline
13. GP Data RFO Testing, Alerting and Monitoring
14. HES_ID and MPS_ID analysis on Historic Episodic Data
15. DPS Core Run & Maintain 20/21
16. IAPT: Output for Casemix Cloud
17. GP Data Post Go-Live Enhancements
18. Data Manager/Production Assistance
19. HESID transition to MPSID in SQL Server

DPS Core 30/09/2020

Added or changed

1. GP Data retokenisation as a separate Databricks job
2. DPS Core Platform Activities 2020/21
3. Mental Health Multiple Submission Window Model Re-submission report
4. Enhanced Splunk Logging and Monitoring
5. Onboard Ambulance Data Set
6. GP Data Post Go-Live Enhancements
7. DPS Core Run & Maintain 20/21
8. Ingest and process Key-worker test results from NPEx for COVID-19
9. PCareMeds Phase 2
10. Data Manager/Production Assistance
11. Further Code Promotion Enhancements
12. IAPT TOS uplift V24
13. COVID-19 SGSS processing
14. Virtual Wards
15. DAE Run and Maintain Improvements
16. DPS - Permission to Contact - Day 3
17. DPS Core Run and Maintain Improvements 2020/21
18. GP Data  De-ID Tokenisation
19. GP Data V&P Testing
20. Data deduplication in AHAS
21. GP Data Run and Maintain Improvements

Fixed

1. GPES-DPS to Deliver GP Clear PID data into DPS - COVID-19 Response
2. IAPT TOS v2.0.23 upgrade work
3. CSDS: Historical data and Index uplift process
4. APC AHAS_FLAT - Readiness for live parallel testing by 7/10
5. CSDS: Provider Post-Deadline Extract
6. AMS001: Create Data Pump between DPS & SQL Clones
7. GP Data Ingestion Parse and Store with XSD Validation
8. POS sending files for processing

DPS Core 27/08/2020

Added or changed

1. CSDS: Validations for pilot
2. Ingest and process Key-worker test results from NPEx for COVID-19
3. DPS Core Platform Activities 2020/21
4. DPS - Permission to Contact - Day 2
5. DAE Run and Maintain Improvements
6. Mental Health Multiple Submission Window Model Core Processing
7. AHAS V HES output comparison
8. DSP Core Tech Debt
9. GPES-DPS to Deliver GP Clear PID data into DPS - COVID-19 Response
10. Privitar token vault priming
11. IAPT TOS v2.0.23 upgrade work
12. DPS Core Run and Maintain Improvements 2020/21
13. GP Data RFO Testing, Alerting and Monitoring
14. GP Data Ingestion Monitoring for New Submissions
15. POS sending files for processing
16. Data Production - Permission to Contact - Day 2

Fixed

1. CSDS: Pre-D Assurance
2. De-ID (utilising Privitar) Tech Debt / Bugs / Tasks
3. CSDS: Pipeline Development
4. Portal: In Hours Releases
5. GP Data Ingestion Parse and Store with XSD Validation
6. IAPT: End of month derivations for pilot
7. CSDS: Remaining Validations (non-pilot)
8. Data Production - Permission to Contact - Day 3

DPS Core 29/07/2020

Added or changed

1. ICNARC Pipeline via GDP
2. Data Production - Permission to Contact - Day 2
3. COVID-19 CHESS processing
4. GPES-DPS to Deliver GP Clear PID data into DPS - COVID-19 Response
5. DAE Run and Maintain Improvements
6. COVID-19: eRS Processing into DPS (tactical)
7. HES_ID and MPS_ID analysis on Historic Episodic Data
8. CDS daily data in ahas_curr_clear - beta phase differences/defects
9. Ingest and process Key-worker test results from NPEx for COVID-19
10. DPS assets for COVID-19
11. COVID-19: CPAS Model Predictions
12. AMS001: Create Data Pump between DPS & SQL Clones
13. DSP Core Tech Debt
14. GP Data V&P Testing
15. DPS - Permission to Contact - Day 2
16. Deloitte Datamine
17. RStudio Shared workspaces
18. COVID-19: SPL on DPS
19. DPS Core Platform Activities 2020/21
20. Generic Data Pipeline
21. IAPT: Provider Pre-Deadline Extracts

Fixed

1. Automate the application of Databricks Groups
2. GP Data MESH Submission Testing
3. Robust Code Promotion Enhancements
4. CSDS: Provider Pre-Deadline Extract

DPS Core 24/06/2020

Added or changed

1. DPS Core Run and Maintain Improvements 2020/21
2. NICOR: Ingestion of datasets

Fixed

1. MHSDS v4.1.9 TOS Updates

DPS Core 29/05/2020

Added or changed

1. IAPT: Remaining Derivations (non-pilot - not End of Month)
2. Mental Health Multiple Submission Window Model Core Processing
3. Ingest and process Key-worker test results from NPEx for COVID-19
4. COVID-19 SGSS processing
5. DAE Run and Maintain Improvements
6. Generic Data Pipeline
7. MHSDS v4.1.9 TOS Updates
8. GP Data RFO Testing, Alerting and Monitoring
9. CSDS: Views and Combined View
10. CSDS: Remaining Derivations (non-pilot)

DPS Core 29/04/2020

Added or changed

1. GP Data RFO Testing, Alerting and Monitoring
2. Managing Databricks Permissions for DAE Users
3. DAE Run and Maintain Improvements
4. General work relating to COVID-19
5. DPS Core Platform Activities 2020/21
6. GP Data - COVID-19 GO-LIVE - Clear Data Release
7. GP Data  De-ID Tokenisation

Fixed

1. Ingest and process Key-worker test results from NPEx for COVID-19
2. PCareMeds Phase 1
3. DPS Core Run and Maintain Improvements 2020/21

DPS Core 01/04/2020

1. Completed Analysts Linkage epic MVP.
2. We completed the assurance for 'Multi Submission Window' so that organisations can submit Mental Health data many times.
3. Created a perinatal view of data
4. Completed the local cross check work on historic Hospital Episode Statistic in preparation for the data to pass through the Master Person Service
5. Implemented changes defined in Mental Health Technical Output Spec
6. Volume and performance testing of the MESH connection
7. Fixed a date problem for Primary Care Medicines data

DPS Core 18/03/2020

1. Code Promotion Ephemeral logs – implementation for Data Managers completed.
2. Maternity DSCRO extracts (includes UPID) produced.
3. Null MAT derivations populated.
4. Spike of MH 4.1 TOS updated.
5. Group Permissions model for DM Toolkit implemented.

DPS Core 11/03/2020

1. HES MPS ID scope defined

DPS Core 04/03/2020

1. MH month end automation completion.
2. MH YTD Extract work completed.
3. MPS ID on HES – MPS request for historic data created.

DPS Core 26/02/2020

1. Implement group permissions model for DM Toolkit.
2. Produce Maternity DSCRO extracts and historic linkage file (including UPID) and disseminate.

DPS Core 19/02/2020

1. Start investigation into XML ingestion into DM Toolkit

DPS Core 12/02/2020

1. Spike re: agree technical approach to code repositories.
2. Remove blocking issues and complete DM Toolkit user onboarding
3. Code Promotion – agree approach on how best to make available to DMs and implement
4. Multiple Submission Window model YTD extract and cross-check functionality – verify and action requirements
5. Commence work on generating MPS ID on HES historic data (test data on ref is required – SB) to enable analysis by the business

DPS Core 05/02/2020

This release has improved our internal capabilities in two key areas:

1. A 'minimum viable product' has been developed to provide better self-sufficiency for our Data Managers to promote the code they have generated to operate in our live service.
2. We have made good progress on the capability to allow Mental Health data providers to submit data multiple times in the year with a path forward working with our internal business.

DPS Core 29/01/2020

In this release we have progressed the following:

1. We have improved the way we process larger file sizes as received for Primary Care Medicines data
2. Engagement has begun with our Data Managers for wider usage of their toolkit for onboarding data
3. Elaborated requirements for improvements to the generation of the Unique Pregnancy Identifier.

DPS Core 22/01/2020

We are continuing to work on the best way to manage the processing of the large primary care medicines file. 

We have started preparatory work to understand requirements to link Data Processing Services to the Data Access Request Service. 

We have improved the way our data analysts can access and manipulate data on the platform.

We have made updates to the Data Manager Toolkit enabling extracts to new locations via the SEFT data transfer system. 

The first phase of the Data Manager Toolkit has been released to data managers.

The internal data services teams are now developing on the platform, promoting mental health data quality and maternity data analysis to the live system. 

We have completed work on perinatal linked data to support data services analysts. We also now derive a 'Universal Pregnancy ID'. 

We are progressing submissions and data validations to enable multiple submissions of mental health data.