Validation reports and data quality

When a data submission is processed, the result will be either an accepted submission or a rejected submission. For a rejected submission, the whole file could have been rejected at the point of submission, or all the records might have been rejected for various reasons. An accepted submission might also have had some of its records rejected. There may also be warning messages that didn’t result in a rejection, but may require some attention to fix some data quality issues.

To see these messages, the portal will generate a Validation file containing the reject messages, and a Data Quality file containing the warning messages. These files can be downloaded from the Summary Reports section of the Submission Summary page. This Submission Summary page can be seen by clicking the View link on the Dashboard page.

There is a guide available dedicated to understanding rejections and warnings:

This guide explains the difference between file, group and record-level rejections.

One of the most important things to understand about rejections is the way they cascade across tables. For example, if a record is rejected in the MPI table, the records for that Local Patient ID will also be rejected from the GP and Referral table, plus any other tables that are associated with that Local Patient ID.

It may therefore appear that you have hundreds or even thousands of individual rejections, but they can all be solved by a simple fix in one table. The Parent/Child table relationships at the end of the SDCS Cloud Data Quality Guide linked above will help to visualise this.

As a general rule when fixing validations, start with the file-level rejects (starting MHSREJ), then move onto the rejections in the Header table (starting MHS000), then work through the tables in turn.

You can find information on each message by searching the validation code in the Technical Output Specification. In the Find and Replace window, where it says within, select workbook, so that it searches all tabs.

Once you have located the code, you can look at the columns ‘Validation Rules’ and ‘Record Level Error/Warning Messages’.

The column Recvd Data Item Blank tells you whether the record will be rejected if you leave this field blank. An N/A in this column means you won’t receive a reject or a warning if you leave that field blank. If a field is mandatory, then Reject will appear in this column, which means that the record will be rejected if you leave the field blank.

The column Format Error is similar, but instead of checking whether the field is blank, it is checking whether what you have entered is in the right format. The right format is specified in the column Format.

The column National Code Error tests whether information entered is one of the options applicable to that field, either using the codes listed in the column National Code or the ODS record of organisation codes (found by searching the ODS Portal  or a full list can be downloaded from ODS data downloads.  

The column Additional Validation Rules describes any rules (in addition to the three columns just mentioned) that are applied to the data. 

The column Record Level Error/Warning Messages explains exactly what each warning or rejection code means. This includes all errors covered by the previous four columns.

We have developed a tool to summarise your validation report into more manageable chunks. This takes your Validations CSV and summarises how many records have each error and provides a further explanation of what the error means.