Check for National Data Opt-outs Service

MESH is NHS Digital's secure messaging service. It supports two-way transfer of data between NHS Digital and health and care organisations. Data is fully encrypted during transit. The service uses MESH to allow users to send a data file containing a list of patient NHS numbers to be checked against the Spine repository, where information about national data opt-outs is stored. The service then uses MESH to send back an updated version of the data file with the NHS numbers of those who have opted out removed.

MESH can be accessed using the MESH client, the most widely-used option, or through systems that have been integrated using the MESH API. This guidance tells you how to set up and configure the MESH client, and how to use the service to check for national data opt-outs.

In most cases systems suppliers or IT administrators will handle the set-up of MESH for each organisation, and end users should get in touch with their organisation's IT contacts with any questions about setting up the service, including installation, access rights or running the MESH client.

If your organisation already has the MESH client installed, you may be able to reuse this instead of installing a new instance. 

To install a new MESH client, follow the instructions on the MESH technical information page. Full guidance and downloads are given in the installation pack.

If you decide to use a new mailbox with an existing client, you need to edit the MESH configuration file, meshclient.cfg, which will be in the directory and folder where the MESH client has been installed.

Add a new client section by including the following code in the configuration file:

<Client>

<ClientIdentity>mailbox name</ClientIdentity>

<ClientAuthentication>authentication key</ClientAuthentication>

<MailboxType>MESH</MailboxType>

<InterfaceRoot>C:\MESH-DATA-HOME\Mailbox2</InterfaceRoot>

<CollectReport>Y</CollectReport>

<TransferReport>Y</TransferReport>

<PollReport>N</PollReport>

<SaveSent>Y</SaveSent>

</Client>

Enter the mailbox name and authentication key you were sent when your MESH mailbox was set up. Make sure the location in the InterfaceRoot field matches the directory location where the MESH client has been installed, and ensure a corresponding folder is created in the root directory location.

See an example of a multi mailbox configuration file

Once you have configured MESH, you can use it to send lists of NHS numbers to be checked, and receive amended lists back.

Before using the service, please review the Check for National Data Opt-outs licence agreement, which notes the rights and conditions upon which your organisation may use the Check for National Data Opt-outs service provided by NHS Digital

Creating data and control files

To send a set of NHS numbers to the Check for National Data Opt-outs service via MESH, you need to create two paired files. These are:

• .dat file – the data file containing the list of NHS numbers that need to be checked
• .ctl file – the control file that contains the information required to identify, transmit and audit the data file

The paired files must have exactly the same name, apart from their respective extensions.

You can use any text editor, including Notepad or Wordpad, to create or edit the files.

Outbound data file (.dat)

This file must only contain a list of NHS numbers, each followed by a comma and  a carriage return (including the final one). Including any other details will stop the file being processed. Files created with a .csv extension type can be renamed to a .dat file. 

See an example of a data file containing dummy NHS Numbers 

Outbound control file (.ctl)

This file tells the MESH server to transfer the .dat file to the Check for National Data Opt-outs service.

SPINE_NTT_UPHOLDING is the workflow identifier for this service. You must copy this exactly to the workflow identifier (WorkflowId) tag, as shown in the example .ctl file below.

The control file must match the format of the following example, with the item highlighted in bold replaced with your own organisation’s details. Without these changes your file will not be successfully delivered and processed.

<DTSControl>

<Version>1.0</Version>

<AddressType>DTS</AddressType>

<MessageType>Data</MessageType>            

<WorkflowId>SPINE_NTT_UPHOLDING</WorkflowId>

<To_DTS>X26HC036</To_DTS>

<From_DTS>XXXXXXX</From_DTS> 

<Subject>NDOP Response</Subject>

 <LocalId>EXAMPLENDOP-14122018-0001</LocalId> 

<Compress>Y</Compress>

<AllowChunking>Y</AllowChunking>

<Encrypted>N</Encrypted>

</DTSControl>

The FromDTS field should contain your organisation's mailbox ID. You should provide a value for the Local Identifier (LocalId) tag in the .ctl file(s) particularly if you're sending lots of different files. Files returned by the service will contain the same LocalId so that you can match returned files with the original source files.

The code <AllowChunking>Y</AllowChunking> tells MESH to allow large files to be processed in 20Mb chunks. Without this configuration value in the .ctl file, a file larger than 100 Mb will make the transfer fail.

Naming the data and control files

Data files submitted to MESH must have unique filenames, so that they can be tracked if there are any issues.

A suggested format to follow is to use your mailbox ID followed by a hyphen then day, month and year of the date the file is produced followed by a hyphen and then a 4-digit sequence number to show what order that file has been produced in that month.

So, for example for an organisation with MESH mailbox id YCLTA53 that had produced a file on 7 February 2019 which was the fifth file produced that month, the file names would be:

• YCLTA53-070219-0005.dat
• YCLTA53-070219-0005.ctl

Sending files to the checking service with MESH

When the MESH client is installed the following default file system directory/folder structure is created. The location of the directory will be dependent upon how the MESH client has been installed at your location.

<MESH-DATA-HOME>

\FAILED

\IN

\OUT

\SENT

\TEMP

\TEMP\RCV

Place your correctly named and configured files together in your MESH OUT folder (put the data file in first if moving them individually).

If the MESH client has not been set to automatically run (if it's not in the start-up services enabled on your computer) then after placing the files in the OUT folder you will need to run the MESH Client.

Timescales

How long it will take to get the processed file back depends on how often the MESH client is configured to connect to the mailbox - this is referred to as 'polling'. By default, this is set up to be every 30 minutes, but you can change this in the meshclient.cfg file. Changing the PollPeriod value will change how often the MESH client connects, so <PollPeriod>40</PollPeriod> will result in the MESH client checking every 40 minutes.

The Check for National Data Opt-outs Service is also set to ‘poll’ every 30 minutes to collect new files to be checked and to provide updated files back to the MESH client. This can mean that if the polling times have not been changed it could take up to 1 hour to get a processed file back after you send it. However, we don't recommend shortening the polling time unless you use the service more than once a day.

We're reviewing the polling period for the checking service, and we expect to reduce it to 1 minute during 2019.

Housekeeping

Once data files are sent via MESH, both the control and data files are automatically copied into the SENT folder. It is the responsibility of your organisation to manage the SENT folder and provide the necessary housekeeping to move, delete or archive these files as required.

Check for failed transfers

If a control file has been completed incorrectly it will be moved to the FAILED folder along with the respective data file. You should check this folder periodically for files that could not be sent via MESH.

Status of files sent to the checking service through MESH

Unlike some other services which use MESH to submit files, you will not receive a delivery status update file from MESH - just a processed data file.

If you need to check on the delivery status of any files submitted via MESH, you can either:

• contact the National Service Desk
• apply for an account to track and trace files sent using MESH by using the MESH Online Enquiry Service (MOLES). 

Receiving files from the checking service through MESH

Files sent back from the service will be placed in the IN folder, along with an updated control file.

Successful transactions

Provided the service has been able to process the file successfully, it will send back a data file with an updated list of NHS numbers - with any that had a national data opt-out recorded removed from the list. This will be placed in the IN folder along with an updated control file. The control file will have a blank subject field, but will include a status record which confirms the transfer of the file back to the organisation, for example:

<DTSControl>

<Version>1.0</Version>

<AddressType>DTS</AddressType>

<MessageType>Data</MessageType>            

<WorkflowId>SPINE_NTT_UPHOLDING</WorkflowId>

<To_DTS>XXXXXXX</To_DTS> *Your mailbox ID

<From_DTS>X26HC036</From_DTS>

<Subject>(blank)</Subject> *blank field, no subject line

<LocalId>EXAMPLENDOP-14122018-0001</LocalId> *LocalID provided in outbound ctl file

<StatusRecord> *Information on the status of the data transfer

   <DateTime>20180928094223</DateTime>

   <Event> TRANSFER </Event>

   <Status> SUCCESS </Status>

   <StatusCode>00</StatusCode>

   <Description> Transferred to recipient mailbox </Description>

</StatusRecord>

</DTSControl>

Error messages

If there was something wrong with the file you submitted, you may get a .dat file sent back containing an error message. Some common ones are listed here, along with what to do when you resubmit the file to try again. If these error messages persist, or you get one that you don't understand, contact the National Service Desk via the weblog or call them on 0300 3035035.

FileHasNoContent

Make sure the data file you provided contains only 10-digit NHS Numbers, separated by a ‘,’.

The file has more than one column

Make sure the data file you provided contains only a single column of 10-digit NHS Numbers, separated by a ‘,’.

The file is missing one or more delimiters

Make sure the data file you provided is comma delimited - each NHS number should be followed by a ',' including the last one on the list. See the example .dat file.

The file contains one or more invalid NHS numbers

Make sure the NHS numbers provided are all 10-digits and just consist of numbers.

Failing to receive a file back

If you don't get a file back, check the file has gone from the OUT folder and been copied to the SENT folder. If no:

• check the MESH client is running
• check the polling period hasn't been changed in the meshclient.cfg file
• check your FAILED folder in case the file was not set up correctly and could not be transferred

If yes:

• check whether any automated processes could be deleting files before you have a chance to collect them

If the problem persists, contact the National Service Desk via the weblog or call them on 0300 3035035.

We've created a small set of test NHS numbers you can test with. You should not use dummy NHS numbers from your own systems, as the service will not recognise them.

Follow the process above with these numbers, to test your processes for preparing, sending and receiving files. Create a .dat and .ctl form and send them to the service. The returned file should then contain the 11 NHS numbers which have no opt-out registered.

There are 15 test NHS numbers available, 4 of which have a national data opt-out registered. The test NHS numbers are shown below, together with some of the associated demographic details and the opt-out status:

NHS_No	DoB	Age	GENDER	FAMILY_NAME	TITLE	Opt-out Registered
9990536627	20050203	14	Female	XXTESTPATIENT-TEXR	MISS	No
9990537313	20030503	16	Female	XXTESTPATIENT-TFAD	MISS	No
9990537364	20010518	18	Female	XXTESTPATIENT-TFAH	MISS	No
9990569479	19760518	43	Female	XXTESTPATIENT-TJIN	MRS	Yes
9990569401	19750721	44	Female	XXTESTPATIENT-TJIH	MS	Yes
9990539685	20010627	18	Male	XXTESTPATIENT-TFIK	MR	Yes
9990539642	20000917	19	Male	XXTESTPATIENT-TFIG	MR	Yes
9990539596	19950125	24	Male	XXTESTPATIENT-TFIC	MR	No
9990539502	19831220	36	Male	XXTESTPATIENT-TFHU	MR	No
9990539537	19731203	46	Male	XXTESTPATIENT-TFHX	MR	No
9990539693	19661027	53	Male	XXTESTPATIENT-TFIL	MR	No
9990539634	19550608	64	Male	XXTESTPATIENT-TFIF	MR	No
9990539553	19540324	65	Male	XXTESTPATIENT-TFHZ	MR	No
9990539499	19410707	78	Male	XXTESTPATIENT-TFHT	MR	No
9990539510	19360512	83	Male	XXTESTPATIENT-TFHV	MR	No

Download the test dataset

Other demographic details have been set to the same values for each of the 15 NHS numbers:

Demographic Detail	Setting (for all 15 NHS Numbers)
GIVEN_NAME	DONOTUSE
OTHER_GIVEN_NAME	-
ADDRESS_LINE_1	C/O NHS DIGITAL TEST DATA MANAGER
ADDRESS_LINE_2	SOLUTION ASSURANCE 1 TREVELYAN SQ.
ADDRESS_LINE_3	BOAR LANE
ADDRESS_LINE_4	LEEDS
ADDRESS_LINE_5	WEST YORKSHIRE
POST_CODE	LS1 6AE
GPP_CODE	Y90001