Impact of methodological changes Improving Access to Psychological Therapies (IAPT) reports

The report complements the Methodological Change Notice for IAPT published in January 2021

Alongside the changes in methodology, the recorded outcomes after the migration have also been affected by some DQ issues, the most significant being

1. Missing scores for some providers This has led to significant reductions in improvement and recovery rates for the affected providers.

2. Missing presenting complaints This has led to a large decrease in the number of referrals with recorded conditions.

3. Internet Enabled Therapy (IET) referrals recorded both as activity logs and care contacts This has led to IET referrals with only one activity log/care contact being counted as finishing a course of treatment.

For further details about these data quality issues, see the IAPT v2.0 DQ Note.

The new dataset has a changed structure with extra tables to allow more flexibility in the recording of data (e.g. separate tables for the scores recorded during care contacts) as well as the inclusion of new data items (e.g. internet enabled therapy logs). The DQ issues listed above have occurred in the incorrect submission or lack of submission of data in these new tables.

The combination of methodology changes alongside DQ issues makes the impact of the methodology changes alone difficult to quantify. Hence, sometimes we will just present data to identify the impact the changes would have had on v1.5 data from before the migration. Other changes will show the comparison of v1.5 and v2.0 data as well.

Patient identification is a key part of both the old and new IAPT datasets. Patients need to be matched in successive months’ submissions so that the care pathways can be tracked for their full duration. Most patients are identified using their NHS number but where this is missing other demographic information is used to try and match a patient across different submissions. This is where the methodology change to patient identification occurs as the new process in v2.0 (called the Master Patient Service or MPS) involves using more demographic data than the method in v1.5 (called SLAB)

To see the impact of the different patient identification methods on the dataset, we need to look at how much the two methods have changed the proportion of referrals that are linked from one month to the next. The referral data between two months is linked by the pathway identifier for both methods. The pathway id itself is made up of the service request id from the provider’s system and the patient id from the SLAB or MPS algorithm. As the service request id must match between months for the same patient referral, a difference in the linkage between the two methods in the same month can only come about because the patient identification methods have differed.

Table 1. Percentage of open referrals at the end of the previous month linked by pathways in next month. Linked pathways include referrals where the patient has moved between providers. England level data, July to November 2020.

Table 1 shows the impact of using the two different methods on linking pathways between successive months from July to November 2020. Up to the implementation of the v2.0 of the dataset on 1st September, the SLAB methodology was used to link referral data across months. From September, the MPS algorithm has been used. However, the MPS algorithm has also been applied to patient data from before September. This allows us to compare the linkage of patient data between the two identification methods for submissions in July and August 2020.

The July and August 2020 columns in Table 1 show that the SLAB and MPS algorithms link data from the previous month’s open referrals at very similar rates with slightly more pathways linked using MPS in both months (189 more in July and 150 in August). This shows the impact of the different patient identification methodology in v2.0 is small, i.e. on the order of 100 to 200 referrals in a month. As the impact on linking data across months is so small, it follows that the impact on activity and outcomes, which involve tens of thousands of referrals every month, is also small.

Table 1 also shows the sharp drop in the proportion of linked pathways with the implementation of v2.0. September 2020 shows a drop to 95.6% as over 15,000 referrals from August were not linked. This is the impact of data quality issues in the submissions to the new dataset. It will include those referrals that weren’t submitted in September at all and those that did not pass the validation procedures during submission. Also, the open referrals from August that weren’t linked in September will include referrals where the demographic data submitted in September meant that the new MPS algorithm couldn’t match the patient details with the same service request id in August

In IAPT publications, outcomes are calculated for referrals that are deemed to have finished a course of treatment. In v1.5 of the dataset, this was defined as ended referrals that had received two attended, treatment appointments. In v2.0, the definition has changed to be ended referrals that had received two attended, non-employment support (ES) treatment appointments.

Table 2. Proportion of finishing referrals in v1.5 dataset with all or all bar one treatment appointments listed as Employment Support (ES).

Table 2 above shows that less than 1% of finishing referrals in the six months up to August 2020 had all or all bar one of their treatment appointments as ES. This indicates the proportion of finishing referrals that were incorrectly classified as completing a course of treatment in v1.5 and now have been corrected in v2.0.

The copyright for using the Impact of Events Scale (IES) questionnaire for PTSD referrals in IAPT was not renewed for v2.0. Providers were asked to submit data using the PCL-5 questionnaire instead. For PTSD referrals that were open at the end of August 2020, this meant that scores at the start of their referral were recorded using IES and later ones using PCL-5. When the referrals finish, outcomes will then be calculated based on the later PCL-5 scores only and so it is likely that that patient improvement will be underestimated as the initial scores are not available. Recovery counts are also likely to be affected.

Table 3 PTSD Referrals finishing and outcomes for months before and after the dataset migration.

Table 3 shows the changes in activity and outcomes for referrals with a provisional diagnosis (v1.5) or primary presenting complaint (v2.0) of PTSD in the lead up to and after the dataset migration on 1st September 2020.

The most striking difference before and after migration is the reduction in PTSD referrals finishing after migration. This is due to a data quality issue where many providers haven’t managed to submit presenting complaint information about the patient since the dataset migration to v2.0.

The impact on improvement and recovery rates is smaller with a reduction of around 3% in improvement and an increase of over 2% in recovery after migration. There is also a significant increase in the number of PTSD referrals starting below case ness in the v2.0 data despite the number of referrals finishing a course of treatment being substantially reduced. Impact of methodological changes. 

This is likely to be due to the change in the PTSD questionnaire which has meant that the first recorded score with the new questionnaire for some open referrals has occurred when the patient’s score has fallen below the case ness threshold.

In IAPT, the outcomes of referrals are calculated using specific questionnaires for certain conditions, namely agoraphobia, social phobia, obsessive-compulsive disorder, PTSD, panic disorder and hypochondriacal disorder. These condition-specific questionnaires are known as Anxiety Disorder Specific Measures (ADSMs).

In v1.5 of IAPT, the outcome of referrals were calculated based on scores from the ADSM questionnaire for the appropriate condition as long as the patient had at least two of those scores (this is known as having paired scores) and started the referral above the caseness threshold for the ADSM measure. If the patient started below caseness or didn’t have paired scores for the ADSM questionnaire, their outcomes would be calculated using the Generalised Anxiety Disorder (GAD7) questionnaire scores

In v2.0 of the dataset, a patient’s recovery is now based on the ADSM questionnaire whether or not their initial scores started above the caseness threshold as long as the referral has paired scores for the relevant ADSM. Otherwise, the referral will again use GAD7 scores to calculate the referral outcomes.

Table 4. Referrals finishing a course of treatment with an ADSM specific condition and recovery outcomes using the v1.5 and v2.0 methodologies for months up to the dataset migration. 

Table 4 shows the number of referrals that finished a course of treatment in the months before the dataset migration with a provisional diagnosis associated with one of the six ADSM used within IAPT. We have not shown the months after migration as the effect of this methodology change will be mixed with that due to the new PTSD questionnaire (see section above).

Table 4 shows that the v2.0 methodology includes between 387 and 529 extra referrals per month that use the appropriate ADSM within the outcome calculations rather than GAD7. The impact of this is to reduce the number of referrals that recover by between 61 and 106 Impact of methodological changes Copyright © 2021 NHS Digital 9 using the v2.0 methodology during the selected months. Recovery rates are also reduced by between 0.3% and 0.7%.