Skip to main content

Impact of methodological changes Improving Access to Psychological Therapies (IAPT) reports

This report discusses the impact of various methodological changes made to the IAPT dataset as part of its migration from v1.5 to v2.0 on 1st September 2020. The impact of each change has been quantified in terms of its effect on activity and/or outcomes.

The report complements the Methodological Change Notice for IAPT published in January 2021


Data quality issues

Alongside the changes in methodology, the recorded outcomes after the migration have also been affected by some DQ issues, the most significant being

1. Missing scores for some providers This has led to significant reductions in improvement and recovery rates for the affected providers.

2. Missing presenting complaints This has led to a large decrease in the number of referrals with recorded conditions.

3. Internet Enabled Therapy (IET) referrals recorded both as activity logs and care contacts This has led to IET referrals with only one activity log/care contact being counted as finishing a course of treatment.

For further details about these data quality issues, see the IAPT v2.0 DQ Note.

The new dataset has a changed structure with extra tables to allow more flexibility in the recording of data (e.g. separate tables for the scores recorded during care contacts) as well as the inclusion of new data items (e.g. internet enabled therapy logs). The DQ issues listed above have occurred in the incorrect submission or lack of submission of data in these new tables.

The combination of methodology changes alongside DQ issues makes the impact of the methodology changes alone difficult to quantify. Hence, sometimes we will just present data to identify the impact the changes would have had on v1.5 data from before the migration. Other changes will show the comparison of v1.5 and v2.0 data as well.


Change in patient identification methodology

Patient identification is a key part of both the old and new IAPT datasets. Patients need to be matched in successive months’ submissions so that the care pathways can be tracked for their full duration. Most patients are identified using their NHS number but where this is missing other demographic information is used to try and match a patient across different submissions. This is where the methodology change to patient identification occurs as the new process in v2.0 (called the Master Patient Service or MPS) involves using more demographic data than the method in v1.5 (called SLAB)

To see the impact of the different patient identification methods on the dataset, we need to look at how much the two methods have changed the proportion of referrals that are linked from one month to the next. The referral data between two months is linked by the pathway identifier for both methods. The pathway id itself is made up of the service request id from the provider’s system and the patient id from the SLAB or MPS algorithm. As the service request id must match between months for the same patient referral, a difference in the linkage between the two methods in the same month can only come about because the patient identification methods have differed.

Table 1. Percentage of open referrals at the end of the previous month linked by pathways in next month. Linked pathways include referrals where the patient has moved between providers. England level data, July to November 2020.

July 2020 August 2020 September 2020 October 2020 November 2020 
Open referrals at end of previous month 340,522 349,069 356,174 375,194 385,310
Linked pathways using SLAB 338,051 346,100      
  99.3% 99.1%      
Linked pathways using MPS 338,240 346,250 340,550 369,580 378,660
  99.3% 99.2% 95.6% 98.5% 98.3%

Table 1 shows the impact of using the two different methods on linking pathways between successive months from July to November 2020. Up to the implementation of the v2.0 of the dataset on 1st September, the SLAB methodology was used to link referral data across months. From September, the MPS algorithm has been used. However, the MPS algorithm has also been applied to patient data from before September. This allows us to compare the linkage of patient data between the two identification methods for submissions in July and August 2020.

The July and August 2020 columns in Table 1 show that the SLAB and MPS algorithms link data from the previous month’s open referrals at very similar rates with slightly more pathways linked using MPS in both months (189 more in July and 150 in August). This shows the impact of the different patient identification methodology in v2.0 is small, i.e. on the order of 100 to 200 referrals in a month. As the impact on linking data across months is so small, it follows that the impact on activity and outcomes, which involve tens of thousands of referrals every month, is also small.

Table 1 also shows the sharp drop in the proportion of linked pathways with the implementation of v2.0. September 2020 shows a drop to 95.6% as over 15,000 referrals from August were not linked. This is the impact of data quality issues in the submissions to the new dataset. It will include those referrals that weren’t submitted in September at all and those that did not pass the validation procedures during submission. Also, the open referrals from August that weren’t linked in September will include referrals where the demographic data submitted in September meant that the new MPS algorithm couldn’t match the patient details with the same service request id in August


Excluding employment support appointments from definition of finishing a course of treatmen

In IAPT publications, outcomes are calculated for referrals that are deemed to have finished a course of treatment. In v1.5 of the dataset, this was defined as ended referrals that had received two attended, treatment appointments. In v2.0, the definition has changed to be ended referrals that had received two attended, non-employment support (ES) treatment appointments.

Table 2. Proportion of finishing referrals in v1.5 dataset with all or all bar one treatment appointments listed as Employment Support (ES).

Month Referrals finishing a course of treatment Referrals with or all bar one EPS appointments Percentage
March 2020  58,050 108  0.19%
April 2020 60,451 156 0.26%
May 2020 50,605 127 0.25%
June 2020 57,060 175 0.31%
July 2020 56,797 179 0.32%
August 2020  47,375 165 0.33%

Table 2 above shows that less than 1% of finishing referrals in the six months up to August 2020 had all or all bar one of their treatment appointments as ES. This indicates the proportion of finishing referrals that were incorrectly classified as completing a course of treatment in v1.5 and now have been corrected in v2.0.


Change of ADSM questionnaire for patients with PTSD

The copyright for using the Impact of Events Scale (IES) questionnaire for PTSD referrals in IAPT was not renewed for v2.0. Providers were asked to submit data using the PCL-5 questionnaire instead. For PTSD referrals that were open at the end of August 2020, this meant that scores at the start of their referral were recorded using IES and later ones using PCL-5. When the referrals finish, outcomes will then be calculated based on the later PCL-5 scores only and so it is likely that that patient improvement will be underestimated as the initial scores are not available. Recovery counts are also likely to be affected.

Table 3 PTSD Referrals finishing and outcomes for months before and after the dataset migration.

PTSD referrals finishing a course of treatment Improved Improve rate % Not caseness Recovered Recovery rates %
April 2020 2,667 1,685 63.2 65 904 34.7
May 2020 1,960 1,245 63.5 47 670 35.0
June 2020 2,363 1,564 66.2 50 878 38.0
July 2020  2,427 1,639 67.5 46 979 41.1
August 2020 2,277 1,522 666.8 57 914 41.2
September 2020 1,458 920 63.1 78 582 42.2
October 2020 1,453 882 60.7 77 538 39.1
November 2020  1,442 916 63.5 62 556 40.3
December 2020 1,273 772 60.6 73 561 41.5
January 2021 1,426 907 63.6 73 561 41.5
Mean v1.5 2,339 1531 65.5 52 869 38.0
Mean v2.0 1,410 869 62.4 71 540 40.3

Table 3 shows the changes in activity and outcomes for referrals with a provisional diagnosis (v1.5) or primary presenting complaint (v2.0) of PTSD in the lead up to and after the dataset migration on 1st September 2020.

The most striking difference before and after migration is the reduction in PTSD referrals finishing after migration. This is due to a data quality issue where many providers haven’t managed to submit presenting complaint information about the patient since the dataset migration to v2.0.

The impact on improvement and recovery rates is smaller with a reduction of around 3% in improvement and an increase of over 2% in recovery after migration. There is also a significant increase in the number of PTSD referrals starting below case ness in the v2.0 data despite the number of referrals finishing a course of treatment being substantially reduced. Impact of methodological changes. 

This is likely to be due to the change in the PTSD questionnaire which has meant that the first recorded score with the new questionnaire for some open referrals has occurred when the patient’s score has fallen below the case ness threshold.


Using ADSMs in the calculation of outcomes for referrals below casenes

In IAPT, the outcomes of referrals are calculated using specific questionnaires for certain conditions, namely agoraphobia, social phobia, obsessive-compulsive disorder, PTSD, panic disorder and hypochondriacal disorder. These condition-specific questionnaires are known as Anxiety Disorder Specific Measures (ADSMs).

In v1.5 of IAPT, the outcome of referrals were calculated based on scores from the ADSM questionnaire for the appropriate condition as long as the patient had at least two of those scores (this is known as having paired scores) and started the referral above the caseness threshold for the ADSM measure. If the patient started below caseness or didn’t have paired scores for the ADSM questionnaire, their outcomes would be calculated using the Generalised Anxiety Disorder (GAD7) questionnaire scores

In v2.0 of the dataset, a patient’s recovery is now based on the ADSM questionnaire whether or not their initial scores started above the caseness threshold as long as the referral has paired scores for the relevant ADSM. Otherwise, the referral will again use GAD7 scores to calculate the referral outcomes.

Table 4. Referrals finishing a course of treatment with an ADSM specific condition and recovery outcomes using the v1.5 and v2.0 methodologies for months up to the dataset migration. 

ADSM referrals finishing a course of treatment ADSM specific scores used in v1.5 ADSM specific scores used in v2.0 method Referrals recovered in v1.5 Recovery rate in v1.5 Referrals recovered using v2.0 method Recovery rate using v2.0 method
April 2020 8,359 3,913 4,400 3,047 37.9% 2,974 37.5%
May 2020 6,137 2,762 3,149 2,411 40.9% 2,350 40.4%
June 2020 7,247 3,468 3,196 2,971 42.6% 2,893 41.1%
July 2020 7,333 3,735 4,264 3,192 45.2% 3,086 44.5%
August 2020  6,661 3,472 3,923 2,872 44.6% 2,810 44.3%

 

Table 4 shows the number of referrals that finished a course of treatment in the months before the dataset migration with a provisional diagnosis associated with one of the six ADSM used within IAPT. We have not shown the months after migration as the effect of this methodology change will be mixed with that due to the new PTSD questionnaire (see section above).

Table 4 shows that the v2.0 methodology includes between 387 and 529 extra referrals per month that use the appropriate ADSM within the outcome calculations rather than GAD7. The impact of this is to reduce the number of referrals that recover by between 61 and 106 Impact of methodological changes Copyright © 2021 NHS Digital 9 using the v2.0 methodology during the selected months. Recovery rates are also reduced by between 0.3% and 0.7%.


Further information

For an explanation of all measures in the Monthly and Quarterly Activity Data File CSVs, see the IAPT Metadata document and the IAPT v2.0 Guidance document.

Last edited: 9 June 2021 12:57 pm