Mental Health of Children and Young People in England, 2023 - wave 4 follow up to the 2017 survey

This appendix provides an update on the methodology used in the Mental Health of Children and Young People (MHCYP) 2023 follow up survey. It contains information on any changes in methodology since the wave 2 survey that took place in 2021, as this is when the last detailed methods report was published. Therefore, for all methodological details, including methodology that has not changed since the wave 2 follow up survey, see the 2021 Survey Design and Methods Report. 

The MHCYP 2023 sample was based on children and young people who took part in the 2017 MHCYP survey. All participants in the 2017 survey who agreed to be recontacted for future research during their interview and continued to agree to be recontacted during the 2020 survey, the 2021 survey and the 2022 survey if they participated in any or all of these, were invited to take part in the 2023 survey.

Consent to recontact was received for 6,898 children and young people aged 8 to 25 years. Contact details were updated using the NHS Patient Register database and any sampled children who were flagged as either passed away or not to be approached for data collection were removed. This resulted in a final issued sample of 6,773 children and young people (Table 1).

The MHCYP 2023 data were collected via Computer Assisted Web Interview (CAWI) and Computer Assisted Telephone Interview (CATI), with the type of questionnaire administered dependent on the age of the sampled child or young person (see Table 2).

Information on a variety of topics were collected either through questions developed specifically for this survey or using standardised measures. The full questionnaire can be found in the questionnaire and materials documentation. The broad themes covered are listed below.

MHCYP 2023 topic coverage for children aged 8 to 16 years in the parent questionnaire:

• Demographics, household composition and tenure
• Employment status
• Strengths and Difficulties Questionnaire (SDQ)
• Self-harm
• Parent’s mental health using the General Health Questionnaire (GHQ-12)
• Loneliness
• Family functioning
• Service contact
• Waiting lists
• Physical health of child
• Education
• Special educational needs or disabilities (SEND)
• Household circumstances and changes
• Feelings about their financial situation
• Eating, sleeping and activities 
• Consent to data linkage and future research

MHCYP 2023 topic coverage for children aged 11 to 16 years in the child questionnaire:

• Strengths and Difficulties Questionnaire (SDQ)
• Short Warwick-Edinburgh Wellbeing Scale (SWEMWBS)
• Loneliness
• Family functioning
• Bullying
• Feelings about their financial situation
• Education
• Service contact
• Eating, sleeping and activities
• Social media

In 2022 and 2023, children aged 16 years and their parents were asked the consent questions on data linkage and future research whereas in 2017, 2020 and 2021 only parents were asked this. 

MHCYP 2023 topic coverage for young people aged 17 to 25 years: 

• Demographics, household composition and tenure
• Education and employment
• Strengths and Difficulties Questionnaire (SDQ)
• General Health Questionnaire (GHQ-12)
• Loneliness
• Self-harm
• Standardised Assessment of Personality – Abbreviated Scale (SAPAS)
• Family functioning
• Service contact
• Waiting lists
• Education
• Physical health 
• Household circumstances and changes
• Feelings about their financial situation
• Eating, sleeping and activities
• Social media
• Feelings about the future and climate change
• Consent to data linkage and future research

In the MHCYP 2023 survey the family connectedness scale was no longer included. All other standardised measures included in the 2021 survey were also included in 2023. These are detailed in the 2021 Survey Design and Methods Report.

Table 3 presents the additional standardised assessment tools that were included in the MHCYP 2023 survey that were not included in the 2021 survey. For the measures used to assess eating disorders see the section on the Eating disorders second stage study.

The Short Warwick-Edinburgh Mental Wellbeing Scale was previously included in the 2020 and 2022 surveys. The Standardised Assessment of Personality – Abbreviated Scale (SAPAS) has not been included in any of the previous MHCYP surveys.

The MHCYP 2023 used the Strengths and Difficulties Questionnaire (SDQ) to assess the likelihood of a possible or probable mental disorder in children and young people. 

Responses provided in the SDQ were combined using the pseudo-diagnostic algorithm to indicate whether each child or young person was unlikely, possibly or probably demonstrating a mental health condition in the following domains: emotional, behaviour and hyperactivity disorders. This algorithm combines data from all available participants. Table 4 presents the participants used to inform the identification of unlikely, possible or probable mental disorder for the 2023 survey by age.

More information on the SDQ and the participants that were used for previous surveys is available in the 2021 Survey Design and Methods Report.

An invitation letter and information leaflet were sent to each sampled address. The letter was addressed to the parent or carer of the sampled child for children aged 8 to 16 years inviting the parent to take part. For those aged 11 to 16 years, the letter also invited the sampled child (via instruction to the parent) to take part. 

For young people aged 17 to 25 years, the letter was either addressed to the parent or carer of the young person or the young person directly, depending on how consent to recontact was obtained in previous surveys. If the young person was old enough to give their own consent to be recontacted the last time they took part in the survey series, then the letter was addressed to the young person inviting them to take part. If the last time the young person had taken part it was their parent or carer that had given consent to be recontacted, then the letter was addressed to the parent or carer asking the young person to take part in the survey.

The letter explained how to complete the questionnaire online. An example of the advance letter is provided in the questionnaire and materials documentation.

A copy of the survey information leaflets (1 aimed at parents and young people, and 1 aimed at children aged 11 to 16 years) is also available in the questionnaire and materials documentation.

The fieldwork took place over 7 weeks (20 February to 10 April 2023). Participants who had not responded 3 weeks into the survey period received a reminder letter. A copy of the reminder letter can be found in the questionnaire and materials documentation.

Telephone interviews started 1 week after the online questionnaire had launched. The telephone unit phoned those participants that had not yet completed the online survey and offered support with the online survey or the ability to complete via telephone. The telephone unit focussed for the first week on ethnic minority participants and those living in the most deprived areas, before opening up to the whole sample. In addition to reminder letters, 2023 participants were also sent text and email reminders (where contact details were available). As a thank you for their participation, a £10 shopping voucher was given to each household that completed the questionnaire.

Table 5 presents the response rate to the survey, by age of the sampled child or young person. Of the 6,773 children and young people aged 8 to 25 years approached to take part, responses were received from 2,370 (35% response rate) children and young people or their parents/carers (following data cleaning and validation). Of these, 437 participants completed the survey via telephone. There were 1,203 children and young people who took part in the 2017, 2020, 2021, 2022 and 2023 surveys.

The response rate for those in the ethnic minority groups was 34% and 35% for those in the white ethnic group.

Table 6 presents the average questionnaire completion length by age of the sampled child or young person.

The time taken to complete the consent questions, collect updated contact details and details for administering the voucher is not included in Table 6 above. This would likely add an additional 2-3 minutes to the total completion time. Caution should be taken when comparing the average length of time taken to complete the questionnaire to previous follow up surveys because the questionnaire timings for previous waves included this.

The survey data were weighted to take account of non-response, so that the results were representative of the population aged 8 to 25 years in England. 

The weighting method was consistent with the method used in 2021 as detailed in the 2021 Survey Design and Methods Report. 

As in previous waves, calibration was used to ensure the weights sum to population totals. Due to the children and young people being older in 2023, the calibration groups were updated, and the achieved sample of 2,370 children and young people were calibrated to 2023 population totals. Calibration was split into 2 partitions. These are outlined in Table 7.
 

The age of the children and young people taking part has increased over time with the analysis based on the age of the sampled child or young person at specific points in time:

• in 2017 they were aged 2 to 19 years based on their age on 31 August 2017
• in 2020 they were aged 5 to 22 years based on their age on 31 August 2020
• in 2021 they were aged 6 to 23 years based on their age on 31 August 2021
• in 2022 they were aged 7 to 24 years based on their age on 31 August 2022
• in 2023 they were aged 8 to 25 years based on their age on 31 August 2023

Therefore, the age groups used for analysis have been updated to reflect this. In the MHCYP 2023 report, the cross-sectional analyses include:

• comparisons between 2017, 2020, 2021, 2022 and 2023 for those aged 8 to 16 years and 17 to 19 years (as these age groups were present at every wave)
• comparisons between 2020, 2021, 2022 and 2023 for those aged 17 to 22 years for some questions
• comparisons between 2021, 2022 and 2023 for those aged 17 to 23 years for some questions
• comparisons between 2022 and 2023 for those aged 17 to 24 years for some questions 
• standalone estimates for those aged 17 to 25 years for 2023 where appropriate (as the 2023 survey was the only one to capture 25 year olds)

The sample sizes used for the analysis in the 2023 report are as follows:

• analysis for 2017 includes the 9,117 children and young people who took part in the survey in 2017 (of these 5,798 were aged 8 to 19 years with 5,796 having a valid mental health category based on the SDQ)
• analysis for 2020 includes the 3,570 children and young people who took part in the 2020 survey (of which 2,919 were aged 8 to 22 years)
• analysis for 2021 draws on data from the 3,667 children and young people who took part in 2021 (of which 3,252 were aged 8 to 23 years)
• analysis for 2022 includes the 2,866 children and young people who took part in the 2022 survey (of which 2,731 were aged 8 to 24 years)
• analysis for 2023 includes the 2,370 children and young people who took part in the 2023 survey 

The characteristics of the sample in all survey waves can be found in Table A in the associated Excel data tables. 

All analysis presented is cross-sectional with no longitudinal analysis included in the 2023 report. Throughout the report 95% confidence intervals have been used to determine if differences between estimates are statistically significant. Estimates are considered very likely to be different if their confidence intervals do not overlap. No additional statistical hypothesis testing has been performed. Further details on confidence intervals are available in the 2021 Survey Design and Methods Report.

Approved researchers seeking to undertake further secondary analysis of data from the MHCYP surveys will be able to access the data from the UK Data Service under Special User Licence. Applicants will need to apply for permission to use the data via NHS England Data Access Request Service. Once approval has been granted, the data can be downloaded from the UK Data Service. You can find out more about access at the UK Data Service webpage and the NHS England Population health surveys webpage.

In 2023 a second stage of the study focused on eating disorders was also completed. The main survey invitation leaflets that were sent to each sampled address (along with the invitation letters) contained information on this second stage. It explained that during the main questionnaire, participants may be asked whether they are happy to complete the second stage of the study and mentioned that this would include questions about eating behaviours and any related concerns the parent or child/young person had. A copy of the survey information leaflets (1 aimed at parents and young people, and 1 aimed at children aged 11 to 16 years) is available in the questionnaire and materials documentation.

In the main survey, parents of 8 to 16 year olds, children aged 11 to 16 years and young people aged 17 to 25 years were asked to complete 5 screening questions taken from the Development and Well-Being Assessment (DAWBA) eating disorder module. Responses to these were used to determine if the child or young person ‘screened positive’ for possible eating problems. The 5-item screen was also included in the 2021 and 2022 MHCYP surveys to screen for possible eating problems.

The screening items ask:

• Have you ever thought you were fat even when other people told you that you were very thin?
• Would you be ashamed if other people knew how much you eat? 
• Have you ever deliberately made yourself vomit (throw up)?
• Do your worries about eating (such as what to eat? where to eat? how much to eat?) really interfere with your life? 
• If you eat too much, do you blame yourself a lot?

‘Screening positive’ for a possible eating problem was defined as scoring above the threshold on these questions; answering yes to 2 or more items where a parent was reporting on a child and answering yes to 1 or more items where a child or young person was reporting on themselves. The difference in threshold reflects the reluctance to disclose information that is often seen among people who have an eating disorder.

The second stage of the study did not include 8 to 10 year olds. Parents of 11 to 16 year olds whose child ‘screened positive’ based on the parent’s responses and all children and young people aged 11 to 25 years who ‘screened positive’ based on their own responses were invited to take part in the second stage (with 563 parents and young people taking part).

A selection of those that screened negative were also invited to take part (with 547 parents and young people taking part). Those that screened negative were randomly selected by the Computer Assisted Web Interview (CAWI) program to be shown the question asking whether the participant wanted to take part in the second stage. A quota was computed in the program that aimed to achieve a 70/30 split by females/males and a 50/50 split of (parents of) 11 to 16 year olds and 17 to 25 year olds. These screen negative quotas were chosen because the aim was to achieve an even spread across the age span, and because eating disorders are more common in young women. 

Participants were encouraged to answer the second stage of the study online, due to the sensitivity of the questions. However, it was possible to take part via telephone if that was the preferred option.

Parents of children aged 11 to 16 years were told that their child may be selected to take part in the second stage of the study and were asked for their consent. Parents could choose to provide consent for just themselves to take part (and not their child), for both of them to take part, or for their child to take part (if the child also agreed to taking part) and not themselves. Young people aged 17 to 25 years provided their own consent to take part.

Taking part in the second stage was voluntary, but those that completed it received a second £10 shopping voucher. Table 8 presents the response rate for the second stage of the study by the age of the sampled child or young person.
 

The questionnaire for the second stage included the full Development and Well-Being Assessment (DAWBA) eating disorder module. The DAWBA is a multi-informant standardised diagnostic assessment which was also included in the MHCYP 2017 survey (for other standardised measures used in the main questionnaire see the section on Standardised measures). 

The DAWBA includes questions about the child or young person’s feelings about eating, their eating behaviours and any concerns the parent or child/young person might have. Questions about the child’s/young person’s height and weight, how they might have been feeling or acting recently, and for girls’ questions about their menstrual cycle were also included. A copy of the questionnaire is available in the questionnaire and materials documentation.

Table 9 presents the average questionnaire completion length by age of the sampled child or young person.
 

The 5 screening items from the DAWBA eating disorders module were asked in 2021, 2022 and 2023 in response to stakeholders’ interest in increases in referrals of children with possible eating disorders to services. Using the DAWBA eating disorders screening items allowed direct comparison with responses to these questions that were also included in the 2017 survey.

Reponses from the screening items can be used to determine if the child or young person ‘screened positive’ for possible eating problems but cannot be used to determine if the child has a clinically impairing eating disorder.

The screening items are designed to route participants who are very unlikely to have an eating disorder on to the next DAWBA module. They are also designed to ensure that all those who might have an eating disorder answer all the eating disorder questions. Therefore, it is not expected that all those that ‘screen positive’ will have a clinically impairing eating disorder. Work on the 2017 sample suggests that the proportion of screen positives who have an eating disorder is low (O’Logbon et al, 2022).

Therefore, in 2023, we invited all screen positives (and a subset of screen negatives) to complete the remaining questions from the DAWBA eating disorder module so that eating disorder rates could be included in addition to rates of possible eating problems.

Parents reported on children aged 11 to 16 years, and those aged 11 years or more reported on themselves. This meant that for most 11 to 16 year olds we can combine reports from parents and children, which improves diagnostic accuracy (O’Logbon et al, 2022). 

The DAWBA eating disorders module includes highly structured questions that relate directly to diagnostic criteria in the International Classification of Disease (ICD-10) and the Diagnostic Statistical Manual, but also asks participants to describe their experience if problems are identified.

As in the baseline 2017 survey, a small team of clinical raters reviewed both structured and qualitative data from all relevant participants about individual children and young people to assign eating disorder diagnoses according to ICD-10. The eating disorder diagnoses were mutually exclusive according to the ICD-10 criteria. This was the same procedure as applied in the clinical rating in the 2017 survey (which used the same rating process for all mental health conditions). This allows us to directly compare the prevalence of eating disorders between 2017 and 2023. 

Data from all available participants was reviewed for each child and young person. For example, for children aged 11 to 16 years this may have included data from the parent only, child only or both the parent and the child. Table 10 presents the possible participants used to inform the identification of eating disorders for each age group.

In 2017, clinical raters had access to greater detail about each child they rated, including reports from parents, children/young people and teachers about a wide range of disorders. Sometimes difficulties relevant to one mental health condition were covered in the qualitative information related to another. In 2023, clinical raters only had access to reports on the eating disorder module.

No screen functions perfectly. Some of those who screen positive will turn out not to have an eating disorder on clinical rating (false positive; see Table 11), while a small number of those who screen negative will have a disorder on further testing (false negative; see Table 11).

So that stakeholders can better understand how our reports of eating problems relate to eating disorders in this and previous years, we invited a selection of those who screened negative to complete the DAWBA eating disorder module; with 547 completing it. This allows us to calculate the negative predictive value which is how many children and young people with screen negative results did not have an eating disorder on clinical rating (see Table 12). Equally, the positive predictive value tells us how likely it is that a child or young person with a positive screen had an eating disorder on clinical rating. 

The sensitivity (true positives as a proportion of all those diagnosed with a disorder on clinical rating) and specificity (true negatives as a proportion of all those clinically rated not to have a disorder) are often in tension so that when one is high the other tends to be low. 

The accuracy of a screen is the proportion of results that are true, this is calculated by adding the number of true negatives and true positives and then dividing this value by the total number of all those screened. Understanding these statistics is important to understanding the eating disorder and eating problems prevalence statistics produced by this survey.
 

Table 12 shows that the DAWBA screening items function as intended to select those with possible problems for further questions. Very few children and young people who screened negative turned out to have an eating disorder on clinical rating (3 in 100 or lower, depending on which participants completed the screening questions (child, young person, parent, or parent and child)).

The majority of children and young people who were screen positive in 2023 did not have an eating disorder on clinical rating. The positive predictive value (the proportion of those screening positive who turned out to have an eating disorder on clinical rating of the full DAWBA module) varied depending on the participant/s completing the screening questions. Positive predictive values in 2023 ranged from 16% to 33%.

In 2017, screen negatives did not complete the rest of the DAWBA eating disorders module and were coded as not having an eating disorder. So that methods are comparable to 2017, for the prevalence rates presented in the main report for 2023 (Table 5.4 in the Excel data tables) those that screened negative, completed the DAWBA eating disorders module and were identified as having an eating disorder, were coded as not having an eating disorder. 

In 2017 when the survey was face to face, all those who screened positive were routed straight through to the DAWBA eating disorders module, whereas in 2023 it was completed as part of a second stage to the study. Therefore, the percentage of screened positives (aged 11 years and over) that completed the DAWBA eating disorders module was lower in 2023 (87%). Respondents who screened positive and did not complete the DAWBA eating disorders module were compared with those who screened positive and did complete the module. It was a small number of respondents and their proportions (by sex / age / probable mental disorder) were similar. Therefore, no additional weights have been created to account for this and so caution is needed when interpreting the results.

Below (Table 13 and Table 14) we present the prevalence rates for 11 to 25 year olds for 2023 where those who screened negative and were identified as having an eating disorder are coded as having an eating disorder (instead of not having an eating disorder as is presented in the main tables). These are presented for interest and cannot be compared with 2017. Again, no additional weights have been created to account for the screening process of the second stage of the study and so caution is needed when interpreting the results (see the section on Weighting and non-response for more details on weighting).

It should be noted that the eating disorder prevalence figures for 2017 in the main report differ slightly from those presented in the MHCYP 2017 report due to slight differences in the weighting of the data. In the 2017 survey, adjustment factors were applied to the weights which were produced when the Development and Well-Being Assessment (DAWBA) was used for mental disorder prevalence estimates. In the follow up surveys the Strengths and Difficulties Questionnaire (SDQ) was used and so adjustment factors were not applied (see the 2017 survey design and methods report for more details on the weighting used in 2017).