This section aims to provide users with an evidence-based assessment of the quality of the Autism Statistics publication by reporting against the European Statistical System (ESS) quality dimensions and principles appropriate to this output.
In doing so, this meets our obligation to comply with the UK Statistics Authority (UKSA) Code of Practice for Official Statistics, particularly Principle 4, Practice 2 which states:
“Ensure that official statistics are produced to a level of quality that meets users’ needs, and that users are informed about the quality of statistical outputs, including estimates of the main sources of bias and other errors and other aspects of the European Statistical System definition of quality”.
For each dimension we describe how this applies to the publication and reference any measures in the accompanying monthly data quality measures report that are relevant for assessing the quality of the output.
Autism StatisticsExperimental statistics
- Publication Date:
- 14 Nov 2019
- Geographic Coverage:
- Geographical Granularity:
- Provider, Country
- Date Range:
- 01 Apr 2018 to 31 Dec 2018
This section aims to provide users with an evidence-based assessment of the quality of the Autism Statistics publication by reporting against the European Statistical System (ESS) quality dimensions and principles appropriate to this output.
This data has been collected and presented as part of a programme of work, led by the Department of Health and Social Care (DHSC) and NHS England and NHS Improvement, to improve health and wellbeing outcomes for autistic children any young people and adults.
The data included in this report are based on the following National Institute for Health and Care Excellence (NICE) guidance:
- NICE clinical audit support tools covering adults and children – NICE Guidance on Autism spectrum disorder in adults: diagnosis and management (CG142);
- NICE Guideline on Autism spectrum disorder in under 19s: recognition, referral and diagnosis (CG128);
- NICE Quality Standard QS51 for autism.
NICE recommend that people who are referred to a professional or team who specialise in diagnosing autism should have an assessment started within 13 weeks of their referral. Data from the Mental Health Services Dataset (v3.0) enables us to measure whether this waiting time recommendation is being met.
MHSDS is a patient level, output based, secondary uses data set which aims to deliver robust, comprehensive, nationally consistent and comparable information for children, young people and adults who are in contact with mental health services, including autism services. People who receive autism spectrum disorder services or who are thought to have an autism spectrum disorder are within the scope of the MHSDS.
Changes in Version 3.0 of the MHSDS, introduced on 1 April 2018, include enhanced data collection in support of timely recording of diagnosis of autism spectrum disorder. From 1 April 2018 providers of mental health, learning disabilities and autism spectrum disorder services must start submitting data related to autism patients, as two new options are available for the primary reason for referral as follows:
- 25 Suspected Autism Spectrum Disorder
- 26 Diagnosed Autism Spectrum Disorder
Initially, autism reporting will focus on the number of new patients referred for suspected autism within the reporting period and waiting times for first appointment (in weeks).
These statistics are classified as ‘Experimental’ and should be used with caution. Experimental statistics are new official statistics undergoing evaluation. They are published in order to involve users and stakeholders in their development and as a means to build in quality at an early stage. More information about Experimental statistics can be found here: https://www.statisticsauthority.gov.uk/wp-content/uploads/2018/02/Code-of-Practice-for-Statistics.pdf
Data limitations and interpretation issues
Work to determine which provider organisations should be submitting data for autism patients is on-going and as such a final list of in-scope autism providers is not yet available. In the absence of an in-scope list of autism providers, we are unable to state how many of the organisations that have submitted data to MHSDS should have submitted autism data.
Submitting Autism Data
The submission of autism data can be very complex as the diagnosis process involves multi-disciplinary teams and take time to agree a diagnosis of autism. As a result, the patient journey can be difficult to represent in MHSDS.
Further complexity can arise where some patients have more than one health condition and can be treated for different conditions under the same referral. This is then also difficult to record and measure over time and may result in data submission issues.
Autism waiting time information relies on submitters providing referral, care contact and diagnosis information in a timely manner, i.e. by the time the submission window closes for each reporting period. If a provider submits any activity data (referrals or diagnosis information) related to the reporting period outside of the submission window, then these are accepted into the dataset if the referral is still open but may not feature in Official Statistics reports, since these extracts are taken at specific intervals. Care contact information cannot be updated retrospectively for any open referrals due to the limitations of the dataset version and the requirements of submitters to send this information promptly. Therefore, there are a significant number of records with a missing (NULL) care contact date and so waiting times between referral and first care contact cannot be derived for some patients.
A missing (NULL) care contact date can mean one of two things: that a care contact occurred but was not submitted before the submission window closed, or that the patient is still waiting for a first appointment. It is not possible to discern the underlying reason from the dataset.
Impact of introducing ‘Suspected Autism’ as a primary reason for Referral
Until 1 April 2018, there was no way to select ‘Suspected Autism’ (code 25) as a primary reason for referral. Instead it was combined within a broader category with various other health conditions under ‘Neurodevelopmental conditions’ (code 24). Even from 1 April 2018, some submitters may still be submitting autism referrals as code 24 if they are not aware of the new referral reason explicitly for ‘Suspected Autism’. As we are unable to identify these as suspected autism records, we are unable to use these records for waiting times between referral and first appointment.
Even since the introduction of ‘Suspected Autism’ as a primary reason for referral, due to the length of time it can take to confirm an autism diagnosis we are not currently able to track the whole autism pathway for the majority of patients from a suspected autism referral to an autism diagnosis. We can only do this for only a small number of records at this stage – where the referral was submitted as suspected autism on or after 1st April 2018 and the diagnosis has been confirmed by the end of December 2018 with all relevant data submitted into MHSDS. Over time, it is expected that the number of records with a whole patient pathway will increase.
The approach for identifying referrals described above is known as a ‘forward model’ – this means all suspected autism referrals within the reporting period are identified and then these are linked to first appointment dates that occurred after the referral date. This measure does not count anyone with a referral reason other than ‘Suspected Autism’ (code 25).
The main limitation of the ‘forward model’ is that there is a reliance on future data to identify whether care contacts have occurred and so this means that the quarters are not comparable.
At the point at which the analysis was undertaken, the latest available full quarter of data is October-December 2018 (quarter 3 2018-19) and so some referrals received in this quarter have not had enough time elapsed to assess whether they meet the 13 week waiting time target to first contact. We therefore expect a higher proportion of referrals with no first contact as illustrated in the chart below:
Another limitation of this approach is that it people with referrals for other things that will end up with a diagnosis of autism.
Future releases of these statistics will seek to develop alternative approaches for identifying possible in-scope patients.
Future dataset versions
MHSDS v.4.0 came into effect on 1 April 2019 but there has been no change in how autism data is collected and reported. However, this version of the dataset is now submitted by providers on a new data platform which is likely to impact on data quality further in future reporting, due to known issues providers had in submitting data to the platform in April to June 2019. Further details about these issues can be found in the data quality reports that accompany the monthly Mental Health Services statistics
This dimension covers the degree to which the statistical product meets user need in both coverage and content.
The statistics in this publication are marked as ‘Experimental’ and may be subject to further change as we develop these statistics. Experimental statistics are a subset of newly developed or innovative official statistics undergoing evaluation. The classification of Experimental Statistics is in keeping with the UK Statistics Authority’s Code of Practice. The ‘Value’ pillar in the Code of Practice advocates that as suppliers of information it is important that we continue “improving existing statistics and creating new ones through discussion and collaboration with stakeholders”.
Accordingly, these are published in order to involve users and stakeholders in their development, and to build quality at an early stage. It is believed that these statistics have immediate value to users; however, those users need to be aware of the statistics’ limitations and related cautions around their use, which are set out in this document, to make informed and qualified use of them. The methods used are still subject to regular evaluation and future modification.
The UK Statistics Code of Practice can be accessed via the following web-link: https://www.statisticsauthority.gov.uk/wp-content/uploads/2018/02/Code-of-Practice-for-Statistics.pdf
Content of this publication
Autism Statistics has been developed to help achieve better outcomes for children & young people and adults with autism. It provides data that will be used to commission services in a way that improves health and reduces inequalities and supports service improvement and clinical quality. This is done by capturing detailed information submitted about the patient (age, gender, ethnicity) along with submitted data at provider level. Further detail on the information captured within the MHSDS dataset is available from: https://digital.nhs.uk/data-and-information/data-collections-and-data-sets/data-sets/mental-health-services-data-set
The initial publication comprises of a set of 2 tables for each financial quarter, in Excel format with a web summary of key information based on those organisations that submitted data.
This release provides the latest information about the people who are in contact with autism services. This publication contains information based on total submissions for England and at provider level, but not all provider organisations are submitting data. Therefore, please treat these figures with caution. Where totals are presented for England, these are aggregate totals of the areas that submitted the relevant information.
There are currently 50+ tables in the MHSDS dataset that each contain information relating to a specific event or type of information that may occur during the mental health care of patients. A link to the dataset model is here for further information: https://digital.nhs.uk/data-and-information/data-collections-and-data-sets/data-sets/mental-health-services-data-set/mental-health-services-data-set-specifications-and-guidance
Only a subset these tables have been used for this release as follows:
- Patient Details (MHS001)
- Referral Details (MHS101)
- Service Type Referred to (MHS102)
- Care Contact Details (MHS201)
- Primary diagnoses (MHS604)
- Secondary diagnoses (MHS605)
Information from these tables have been used to produce the following tables:
Table 1: Number of new patients referred for suspected autism within the reporting period and waiting times for first appointment (in weeks) by age, gender and ethnicity
Table 2: Number of new patients referred for suspected autism within the reporting period and waiting times for first appointment (in weeks) by provider
Reporting will change to include more data tables as data quality and coverage improves and hence increase the relevance of this report to reflect wider user needs.
Coverage – are all relevant providers submitting data?
All providers of NHS-funded specialist mental health, learning disability and autism services should submit to the MHSDS. At present only a small number of independent sector providers are making submissions, and this has an impact on coverage.
Generally, the number of service providers who have provided an MHSDS submission is monitored and reported in the MHSDS Submission Update which shows the number of providers submitting data each month and the number of records by provider and by table. These reports are available at:
The submission report shows those providers providing a submission compared to those NHS Digital currently consider to be in scope for the collection. The providers considered in scope for the MHSDS have been determined through intelligence gathered from other national datasets and from information provided by national and local stakeholders. This list is subject to revision based on new intelligence received.
In order to be counted as providing a submission, only two of the data tables are mandated to flow each time any activity is reported within the MHSDS (MHS001; patient demographics and MHS002; GP registration). Completion of the remaining tables is only required when activity has occurred that is captured within these tables.
All providers of autism services are required to submit data if they provide this care. Diagnosis takes place across mental health, community and paediatric services and so it is difficult to determine an exact number of organisations that should be submitting autism data to the MHSDS. Work to determine which provider organisations should be submitting data for autism patients is ongoing, using other sources of information (e.g. National Autistic Society directory) as well as contacting providers directly. This will help build a clearer picture for future reporting.
As part of the MHSDS monthly statistics publication, national and organisation level data quality measures are shown that validate a selection of key data items by provider. These show the proportion of records as counts and percentages which have ‘valid’, ‘other’, ‘default’, ‘invalid’ and ‘missing’ values for key elements of the dataset, such as Primary Reason for Referral. These DQ measures need to be considered when interpreting these autism statistics.
Completeness of submissions varies month-on-month by provider. Local knowledge may be required to assess the completeness of a submission, based on information about local caseload. This release provides detailed information about activity, which providers and commissioners are encouraged to review to ensure that submissions accurately reflect the local situation.
We have been able to group submitters into five categories:
- Non-submitters for anything autism related but do submit to MHSDS for other types of patients
- Partial submitters that submit autism referrals but nothing else
- Partial submitters that do not submit a primary reason for referral but do submit autism diagnoses (mostly with care contact dates)
- Irregular submitters i.e. don’t submit each month (this can include full and partial submissions).
- Good submitters that submit autism referrals, care contact dates and diagnoses information each month
Providers have been contacted in relation to non-submission, partial completion and low submission. Responses were summarised as follows:
- Confirmation that they do not provide autism services;
- Uncertainty around how to submit the data and advice has now been given, signposting to the autism guidance;
- That they do not have the relevant IT systems in place to provide all the relevant information for submission especially around coded diagnosis. For some providers, new systems are being introduced;
- Some providers assess patients as having suspected autism but seems there is confusion around recording this as a referral or onward referral especially where providers do not treat or formally diagnose patients for autism;
- Some providers are not able to record autism-specific activity in their local systems and still using broader category of ‘Neurodevelopment conditions’;
- Community hospitals have submitted data to MHSDS for autism but not sure if they should. This is where patients are presented to them with more than one health condition including autism with some of the patient pathway recorded in the Community Services Dataset.
As a result of the above findings, NHS Digital plan to:
- Review the current autism guidance for data submitters;
- Wait to see whether certain providers submissions improve following advice and guidance or following the introduction of improved data collection and IT systems;
- Produce an in-scope list of those organisations that provide autism services, so coverage and completeness are better understood.
Reporting requirements that cannot be yet met due to very low completeness are:
- Post-diagnostic outcomes which require completion of the Onward Referral table and an increased number of submissions for referrals for ‘diagnosed autism’ (code 26).
- Robust waiting times between referral and first care contact/appointment - current waiting times are based on length of time between the referral date and earliest care contact date submitted for each patient. This could be an initial assessment/triage appointment or a first treatment appointment. First treatment appointments could be a more robust measure for stopping the clock on waiting times but there is very low completion of the Consultation Type field which indicates whether the first appointment was an initial assessment/triage or a first treatment appointment. Therefore, all care contact dates have been used to stop the clock on waiting times.
Clinical codes used for autism diagnoses
In terms of diagnoses information, different clinical classifications have been used across providers. Some providers have submitted International Classification of Diseases Version 10 codes (ICD-10), some READ codes and some SNOMED codes. This analysis captures all classifications. For ICD-10 submitters, all F84.0 codes have been included which are all codes for autistic spectrum disorder. A list of all ICD-10 codes can be found here: https://www.who.int/classifications/icd/icdonlineversions/en/
For SNOMED, the following concept IDs have been used:
Active but odd autism (disorder)
Active infantile autism
Autistic spectrum disorder
High-functioning autism (disorder)
Pathological demand avoidance
Residual Asperger’s (disorder)
Residual infantile autism
The above codes have been mapped to READ codes to ensure all READ V2.0 and Read CTV3 codes are also captured.
From 1st April 2020, SNOMED completion will become mandatory which will help to standardise coding and reporting.
Accuracy and reliability
This dimension covers, with respect to the statistics, their proximity between an estimate and the unknown true value.
The MHSDS is a rich, person-level dataset that records packages of care received by individuals in contact with mental health secondary care services. NHS Digital provides different reports at different stages in the data flow to ensure that the submitted data reflects the services that have been provided.
At the point of submission, providers receive immediate feedback on the quality of their submission through a validation file. This file includes record-level reports of any submission errors, giving the data providers detailed information of which records produced which errors. Providers should then, where possible, use these errors and warnings to rectify these issues before resubmitting their data via a refresh submission up to one month later.
A variety of data quality checks are run as part of the validation and load process for monthly data, prior to production of any monthly release. Where there are concerns about data quality, providers are contacted directly so that any issues with local data extraction processes can be addressed for a future submission. These checks are currently limited to key elements of the dataset. Additional checks may be developed as part of future submissions.
Initial reporting may contain data quality issues due to inaccurate content being submitted. While steps are to be taken to reduce data quality issues in future reporting, some of these issues affect the current outputs. However, as data providers become accustomed to additional data quality requirements and further developed autism guidance over the coming reporting periods, the effect on outputs should lessen.
Whilst local systems may be continuously updated, the MHSDS submission process currently provides two opportunities for data relevant to each month to be submitted. The primary submission window opens for around one month immediately following the end of the reporting month and the refresh submission window is open in the subsequent month. This means that the timeliness of recording all relevant activity on local systems has an impact on the completeness of the MHSDS submission. For example, a referral made in December 2018, but not entered onto the local system until April 2019, will not be included in reporting.
Providers should use the submission summary reports produced at the point of submission to ensure that all relevant data has been included. Statistics published in this report are calculated using data taken after the refresh submission window has closed.
Timeliness and punctuality
Timeliness refers to the time gap between publication and the reference period. Punctuality refers to the gap between planned and actual publication dates.
The analysis in this report is based on data submitted by providers during both the primary and refresh submission windows. Any data which are re-submitted by a provider will be used in place of an earlier submission only within the submission window.
These quarterly autism publications are based on the most recent available final data for Q1-Q3 2018. Q4 data has deliberately not yet been published as it relies on data not yet available at the time of this initial publication from MHSDS v4.0 (due to a move to a new data platform) to be able to derive whether patients have been seen within the recommended 13 weeks.
We have waited to publish three quarters together to allow time for providers to become familiar with the new autism submission requirements. We are now able to start the early evaluation of autism data with users and stakeholders so we can continue to develop these new experimental Statistics so they can eventually become Official Statistics.
Information regarding submission deadlines for MHSDS is published here: https://digital.nhs.uk/data-and-information/data-collections-and-data-sets/data-sets/mental-health-services-data-set/how-do-i-submit-data-to-the-mental-health-services-data-set#what-are-the-submission-dates-
The Autism Statistics quarterly publications will be published on pre-announced dates and is therefore deemed to be punctual.
Accessibility and clarity
Accessibility is the ease with which users are able to access the data, also reflecting the format in which the data are available and the availability of supporting information. Clarity refers to the quality and sufficiency of the metadata, illustrations and accompanying advice.
The format of this publication meets user needs in HTML format with accompanying Excel data tables.
This report is made freely accessible via the NHS Digital website. The data files are also available on the data.gov website: https://data.gov.uk/data/search
Re-use of our data is subject to conditions outlined here: https://digital.nhs.uk/about-nhs-digital/terms-and-conditions
The quarterly data file is presented as a set of Excel data tables. Definitions will be developed, and further guidance provided in future editions of this publication series.
Full details of the way that MHSDS returns are processed, which will be of use to analysts and other users of these data, are provided in the MHSDS User Guidance, available on the NHS Digital website: https://digital.nhs.uk/data-and-information/data-collections-and-data-sets/data-sets/mental-health-services-data-set/mental-health-services-data-set-specifications-and-guidance
Coherence and comparability
Coherence is the degree to which data which have been derived from different sources or methods but refer to the same topic are similar. Comparability is the degree to which data can be compared over time and domain.
Prior to 1 April 2018 (or in MHSDS v2.0), autism was included in the wider ‘code 24’ for Neurodevelopmental Conditions option for primary reason for referral. The changes in v3.0 allow for autism referrals to be separately identified to aid analysis through the introduction of codes 25 and 26 for ‘Suspected autism’ and ‘Diagnosed autism’ respectively.
Prior to publication, NHS Digital did some scoping work to check how best to report on these new measures. We found that data could be presented via two methods:
- The ‘forward model’ approach – which captures all suspected autism referrals (code 25) within a reporting period and then we search later periods of data to check whether those patients have had a first appointment/care contact and in turn, whether they have received an autism diagnosis.
- The ‘reverse model’ approach – which captures all autism diagnoses within a reporting period and then we search earlier periods of data to check when those patients had their first appointment/care contact and when they were referred.
We have initially based these statistics on the forward model approach because of the introduction of ‘suspected autism’ occurring only from April 2018. Subsequent releases will present additional data based on the reverse model approach with a view to moving wholly to this model once submissions are more reliable and complete. One of the limitations of the data is that generally not enough time has elapsed from 1 April 2018 to follow the whole patient pathway from referral date, as some patients will not have been diagnosed yet. However, this methodology captures those patients referred with new code 25 for suspected autism and whether they had attended a first appointment.
Diagnosis and treatment of autism can happen in different settings, such as primary, secondary and tertiary care. There isn’t a single dataset that collects information from all these settings to help with a more comprehensive picture of all autistic patients or autism prevalence.
Patients in MHSDS can have a recorded diagnosis of autism in mental health secondary care services. These statistics do not include people diagnosed with mental health conditions who are only in contact with primary care services, such as their General Practitioner (GP). Therefore, the autism data presented in this release only presents a partial picture of the number of people with an autism diagnoses from those providers that have submitted data.
Our other data collection, called Assuring Transformation, can provide figures for inpatients only who have autism and can be broken down by age and gender. As this dataset only holds information on inpatients, this would not show the whole population and would also provide a partial picture. This publication series is available here:
Our most recent adult survey (from 2014) is the Adult Psychiatric Morbidity Survey (APMS) series, which covers the general population and gives the latest and best estimate of autism prevalence by age and gender. See Chapter 6 in the following link: https://digital.nhs.uk/data-and-information/publications/statistical/adult-psychiatric-morbidity-survey/adult-psychiatric-morbidity-survey-survey-of-mental-health-and-wellbeing-england-2014
The APMS series is England's key data source for the prevalence of treated and untreated mental disorders in the general population. APMS is designed to provide an estimate of the prevalence of mental disorders in the population regardless of whether these have been diagnosed by a clinician. As such these numbers represent people who have been assessed rather than those who have received a diagnosis.
Our most recent child survey (from 2017) is the Mental Health Survey for Children and Young People, which covers the general population and gives the latest and best estimate of autism prevalence. This series is England's key data source for the prevalence of treated and untreated mental disorders. It is designed to provide an estimate of the prevalence of mental disorders in the population regardless of whether these have been diagnosed by a clinician. As such these numbers represent children who have been assessed rather than those who have received a diagnosis.
The link to this survey report can be found here: https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-of-children-and-young-people-in-england/2017/2017
Trade-offs between output quality components
This dimension describes the extent to which different aspects of quality are balanced against each other.
This publication presents new autism waiting times measures using data in the MHSDS (from April 2018) following the publication of new autism submission requirements and guidance earlier in 2019.
Because this is relatively new, some providers will continue to experience issues making a comprehensive submission within the permitted timescales. We expect a more complete and accurate picture to emerge over time. This analysis presents an early view and is subject to caveats both in terms of provider coverage and completeness of the submission and the limits of the data that has been provided around pathways into services to support monitoring of waiting times.
The format of this publication has been determined to enable timely reporting of key initial measures while adjusting the scope of analysis to be achievable within NHS Digital resources and production time. The scope of analysis will be expanded in future editions of the publication series to increase the usefulness and usability of these statistics for different users. By publishing an increasing range of clearly defined measures in a timely fashion we hope to support discussions between providers and commissioners about caseload and activity and promote a virtuous cycle of improving data quality through use.
Assessment of user needs and perceptions
The development of these autism statistics is expected to become a key driver in helping achieve better outcomes for patients. These new quarterly reports provide autism service providers, commissioners and other stakeholders with timely information about activity.
This publication meets initial user needs in terms of reporting on the number of patients referred for suspected autism with associated waiting times, subject to a number of caveats around coverage and completion.
Therefore, reporting is currently shaped by the coverage and completeness of the data, but as data quality improves these reports will change to present more information.
NHS Digital is keen to gain a better understanding of the users of this publication and of their needs. Feedback is very welcome via our firstname.lastname@example.org address (please quote ‘Autism Statistics’ in the subject line). Alternatively, you can call our Contact Centre on 0300 303 5678 or write to: NHS Digital, Learning Disabilities Team, 1 Trevelyan Square, Boar Lane, Leeds, LS1 6AE.
Performance, cost and respondent burden
This dimension describes the effectiveness, efficiency and economy of the statistical output.
As a 'secondary uses' data set, the MHSDS does not require the collection of new data items by autism service providers. It re-uses existing clinical and operational data for purposes other than direct patient care.
Providers are not required to submit data held only on paper records as no provision has been made in the MHSDS for the cost of transcribing these records to an electronic format.
Only two of the data tables are mandated to flow each time any activity is reported within the MHSDS (MHS001; patient demographics and MHS002; GP registration), completion of the remaining tables is only required when activity has occurred that is captured within these tables.
Confidentiality, transparency and security
The procedures and policy used to ensure sound confidentiality, security and transparent practices.
In order to prevent disclosure of identities or information about service users, all figures (except national) for all organisations which submitted, are rounded to the nearest five. All figures between 0 and 4 and all percentages calculated from values between 0 and 4 are suppressed (*).
All publications are subject to a standard NHS Digital risk assessment prior to issue. Disclosure control is implemented where deemed necessary.
Please see links below to relevant NHS Digital policies:
Freedom of Information process:
A Guide to Confidentiality in Health and Social Care:
Privacy and Data Protection: