Summary and recommendations

The section presents a brief summary of the feasibility of collecting the complex health data required for Health Survey for England (HSE) via remote data collection methods.

Surveys using remote data collection methods are relatively cost effective and can be conducted within a short amount of time. However, they are compromised by the far lower response rates achieved compared with the HSE face-to-face survey (the household response rate achieved on the  Health Survey for England Feasibility Study (HSE FS) was 25% compared with 60% on HSE 2019). This is also true for individual consents to a nurse visit (52% in the HSE FS compared with 85% in HSE 2019), data linkage and follow-up research.   

As well as the lower response rates achieved on the HSE FS, there were differences in the key estimates compared with HSE 2019. There are a number of possible explanations for the differences: mode effects, differences in the profile of the achieved samples and also, genuine changes. Whilst differences cannot be fully disentangled, it is possible to make some informed observations.  

Responding adults in the HSE FS were more likely to be older, from white backgrounds, living in less deprived areas of the country and less likely to be renting their home compared with those in the face-to-face survey. Whilst data were weighted to control for a number of demographic characteristics, it is likely that the achieved HSE FS sample underrepresented some subgroups for which weighting the data did not fully adjust. These sample biases, as well as a lower overall response, reduce the efficiency of the weights.

Specifically for this survey, results across a number of questions suggest that the HSE FS may have missed less healthy individuals and individuals who had less healthy lifestyles: the proportion of adults reporting a longstanding illness, being a current smoker or not meeting the 5 A DAY recommendation were all lower on the HSE FS compared with HSE 2019 or 2018. It is possible that some individuals, including the less healthy, may only take part in a health survey because an interviewer encourages them to do so (for example, by emphasising the importance of different types of individuals taking part) and also, facilitates their participation (by administering the questionnaire and other survey elements).

The HSE face-to-face survey is complex, collecting health data using lengthy questions and instructions which require cognitive effort and engagement. The length, wording and complexity of the questions have been developed with interviewer-led administration in mind, allowing the use of prompting at more complex questions. As such, many of the questions are not suited to the format needed for successful remote data collection where good practice suggests that question wording should be kept simple and avoid complex routing. Whilst routing and automated checks could be incorporated into the online survey easing some of this complexity, it was more difficult to do this in the paper mode.  

Despite efforts to reduce measurement error in the HSE FS through ensuring questions, response options and format were identical to the face-to-face survey, it was not possible across every question or mode. Some questions, such as those collecting the amount of fruit and vegetables consumed, had to be specifically adapted for remote data collection methods. Others, such as those collecting data on the types and quantities of alcoholic drinks consumed, were asked in a different format. Such differences may have impacted on the results reported (e.g. increasing the number of fruit and vegetables reported).

The visual design of the paper questionnaire, including font and layout, is crucial, as participants are completing it with no guidance from an interviewer. The complexities of the questions and the length of the questionnaire could have resulted in some individuals being put off starting or completing the survey.

A paper questionnaire was included in the HSE FS design to maximise response and sample representativeness amongst groups of the population who would otherwise be underrepresented in the survey because they are not able to respond online or are less comfortable doing so. In the HSE FS, a higher proportion of older adults, those living in the most deprived areas and those from an ethnic minority completed the survey on paper rather than online.  

Sending a paper questionnaire may also have encouraged some people who did not have barriers to completing online to fill out the questionnaire, as they may have viewed completing paper questionnaire as more convenient. It is also possible, that some people did not open or notice the first two communications, instead opening the bulkier second reminder letter as it is more likely to stand out from the regular mail.

Whilst an alternative data collection mode was necessary (and was taken up by 30% of adults at stage one), the inclusion of a paper questionnaire created complexities such as having to edit the data to deal with mistakes and inconsistencies. There was also a need to remove duplicate responses and to ensure consistency between participants across household-level variables.      

The HSE FS adopted a two-stage design to ensure the random selection of children for stage two if a household contained more than two children in the eligible age range. However, agreement to be re-contacted for the stage two child survey was low among eligible adults. This resulted in a smaller than expected achieved sample of children. It may be worth considering incorporating the stage two survey as part of stage one, by building in child selections or eliminating them altogether. This would have its limitations in terms of gaining accurate information about the household composition, but it would eliminate the need to produce a complex sample file requiring editing to resolve any inconsistencies.        

• The HSE FS achieved a less representative sample of participants than the face-to-face surveys which weighting may not completely compensate for. Further consideration about how to engage and secure the participation of groups that are underrepresented in remote data collection modes is needed.
• A change to remote data collection modes would necessitate a review and modification of questionnaire content to ensure that questions can be fully understood without the presence of an interviewer to explain and probe. Examples of the more complex questions include those asking about household composition, limiting long standing illness and fresh fruit consumption. Other modules such as those on smoking and drinking, worked well online but did not work so well on paper due to the complex routing.
• It is also worth considering whether stage one and two surveys could be combined with all children included in the survey (thus eliminating the need for child selection between the two stages). Parents completing the survey online could be directed to complete child questionnaires (or provided with a link for older children to complete on their own) after they have completed their own questionnaire. Child paper self-completion questionnaires could be sent with adult paper self-completions. This should lead to a larger number of children taking part and to reducing the time needed to carry out the survey.

Further discussion with stakeholders would be needed to test whether modifications are acceptable, because the change in mode and the level of detail that could be collected would result in a notable break in time series trend data.  

Thank you to all the participants that generously gave their time to participate in the study.

Thank you to everyone at NatCen involved with developing the feasibility study and report, in particular to Beverly Bates, Dhriti Mandalia, Katie Ridout, Mari Toomse-Smith, Elizabeth Fuller, Laura Brown and Chloe Robinson. At NHS Digital thank you to Vicky Cooper, Sabeehah Rafiq, Alison Neave, Asif Mehmood, Nicholas Fox, Michael Akintunde and Cher Cartwright.