Data analysis and reporting

The commentary in this report refers to key estimates on the health of adults aged 16 and over.

The Health Survey for England Feasibility Study (HSE FS) collected information from a sample of the population living in private households in England. The sample was designed to represent the whole population as accurately as possible within practical constraints, such as time and cost. Consequently, statistics based on the survey are estimates, rather than precise figures, and are subject to a margin of error, shown as a 95% confidence interval. For example, the survey estimate might be 24% with a 95% confidence interval of 22% to 26%. A different sample might have given a different estimate, but we expect that the true value of the statistic in the population would be within the range given by the 95% confidence interval in 95 cases out of 100.

Where differences are commented on in this report, these reflect a particular minimum degree of certainty that these differences are real, and not just within the margins of sampling error. These differences can be described as statistically significant.

The Health Survey for England (HSE) is a cross-sectional survey of the population. It examines associations between health status individual/household characteristics and behaviour. However, such associations do not necessarily imply causality. In particular, associations between current health status and current behaviour need careful interpretation, as current health status usually reflects past, rather than present, behaviour (for instance, current liver disease may reflect previous heavy drinking, although no alcohol is currently consumed). Similarly, current behaviour may be influenced by advice or treatment for particular health conditions. For instance, not smoking currently because of advice relating to existing lung disease caused by previous smoking that is exacerbated by continuing smoking or not drinking alcohol because of the risks of hypoglycaemia (too low blood sugar) in someone with diabetes.

Estimates presented in the text are rounded to the nearest whole number. Where categories are combined, the sum of two estimates may sometimes appear to be greater or less than the expected value. This reflects the effect of rounding; for example, estimates of 10.6% and 12.7% would round respectively to 11% and 13%, but the sum (23.3%) will round to 23% rather than 24%. The charts are based on unrounded estimates. As a result, values given in the text may appear different in the corresponding chart. For example, an estimate of 10% in the text may represent a value between 9.5% and 10.4%, and it is the unrounded value that would be reflected in the chart data points.

Significance testing was carried out on the results in this report. The term ‘significant’ refers to statistical significance at the 95% level and is not intended to imply substantive importance. The significance tests were carried out to test the relationship between variables in a cross tabulation, usually an outcome variable nested within sex, cross-tabulated with an explanatory variable such as age (in categories). Where comparisons are made between key estimates from the HSE FS and HSE 2018 or 2019, these have been tested in the same way, using data from both sources. 

Comparisons for key estimates are made between the HSE FS and HSE 2019, the latest fieldwork year for which HSE data are available. As HSE 2019 employed face-to-face, in-home data collection methods and the HSE FS employed self-completions, these comparisons should not be used to measure any true change in these estimates over time. It is not possible to determine whether differences in survey estimates are a result of differences in the profile of the achieved sample or due to mode effects, or even due to changes in behaviours because of the COVID-19 pandemic, as these factors are confounded. The following points should be considered when interpreting these comparisons.

Survey estimates could be influenced by sample biases in the demographic profile of those who responded to the HSE FS. For example, the difference between the levels of response by area deprivation was greater than in the HSE 2019 face-to-face survey, with fewer people in deprived areas taking part than is usually the case. Although weighting adjusts for this, the low response rate achieved on the HSE FS (individual response rate of 17%) and the lower number of complete questionnaires achieved amongst the very deprived could explain the differences. Hence, for some survey estimates the results may not reflect the true prevalence.

Differences between survey estimates could arise from mode effects. Mode effects are differences in responses that happen because people respond differently to interviewer-administered questions (face-to-face) than to self-complete questions (online and paper). The physical presence of an interviewer can improve accuracy in survey measurement, by reducing question difficulty and increasing participant motivation to answer questions through probing, clarifying questions and instructions.

Conversely, although the physical presence of an interviewer can improve accuracy, social desirability reporting can have an impact on the data collected leading to systematic biases. For example, the presence of an interviewer can lead to participants giving answers that are perceived as more acceptable than in self-completion modes.

In self-completion modes the risk of participants providing the minimum information required or expected is greater for questions perceived to be difficult to answer such as questions involving calculations. ‘Short-cutting’ strategies can be adopted, such as rounding, which could affect the accuracy of answers and the accuracy of the survey measure.

The complexity of questions instructions and routing may impact on data quality and the usability of the survey. Routing in paper questionnaires can be difficult for participants to follow and can lead to item non-response, or even to question order effects as they will see questions which they are not required to answer. Hence, participant error is a particular issue for those completing the questionnaire on paper. Face-to-face and online surveys minimise this risk with built in automated checks and routing minimising participant error.

Participants’ willingness to take part in a health survey is also likely to be influenced by their own health-related factors, biasing the sample, irrespective of mode.

Lastly, the short data collection period means that it was not possible to monitor changes in survey estimates that can occur as a result of seasonality. It was also difficult to assess whether changes occurred as a result of the restrictions being imposed or lifted during the COVID-19 pandemic. 

Ethical approval for the HSE FS was obtained from the NatCen Social Research Ethics Committee.