In recent months I’ve been concentrating my efforts toward researching registered blind and low vision users particularly focused on their use of NHS login, an authentication service people can use to access various health and care services with a single reusable login credential. Making this service accessible and inclusive for all has been a pillar of the programme from the start.

In the past, our research activities with registered blind and low-vision users have been in-person, usually within usability lab facilities or visits to community outreach centres and charities such as Kirklees Visual Impairment Network or RNIB.

There is a lot of contextual richness that comes with being in the same environment as a research participant with access needs – you not only get a more intimate understanding of the assistive technologies they might be using, but also the observation of idiosyncrasies and workarounds users may have with a service. For many researchers, it is also easier to build rapport with a participant when able to have a face-to-face conversation.

The practicalities and consequences of conducting evaluations remotely have been explored in various literature, but rarely with people with disabilities. Due to the lack of depth in any literature, and because at NHS Digital we’ve primarily done user research with people that have access needs in person, I was a bit hesitant in going fully remote.

As we entered lockdown at the beginning of the pandemic, however, we had no alternative. People were relying on digital services to access health and care during the pandemic and it was more important than ever to make sure blind and low-vision users were not left behind.

In early summer, I ran a large cycle of moderated and unmoderated research sessions remotely using Microsoft Teams with registered blind and low vision users. 22 people took part in the study, 10 in the unmoderated condition and 12 in the moderated condition. 

I’ve also been working with Professor Helen Petrie at The University of York. We wrote a paper that was accepted by the Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion Conference, outlining the research methodologies used at NHS Digital for this particular cycle of research and some of the strengths and weaknesses that come with remote evaluations within the space of online identity authentication with this demographic of users.

User recruitment

A major advantage of remote evaluations is the recruitment process. Remote evaluations not only allow recruitment across a broader geographic area which can be helpful in recruiting participants with particular characteristics such as disabilities, but also allows the inclusion of participants who may find it difficult to travel to an evaluation site due to their disability.

We found that recruiting blind and low vision users remotely was faster and more efficient, as opposed to recruitment timelines in the past with similar demographics for in-person research. Several participants dropped out of the study for different reasons, but it was fairly quick and easy to recruit replacement participants.

User control and freedom

An interesting insight coming out of this research was that a remote approach gave research participants more control over things that they’d rather not have observed by a moderator or formally recorded. Being that NHS login sometimes requires users to authenticate their email, enter a password and upload an identity document, we found some users preferred to not have this information recorded or observed. Some research participants therefore chose to turn off the screen-sharing feature during certain sections when sensitive and confidential information was being inputted.

Getting the team involved

Another advantage of taking a remote approach was that other members of the development team at NHS Digital were able to observe sessions easily. Of course, permission was obtained from the participants for this first.

It proved much easier to get development team members involved, who benefit greatly from watching participants interact with the service. Oftentimes, getting team members to observe sessions can be difficult because they might be busy with other priorities or the commute time to a usability lab may be prohibitive. With a remote approach, this is much easier, as team members just need to access the session link and they are able to fit observable research hours in with their other work in an unobtrusive manner.

Streamlined analysis

As mentioned, these sessions were carried out on Microsoft Teams. A useful feature with Teams is the integration of video files being stored directly to the cloud, as well as the built-in speech-to-text feature. I found this feature very useful for analysis, in that I could jump directly to different areas within a research session and edit findings to share with the wider team quickly and effectively.

Consent

Gaining informed consent from participants turned out to be a little more problematic than expected for blind and visually impaired users. Participants were sent an information sheet and informed consent form before their session and asked to sign it and return it, following NHS ethical requirements. Participants often found this requirement difficult and tedious and had not done this prior to the start of the session, so an alternative was developed. In the moderated condition, the researcher read the points in the information sheet and consent form, and then the participant gave a verbal agreement. Participants said this was a much easier and clearer process for them.

Tech as a barrier

A further set of technical issues for both conditions, but particularly important for the moderated condition, was the quality of the audio, as well as the quality of the internet connection.

Participants’ devices, particularly microphones, were of varying quality, and this could make communication difficult. This was particularly the case when participants were using a screen reader. At times I was trying to understand both what the participant was saying and the screen reader speech. It was sometimes necessary to ask the participant to pause the screen reader while they articulated their thoughts, so I could hear what was being said. This is probably more critical in a remote situation than in a face-to-face situation when the sound transmission is optimal.

In the moderated sessions, there were sometimes lags or breaks in the internet connection, and the participants were not always aware of these. When this happened, I had to ask the participant to repeat actions or information which was not ideal because you want the participant to forget about the remote situation as much as possible and concentrate on the evaluation, tasks and questions. Whenever possible, it is better to avoid drawing attention to the technicalities of the situation.

Run a tech check first

The last thing you want is to have a participant inform you they are not able to use the research platform or access the app right at the moment the study is about to happen (I learned this the hard way).

I found it useful within the recruitment process to have it explicitly outlined to users that a short check-in process needed to happen in the days leading up to the study. This short 5 to 10 minute informal check-in was invaluable to see if Microsoft Teams, the screen-sharing options and their device were compatible, updated, and working properly.

We found that during this check-in process, a few users needed to update their device software or had to enable hard-to-find device permissions to gain access. By identifying and mitigating factors that would become a barrier for users in the actual research sessions, it made for more efficient remote research that didn’t waste time on getting the tech right.

Have a dry run with your team

Having a testing process where the effects of a possible failure are mitigated is good research hygiene.

For this particular research endeavour with blind and low-vision users, a mix-methods approach was used, using both moderated and unmoderated methods as well as pre- and post-session surveys. Putting the moderated discussion guide, unmoderated task instructions and survey questions under scrutiny within the team I was working with prior to sessions with users ensured that there was no confusion or unintended mistakes as a result of poorly constructed instructions or questions. User research is a team sport.

Do the hard work to make it easy for users

Users taking part in remote research might be a bit nervous or intimidated by this approach. If someone is taking part in remote research for the first time and is unfamiliar with the process, or is not confident in their ability, it could be anxiety-provoking.

To ease any anxieties or confusion, I sent detailed instructions on how to set up and use Microsoft Teams and let users know they could contact me prior to the study via email or work phone if they had any questions about the methodology or technology.

Have a look at some recommended reading

A very useful book that I’ve found myself frequently referencing since the outbreak of COVID-19 is Remote Research by Nate Bolt. Although this book was published back in 2010, most of the advice and insights hold true more than a decade later when it comes to running successful remote research.

This cycle of research with blind and low-vision users was just a portion of the work I’ve been doing at NHS Digital and blending with an on-going MSc at the University of York. As a full-time user researcher at NHS Digital and part-time Human-Computer Interaction MSc candidate at the University of York, I’ve found the collaboration happening between NHS login and the university to be very impactful. For example, the computer science department at the University of York carried out a collaborative heuristic evaluation of NHS login, which gave our team valuable feedback to further iterate and improve the service for users.

We work in an agile manner at NHS Digital and are always making rapid improvements to the service based on user research. The addition of academic perspective has been not only useful but welcomed in our fast-moving work environment. My supervisor, Professor Helen Petrie, is a world-leading expert within the domain of inclusive design of technologies for disabled people. Getting her feedback on how to make the service more accessible has been incredibly helpful.

I’ve found that industry and academia teach you different yet complementary skills. By alternating and applying different skills in different contexts, you can use insights from one to apply to the other in a way that compounds.