Remote User Testing for Communities

by guest author, Cade Diehm, founder of New Design Congress

All design practice requires user testing. Regardless of the field or medium of design, user testing should be a core part of the design process. User testing ensures that your proposed solution can be understood and operated by people. It is critical in all stages of design, from prototype to release candidate. This chapter covers practices that can be used in-person or remotely for user testing, while understanding the opportunities and barriers of remote testing for your product or service.

In-person user testing is often done in controlled environments, where a design team selects sections of their current work to test or tests the entire experience. The design team would establish an on-site environment with team-controlled devices and observational equipment. The team then recruits testers, scheduling one-on-one or group appointments between test facilitators and a cohort of users. User testing is qualitative, in that it focuses on individual users and their experiences with what you are testing. This is true whether the testing is conducted as one-on-one sessions, or as participatory testing in larger groups or communities.

Remote user testing uses a similar process to its in-person equivalent, except conducted online. User feedback during on-site user testing is usually conducted verbally and observations of the user testing session is conducted and documented visually. In remote user testing, the entirety of the test is facilitated via digital platforms and technologies. Remote user testing differs significantly from its “local” counterpart, despite having the same objectives and, in many cases, producing similar data.

Note: User testing — whether local or remote — is not the same as bug reporting or beta testing. The purpose of both on-site and remote testing is not to identify and document software bugs, design flaws or hardware issues. Finding flaws in a product is handled by Quality Assurance, which focuses on discovering unexpected behaviors or issues in designs that are finalized and occur closer to launching the product or service.

User testing is the process of testing the ideas produced by a design team. Although the motivation of user testing remains fairly common across projects (to see if the design meets the expectation and comprehension of users), the goals of user testing can vary. Goals for user testing might be to:

Good user testing — especially testing as part of a participatory design, human-rights centered design, or an anti-weaponised design process — accomplishes its goals through collaboration between the design team and test participants. The design team may have specific goals in mind, but the main goal of the team is to present the testing and user walk-through of the design in an unstructured way, balancing direct instruction with exploration and input from the participants. User testing is extremely effective when the test design and facilitation process incorporates the participant and community in such a way it embraces the expertise of participants and their social environments.

Remote user testing is the practice of facilitating testing with users and communities where the facilitators and participants are not in the same physical space. Instead, the design team uses digital platforms and tools — video calling, remote access, digital whiteboards, live prototypes, etc — to facilitate the testing and research.

In situations where you are unable to physically access communities, such as the Pandemic, remote testing is often the only solution. Testing with users in vulnerable regions, where physically meeting with participants may put them at risk, is a similar situation where remote user testing may be required. Much like all ethnographic and/or social research, the safety of participants is paramount. Remote user testing, combined with strong ethical controls and security practices, makes the safe facilitation of user research more likely.

Remote testing is also appropriate in situations where the economic conditions of either the design team or the user base make it impractical to perform in-person testing. This may be due to the design team dispersed globally, where the logistics of travel make in-person testing impractical. Remote testing is also useful for under-resourced teams, such as projects staffed by volunteer designers.

Building multicultural user experience and testing against different audiences is another good use case for remote testing. Testing internationally, in which the design goal of localizing the design or cultivating multicultural user experience, can be difficult without the privilege of a large supporting budget. Remote testing may not solve the challenges of localization or communication between facilitators and participants, but offers low-barrier access to communities across the globe.

Remote testing can also be institutionalized as part of an organization or project’s participation with the community, offering regular structured feedback sessions with communities or participants who have a stake in the design team’s work. In many cases, this is incorporated into an “early adopters” program as part of a community engagement and outreach plan.

Remote user testing is often selected due to the circumstances of the design team. Despite this, the approach offers a wide variety of benefits that are common to almost all remote testing projects, regardless of motivation:

Although remote user testing has significant benefits, the process also has some overall potential drawbacks. The challenges of test logistics, the legality or ethics of using participant devices, testing complexity, and other issues can compromise remote user testing. Many (not all) of these can be mitigated via careful planning by the design team, as well as regular review of test structures and dedicated feedback rounds with participants.

The logistics of testing with an individual participant is often shared between that participant and the test facilitators. For participants with less technical expertise, this can be a significant or confusing burden. Working with a participant to set up a test environment, install test software, or gain access to the designs can result in communication, frustration, and stress issues for both the participant and facilitator. This is the antithesis of participatory design. Rather than meeting participants and communities where they are, remote user testing logistics often create experience silos where the individual participant is on their own with the facilitators and shouldering additional responsibility.

Additionally, the reliance on the user’s own devices to complete a test session can be variable. Unless a design team is particularly well resourced, it is often infeasible to distribute test devices remotely to participants or communities. As a result, remote user testing occurs mainly with devices the participants have on hand. This can have significant implications. Not only are there ethical considerations in the economic and/or power dynamic of a test participant and their devices accessing the design, but participants potentially open themselves up to intellectual property or copyright liability by having a copy of the test on their devices. The ethics of deploying test software on user devices is problematic without careful consideration or mitigation. There are issues of code stability, digital safety, and other concerns for the participant. Often, participants are unfamiliar with the design and its broader social dynamics and goals, and as such are unaware of the risks they are consenting to. Depending on the sensitivity of the project, these test designs can be used to target the participant simply by participating online in a way that can be observed by a third party.

Testing on participant hardware often allows facilitators access to niche hardware that is used by the individuals for great accessibility online. However, the lack of familiarity of this hardware from the facilitating design team carries substantial risk. These tests can encounter unexpected obstacles, where tools for observing participants interfere with their accessibility hardware, or render either the test or the participant environment completely inaccessible.

Depending on the project, remote user testing can benefit by being closer to a real world test case. But the nature of today’s personal and mobile computing prioritizes a “1 user, 1 device” paradigm. This means that, in many cases, the remote test itself is severely limited, as the test is filtered through platforms that adopt the 1:1 user/device design model. For participatory design, this is particularly tricky, as performing remote user testing simultaneously with inter-community participants is often overwhelming and difficult without careful test design.

While drawbacks are inevitable in remote testing, having good feedback loops between participants and facilitators where they are able to properly evaluate the tools and platforms that may be used as part of the test prior to testing can help minimize setbacks.

User research is conducted on people and communities and is subject to research ethics. For in-person testing, many of the human research ethics questions have been extensively researched in the adjacent areas of social science and philosophy. For remote user testing, research ethics are more complicated and existing recommendations are less useful.

Beyond the expectations set out in existing ethical frameworks for researching with humans, the ethical implications of technologies, networks and data remain vague and under-explored. For example, most testing tools are built by Silicon Valley companies that have economic relationships with data brokers, who will sell the data associated with your testing to their customers. In testing plans that include multiple platforms, this multiplies the number of privacy violations exponentially. A design team may think that they have nothing to hide, and not worry about the issues related to platform surveillance. Other teams may choose to only use platforms that allow them greater control. In all cases, the design team is making a judgment call on the safety of their participants. Their decisions may have profound and unforeseen consequences for their participants, especially as societies change, companies change leadership or even a participant’s individual circumstances change over time. A good general rule is to collect as little data as needed, and nothing more. This can be applied to both the research data generated and archived by the facilitators, but also the data generated for the platform operator by the testing session.

Remote user testing can also require test participants to install intrusive software on their computers. These software tools include remote control software (like TeamViewer or similar applications), proxy software (to enable a participant to access confidential parts of a test), and other surveillance tools. Almost all of these tools have the potential for abuse, and their use can help normalize the opportunities for abuse. For example, TeamViewer is a remote screen sharing and desktop control tool that can streamline testing sessions, but also an iconic example of weaponised design. Its aesthetics, feature set and interface makes it a staple software application for technical support phone scams, where fraudsters deploy TeamViewer on a victim’s computer as part of a sophisticated attack to control their computer and empty their bank accounts. Design teams should think carefully about the software they use and how they communicate the harm to participants, as well as instructions for proper removal after the testing.

Remote user testing also has a significant power imbalance between participant and facilitator. The inability for either side to see each other fully, combined with the observation of the participant by facilitators creates a hierarchy of visibility that can have serious ethical consequences. For example, participants are in many cases unable to see what the facilitators are doing, and must trust the facilitators to communicate when they start or stop the session recording. With in person testing, there is often a stronger implied social contract and the physical presence of facilitators and testers helps to lower this power imbalance. Other issues also contribute to this power dynamic, such as data governance and privacy of the test data produced during a session.

Participants are also sometimes asked to run the code of the design being tested, either on their computer or within their web browser. This means, in many cases, the design you are testing will leave the custody of the design team and be copied onto the participant’s device. This creates an additional concern for the participant, who in many cases has to commit to removing the test ephemera (and know how to) once the test is concluded. Teams should think carefully about asking participants to commit to this, and explore the potential risks associated with making this a component of their remote user test.

During in-person testing, the test facilitators are able to deploy and completely control the testing environment. Device administration, observation equipment, and other tools are all expected to be supplied by the facilitators, or in cases where participants bring their own devices, the on-site support from facilitators during the test is invaluable. The opposite is true for remote testing, and this creates perhaps one of the most common and significant challenges encountered during remote testing. Setting up the test environment is a collaborative process between the participants and the facilitators. This takes time and can be fatiguing to the participant. In the worst case scenario, a complex setup process may be beyond the abilities of the participant, and prevent the test from happening. When planning remote user testing, facilitators should minimize the setup obligations they place on the participant, and provide ‘off-ramps’, or alternative means to complete the testing session in cases where the participant encounters obstacles at the beginning of the test.

Hardware, software, and connectivity requirements have an under-appreciated but significant impact on the viability and accessibility of remote user testing. As computing power has increased over time, and the reach of wireless connectivity expands, software platforms too have expanded to fill these newly available resources. Remote user testing can require a specific set of applications running simultaneously. When combined with intensive tasks such as screen broadcasting or group calling, or the need for constant connectivity from participants’ connections, can create technical limitations for participants. It is naive to assume that this problem is solely driven by economic factors — many popular Apple laptops from the last five years are incapable of running many resource intensive tasks at the same time without encountering significant performance issues. These technical accessibility concerns have a direct impact on a remote testing session. Teams can prevent this by refraining from running multiple resource-intensive platforms simultaneously, or offering flexible testing pathways for participants.

Remote testing platforms or the act of using a platform familiar to a participant in an unfamiliar way can also affect the accessibility of the test. The dominant paradigm of user experience design, Don’t Make Me Think, has created a collection of interaction paradigms that emphasize simple, inflexible means of operating software. When these are combined in unusual ways, or when moving between two incompatible platforms during the testing process, the changes in user expectation can be jarring. This can have a significant cognitive cost for the participant, and is something that can be supported with ease during in-person testing. Teams can prevent this accessibility problem by exercising restraint in their choice of platforms, and by setting exploratory expectations with participants at the beginning of the test session.

In cases where a design team is testing with an inter-sectional sample of participants with diverse accessibility requirements, the testing environment can conflict with accessibility software and hardware used by the participant. For example, a test environment may not follow accessibility legislation, and as a result the participant’s screen reader cannot render the platform’s interface in a comprehensible way. Or, issues with the testing platform could act as an interference layer, preventing the participant from interacting with and evaluating the design, causing the test to fail. Mitigating conflicts between testing platforms, design and accessibility tooling requires careful planning and preparatory research from the design team. In cases where the team has advanced awareness of test recruits with accessibility peripherals, scheduling short platform tests or contacting the developers of a proposed test platform is a starting point for preventing unrecoverable accessibility obstacles during testing.

Testing A Tool For Recording Interviews

In a recent project, my design team sought to test an experimental user experience tool for navigating and exploring recorded interviews that was designed during the pandemic. The design team wished to test the tool’s design, differences experienced across cultures, and the relationship between spoken word recordings and text.

The team structured the remote user test as a shared participatory workshop activity. Test participants were selected via a call for participation online, where both the workshop activities and the goals of user testing were explicitly detailed. Participants were selected from the broader international network available to the design team. The team identified a single design they wished to test, and structured the entire remote-testing process around this design. Participants signed up for one of two workshop choices, structured to cover different time zones.

The remote user testing sessions were held over a group Zoom call. After an introductory icebreaker, the two test facilitators detailed the consent and ethical commitments from the team, and other introductory processes. The design team’s two facilitators described the design scenario, and the participants collectively operated the tool, giving verbal feedback and collective reflections in real time. The test facilitators recorded this feedback. Each test session / workshop took 1.5 hours to complete.

At the conclusion, participants were invited to reflect collectively on the test process. A few days after the test sessions, the design team contacted participants individually for written feedback. Facilitator notes were combined with summaries of the recordings of the test sessions, and presented back to the design team.

User testing is a core design practice and is used to uncover everything from unrealized socio-technical security threats to cognition difficulties in a designed system. Remote user testing, the practice of performing user tests over a network, allows designers to reach a wider range of people and communities to collaborate with, but not without significant challenges inherited from limitations of the internet and personal computing. Careful planning and sensitivity to issues of accessibility, colonialism, institutional power and safety can create the conditions for successful remote testing that can compare measurably to a design’s consequence in the world beyond the bounds of its creators.

Further Reading The Limits to Digital Consent