In recent years, as user experience research has become more critical or even standard practice, there is a clear trend towards measuring user experience and usability using quantitative research techniques. Brands with a strong online presence want to conduct user experience benchmarking projects collecting feedback from hundreds of users who participate in their natural context, from geographically dispersed locations. The results help these brands complement the already well-known and widely adopted qualitative research (such as usability testing in the Lab). Moreover, today’s Web is a changing medium and, as such, understanding how users use it requires a mix of innovative technologies and research methods.
Fortune Global 2000 and Internet 200 companies such as Google, Yahoo, eBay, Monster, Verizon, Sprint, IBM, Travelocity, Continental Airlines, Orange or Telefónica among many others all used or are currently using Unmoderated Remore Usability Testing (URUT) as part of their user experience and usability research toolkit. URUT can be both simple and quite sophisticated. With this article our mission is to explain it in a way that will hopefully help people better understand it.
Why, when and how would you use URUT?
1. To quantify your usability research: Consider you have a huge customer base. This customer base includes different personalities, different usage patterns, and different perspectives. Quantifying your usability is the only way you can ensure that you are reaching a true representation of your diverse population. Not only can you gain valuable data that solidifies your true population but also you can validate your lab findings or alternatively target which critical tasks you need to be probing in a lab-based study.
2. To conduct benchmark studies: URUT allows researchers to obtain statistically significant usability metrics on how a website performs vs. other versions of the site or vs. competing sites. Therefore, it’s a great way to take usability testing a step further to actually measure user experience and compare results either across time or through ‘industry benchmarking’.
3. To test users in their natural context: My computer and environment is different from my friend’s computer and environment and most likely different than a good portion of the population. Testing participants in their natural context accounts for different systems, configurations, and setups. The data you gain not only accounts for a mix of these various environments and setups but also encourages participants to act as they normally would, as they are not being “observed.”
4. To understand users’ behavior: You want to understand why users are coming to your site and what they do once they come there. URUT uses a combination of web analytics (where users go) and surveys (the why) to create a complete picture and provide valuable data in providing the best user experience for your site.
5. To validate or define your lab-based research: You want to ensure that the research you are currently conducting is valid and a true representation. With URUT not only do you gain valuable data that quantitatively solidifies your current research but also alternatively you can use URUT to target key critical issues and tasks to bring in the lab for further probing.
6. To test internationally without traveling: International research is very expensive and at times put aside due to the cost and time commitment. URUT allows you the flexibility to conduct a study in many internationally locations without taking a step out of your home. Not only does it removes the expense of travel but also removes the need for all your data collected to be translated in order to analyze it. URUT removes the barriers that have traditionally impeded this very critical research.
Other FAQs and discussions on URUT
1. Should I conduct URUT always combined with moderated testing?
URUT is a very valuable method that, if used correctly, can definitely be used as a stand-alone method. However, this does not devalue other methods such as qualitative lab, focus groups, contextual inquires which all can complement URUT nicely and each have their place and purpose in the UX ecosystem. We firmly believe in the concept that you must use the right method and tool for the right goal or data needed. In user experience, very often a combination of methods + tools yields the best results.
2. How do you resolve the issue of recruiting the right sample of participants and guarantee the validity of their behavior and feedback?
As with all recruiting methods, the key is in a top-notch screener. Without that, no one can guarantee quality. In addition having sample quality controls can guarantee the validity of your results and sample. There is no difference from recruiting with any research method you use.
3. How often should I use an URUT method for research?
As with any research method, URUT should be an integrated part of your usability roadmap. It also depends on the research needs. URUT can easily be used for international research, iterative testing, prototype testing, pre-code freeze research, ‘quick and dirty’ usability research, and much more. That said, URUT should be used regularly to make the most of your research and ultimately your ROI.
4. How easy is an URUT tool to use?
Actually quite easy. As with any new software tool, you need to spend time with it to get familiar with it and all its functionalities. With our tool many of our clients are up and running in a matter of hours with a study without assistance. URUT are made for researchers, and in our case, by researchers so the tool is made with ease of use and with a great user experience in mind.
5. Can URUT track web 2.0 sites built on Flash or Ajax?
Yes! Most sites today such as Flash and Ajax sites are built trackable. What that means is the elements are tagged so that they are tracked by web analytics tools which would include URUT.
As the web becomes a more complex, mature place and users use it and interact in different ways, user experience and usability testing and measurement must evolve and continuously innovate. URUT is an example of this innovation and has proven its worth for the past 6 or 7 years, since various tools came out to the market. The key to solid research lies not only in proper execution and the right technology, but also in the ability of the research team to understand that different data comes from different methods and tools, and that each should be used with a purpose and to meet specific goals (what, why, when and how). The combination of methods and tools is often the best way to go. URUT is a great choice for specific purposes and, if well executed, can become an invaluable source of data about user experience.
Alfonso de la Nuez & Kim Oslob
Founding Partner & Director of Research and Product Strategy
Userzoom