Study Finds Troubling Trends in Virtual Reality Research

Virtual reality (VR) technology and applications have exploded over the past several years, with estimates of VR growing to a $160 billion industry by 2023.

Academic research on virtual reality is also common, but what insights can be gleaned from this large body of work? A new study by a research team (including Frank Waddell, Journalism assistant professor at the University of Florida College of Journalism and Communications) finds troubling trends in published research on virtual reality.

First, the authors suggest that many published studies on VR may be too small given the topics that they study. VR research often investigates attitudes and behaviors that would only be expected to change a relatively small amount in a short research study. To properly investigate these types of questions, much larger studies with more people are needed. With the current size of studies in past work, only about 17 percent of studies would be capable of finding these types of “small effects” with the typical sample size used in past work.

Second, the authors also found that many past studies on VR have either improbable or even incorrect statistical evidence. For example, 15 percent of past studies included mismatched statistical values, in some cases meaning the error could lead to incorrect conclusions about the study. Other problems include a high number of values just below the tipping point for claiming a study had made a discovery. This over-representation of certain statistical values could suggest that researchers and journals are “stacking the deck” towards publishing research that finds VR is a valuable tool, which may not actually be the case.

Finally, the researchers also show that most studies on VR do not share the materials and VR applications that were used to conduct their research. Specifically, only two out of sixty-one published studies reviewed included the materials that were used or the data set that was analyzed. Without these materials, it becomes more difficult for the scientific community to study similar questions or double check the work done by others.

Fortunately, there are clear paths to rectify the situation such as increasing the size of VR studies, encouraging scholars to document their predictions before research begins, and including the supporting materials and data that were used in the study for the general public.

The original research paper, “Virtual Reality Check: Statistical Power, Reported Results, and the Validity of Research on the Psychology of Virtual Reality and Immersive Environments,” appeared in Computers in Human Behavior, Volume 100, November 2019.

 Authors: Madison Lanier, Northrup Grumman Corp.; Frank Waddell, University of Florida; Malte Elson, Ruhr University; Daniel Tamul, Virginia Polytechnic Institute and State University; James D. Ivory, Virginia Tech; Andrew Przybylski; Oxford University.

This summary was written by Dana Hackley, Ph.D.

Posted: July 31, 2019
Tagged as: Frank Waddell, virtual reality, vr