Remote Usability Testing using Wonderland

February 24, 2011

Kapil Chalil Madathil and Dr. Joel Greenstein conducted an interesting study analyzing the feasibility of using Open Wonderland for synchronous remote usability testing.

Kapil is currently a doctoral student at Clemson University working with Dr. Joel S. Greenstein. Dr. Greenstein is an Associate Professor in the Department of Industrial Engineering and the Director of the Human-Computer Systems Laboratory at Clemson University.

Here they share some excerpts from their work that will be published at the CHI 2011 conference in Vancouver, Canada.

CHI 2011 Preview:  A New Perspective to Remote Usability Testing using Wonderland

The emergence of high speed internet technologies has resulted in the concept of the global village and next generation web applications addressing its needs. In such a scenario where usability evaluators, developers and prospective users are located in different countries and time zones, conducting a traditional lab usability evaluation creates challenges both from the cost and logistical perspectives. These concerns led to research on remote usability evaluation, with the user and the evaluators separated. However, remote testing lacks the immediacy and sense of “presence” desired to support a collaborative testing process. Moreover, managing inter-personal dynamics across cultural and linguistic barriers may require approaches sensitive to the cultures involved.  Three-dimensional (3D) virtual world applications may address some of these concerns.

Collaborative engineering was redefined when Open Wonderland integrated high fidelity voice-based communication, immersive audio and screen-sharing tools into virtual worlds. Such 3D virtual worlds mirror the collaboration among participants and experts when all are physically present, potentially enabling usability tests to be conducted more effectively when the facilitator and participant are located in different places.

We developed a virtual three-dimensional usability testing laboratory using the Open Wonderland toolkit.

We then conducted a study to compare the effectiveness of synchronous usability testing in a 3D virtual usability testing lab with two other synchronous usability testing methods: the traditional lab approach and WebEx, a web-based conferencing and screen sharing approach.

The study was conducted with 48 participants in total, 36 test participants and 12 test facilitators. The test participants were asked to complete 5 tasks on a simulated e-commerce website. The three methodologies were compared with respect to the following dependent variables: the time taken to complete the tasks; the usability defects identified; the severity of these usability defects; and the subjective ratings from NASA-TLX (NASA Task Load Index), presence and post-test subjective satisfaction questionnaires.

The three methodologies agreed closely in terms of the total number of defects identified, the number of high severity defects identified and the time taken to complete the tasks. However, there was a significant difference in the workload experienced by the test participants and facilitators, with the traditional lab condition imposing the least and the virtual lab and the WebEx conditions imposing similar levels. It was also found that the test participants experienced greater involvement and a more immersive experience in the virtual world condition than in the WebEx condition. The ratings for the virtual world condition were not significantly different from those in the traditional lab condition.  The results of this study suggest that participants were productive and enjoyed the virtual lab condition, indicating the potential of a virtual world based approach as an alternative to conventional approaches for synchronous usability testing.

We will be presenting the full details of our study at CHI 2011 in Vancouver, Canada.

Hope to see you there!!

Kapil Chalil Madathil and Dr. Joel S. Greenstein


Association of Virtual Worlds Meet Up

February 16, 2011

I’ll be hosting this month’s Association of Virtual Worlds (AVW) Meet Up event on Friday, February 18th on the Wonderland Community server:

AVW Meet Up! Friday Feb 18 at 4 EST in Open Wonderland

(see time in your timezone)

You need to be a member of the AVW to attend, but membership is free. There’s a sign up button on the announcement page.

It would be great to have some experienced Wonderland users at the event to help show the newcomers around.

Wonderland Wednesday EZScript Demo

February 13, 2011

At the February 9th Wonderland Wednesday session, Ryan Babiuch (aka Jagwire), a member of the iSocial team at the University of Missouri, demonstrated his new EZScript module. This module integrates JavaScript as a capability that can be applied to any in-world object. While this is still a work in progress and not yet available in the Module Warehouse, it is now functional enough to experiment with. Here’s a demo of how it works:

And here’s Ryan presenting at the Wonderland Wednesday session:

For those who want to experiment, Ryan has provided code below for the four examples shown in the videos, plus another script that places a message in the HUD (heads-up display) when the object is clicked. The source code for the EZScript module is currently in the wonderland-modules/unstable directory hosted on google code. The full path is: wonderland-modules/unstable/EZScript. If you want to try out EZScript on your own server, you can use Subversion to check out the code.

EZScript Examples

// Hover
for(var x = 0; x < 2; x++) {
   animateMove(cell, 0, 0.5, 0, 1);
   animateMove(cell, 0, -0.5, 0, 1);

// Play animation already associated with an object

// Spin 
function s() {
   spin(cell, 5, 5);
ScriptContext.onApproach(s, false);

// Move NPC

var x = 0;

function walk() {
   MoveNPC(cell, x, 0, 0);
ScriptContext.onClick(walk, false);

// Show message in HUD
function show() {
   ShowHUDMessage("Welcome to 
        Open Wonderland!");
ScriptContext.onClick(show, false)

First P2PU Open Wonderland Class

February 4, 2011

Two weeks ago José Domínguez (aka Josmas Flores) announced an Open Wonderland course that he organized for Peer to Peer University (P2PU). He anticipated only 5 students, but ended up with at least 13, with several others auditing the course. The class met as a group for the first time today.

First P2PU class

First P2PU Open Wonderland class

This meeting was primarily an organizational meeting, with the real work getting underway next week as the class participants show off their “shape cells” created by following the steps in the “Developing a New Cell” tutorial.

One of the great things about this course is that all the content is freely available to anyone, including the course forum. Refer to the course home page for more information. In addition, a requirement of the course is that all enrolled participants blog about their experience. In the first session today, José shared a link to a web page that aggregates all the student blogs.

From skimming the blog posts, I got a sense of who was taking the class. It’s quite a diverse group. There are a few Open Wonderland community regulars, including Nina, a developer and virtual world evangelist from Lockheed Martin; Michel, a French entrepreneur running a business based on Open Wonderland, and Morris, a retired programmer who is doing Open Wonderland development for the fun of it. Others in the course include students from University of Essex in the UK, University of Lyon in France, University of Missouri in the States, Siena University in New York, and Karunya University in Coimbatore. There’s also a developer who works on the TEALSim project at MIT, an architect (of buildings and software), and a gentleman from Wells Fargo.

José said to encourage anyone who wishes to follow along to monitor the course forum, read the student blogs, and try out the assignments.

Virtual Worlds Best Practices in Education

February 2, 2011

I just noticed that the deadline for submitting abstracts to the Virtual Worlds Best Practices in Education conference has been extended to February 15th. The submission only requires a 50 – 100 word abstract. The conference takes place in Second Life March 17th – 19th, so people from around the world can participate.

VWBPE LogoVirtual World Best Practices in Education
2011 Call for Proposals

It would be great to see some Open Wonderland presentations on the program. It’s not too competitive to get accepted, so please consider submitting a talk abstract.

%d bloggers like this: