From Living Lab

Mobile Living Labs 09: Methods and Tools for Evaluation in the Wild

A one-day workshop at MobileHCI09, September 15th, 2009, Bonn, Germany.
http://mll09.novay.nl

Background

Mobile devices have become truly ubiquitous computing platforms, offering a wide range of functionality from traditional voice/text communication to GPS navigation and mobile friend finding. Much of their value stems from their ability to be taken anywhere offering quick, convenient access to applications and services in a variety of contexts. These “contexts of use” are even more important now as context-aware applications (such as location-aware services) are becoming a reality in many contemporary smartphones. This context of use, however, makes it difficult for researchers to evaluate mobile device applications and interactions using traditional human-computer interaction methods. Whereas some aspects of the user experience can be evaluated with lab experiments, interviews, focus groups and/or surveys, many other aspects are harder to investigate if taken out of the natural context of use (e.g., out in the wild).

Instead of focusing solely on bringing people to the lab, researchers who want to evaluate mobile devices and services are increasingly doing the opposite: bringing the lab to the people [1]. This is a key ingredient of the “Living Labs” approach [7], which employs a range of methods, including: self-report methods (e.g. experience sampling, diaries, day reconstruction), measurement (e.g. application usage logging, context logging), as well as observation (e.g. ethnography).

For Mobile Living Labs, smartphones are often used, not only as a platform to deliver new mobile services, but also as a platform for data collection tools. Contemporary smartphones can be utilized to collect all kinds of data (audio, video, user input), according to a wide range of methods. These methods range from automatic data collection without user intervention (logging) to asking users small questions at random moments (experience sampling). Some tools used in Mobile Living Labs recently include ContextPhone[6], MyExperience[1], Xensor[3], RECON[4], and BeTelGeuse[5]. Of course, data gathered by such tools is different from observations by researchers (e.g. well-trained ethnographers), but using smartphones enables studying more persons, longer and at times and locations where observations by researchers would have been difficult.

References

[1] Connelly, K.; Siek, K.A.; Mulder, I.; Neely, S.; Stevenson, G.; Kray, C. (2008). Evaluating Pervasive and Ubiquitous Systems , IEEE Pervasive Computing, 7(3), 85-88.
[2] Froehlich, J., Chen, M., Consolvo, S., Harrison, B., & Landay, J. (2007) MyExperience: A System for In Situ Tracing and Capturing of User Feedback on Mobile Phones. In Proceedings of MobiSys 2007, San Juan, Puerto Rico.
[3] Hofte, G.H. ter (2007). What's that Hot Thing in my Pocket? SocioXensor, a smartphone data collector. In: Proceedings of e-Social Science 2007, the Third International Conference on e-Social Science, October 7-9, 2007, Ann Arbor, MI, USA.
[4] Jensen, K. 2008. Evaluating Mobile and Ubiquitous Applications in the Field by Automated Capture and Analysis of Reality Traces. In Proceedings of 8th Danish HCI Research Symposium.
[5] Nurmi, P., Kukkonen, J., Lagerspetz, E., Suomela, J., and Floréen, P. 2007. BeTelGeuse – a tool for Bluetooth data gathering. In Proceedings of BodyNets 2007.
[6] Raento, M., Oulasvirta, A., Petit, R., Toivonen, H. (2005). ContextPhone: A Prototyping Platform for Context-Aware Mobile Applications. IEEE Pervasive Computing, 4, 51-59.
[7] Schumacher, J, & Niitamo, V.P. (Eds). (2008). European Living Labs: A new approach for human-centric regional innovation. Wissenschaftliger Verlag, Berlin, Germany.