Mobile Community Design

Research and design information for mobile community developers.

Index of this document:

  1. Mobile Research Methods
  2. Design Philosophies
  3. How People and Society Interact With Technology
  4. Applicable Timescales

Mobile Research Methods
A comparison of methods for understanding mobile behavior to inform technology design.

Last updated October 2004

Name Description Strengths Weaknesses Usable
remotely?
Usable while
moving?
Usable for groups? Supporting
Technologies
Pictures
Case Studies High-level review of the design, implementation or usage of a product or situation to evaluate it and inform future processes.

Very comprehensive
Can explain systemic issues
Useful for process improvement

Time intensive
Can be very biased in terms of analysis
Is an evaluative method
NO unless online community. YES YES Depends  
Diary Studies Gathers self-reported information from participants in the context of their problem without an observer. Traditionally this used paper diaries but has branched out to use other media.

Self-reported data without observer
Can be very detailed and long term
Doesn't rely on participants to remember interaction details

Participants may not report some information
Information can be distorted
Reporting detailed activities can be tedious for participants

YES, only remotely YES, although text entry while moving can be problematic YES

Notepads
Voicemail
Pens
Sticky notes
Diaries

4
  Cultural Probes A variation which has used cameras and voice recorders to gain a more holistic understanding of users' culture. May provide a more open-ended way for participants to express themselves May make data analysis difficult see above see above see above Disposable cameras
Portable voice recorders
Notepads
 
Ethnography Longer-term studies where an observer takes notes while having minimal impact on the situation under study. Very rich accounts of behavior
Understanding of context
Longitudinal format allows seeing trends
Very time intensive
Record notes while moving difficult
Results in large quantities of unstructured descriptive data
Does not test future scenarios
NO Probably NO YES

Notepads
Voice recorders
Sometimes video
Cameras

Field Studies
(Observational Fieldwork)
A broad class of methods which occur in the participants' environment and tend to be short term studies. Relatively fast
In-context
Allow observation of environment
Provide rich accounts of behavior
Can produce large data sets
Hard to record detailed behavior sometimes
May not show trends in behavior since short term
Sometimes YES YES Video
Audio record
GPS, RFID, cell-positioning, wi-fi
Notepads
Wearables
 
  Natural Not common, but may be possible via discreet recording mechanism (e.g. hidden observation cameras). Very little to change behavior of participants May have logistical or ethical problems YES Possibly Possibly Surveillance equipment
Common technology which records usage
 
Artificial observer An observer is introduced to the environment being studied.

The experimenter can get a rich, holistic understanding of the activity and experience
Experimenter can record contextual information more selectively than automated methods

Observer (or group of observers) may affect participant behavior, change context and behavior of others
Asking questions can distract users from a task
NO YES, data entry is problematic YES, but watching dispersed groups is problematic Notepad
Wearables
Video
Cameras
1
Artificial activity An activity (possibly a typical one or one of interest to experimenters) is introduced. An activity which occurs infrequently can be examined easily It may be less spontaneous or realistic than the real thing NO, unless online community YES YES see above  
Artificial technology (can be similar to tech probe) A new (possibly prototype) technology is introduced. Behavioral responses to future technologies can be gauged in-context before they are full developed

Long-term usage is hard to predict
Social responses are hard to observe/predict

YES YES YES see above  
Artificial duration An artificial time span is introduced (usually for practical reasons). Allows in-context observations under realistic development timelines Restricted time can alter natural behavior
May inform about only beginning phase of longer process
YES YES YES see above  
Participant observer Either the researcher is already accepted member, or possibly a member could record data for a researcher.

The researcher theoretically has less effect on surrounding people's behavior
The researcher understands more about the activity and meaning of actions within it

Due to notetaking, their behavior may change
Recording data while engaged in a task may be problematic

NO, unless online community YES YES see above  
Interviewing Talking with participants and asking questions usually one on one. Good way to get detailed information
Can be done in many locations
Can be time consuming to do and to analyze audio data
Participants may not tell the truth or withhold information
Participants may not be aware of information or misremember it
YES, which probably reduces credibility of data Probably NO, or difficult to do so YES, but time consuming if separate Audio recording
Notepad or laptop
Video
  Out-of-context Interviews The interviews are held at an artificial location (e.g. the office of the researcher.) May be more convenient to researcher Artificial context may make participants less forthcoming and more uncomfortable
Context means they may not be able to refer to objects while they talk
NO NO YES see above  
Contextual Interviews Interviews held in the location where the user typically does the activity being studied.

Participants may be more comfortable
Rapport may be reached faster
Participants can demonstrate objects or environmental issues easier

Can be more difficult to arrange
Some environments are unrealistic to conduct interviews in
YES, but experimenter isn't in-context, which is the point. YES, but probably during temporary stops during movement YES, but time consuming if separate see above  
Structured vs Unstructured Some interviews ask specific questions while others use open-ended questions. Structured can give precise answers for comparisons
Structured can make interviews faster
Unstructured allows finding unknown concepts and gives time to explore them see above Unstructured probably easier see above see above  
Focus Groups One researcher talks with a group of people. Allows quick gauging of group opinion
Gets feedback from many people
Is only realistic method in some group settings
Group effects bias data
Assertive participants can intimidate or out-talk less assertive members
Conformity effects
Not enough time to get each person's detailed feedback
YES, but not common Difficult YES see above 4
Laboratory Experiments Testing of an interface in an artificial, controlled environment. Allow precise metrics
Comparisons and evaluations are more accurate
External factors are less likely to affect results
Artificial environment may make people uncomfortable or act differently
Labs can be expensive
Some activities are not feasible to do in a lab
NO NO, could be simulated YES Video
Video-mixers
Audio recording
Laptops
Eye-trackers
Movement trackers
 
  Static Usually in an office or laboratory with recording equipment. Mobility must be simulated. See above See above See above See above See above See above 1
Contextual (sometimes called mobile) Lab equipment is brought to the users' natural environment where testing occurs.

It is unrealistic to take some participants away from work for practical or security reasons, this is an alternative
Participants may be more comfortable testing products in their natural environment

Some equipment is not small enough to be portable
Difficult to avoid interruptions in some testing environments
NO YES, but only between locations, not for mobile users YES Video mixers
Video cameras
Laptop
3
Normative Writings Higher level discussion about the research area and surrounding issues. Useful to give understanding of where research is heading
Important to See unexplored areas or emerging trends outside academia
Can be abstract and theoretical
Can lack a grounding in real usage and practical problems
NO YES, can be based on mobile experiences YES, good for describing group trends Notepad
Normal observations
Press articles
Movies and other media
 
Survey Research Usually involves handing out questionnaires to users with questions about usage and demographics. Can cover a large number of people
Allows comparisons and statistics
Can be done rapidly
Usually very targeted questions allowing little explanation or discussion
Not in-context
Evaluating credibility is difficult
YES NO YES Web forms
Paper forms
Electronic documents via e-mail
Technology Probes Uses a prototype technology with limited open-ended usage possibilities, used in-context for long periods which records how it is used. Collects data without a highly visible presence watching participants
Can be used in mobile locations easily
Can be expensive
Time consuming to build
Difficult to build open-ended functionality
Can be time-consuming
Data may be de-contextualized and hard to understand
YES YES YES Prototype device 5

The study by Kjeldskov, J., & Graham C. (2003), helped inform the above taxonomy.


Design Philosophies

These are some common guiding theories or frameworks for design, which often use some of the methods shown above.

Name Description Strengths Weaknesses
Action Research Uses research in the field to help users with solve real problems. Results are taken and fed back into new research studies.

Theory is immediately tested
Allows building rapport with users
Easy for researcher to see results and understand design space.

Researcher may lose objectivity
Can be indistinguishable for normal business development
May not allow research time due to practical constraints

Ethnomethodology (EM) Similar to ethnography but with more emphasis on understanding the participants' generation of meaning and removing observer interpretation. Provides rich understanding of perspective of user
Explains how users interact with environment
Focused on what behavior accomplishes
Doesn't support abstraction or patterns
Doesn't support cogsci explanations (e.g. cognitive load)
No support for external explanations for behavior
Time-intensive
No behavior prediction
Participatory Design (PD) Advocates including users in the design process to greater or lesser degrees. Typically includes working on-site and forming long-term liasons between designers and users. Builds strong relationships with users
Gets in-context evaluation of ideas
Can be very time intensive
May not result in technologies which are desirable but which users don't want
May require site-visits
Places more responsibilities on users for designs
User Centered Design (UCD) An iterative process which advocates focusing product development on user needs and increasing time spent on research and design before development phases begin. Methods are built to focus on user needs
Has many methods suited to different types of design
The process allows repitition and sufficient design refinement before programming begins.
Reduces risk
Increases customer retention and lowers support costs
Can be more expensive or time-consuming in the short-term.
Getting access to users can be problematic.
Some methods may be more effective than others.

 


How People and Society Interact With Technology

It may be useful to visualize two feedback loops taking place during the use of a technology:

This diagram borrows from discussion in Brown, B., & Randell, R. (2004) and Dix, A. (2003).

Technology permits actions (resulting in affordances) that may have been impossible before. Users then decide whether to do these actions. Sometimes these affordances are forseen by the designers and sometimes they are accidentally permitted. When people do actions, they are seen by others, who then either allow or dissuade this behavior by a variety of complex social methods. This affects norm development and can control how a technology is used by a society just as much as what the affordances allow.


Applicable Timescales

Design research methods should in theory help inform the design of future products. Some do this by focusing on understanding existing behavior, while others look at predicting future behavior. This is a comparison of where methods are most suitable along the temporal spectrum.

Four major types of behavior can be used to inform new designs:

  1. Past behavior (using previous technology, See A,B below)
    How we used technologies with certain affordances previously.
  2. Common present behavior (using existing technology, see C below)
    Technology usage we commonly see around us.
  3. New present behavior (using existing technology, See D below)
    Where groups such as teenagers who are doing new and unexpected actions with a technology's affordances, who are probably doing behavior other groups will emulate.
  4. Snapshot of future behavior (using prototypes of future technologies, see E below)
    Where people use future technologies at various levels of fidelity to get an idea of how they will choose actions in response to new affordances.

The following is a rough estimate of the temporal focus of different research methods.

Applicability of methods for understanding different temporalities of usage
Name Past Present Future
  (A) Far (B) Recent (C) Common (D) New (E) Future snapshot
Case Studies          
Diary Studies          
Ethnography          
Field Studies          
Interviewing          
Laboratory Experiments          
Normative Writings          
Survey Research          
Technology Probes          

 



References

  1. Kjeldskov J. and Stage J. (2004) New Techniques for Usability Evaluation of Mobile Systems. International Journal of Human Computer Studies (IJHCS) Elsevier, 60(2004):599-620.
  2. Kjeldskov, J., & Graham, C. (2003). A Review of Mobile HCI Research Methods. In L. Chittaro (Ed.), Mobile HCI (pp. 317-335): Springer-Verlag.
  3. http://www.normwilcox.com/lab-in-a-bag.htm
  4. Bidwell, N. J. (2004). Pictures Made for Walking: Pilots & Orienteers 1. In OzChi 2004. Wollongong, Australia.
  5. Axup, J., Bidwell, N. J., & Viller, S. (2004). Representation of self-reported information usage during mobile field studies: Pilots & Orienteers 2. In OzCHI 2004: Supporting Community Interaction: Possibilities and Challenges, Wollongong Australia.
  6. Brown, B., & Randell, R. (2004). Building a context sensitive telephone: Some hopes and pitfalls for context sensitive computing, from www.dcs.gla.ac.uk/~barry/papers/context.pdf
  7. Dix, A. (2003). Human-computer interaction (3rd ed.). Harlow, England ; New York: Pearson/Prentice-Hall.