introducing evaluation

evaluation

integral to design process

collect and analyze data when interacting with a design artifact

goal: improve artifacts design

usability and user experience

design assumptions

designing for self or colleagues

assumptions about what users will use

evaluation enables checking that a design is correct and get at the root cause

root cause

why evaluate?

users expect more than just a usable system

pleasing and engaging experiences

well-designed products sell

make informed design decisions rather than debating

what to evaluate?

ranges for low-fidelity prototypes to complete systems

depends on the stage of design and the context of use

different types of evaluation are needed

main criteria: what the user needs or wants

where to evaluate?

depends on what is being evaluated

controlled lab studies vs in-the-wild studies

remote studies

when to evaluate?

depends on the type of product and the development process

new concept or upgrade on existing one

formative vs summative evaluations

rapid iterations

types of evaluation

controlled settings with users

control what users do, when they do it, and for how long

reduce outside influence and distractions

evaluate software applications with seated participants

usability testing

fundamental HCI process

evaluating user interfaces in controlled setting

primary goal: determine whether an interface is usable by intended users

recordings of interactions

crowdsourcing

hundreds, thousands, or millions

evaluate a product or participate in study

short, simple tasks

living labs

evaluate people's everyday lives

more controllable than in the wild

smart homes

natural settings with users

field studies

help identify opportunity for new technology

establish the requirements for a new design

facilitate the introduction of technology

inform deployment of existing technology in new contexts

field studies: methods

methods: observation, interviews, and interaction logging

data: events and conversations

goal: be unobtrusive, avoid influence on behaviors

influence may be inevitable

in the wild

look at how new technology or prototypes deployed and used

give up control of what is being evaluated

slow technology

analytics

logging and analyzing interaction data

web analytics

google analytics

opportunistic evaluations

contrast detailed and planned studies

typically early in design process

informal

minimal resources

combining methods

obtain richer understanding

usability testing combined with observations in the wild

different pros and cons

choice of methods

is the method reliable?

consistency

reproducible / replication

high: controlled experiments

variable: observations

validity

measures what is intended

depends on goals of study

ecological validity

how environment influences results

controlled experiments

participant awareness of being studied

bias

results are distorted

observational data

interviews

scope

how generalizable are the findings

narrow vs wide

overstating results

questions?

reading for next class

Chapter 9: Data Analysis, Interpretation, and Presentation
Interaction Design: Beyond Human-Computer Interaction