Last updated: August 19, 2015.
You should think of your batch evaluation and your user study as ways of answering different aspects of the same question: "How well does our system help the user achieve their goals?" Because user studies are generally more expensive than batch evaluations (i.e., they take more time to plan, execute and analyze if done well), it is prudent to focus your user study on aspects of the question that are not amenable to batch evaluation. Some examples of these kinds of questions are:
Once you have decided what you want to know, you are ready to choose a study design. There are two basic kinds of study designs:
Once you have chosen goals for your study and the kind of study design you want to use, you should consult one or more examples of studies using that kind of design. Here's one of each that you could start with:
Once you have a study design, you should test it on someone who did not contribute to your study design and who will not be one of your actual study participants. This is called a "pilot study", and you are sure to learn of the need for some improvements (e.g., in how you do training, or in how you collect data). You can go do this with almost anyone since you are not studying them, you are simply trying out your study design. But it is best if you don't use a member of your team (because they know too much!).
Normally you would want to recruit study participants who are representative of the people who would really use a system like the one you are building, and if you can find such people and motivate them to participate that would be great (but see below on IRB requirements if you want to make a publishable study out of this). But in practice, most of you will actually recruit your classmates. The only restriction on this is to avoid over-using same members of a team because they can get a learning effect.
With that as background, here is what should be in your plan:
As general guidelines, your study design should fit comfortably on four pages, and (unless all you are doing is a user study!) your study should require around 15% of the total time available for your project, including planning, execution, and analysis (9 weeks * 6 hours per week * the number of people on your team * 15% ... for a 6 person team this number is around 50 hours).
If you plan to publish the results of your study, advance approval from LSU Institutional Review Board (IRB) is required. If your results will not be published, that step is not required.
Acknowledgement to Doug Oard (LBSC 796/INFM 718R Spring 2011).