Unlike other areas in computer science, the field of HCI or VIS usually involves real users in the research. User study is an essential means of validating the proposed systems, techniques, or ideas. There exist many methodologies for the design, development, and evaluation with users in the HCI literature. For example, one common method is iterative user-centered design. You need to get familiar with the contents and materials of CS449/649 or the following books (accessible online from the University of Waterloo library).

<aside> 📖 *Research Methods in Human-Computer Interaction by Lazar et al.*

</aside>

<aside> 📖 *Interaction Design: Beyond Human-Computer Interaction by Sharp et al.*

</aside>

Preparation

The TCPS 2 Tutorial Course on Research Ethics (CORE), also known as the TCPS2 tutorial, is mandatory for all researchers who intend to engage in research with human participants. You should also get an ethics clearance before conducting any user studies, required by the University Office of Research Ethics. You will need to register on the Research Ethics System to apply for ethics clearance of your study protocol before you do any studies. It usually takes about 1.5 months and multiple rounds of revisions, so be mindful to take this into your consideration of your project timeline.

More information can be found on their website. Please consult with me regarding the ethics application process and ask me for sample applications.

Design

Before running a user study, you should have a robust design of the study and prepare the following materials:

• Study overview: Goals, design (within- or between-subjects), tasks, hypotheses, conditions and ordering (for controlled studies), procedure.
• Study scripts: Written scripts for explaining the systems and tasks, to keep the study consistent among participants.
• Study instruments: Questionnaires, interview questions, etc. For questionnaire design, you may refer to a few standard benchmarks; however, your study instruments design should follow your research questions.
  • NASA TLX and ARWES/CSS for evaluating workload
  • SUS, UMUX-LITE, and UMUX for evaluating system usabilities
  • OATS and DSoT for evaluating training effectiveness
  • TOAST for evaluating trust
  • ResQue for evaluating recommender systems
  • Creativity Support Index for evaluating digital tools (also see the Appendix A of this)
  • Self-Assessment Manikin (SAM) for assessing emotion
  • For Likert scales, here is a list of example anchors.
• Apparatus or study systems: Datasets, hardware, and software, for data capturing and running your system
• Recruitment: Materials and channels for getting participants. To determine a suitable sample size of participants, refer to the G*Power tool, while 12 is usually the convention of most HCI research studies.
• Condition order list (if controlled experiments): A list of sequenced conditions that each participant will experience. Refer to the Balanced Latin Square Generator for reducing the ordering effect.

I highly recommend you to read Chapters 8, 14, and 15 in *Interaction Design: Beyond Human-Computer Interaction* as well as Chapters 3, 5, and 6-10 in *Research Methods in Human-Computer Interaction* to get a deeper understandings on different HCI research methods and experimental design.

Lastly, it is always to do several (2 or 3 participants) pilot studies before running the real ones. During the pilot study, you can examine your task design, study duration, questionnaires, interview questions, and many other uncertain factors for the study. You can then adjust your study design accordingly.

Analysis and report

There are usually two types of data that you can collect from a user study: quantitative data (e.g., task completion times, task accuracy, questionnaire ratings), and qualitative data (e.g., interview results and observation notes). For example, you can refer to the ChartSeer paper for examples of describing the studies and reporting the results, on both quantitative and qualitative aspects. The Latex macros at the end of How to write a research paper page provides you with some templates for reporting the results.