Archive | August, 2011

A Journal-Based Approach to User Research with Students

3 Aug

Market/user research usually involves instructors because they are the prime prospects that the publishing companies sell products to. User research involving students are few and far between and when that happens, the questions are limited to “what” they do and “what” they think. The research doesn’t focus on “why” and “how” the students do something or what would make them do things that they currently don’t do. And this results in products that are good but are not compatible with the students’ requirements.

A common user research finding is that “the students don’t read books”. None of the studies go beyond this statement to find out why the students don’t read books. Do they not buy the books because it’s expensive? If they buy books, are the chapters too long and take a lot of time to read? If they start reading the books, do they not understand the content? Is the content too boring and not engaging enough to hold their attention? There are currently 19 million college students in the US. It’s obvious that many students do read books. What makes them read their books when many others do not read books?

The user research proposed below is intended to find answers to these and similar questions and aimed at the day-to-day behavior of the students with respect to their school, program, and courses.

Common User Research Methods

  • Focus Groups (colocated)
  • Webex sessions (distributed focus group)
  • Online survey (Qualtrics, Zoomerang, Survey Monkey)
  • Interviews (in-person or telephonic)
  • Ethnographic (observation and interview)
  • Literature review

Selection of a Research Method
Each research method has its pros and cons. The best research method used in a situation depends on several factors; for example, physical proximity, travel costs, availability of participants, budget for paying the participants and other expenses, time, depth of information required, availability of previous research data, number of participants, and qualitative or quantitative data required.

Limitations or Issues with Research Methods
Focus groups are intended to bring together people from the same field that can share thoughts and experiences and provide their expert insights into issues at hand. But do people freely share their thoughts in a group setting without getting influenced?

The best group is one where people having diverse backgrounds, experiences, and educational qualifications share their knowledge independently from each other. What happens when people in a group do not act independently? The strong personalities dominate the conversation and the group opinion flows with them. Some people might not find it comfortable to disagree openly and express their ideas. Some people might feel it awkward to convey negative opinions. Someone is biased, while someone else is not interested. Sometimes, people say what is considered correct or do not know what they do not know. Or they do not know the best answer.

Example 1
Ebooks: When a group of instructors is shown a new ebook and its benefits explained, will the instructors be in a position to figure out whether the ebook meets their needs or will it be beneficial in the long term or will it be an effective learning tool for their students? If they have not seen anything that looks similar, will they be able to immediately and properly make an assessment? If they see two ebooks, they will be able to compare the two, and even suggest improvements, but will they know what is missing in both ebooks? Will they be able to think of an entirely new feature in the ebook?

Example 2
Games: Current market research suggests that the students like educational games and the games help them learn and prepare for the test. However, there is no data to prove that the games have any instructional value.

If the students are having fun playing the game, it does not mean that it is helping the students by motivating them. Does fun correlate to motivation? If the students are having fun, does it motivate them to spend more time studying or taking a quiz? The game is very different from the test, so it doesn’t prepare the students for the test. Research proves that the context is very important for learning with games. If the students are playing an economics game, the game should be set in a subject-matter-related context, and not, for example, a jeopardy game.

Benefits of observational research
Observation removes many of the biases and limitations that are inherent to other research methods. Observing people in their natural settings doing things that they do is better than asking them what they do, which might not elicit the correct response. However, observation has its own challenges. If people know that they are being observed, it might alter their behavior. Or, we might interpret things differently than what’s actually happening or intended. A combination of observation and interviews might help correct some of the collected information.

It is impractical to observe 15-20 people the whole day for a month. Fewer people or lesser observation time would yield results that would be statistically insignificant. For drawing conclusions based on collected data, it’s necessary to target people in diverse user groups and collect data over a long period of time.

Daily journal as a user research tool
An alternative to observational research is the daily journal. People keep their journal and make entries every day, several times a day, for the activities they are engaged in. It’s not clear if the entries should be only related to learning activities; for example, reading books, attending lectures, asking friends questions, online search, or the daily activities not related to learning should also be recorded. Though it seems that the unrelated activities will not yield useful information, we don’t yet know if these activities have any effect on studies; for example, surfing the internet on the iPad, though seems unrelated, might affect the time spent on studying or give us information on people’s preferences for a digital tool.


  • For the pilot study, four or five students will be selected from different target user groups; for example, two-year and four year, economics major and accounting major, a moderately high-ranked and low-ranked school, male and female, smartphone/tablet owner and non-smartphone/tablet owner.
  • The purpose of the study will be explained to them and a consent form signed. A journal will be given to them that already include categories of activities and questions (to be decided later).
  • The students enter notes in the journal several times a day for 30 days. Alternatively, they can enter the notes at the end of each day (though this is discouraged).
  • At the end of 30 days, the journals will be collected from the students and their notes read and analyzed. The students will be contacted if there are any follow-up questions.
  • Based on the results, the process will be refined and expanded to more students after the pilot.


The initial cost breakdown for the pilot is as follows:

  • 30 min a day writing the journal x 30 days = 900 minutes/60 = 15 hours
  • Rate of pay = $15 an hour
  • Cost per student = 15 hours x $15 an hour = $225
  • Total cost for five students = $225 x 5 = $1125

To avoid the pitfalls of traditional methods of user research and learn more about the behavior of the students, the journal method of research can be tried and tested. If the pilot is successful, the research can be expanded to more students and if possible, instructors. It is expected that the data would give valuable insights into the mindset of students and their behavior and preferences. This will provide useful information to help design technology-based learning products and assets that not only provide instructional value but also motivate the students to use them.