An in-depth insight on users’ real and recurring issues

How should we start a diary study?

Source: www.webcredible.com

We want to know what users do when we are not there with them: what are their recurring problems, what are the actual situations they experience. Ask them to take notes of what they do (preferably in digital form, for example in a private blog, Google Docs, etc.) then share it with us. With this method, we can identify problems and get to know user-related issues that we would not know otherwise be since we cannot sit next to the user all the time. This is because users use a given interface sporadically and randomly — for instance, a small business owner would be using the online bank at 11 pm. Clearly, no one will appear all of the sudden to monitor each move. User tests do not necessarily reveal users’ actual concerns.

Although a diary study requires a lot more time and attention than simpler interview or UX research, its benefit is that it can provide some insight on the subject’s real/reported behaviour and allow us to understand the context, the longer-term processes and behaviours. What one should know about the off-line and online application of the method: what is the right way to create such a diary and finally, what is the best solution, only suitable for researchers working with strict methodology, ESM.

We can thus gain insight on what they say and do. But if we are not only interested in what they say and remember, but also want to know exactly what they do, we need to measure with an instrument (for instance Lookback, addressed later on) or ask them to report frequently and very precisely, and also talk to them regularly.

How diary study works?

Researchers have developed several methods to get to know users’ usual daily activities. These can also be assessed by filling out various questionnaires or telephone inquiries. But we can obtain more relevant data if the users report their behaviour while doing the activity or shortly after.

“Please take notes in your radio diary.”

One of the methods is to keep a diary study, a fairly old and well-established method for a person or household to take notes of the events relevant to the studied topic. For instance, to which radio station they listen at what time or what they bought at the store. This diary study is used for measuring the listener’s ratings of radios or the household panel, where the households involved in a diary study on daily purchases.

In the previous case, the volunteer listeners (31,000 households) log what radio station they listened to and the approximate time. They keep the diary for one week and indicate which station they listened to and where in quarter-hour time bands.

In general, researchers consider subsequent reporting of earlier activities not very reliable. Since we cannot be sure whether someone is accurately recording what and when they have listened to with quarter-hourly precision when measuring radio listening figures, radio logging is not necessarily the most reliable method either. Only very few people would be able to accurately report their activities; this degree of awareness cannot be expected from anyone. But with the help of logging, researchers can obtain information which would be virtually impossible to observe in a person on a regular basis (e.g. prohibitive costs). So in areas where test users are expected to log their activities, they are often given a phone call, for instance the next day, to discuss the events they logged.

The DAR test enables a presumably more accurate measurement, where they reconcile the users’ radio listening activity of the previous day in person or by phone via an interviewer. As part of this interview, they discuss which radio stations the user listened to and at when, broken down to quarter-hourly blocks, in what location and whether they did this alone or with someone. The advantage of this method is that the measurement is more accurate, and the interviewer can also ask questions over the phone; but unfortunately, the above-mentioned lack of awareness remains a drawback. Another shortcoming is that it is substantially more expensive. In Hungary no such survey is prepared, but in Norway or in Italy they do prepare surveys of this type.

If we are not interested in users’ own activity report, but want to know the exact facts instead, then the ideal solution in the off-line world is the Watch-meter. The chip built into the watch records the frequency of the radio station the user is listening to. Television watching could also be measured with the help of a similar instrument. When the Watch-meter is used, the users under research wear the watch during one week, then return it to the researcher who copies and analyses the data stored on the chip. This could ensure the most precise measurement. But its disadvantage is that it entails high investment costs; moreover, the fact that it is very accurate may also be problematic as it would measure lower than expected listening and viewing times.

Previous Hungarian developments included the concept of a bracelet which also measures psychological indicators (such as breathing, pulse, emotional state), but this plan stopped at the conceptual level while seeing the received data and their expected accuracy/inaccuracy, let's be honest: we all know that we work with somewhat inaccurate data, but we believe them or at least we take them into consideration. Among researchers, this behaviour is called silent white lie.

Logging in the off-line world: evaluation is not easy either

Source: kimberlychang.net

The online environment is once again more precise, but why?

As researchers we are not interested in the story telling of the users about their activities, but rather we want to know what they actually did. Google Analytics or Lookback enable a more precise software-based data collection for any webpage or application. Diary study can then ensure understanding how the users use the interface in their natural environment. Therefore, designers can observe exactly how users use the product. By merging the logs and the data, we can obtain very useful information which we can be forwarded to our testers in order to take some qualitative sampling and share our experiences or positive and negative impressions with them. We can do that on a one-on-one basis or in a group.

The Lookback interface

Source: lookback.io

The clear and obvious advantage of dairy studies is that we do not have to be afraid that the collected data will be obtained by any unauthorized people, as the content is password-protected. It is very simple to format, update or organize the collected information, and the author (the person who kept the diary) and the researcher alike always have access to them.

Google Docs or Microsoft One Note interfaces are clearly suitable for online logging. Their advantage is that no local measurement is needed, each piece of information is automatically stored in the cloud. Moreover, they provide a password-protected service, and arranging the collected information into a diary is easy.

As the saying goes, it is better to prepare a simpler research than not to research at all, but diary study goes beyond this simplistic approach. It is an excellent method for understanding processes in the long run. If we can accept the listed shortcomings, optimize, adjust them and end up opting for diary studies, here is a short guideline on how to do it.

If we do it right, it can be good

Case study preparation is just as important as preparing for any other research method: we formulate the research question, i.e., what we want to find out. Do we want to understand the behaviour of the user? How do they use an application? How do users start to use it, and how their usage changes once they acquired the system’s operation and know it inside out? We generally want to know how users use an interface or we are interested in a specific area, for instance, how do users order a pizza or buy a theatre ticket?

Source: chiafangluecmuthesis.wordpress.com

We can then start the detailed design of the research and the preparatory work: we select the appropriate, most reliable method for the diary studies. We could (actually we should) conduct a pilot research enabling us to find out some minor details that may influence the research script.

It is definitely recommended to meet the research subjects prior to starting the study: we must explain them the exact process of case study and also discuss for how long (and how) we want them to keep the diary and what exactly are the activities we want them to record. It is also important to mention when they should take note of the events relevant to the research — when the events are actually happening or later? They should also write down their impressions and feelings and provide any other details.

It is crucial that we also think these aspects through ourselves and discuss them with the client, since a typical mistake is finding out at the end of the study, when presenting the results, that we omitted a lot of relevant information that we can only obtain in a next study.

While the research is being conducted, the recorded available data must be regularly followed up and research subjects reminded to keep on recording the data. We can also ask them to record their general impressions and remarks that are likely to be forgotten by the end of the study without taking note from them.

After the research is finished, once we have all the data, but prior to the detailed data analysis, we should contact the research subjects in person and ask for additional qualitative feedback. Simply talk with them about their impressions. It is almost certain that they have some additional remarks that are not featured in the collected data. Only once we have collected all of the above information can we start preparing the summary analysis.

And what is even better: ESM

The ESM, which stands for Experience Sampling Method, is an experience sampling procedure enabling a combined qualitative and quantitative survey.

“The main idea of the method is that research subjects give account of their cognitive, emotional and motivational state of mind and of the circumstances of their activities performed at the time of the signal received randomly and unexpectedly several times per day, during several days, by completing a structured questionnaire.” (GYÖNGYI KÁLLAI: “In the footsteps of workplace flow” analysis of the flow experience in the workplace environment and the description of the factors potentially influencing the frequency of the flow experience within the private and public spheres. In: DOCTORAL (PhD) THESIS 2016 (Pp 64, Et.al.) DOCTORAL (PhD) THESIS 2016 (Pp 64, Et.al.) They can do this in an offline notebook or on the online interface. “The survey typically covers a one-week period and the random timing of the sampling ensures the unbiased nature of the samples. This method guarantees broad and representative analysis of the experiences and allows the analysis of the effect of the context on the experience itself.

The ESM underwent a number of developments over time. Giving feedback to research subjects about measurement results is among the most cutting edge methodology innovations. The ESM tool is broadly applied in many different areas of learning, work, arts and free time activities.”

They recommend combining the ESM method with preliminary and subsequent (deep) interviews and other qualitative and quantitative methods.

Therefore, ESM is a more accurate, precise and more detailed version of the diary study. Equipped with a thorough research plan and methodological framework, we can accurately monitor the behaviour of our research subjects and the change in their opinions or moods even in the long run which is much more than a simple diary kept by our research subject based on their recollections and memories.

Recommended literature:

Earl Babbie: Practical social science research (Et.al.) Budapest, Balassi Publisher, 2008

Dumas, Joseph S.– Loring, Beth A. (2008): Moderating usability tests. Principles and Practices for Interacting. Morgan Kaufmann, Burlington.

Krug, Steve (2008): Don't make me think, free download
https://www.nngroup.com/articles/diary-studies/
https://www.quora.com/User-Research-What-software-do-you-use-for-a-diary-study http://uxmastery.com/diary-study-extending-ux-research/
http://www.leanresearch.co/chapter-4
https://lookback.io/blog/using-lookback-for-ios-diary-studies