Modified contextual design as a field evaluation method Export

@inproceedings{citeulike:1152863, abstract = {Downstream utility is a critical success factor for usability evaluation methods, in terms of the extent to which they can deliver value. In this paper we argue that field methods can significantly improve downstream utility through the added value they provide in terms of the range of usability problems they uncover and the contextual information they yield on user difficulties and their causal explanations. By way of an example we describe our experience of applying an adaptation of Rapid Contextual Design called Rapid Contextual Evaluation in a small scale field evaluation of a course administration system.}, address = {New York, NY, USA}, author = {Mcdonald, Sharon and Monahan, Kelly and Cockton, Gilbert}, booktitle = {NordiCHI '06: Proceedings of the 4th Nordic conference on Human-computer interaction}, citeulike-article-id = {1152863}, citeulike-linkout-0 = {http://dx.doi.org/10.1145/1182475.1182531}, comment = {WHY FIELD EVALUATION? "field methods for usability evaluation...reveal a broad and very different range of problems that could not be achieved through lab testing. Second, they provide a better basis for understanding the causes of usability problems." pp. 437 "...they detail usability problems and the evidence for their existence in ways that motivate developers to make fixes." pp. 437 "In lab studies it is our understanding of the design that typically gets iterated, whereas in field testing our understanding of the context and intended value are also iterated." pp. 437 ”...roughly 2/3 of the problems identified were not within the OVLE system itself, but were related to the context of use of the system. Such a broad range of problems could not have been identified by laboratory testing.” pp. 439 ---=note-separator=--- "if field methods are to enter widespread use we need to devote research energy into investigating how they might be systematized and made more accessible to practitioners and researchers alike." pp. 437 ---=note-separator=--- "Although RCD is a design method, its frequent use in situations where there is a pre-existing system means that it is highly relevant to the process of evaluation. Consequently it could be easily modified to make it Rapid Contextual Evaluation (RCE)." pp. 437 ---=note-separator=--- "We suggest that the use of Structured Problem Reports will facilitate the extraction of usability problems." pp. 437 ---=note-separator=--- METHODOLOGY: OBSERVATION SESSIONS "The data was collected during Contextual Interviews, each involving a single researcher and participant. ...Observation sessions were unstructured; the participant was asked to carry out their normal work tasks (including answering the phone or dealing with other people) while commenting on their activities. The researcher observed the participant performing their tasks, occasionally interrupting to ask questions for clarification purposes. The researcher made written notes, which included a record of the participant's actions and tasks, information about any problems, interruptions or task breakdowns. Interviews lasted for between 30 minutes and 1.5 hours..." pp. 438 "Sessions were recorded using a digital recorder. Digital photographs were taken and drawings were made of the physical environment. Important artefacts were copied." pp. 438 * Taking photos and using a digital recorder cannot be done in a commercial environment, for legal reasons! ---=note-separator=--- METHODOLOGY: DATA ANALYSIS The data analysis involved group interpretation sessions, each of which analysed a single contextual interview. 1) ...constructing a user profile, including information such as demographics, role profile, work tasks, etc. 2) ...construct an organisation profile in a similar manner, including information such as demographics, number of employees, etc. 3) ...The physical model was then constructed with reference to the photographs and drawings from the interview, paths (places where people frequently walk to and from) were marked on the model. 4) ...Finally the interviewer led the group members through the interview, using the written notes from the interview and a transcription of the audio recording. A sequence model was created for each task observed during the course of the interview. Affinity notes were written during the entire process of developing the profiles and models; these recorded important issues such as use of artefacts, problems, opportunities, breakdowns, questions, etc. Each interpretation session generated questions, so follow-up interviews were conducted with the participants in order to find answers. Each follow-up interview took approximately 20 minutes, and was recorded with a digital recorder and was subsequently transcribed. 5) The final stage of data analysis was the consolidation session, in which the affinity notes from all the interpretation sessions were used to build an affinity diagram. pp. 438 ---=note-separator=--- METHODOLOGY: DATA ANALYSIS, cont. "In Contextual Design the team ” walk the wall” in order to generate design ideas; they write them on yellow Post-it notes and attach them to the wall near the relevant affinity notes. In this case, the wall walk was used for usability problem extraction; any team member could identify a suspected problem. Problems were written on yellow Postit notes and attached to the wall beside the affinity notes that generated them. Each suspected problem was then written on a problem report sheet...[which] asked for details on the likely difficulties a problem could cause, the specific context in which it would occur, and the assumed causes of the problem." pp. 438-439 ---=note-separator=--- "...it would appear that...there is evidence to suggest the iterative potential of field methods covers not only our understanding of the design, but also our understanding of the product value, and our understanding of the context of use." pp. 439 ---=note-separator=--- "The application of Structured Report Formats to field testing in this study enhanced the process of problem extraction, grounding problems in the data presented on the affinity diagram." pp. 440 ---=note-separator=--- "Some team members commented that the process of completing the Structured Problem Reports actually improved their understanding of the data presented in the affinity diagram, and their understanding of the nature of the problems they identified." pp. 440 ---=note-separator=--- "...it is difficult to establish before hand the importance or usefulness of certain information types. The interpretation sessions served as a mechanism to understand the data, but also as a means of heightening the researcher's awareness of the importance of different information types, or issues that needed to be followed up, thereby enhancing the researcher's interviewing skills iteratively." pp. 440 * See Grounded Theory!}, doi = {10.1145/1182475.1182531}, isbn = {1595933255}, keywords = {context, evaluation, methods}, pages = {437--440}, posted-at = {2007-03-10 14:45:35}, priority = {5}, publisher = {ACM Press}, title = {Modified contextual design as a field evaluation method}, url = {http://dx.doi.org/10.1145/1182475.1182531}, year = {2006} }

TY - CONF ID - citeulike:1152863 L3 - citeulike-article-id:1152863 N2 - Downstream utility is a critical success factor for usability evaluation methods, in terms of the extent to which they can deliver value. In this paper we argue that field methods can significantly improve downstream utility through the added value they provide in terms of the range of usability problems they uncover and the contextual information they yield on user difficulties and their causal explanations. By way of an example we describe our experience of applying an adaptation of Rapid Contextual Design called Rapid Contextual Evaluation in a small scale field evaluation of a course administration system. SN - 1595933255 AD - New York, NY, USA EP - 440 TI - Modified contextual design as a field evaluation method PB - ACM Press T2 - NordiCHI '06: Proceedings of the 4th Nordic conference on Human-computer interaction SP - 437 KW - context KW - evaluation KW - methods N1 - WHY FIELD EVALUATION? "field methods for usability evaluation...reveal a broad and very different range of problems that could not be achieved through lab testing. Second, they provide a better basis for understanding the causes of usability problems." pp. 437 "...they detail usability problems and the evidence for their existence in ways that motivate developers to make fixes." pp. 437 "In lab studies it is our understanding of the design that typically gets iterated, whereas in field testing our understanding of the context and intended value are also iterated." pp. 437 ”...roughly 2/3 of the problems identified were not within the OVLE system itself, but were related to the context of use of the system. Such a broad range of problems could not have been identified by laboratory testing.” pp. 439 N1 - "if field methods are to enter widespread use we need to devote research energy into investigating how they might be systematized and made more accessible to practitioners and researchers alike." pp. 437 N1 - "Although RCD is a design method, its frequent use in situations where there is a pre-existing system means that it is highly relevant to the process of evaluation. Consequently it could be easily modified to make it Rapid Contextual Evaluation (RCE)." pp. 437 N1 - "We suggest that the use of Structured Problem Reports will facilitate the extraction of usability problems." pp. 437 N1 - METHODOLOGY: OBSERVATION SESSIONS "The data was collected during Contextual Interviews, each involving a single researcher and participant. ...Observation sessions were unstructured; the participant was asked to carry out their normal work tasks (including answering the phone or dealing with other people) while commenting on their activities. The researcher observed the participant performing their tasks, occasionally interrupting to ask questions for clarification purposes. The researcher made written notes, which included a record of the participant’s actions and tasks, information about any problems, interruptions or task breakdowns. Interviews lasted for between 30 minutes and 1.5 hours..." pp. 438 "Sessions were recorded using a digital recorder. Digital photographs were taken and drawings were made of the physical environment. Important artefacts were copied." pp. 438 * Taking photos and using a digital recorder cannot be done in a commercial environment, for legal reasons! N1 - METHODOLOGY: DATA ANALYSIS The data analysis involved group interpretation sessions, each of which analysed a single contextual interview. 1) ...constructing a user profile, including information such as demographics, role profile, work tasks, etc. 2) ...construct an organisation profile in a similar manner, including information such as demographics, number of employees, etc. 3) ...The physical model was then constructed with reference to the photographs and drawings from the interview, paths (places where people frequently walk to and from) were marked on the model. 4) ...Finally the interviewer led the group members through the interview, using the written notes from the interview and a transcription of the audio recording. A sequence model was created for each task observed during the course of the interview. Affinity notes were written during the entire process of developing the profiles and models; these recorded important issues such as use of artefacts, problems, opportunities, breakdowns, questions, etc. Each interpretation session generated questions, so follow-up interviews were conducted with the participants in order to find answers. Each follow-up interview took approximately 20 minutes, and was recorded with a digital recorder and was subsequently transcribed. 5) The final stage of data analysis was the consolidation session, in which the affinity notes from all the interpretation sessions were used to build an affinity diagram. pp. 438 N1 - METHODOLOGY: DATA ANALYSIS, cont. "In Contextual Design the team “walk the wall” in order to generate design ideas; they write them on yellow Post-it notes and attach them to the wall near the relevant affinity notes. In this case, the wall walk was used for usability problem extraction; any team member could identify a suspected problem. Problems were written on yellow Postit notes and attached to the wall beside the affinity notes that generated them. Each suspected problem was then written on a problem report sheet...[which] asked for details on the likely difficulties a problem could cause, the specific context in which it would occur, and the assumed causes of the problem." pp. 438-439 N1 - "...it would appear that...there is evidence to suggest the iterative potential of field methods covers not only our understanding of the design, but also our understanding of the product value, and our understanding of the context of use." pp. 439 N1 - "The application of Structured Report Formats to field testing in this study enhanced the process of problem extraction, grounding problems in the data presented on the affinity diagram." pp. 440 N1 - "Some team members commented that the process of completing the Structured Problem Reports actually improved their understanding of the data presented in the affinity diagram, and their understanding of the nature of the problems they identified." pp. 440 N1 - "...it is difficult to establish before hand the importance or usefulness of certain information types. The interpretation sessions served as a mechanism to understand the data, but also as a means of heightening the researcher’s awareness of the importance of different information types, or issues that needed to be followed up, thereby enhancing the researcher’s interviewing skills iteratively." pp. 440 * See Grounded Theory! AU - Mcdonald, Sharon AU - Monahan, Kelly AU - Cockton, Gilbert PY - 2006/// UR - http://dx.doi.org/10.1145/1182475.1182531 ER -