Assignment 3, Due Monday, July 23rd

  

Part 1: Work Out the Filling of the Roles in the Study

Roles to Be Filled

• Demonstrator: The one who demonstrates the mockup and the scenario.
• Interviewer: The one who asks the interview questions.
• Observer/ Note Taker: The one who observes the participant and takes notes during the demonstration and the interview.

Possible Multiple Roles

The Demonstrator can also be the Interviewer.

Having just one person filling all three roles is difficult, since it would require too much multitasking.

If it is necessary (e.g., because the participant is a family member who cannot conveniently meet with two or more group members), it is best to make an audio recording, so that you can make notes after the meeting with the participant is over.

Rotating The Filling of Roles

Avoid having the same student in your group fill the same role with all participants; it will be valuable for each student to gain experience with each role. 

Part 2: Select and Invite the Participants

The number of participants should be twice the number of members in your group (e.g., each group member recruits 2 participants).

Whom to ask:

• The best participants are friends and family members who do not study Media Informatics or Computer Science but who have at least some experience in searching and browsing the web.
• They should also have some interest in the domain that you show for your mockups and scenarios: That is, they should be able to imagine that they are the user described in at least one of the scenarios.

You can tell participants that their participation will take under half an hour and that it should be fairly interesting.

You can arrange to meet each participant at some convenient and appropriate location (e.g., at the university or at home). 

Part 3: Prepare the Mockups for Your Two Tasks

Select the Mockups and Scenarios

Default procedure: Use the two mockups and scenarios that you developed for Assignment 2, making any changes that are necessary to make them usable for your study (see the instructions below and the feedback on your report).

Alternative procedure: Start with (and perhaps adapt) the mockups and scenarios provided in the next subsection, which have been copied from one of the especially well-done reports for Assignment 2.

The feedback on the report for Assignment 2 will include a recommendation of one of these two options.

Recommended Alternative Mockups and Scenarios

• CogTool project with mockups
• Scenarios

Notes on the Mockups

The designers of these mockups added a couple of features that weren’t present in the originally supplied mockup. These involve the multiple selection of items in the left-hand pane and the right-hand pane. Please read the scenarios and study the mockups carefully to make sure you understand these features before you demonstrate the mockup.

The mockup and the scenarios are in German. Most students and potential participants can presumably at least understand enough German to be able to understand the mockups and scenarios. If it seems better, you can explain the scenarios to the participant in English (note that the participant does not need to see the written version), along with the few words in the mockup that a non-German speaker might not understand (e.g., “Veranstaltungsort”).

These particular mockups do not work well in HTML (see the next subsection), because in CogTool’s conversion process the green color gets lost and some special characters are not handled correctly. But a good demonstration can be given in CogTool.

Prepare the Mockups for Use

The mockups have to be in such a state that you can demonstrate them to your participants as a way of illustrating your scenario; but they don’t have to be so self-explanatory that the participant can operate a mockup themselves.

Unless you have a reason to prefer using CogTool itself to demonstrate the mockups, export the relevant design(s) as HTML:

1. To enable the function for exporting a CogTool mockup as HTML, in File / Preferences / Research Commands, check the box “Enable research commands”.
2. From then on, in the File menu, you will see a command “Export Design to HTML”.

   Invoking this command creates a lot of small files within a folder that you can specify.

When you open the file “index.html”, you will see how to demonstrate the mockup by clicking in the right places.

Note that, depending on how your mockup is designed, it may not be obvious from the appearance of the screen where you need to click next; so you should practice the demonstration and if necessary create some sort of reminder of where to click when.

If for some reason the HTML version of your mockup does not work well, you can do the demonstration within CogTool itself: Step through a previously created “demonstration” of each task. Before starting the demo, choose “Zoom to Fit” to ensure that the whole demo will fit into the main screen.

Prepare the Scenarios for Use

Prepare the texts of the scenarios so that you can read (or recite) them while demonstrating the mockup.

Make sure that the scenario is consistent with what you can show in the mockup.

Adapt each scenario text in accordance with any recommendations made in the feedback on your report for Assignment 2. 

Part 4: Prepare the Questionnaire and the Interview Questions

Create the Questionnaire Questions

Formulate closed questions of the following types for inclusion in your questionnaire:

1. Questions about the participant’s background

   Formulate 2 or 3 closed questions that will help us to understand what sort of knowledge and experience this participant has that might be relevant to Parallel Faceted Browsing.

2. Questions about how well the participant understood the demo/scenario.
3. Questions about the extent to which the participant would be interested in using Parallel Faceted Browsing (assuming the availability of a well-designed, fully functional system) in situations like the one(s) demonstrated or in other situations.
4. (Any other relevant questions that occur to you.)

You may find it convenient to formulate all questions except for those in the first category as Likert-scale questions.

Write the Interview Questions

Formulate several questions that address the following issues:

• How does the participant perform tasks like these using the websites that they are already familiar with?
• Would the participant want to be able to use Parallel Faceted Browsing (instead of the sites they are already familiar with) for performing tasks like the ones you demonstrated to them? Why or why not?
• Can the participant think of any extensions or improvements of the method (e.g., things they would like to be able to do but cannot do with the demonstrated mockup)?

  Note: You probably don’t want to encourage the participant to comment on things like the details of the graphic design, since such details are not in the focus of this study.

Realize the Questionnaire and Interview in Google Forms

Formulate the questionnaire and the interview as two parts of a Google Form:

In Google Docs, “Create” a new item of the type “Form”. You can then formulate your questions in that form and have people enter answers into the form, which will be collected in a spreadsheet.

The procedure for creating and using a form should be self-explanatory.

In addition to entering the questions described above, include at the end the following three open “questions” for yourselves, which will enable you to enter your notes into the form (either during or after the meeting with the participant):

1. What interesting things did the participant do or say in response to the first mockup/scenario?
2. (Same question with regard to the second mockup/scenario.)
3. What new ideas did the participant mention when you asked about their answers to the questionnaire questions?

 

Part 5: Conduct a Small Pilot Study

Go through the steps listed in the next part at least once to debug the materials and procedure.

The “participant” in this pilot study can be a member of your group (or of another group, if it’s more convenient). 

Part 6: Conduct the Study

1. Explain the purpose of the small study

   You can explain the study in whatever style seems most natural for the participant in question, but be sure to mention the following points in some way or another:

   We are exploring a new approach to finding information on the web. We want to see:

   • ...whether you would find this approach useful in a particular situation - and/or in other situations;
   • ... what you think of the particular way that we have worked out of realizing this approach;
   • ... what ideas you might have about how the approach should be extended or improved.
2. The Demonstrator demonstrates the first mocked-up task, explaining it in the way specified in the textual scenario.

   Example: “Suppose you are Peter, a poor student who needs to buy a second-hand car. You visit this website ...”.

   The Observer makes notes on anything potentially relevant that the participant says or does.

3. The Demonstrator does the same thing for the second mocked-up task and scenario.
4. The Interviewer administers the questionnaire.

   Ask the participant to answer the questionnaire questions in a web browser using the Google Form.

5. The Interviewer conducts the semistructured interview

   Start by going over the participant’s answers to the questionnaire questions and ask follow-up questions where it seems useful to increase your understanding of their answers.

   Present the interview questions orally and make notes on the answers. You can put these notes into the Google Form right away or (if you can’t type that fast) after the meeting with the participant.

6. Thank the participant and answer any questions that they may have.

 

Part 7: Write and Submit a Report on Your Study

Note: What you need to submit for this assignment is (a) the main document with this report and (b) the other items listed below as appendixes.

Introduction

List the main questions you are trying to answer with your study. Assume that the reader of this report already knows what Parallel Faceted Browsing is and what general issues it raises.

Methods

Participants

Describe each participant without revealing his or her identity. Use what you know about the participant plus their answers to the self-descriptive questions in the questionnaire.

Materials

Very briefly describe the mockup, the scenarios, and the questions that you prepared for the questionnaire and for the interview.

One or two sentences about each of these items should be enough, since you will be including them as appendixes (see below).

Procedure

Briefly describe the steps that you went through, in order.

Results

1. For the notes taken during the demonstration: Describe what noteworthy things were observed and how you interpret them.
2. For each question on the questionnaire (except for the questions about the participant), prepare a suitable graph or chart showing the answers.
3. For each interview question, write a qualitative discussion in which you mention the most important points made by the participants and what can be learned from them.

Discussion

Summarize what you have learned about the questions that you listed in the Introduction.

Appendix A, Scenarios

Reproduce in this appendix the two scenarios that you used.

Appendix B, Mockup

“Share” your CogTool mockup.

Appendix C, Raw data

“Share” your Google Form and the associated spreadsheet.