Adapted from James Allan's CMPSCI 646 course (Fall, 2004) at U. Mass. and Doug Oard's LBSC 796/INFM 718R at UMD.
This Excel spreadsheet (based on your hw2) contains three separate sets of judgments for the hits examined in Homework 2. For one of the two topics (i.e., Q1 and Q2), you will:
1. Agreement on Relevance Judgments
The above spreadsheet should contain three sets of judgments for every document (Web page). The first question you'll answer is: How often do judges agree on relevance? There are four possibilities:
(1.1) For your chosen topic, figure out how often each case happens (both in terms of counts and in terms of percentage). Turn this information in. [10 points]
(1.2) For your chosen topic, pick three cases where judgments about a particular hit are not uniform, and briefly speculate why this may be so. The three cases can be selected either from one system or from two systems. Try to employ the concepts of relevance discussed in lecture. Turn this in. [10 points]
2. Adjudication
Adjudication is simply the process of reconciling inconsistent judgments. Do this by simple majority voting; for example, the adjudicated result of NNR is N (based on majority voting). You do not need to turn anything in for this, but you will need the results for the third question.
3. Evaluation of Bing and Google
(3.1) Now, evaluate Bing and Google using the adjudicated relevance judgments you just created (for the topic you chose). Make sure you are pooling judgments from both systems! [10 points for pooling]
Turn in the following information for both search engines:
4. In addition, answer the following questions:
Please post your assignment on your Website.