Assignment adapted from James Allan's CMPSCI 646 course (Fall, 2004) at U. Mass and Doug Oard's INST 734 at UMD.
Last updated: August 19, 2015.
The purpose of this exercise is to gain some "hands-on" experience in the process of evaluating information retrieval systems. You will be assessing documents that are retrieved in response to two "topics" (statements of information needs). The two search engines we'll be comparing are Google and Bing.
online degrees benefits tuition quality legitimacy
Many universities are now offering online degrees in addition to campus degrees.
What are the benefits of getting online degrees? Relevant Web pages must address tuition, quality,
and legitimacy (i.e., acceptability by employees) of online-degree programs.
one-day tour in New Orleans in summer
A French friend of yours is coming to
attend a conference in New Orleans for his first time next summer. He wishes to spend one
day in New Orleans to tour the city. What is a good plan for his one-day tour in New Orleans next summer?
To ensure that everyone evaluates the same hits, results from each search engine have been cached for you. See the Excel spreadsheet below. You will evaluate the relevance of a subset of these hits. If your social security number (or personal identification number) ends in an even digit (i.e., 0, 2, 4, 6, 8), evaluate the first 15 hits of each query. If your social security number (or personal identification number) ends in an odd digit (i.e., 1, 3, 5, 7, 9), evaluate the last 15 hits of each query.
Use this Excel spreadsheet to keep track of your relevance judgments. Note that there are 4 spreadsheets in the file, one spreadsheet for each query (i.e., Google Q1, Google Q2, Bing Q1, Bing Q2). In the column marked "Relevance", enter "R" if you think the document is relevant. Enter "N" if you think the document is not relevant. Unaccessible Web pages are considered non-relevant. Add the spreadsheet to your homework Web page. Change the name of the spreadsheet to your last name followed by your first initial so that we don't wind up with a dozen files with the same filename.