next up previous
Next: Closed World Assumption Up: Unstated Assumptions Previous: Users queries are ``well-formed''

Users make ``perfect'' choices

In the evaluation of RF systems, it is assumed that the user makes a ``perfect choice'' in that all documents that are relevant are marked as such. In operational settings, this is unlikely, and thus the empirical results only give an upper bound on performance. However, while a user may not select some relevant documents for feedback, it is typically assumed that the user will not mark an irrelevant document for feedback. In addition to revisiting feedback in the context of limited relevance information, it would seem that evaluating different feedback techniques in the context of noisy feedback information is warranted, especially as users interact with new, unfamiliar IR systems employing some form of RF.



Erik Selberg
Wed Aug 6 12:24:17 PDT 1997