Term Weighting in a Probabilistic Model

 

Haines and Croft implement Relevance Feedback using an inference network model in the INQUERY system [16]. They accomplish this by attaching new Query nodes to new terms and by altering existing nodes' link matrices, although it is not specified in their paper how they do this. Once the nodes are added and the matrices adjusted, they are able to recalculate and repropagate all probabilities to the I node. This cycle is repeated until the user is satisfied with the results.

In order to evaluate if Relevance Feedback can be implemented in an inference network model and what degree of improvement it could have, Haines and Croft carry out an extensive set of experiments. They evaluate their system using two test collections: the CACM collection comprised of 3,204 titles and abstracts with two different 50 query sets, the second being a custom set, and the WEST collection, containing 11,953 full text legal documents and the standard 34 query set. Their system is evaluated on several axes. They examine a variety of methods to select terms to add to the query. Furthermore, based on Harman's result described in Section 4.2 as well as subsequent results [18], they experiment with adding between 0 and 150 terms for the CACM collection and 0 to 100 terms for the WEST collection. These results will be discussed more extensively in Section 5.2.

In addition, they also carry out some experiments using structured queries. The results from these experiments are beyond this paper's scope to document fully, but they do confirm that Relevance Feedback can done on structured queries, but the performance increase isn't as large as keyword or natural language queries.

Haines and Croft also examine methods of term weighting, which in their model correspond to re-estimating the probabilities given prior knowledge. They make use of and as defined in Section 4.2. They choose to calculate the weight of term using one of the following two functions:

rtf

Frequency of the term in relevant documents.
 

rtf * idf

rtf multiplied by the Inverse Document Frequency (idf). The idf factor attempts to correct for terms found in both relevant and irrelevant documents.
 

A selection of their results are summarized in

 
Table 2:  Average percentage increase in performance in INQUERY.

Table 2, which measure the performance of each selection formula by the average increase in precision at a fixed level of recall. The conclusions they are able to reach from their experiments regarding term weighting are:

1. Term reweighting increases performance;
2. The increase in performance by term weighting is dependent on the collection used.

In particular, it is interesting that Haines and Croft are able to obtain an 83.6% increase in performance on average using the CACM collection of abstracts, compared with just a 30.5% increase using the full-text WEST collection.

In addition, they also examine whether it is better to change the relative weights of pre-existing terms and new terms, rather than to evaluate each term's weight regardless of whether it is selected initially or subsequently. In vector model terms, they evaluate Equation (8) using different values for and . They find that the best overall relative weighting is 65%/35% old/new. However, results from 60%/40% through 80%/20% were comparable. The conclusions reached from this experiment indicate that weighting older terms more than newer terms does increase performance, but it is unclear what those weightings should be.

The results obtained by Haines and Croft demonstrate that Relevance Feedback can be adapted to probabilistic models, including those that use a more structural form than the mathematical models originally suggested by Robertson and Sparck Jones [38]. Furthermore, they indicate that term weighting also affects probabilistic models, and that judicious use of term weighting can dramatically affect performance.