[TR-301] Negative requirement not functioning correctly Created: 29/Apr/14  Updated: 16/Jun/14  Resolved: 27/May/14

Status: Resolved
Project: Terrier Core
Component/s: None
Affects Version/s: 3.6
Fix Version/s: 4.0

Type: Bug Priority: Major
Reporter: Craig Macdonald Assignee: Richard McCreadie
Resolution: Fixed  
Labels: None

Attachments: Text File TestManager.java    

 Description   
Documents:
"The quick brown fox jumps over the lazy dog", //doc1
"how much is that dog in the window" //doc2

Query:
 "dog -fox"

Erroneously returns two results. I think Manager.runPreprocessing() should encompass negative requirements?

Attached is an updated TestManager.java which demonstrates the problem.

 Comments   
Comment by Richard McCreadie [ 22/May/14 ]

I can confirm that these types of queries do not work. I don't think that they have ever been supported.

While RequirementQuery specifies that it should work with both '+' and '-' terms, I think that it only covers the '+' case. q.getTermsOf(RequirementQuery.class, requirement_list, true); in Manager returns nothing for '-' queries.

I guess that the antlr compiled lexer will need to be updated, followed by a NegativeRequirmentQuery class.

Comment by Richard McCreadie [ 27/May/14 ]

Added a fix and additional unit test for negative requirements in fields. Committed r3871

Main changes to core below:

RequirementQuery.java
public void getTermsOf(Class<? extends Query> c, List<Query> alist, boolean req) {		
		if (PhraseQuery.class.isInstance(child) && !MustHave)
			return;

		int required = 0;
		if (this.toString().startsWith("+")) required=1;
		if (this.toString().startsWith("-")) required=-1;

		//System.err.println("Requirement for "+this.toString()+" = "+required);
		List<Query> termsOfThisType = new ArrayList<Query>();
		if (c.isInstance(this)) this.getTerms(termsOfThisType);
		child.getTermsOf(c, termsOfThisType, req==MustHave);
		try {
			for (Query q : termsOfThisType) {
				((SingleTermQuery)q).required=required;
			}
		} catch (Exception e) {
			System.err.println("RequirementQuery: Failed to set query requirement, this probably failed since getTerms() returned a query type other than SingleTermQuery");
			e.printStackTrace();
		}
		alist.addAll(termsOfThisType);
	}
Manager.java
// Issue TREC-370
q.getTermsOf(RequirementQuery.class, requirement_list_all, true);
for (Query query : requirement_list_all ) {
	if (((SingleTermQuery)query).required>=0) {
		requirement_list_positive.add(query);
	}
	else {
		requirement_list_negative.add(query);
	}
}
for (Query negativeQuery : requirement_list_negative) {
	//System.err.println(negativeQuery.toString()+" was a negative requirement");
	mqt.setTermProperty(negativeQuery.toString(), Double.NEGATIVE_INFINITY);
}


......


mqt.setQuery(q);
mqt.normaliseTermWeights();
try{
	ResultSet outRs = matching.match(rq.getQueryID(), mqt);
				
	//check to see if we have any negative infinity scores that should be removed
	int badDocuments = 0;
	for (int i = 0; i < outRs.getResultSize(); i++) {
		if (outRs.getScores()[i] == Double.NEGATIVE_INFINITY)
			badDocuments++;
	}
	logger.debug("Found "+badDocuments+" documents with a score of negative infinity in the result set returned, they will be removed.");
				
	//now crop the collectionresultset down to a query result set.
	rq.setResultSet(outRs.getResultSet(0, outRs.getResultSize()-badDocuments));
} catch (IOException ioe) {
	logger.error("Problem running Matching, returning empty result set as query"+rq.getQueryID(), ioe);
	rq.setResultSet(new QueryResultSet(0));
}
Comment by Richard McCreadie [ 27/May/14 ]

Unit tests pass

Generated at Sun Dec 17 13:58:36 GMT 2017 using JIRA 7.1.1#71004-sha1:d6b2c0d9b7051e9fb5e4eb8ce177ca56d91d7bd8.