Package org.terrier.evaluation

Class TRECQrelsInMemory

  • public class TRECQrelsInMemory
extends java.lang.Object
    Loads the relevance assessments in memory, for performing evaluation of runs.
    Author:
    Ben He & Vassilis Plachouras & Craig Macdonald

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  TRECQrelsInMemory.QrelsHashSet
      Models the set of relevant documents for one query.

    • Constructor Summary

      Constructors 
      Constructor Description
      TRECQrelsInMemory()
      A default constructor that creates an instance of the class and loads in memory the relevance assessments from the files that are specified in the file specified by the property TREC_QRELS.
      TRECQrelsInMemory​(java.lang.String qrelsFilename)
      A constructor that creates an instance of the class and loads in memory the relevance assessments from the given file.
      TRECQrelsInMemory​(java.lang.String[] qrelsFilenames)
      A constructor that creates an instance of the class and loads in memory the relevance assessments from the given files.

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      gnu.trove.THashSet<java.lang.String> getAllRelevantDocuments()
      Get the identifiers of all relevant documents in the pool.
      java.lang.String[] getAllRelevantDocumentsToArray()
      Get the identifiers of all relevant documents in the pool.
      int getGrade​(java.lang.String qid, java.lang.String docno, int def)
      Returns the grade of a document for a given query.
      gnu.trove.THashSet<java.lang.String> getNonRelevantDocuments()
      Get all the pooled non-relevant documents.
      gnu.trove.THashSet<java.lang.String> getNonRelevantDocuments​(java.lang.String queryid)
      Get the pooled non-relevant documents for the given query.
      java.lang.String[] getNonRelevantDocumentsToArray()
      Get all the pooled non-relevant documents.
      java.lang.String[] getNonRelevantDocumentsToArray​(java.lang.String queryid)
      Get the pooled non-relevant documents for a given query.
      int getNumberOfQueries()
      Returns the total number of queries contained in the loaded relevance assessments.
      int getNumberOfRelevant​(java.lang.String queryid)
      Returns the numbe of relevant documents for a given query.
      java.lang.String[] getQueryids()
      Get ids of the queries that appear in the pool.
      gnu.trove.THashSet<java.lang.String> getRelevantDocuments​(int grade)
      Get the identifiers of all relevant documents in the pool with a given relevance grade.
      java.lang.String[] getRelevantDocuments​(int[] grades)
      Get the identifiers of all relevant documents with the given relevance grades.
      gnu.trove.THashSet<java.lang.String> getRelevantDocuments​(java.lang.String queryid)
      Get the pooled relevant documents for the given query.
      gnu.trove.THashSet<java.lang.String> getRelevantDocuments​(java.lang.String queryid, int grade)
      Get the pooled relevant documents for the given query.
      gnu.trove.THashSet<java.lang.String> getRelevantDocuments​(java.lang.String queryid, int[] grades)
      Get the pooled relevant documents for the given query.
      java.lang.String[] getRelevantDocumentsToArray​(int grade)
      Get the identifiers of all relevant documents in the pool with the given relevance grade.
      java.lang.String[] getRelevantDocumentsToArray​(java.lang.String queryid)
      Get the pooled relevant documents for a given query.
      java.lang.String[] getRelevantDocumentsToArray​(java.lang.String queryid, int grade)
      Get the pooled relevant documents for a given query.
      java.lang.String[] getRelevantDocumentsToArray​(java.lang.String queryid, int[] grades)
      Get the pooled relevant documents for the given query.
      boolean isRelevant​(java.lang.String queryid, java.lang.String docno)
      Check if the given document is relevant for a given query.
      protected void loadQrelsFile()
      Load in memory the relevance assessment files that are specified in the array fqrels.
      static java.lang.String parseTRECQueryNo​(java.lang.String queryid)  
      boolean queryExistInQrels​(java.lang.String queryid)
      Checks whether there is a query with a given identifier in the relevance assessments.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • logger

        protected static final org.slf4j.Logger logger

      • qrelsPerQuery

        public TRECQrelsInMemory.QrelsHashSet[] qrelsPerQuery
        Each element in the array contains the docids of the relevant documents with respect to a query.

      • fqrels

        protected java.io.File[] fqrels
        An array with the qrels files.

      • totalNumberOfRelevantDocs

        public int totalNumberOfRelevantDocs
        The total number of relevant documents.

    • Constructor Detail

      • TRECQrelsInMemory

        public TRECQrelsInMemory​(java.lang.String qrelsFilename)
        A constructor that creates an instance of the class and loads in memory the relevance assessments from the given file.
        Parameters:
        qrelsFilename - String The full path of the qrels file to load.

      • TRECQrelsInMemory

        public TRECQrelsInMemory​(java.lang.String[] qrelsFilenames)
        A constructor that creates an instance of the class and loads in memory the relevance assessments from the given files.
        Parameters:
        qrelsFilenames - String[] The full path of the qrels files to load.

      • TRECQrelsInMemory

        public TRECQrelsInMemory()
        A default constructor that creates an instance of the class and loads in memory the relevance assessments from the files that are specified in the file specified by the property TREC_QRELS.

    • Method Detail

      • getQueryids

        public java.lang.String[] getQueryids()
        Get ids of the queries that appear in the pool.
        Returns:
        The ids of queries in the pool.

      • getRelevantDocuments

        public java.lang.String[] getRelevantDocuments​(int[] grades)
        Get the identifiers of all relevant documents with the given relevance grades.
        Parameters:
        grades - The relevance grades.
        Returns:
        The identifiers of all relevant documents with the given relevance grades.

      • getAllRelevantDocuments

        public gnu.trove.THashSet<java.lang.String> getAllRelevantDocuments()
        Get the identifiers of all relevant documents in the pool.
        Returns:
        The identifiers of all relevant documents in the pool.

      • getRelevantDocuments

        public gnu.trove.THashSet<java.lang.String> getRelevantDocuments​(int grade)
        Get the identifiers of all relevant documents in the pool with a given relevance grade.
        Returns:
        The identifiers of all relevant documents in the pool.

      • getAllRelevantDocumentsToArray

        public java.lang.String[] getAllRelevantDocumentsToArray()
        Get the identifiers of all relevant documents in the pool.
        Returns:
        The identifiers of all relevant documents in the pool.

      • getRelevantDocumentsToArray

        public java.lang.String[] getRelevantDocumentsToArray​(int grade)
        Get the identifiers of all relevant documents in the pool with the given relevance grade.
        Returns:
        The identifiers of all relevant documents in the pool.

      • getNonRelevantDocuments

        public gnu.trove.THashSet<java.lang.String> getNonRelevantDocuments​(java.lang.String queryid)
        Get the pooled non-relevant documents for the given query.
        Parameters:
        queryid - The id of the given query.
        Returns:
        A hashset containing the docnos of the pooled non-relevant documents for the given query.

      • getNonRelevantDocuments

        public gnu.trove.THashSet<java.lang.String> getNonRelevantDocuments()
        Get all the pooled non-relevant documents.
        Returns:
        All the pooled non-relevant documents.

      • getNonRelevantDocumentsToArray

        public java.lang.String[] getNonRelevantDocumentsToArray()
        Get all the pooled non-relevant documents.
        Returns:
        All the pooled non-relevant documents.

      • getRelevantDocuments

        public gnu.trove.THashSet<java.lang.String> getRelevantDocuments​(java.lang.String queryid)
        Get the pooled relevant documents for the given query.
        Parameters:
        queryid - The id of the given query.
        Returns:
        A hashset containing the docnos of the pooled relevant documents for the given query.

      • getRelevantDocuments

        public gnu.trove.THashSet<java.lang.String> getRelevantDocuments​(java.lang.String queryid,
                                                                 int grade)
        Get the pooled relevant documents for the given query.
        Parameters:
        queryid - The id of the given query.
        Returns:
        A hashset containing the docnos of the pooled relevant documents for the given query.

      • getRelevantDocuments

        public gnu.trove.THashSet<java.lang.String> getRelevantDocuments​(java.lang.String queryid,
                                                                 int[] grades)
        Get the pooled relevant documents for the given query.
        Parameters:
        queryid - The id of the given query.
        Returns:
        A hashset containing the docnos of the pooled relevant documents for the given query.

      • getRelevantDocumentsToArray

        public java.lang.String[] getRelevantDocumentsToArray​(java.lang.String queryid,
                                                      int[] grades)
        Get the pooled relevant documents for the given query.
        Parameters:
        queryid - The id of the given query.
        Returns:
        A hashset containing the docnos of the pooled relevant documents for the given query.

      • getNonRelevantDocumentsToArray

        public java.lang.String[] getNonRelevantDocumentsToArray​(java.lang.String queryid)
        Get the pooled non-relevant documents for a given query.
        Parameters:
        queryid - The id of the given query.
        Returns:
        An array of the docnos of the pooled non-relevant documents for the given query.

      • getRelevantDocumentsToArray

        public java.lang.String[] getRelevantDocumentsToArray​(java.lang.String queryid)
        Get the pooled relevant documents for a given query.
        Parameters:
        queryid - The id of the given query.
        Returns:
        An array of the docnos of the pooled relevant documents for the given query.

      • getRelevantDocumentsToArray

        public java.lang.String[] getRelevantDocumentsToArray​(java.lang.String queryid,
                                                      int grade)
        Get the pooled relevant documents for a given query.
        Parameters:
        queryid - The id of the given query.
        Returns:
        An array of the docnos of the pooled relevant documents for the given query.

      • getNumberOfQueries

        public int getNumberOfQueries()
        Returns the total number of queries contained in the loaded relevance assessments.
        Returns:
        int The number of unique queries in the loaded relevance assessments.

      • getNumberOfRelevant

        public int getNumberOfRelevant​(java.lang.String queryid)
        Returns the numbe of relevant documents for a given query.
        Parameters:
        queryid - String The identifier of a query.
        Returns:
        int The number of relevant documents for the given query.

      • loadQrelsFile

        protected void loadQrelsFile()
        Load in memory the relevance assessment files that are specified in the array fqrels.

      • parseTRECQueryNo

        public static java.lang.String parseTRECQueryNo​(java.lang.String queryid)

      • queryExistInQrels

        public boolean queryExistInQrels​(java.lang.String queryid)
        Checks whether there is a query with a given identifier in the relevance assessments.
        Parameters:
        queryid - String the identifier of a query.
        Returns:
        boolean true if the given query exists in the qrels file, otherwise it returns false.

      • isRelevant

        public boolean isRelevant​(java.lang.String queryid,
                          java.lang.String docno)
        Check if the given document is relevant for a given query.
        Parameters:
        queryid - String a query identifier.
        docno - String a document identifier.
        Returns:
        boolean true if the given document is relevant for the given query, or otherwise false.

      • getGrade

        public int getGrade​(java.lang.String qid,
                    java.lang.String docno,
                    int def)
        Returns the grade of a document for a given query.
        Parameters:
        qid - String the identifier of the desired query.
        docno - String a document identifier.
        def - The default value to be returned if the document is not found.
        Returns:
        int The grade of the document for the given query.