One of the common use-cases for Terrier is as a search component within a larger application. For example, you might want a custom search service within an email management system, or use the search results produced for one or more queries as input to another system, such as a classifier. This page will describe a quick way to integrate Terrier into an existing Java application, programatically index documents and issue search requests.
In Terrier, the textual items to be indexed, e.g. emails or tweets, are represented using the Document class. FileDocument is a useful Document class implementation that can wrap arbitrary text strings.
String text = "It is easy to index documents in Terrier";
Document document = new FileDocument(new StringReader(text), new HashMap(), Tokeniser.getTokeniser());
An Index is the storage structure that saves (indexes) each document. MemoryIndex is a simple type of index structure to get started with. It simply stores each document in your local machine's memory.
MemoryIndex memIndex = new MemoryIndex();
MemoryIndex belongs to a class of updatable indices, which means that it implements an indexDocument() method, allowing us to add new documents to the index via a single line of code.
memIndex.indexDocument(document);
To search our index, we need to use a querying Manager, which does the work of scoring each document for your query. Your query is stored in a SearchRequest object that the Manager can generate for you. We also need to specify which scoring function to use when ranking documents via the addMatchingModel() method (in this case we are using BM25).
Manager queryingManager = new Manager(memIndex);
SearchRequest srq = queryingManager.newSearchRequestFromQuery("my terrier query");
srq.addMatchingModel("Matching","BM25");
Finally, we issue the search:
queryingManager.runSearchRequest(srq);
Once the search is finished, the results are stored in the SearchRequest as a ResultSet.
ResultSet results = srq.getResultSet();
If you don't have Java 1.8 or later, download the current Java Development Kit (JDK) here. To check if you have a compatible version of Java installed, use the following command:
java --version
Apache Maven is a dependency management and automated build tool for Java-based projects. Install or update Maven to the latest release following their instructions for your system. Maven is widely used among Java developers and is the recommended way of integrating Terrier into your project, as it handles the import process for all of Terrier's dependencies automatically. Terrier requires Maven 3. You can also use other build tools such as Ivy and Gradle (however the use of these other build managers is not covered in the Terrier documentation).
As mentioned above, to import Terrier into your application we recommend that you use the Maven dependency manager. In effect, Maven is an alternative way to build your project that uses a special pom.xml file to control the importing of other pieces of software that your project depends on at compile-time. If you are unfamiliar with Maven, then try working through this tutorial.
If your project is not yet a Maven project, then we recommend that you convert it to one, following the steps in the above tutorial. If you are using an integrated development environment (IDE) such as Eclipse or IntelliJ, then they support plugins to help you use Maven from within the IDE, such as M2Eclipse.
Once you have your project setup with Maven, its time to add Terrier to your project. To do this, you will need to modify the Maven pom.xml file. In particular, you will need to copy/paste the following dependency definition into the <dependencies>...</dependencies>
block of your pom.xml (if you do not have a dependencies block, you will need to add one).
<dependency>
<groupId>org.terrier</groupId>
<artifactId>terrier-core</artifactId>
<version>4.2</version>
</dependency>
Save the pom.xml file and then trigger the building of your project. How you do this will depend on whether you are using the command line or an IDE. During the build process, Terrier along with all of its dependencies will be downloaded and compiled. If you are using an IDE, then Terrier and its dependencies should also be automatically added to your Java classpath. At this point you should be ready to start coding!
The first step when creating an index is to convert each file or piece of text we want to search into a form that Terrier can understand. This is done by converting all files and/or pieces of text into Terrier Documents. A Terrier Document is comprised of three main components:
There are a variety of different Terrier Document implementations provided out-of-the-box. The reason for having different Document implementations is to make extracting (useful) text/metadata from different types of common document formats easier. FileDocument is the simplest Document implementation, it simply stores all text read from an input reader. In contrast, TaggedDocument is designed to perform text extraction from html/xml tagged documents and as such, only stores texts within tags named by the user. For instance, given the following text file (test.html):
<html>
<head></head>
<body>
<p>This is a sample HTML document</p>
</body>
</html>
We can convert this to a Terrier Document using the FileDocument class as follows:
Document document = new FileDocument(Files.openFileReader("test.html"), new HashMap(), Tokeniser.getTokeniser());
However, if we do so the stored text String will read:
"<html><head></head><body><p>This is a sample HTML document</p></body></html>"
On the other hand, if we instead convert this to a Terrier Document using the TaggedDocument class the stored text string will read:
"This is a sample HTML document"
since TaggedDocument (by default) only stores text contained within the tags, not the tags themselves. In practice, we recommend that you use a Document implementation that stores only the text that you are interested in searching for within each document, as this will often result in better search effectiveness, consume less memory and make searches faster.
When we add a document to a Terrier index, it is automatically assigned a numeric (integer) identifier (known as a docid). However, in practice, we nearly always want to define a different identifier for each document that is more meaningful. For instance, when indexing a collection of webpages, we might want to refer to each page by its URL. By default, Terrier usually assumes there is a docno which serves as each document's unique identifier.
To do this, we need to do two things:
The Terrier configuration is stored in a static class called ApplicationSetup. We can add new configuration settings to Terrier using the ApplicationSetup.setProperty(key, value)
method. Earlier, we noted that each Terrier Document optionally has a String->String map that contains metadata about that document. When a document is indexed, Terrier checks the keys in this map against a pre-defined list of keys to store. If it finds a match, it records the value of that key within the index itself. You can specify which keys to store via the indexer.meta.forward.keys
and indexer.meta.forward.keylens
properties as shown below:
ApplicationSetup.setProperty("indexer.meta.forward.keys", "docno");
ApplicationSetup.setProperty("indexer.meta.forward.keylens", "30");
In this case, we are specifying that each document will have a key called 'docno' that we want to store and that the maximum 'docno' length is 30 characters. Both indexer.meta.forward.keys
and indexer.meta.forward.keylens
are comma-delimited lists, such that you can specify multiple keys to store.
Troubleshooting Tips:
- ApplicationSetup.setProperty() must be called before the index is initialized, i.e. before
new MemoryIndex()
is calledindexer.meta.forward.keys
andindexer.meta.forward.keylens
must have the same number of entries
Once we have configured Terrier, we also need to add the new identifiers to each document. Assuming we have already created a document called 'document', we can add an identifier as follows:
document.getAllProperties().put("docno", "This-is-test.html");
Now that we know how to convert files or pieces of text into Terrier Documents, the next step it to create an index and add those documents to it. The index is a storage structure that contains the documents that are to be made available for search. Indices extend the Index class in Terrier. There are three categories of index currently supported in Terrier, namely: IndexOnDisk, IncrementalIndex and MemoryIndex. For this quickstart we will be focusing on the MemoryIndex.
MemoryIndex is a convenient index class to use when you have a relatively small number of documents that you need to search over, e.g. 10,000 to about 100,000 web pages. The memory index stores documents as a series of arrays in local memory (RAM).
A new memory index can be made using the default MemoryIndex constructor:
MemoryIndex memIndex = new MemoryIndex();
Once we have an index, new documents can be easily added using the indexDocument()
method:
memIndex.indexDocument(document);
import java.io.File;
import java.io.FileReader;
import java.util.HashMap;
import org.terrier.indexing.Document;
import org.terrier.indexing.TaggedDocument;
import org.terrier.indexing.tokenisation.Tokeniser;
import org.terrier.realtime.memory.MemoryIndex;
import org.terrier.utility.ApplicationSetup;
public class IndexingExample {
public static void main(String[] args) throws Exception {
// Directory containing files to index
String aDirectoryToIndex = "/my/directory/containing/files/";
// Configure Terrier
ApplicationSetup.setProperty("indexer.meta.forward.keys", "docno");
ApplicationSetup.setProperty("indexer.meta.forward.keylens", "30");
// Create a new Index
MemoryIndex memIndex = new MemoryIndex();
// For each file
for (String filename : new File(aDirectoryToIndex).list() ) {
String fullPath = aDirectoryToIndex+filename;
// Convert it to a Terrier Document
Document document = new TaggedDocument(Files.openFileReader(fullPath), new HashMap(), Tokeniser.getTokeniser());
// Add a meaningful identifier
document.getAllProperties().put("docno", filename);
// index it
memIndex.indexDocument(document);
}
}
}
Before you start setting up searches and running them, it is important to enable any enhancements or transformations that we want Terrier to apply to the search results that will be generated. There are a variety of enhancements/transformations that Terrier can apply out-of-the-box, such as re-ranking the results based on user-defined features or generating query-biased document snippets.
For this quickstart guide, we will cover enabling one very common type of search result enhancement, which is known as decoration. The goal of decoration is to copy metadata about each individual document into the search results returned, such as the 'docno' identifier that we configured earlier. Decorate (or more precisely the org.terrier.querying.SimpleDecorate
class) belongs to a group of functions in Terrier known as post filters. To enable decorate functionality, we need to update the Terrier configuration that is stored in the static ApplicationSetup class. In particular, there are two configuration parameters that need to be set: querying.postfilters.controls
and querying.postfilters.order
. Simply, these parameters specify the 'what' and 'when' any post filters should be applied. querying.postfilters.controls
is a comma delimited list of name:class
pairs, where name
is a short identifier for a post filter and class
is the full classname. Meanwhile. querying.postfilters.order
is a comma delimited list of post filter class names, where the order of the class names defines the order in which they are executed. Hence, we can enable the org.terrier.querying.SimpleDecorate
post filter class by setting the Terrier configuration as follows:
ApplicationSetup.setProperty("querying.postfilters.controls", "decorate:org.terrier.querying.SimpleDecorate");
ApplicationSetup.setProperty("querying.postfilters.order", "org.terrier.querying.SimpleDecorate");
In this case, we have specified that org.terrier.querying.SimpleDecorate is a post filter we want to have access to, we have given it the name i.e. 'decorate' and we have added it to the list of filters to run.
Troubleshooting Tips:
- ApplicationSetup.setProperty() must be called before the search Manager is initialized, i.e. before
new Manager(index)
is called.- A post filter must appear in both
querying.postfilters.controls
andquerying.postfilters.order
before it will be used.
Within Terrier, searches are performed using a Manager class. This class performs the nuts and bolts of actually matching your query against the documents that were indexed. If you are running multiple queries, you need to only create a single manager and use it multiple times. There is only one Manager implementation in Terrier, which you can instantiate as follows:
Manager queryingManager = new Manager(memIndex);
This creates a new querying manager with a default configuration and sets the index to be searched (to our 'memindex' in this case). The next step in the process is to create a SearchRequest, which contains both our query as well as some other information about how we want the search to be processed. The Manager can generate a SearchRequest for you with default settings as shown below:
SearchRequest srq = queryingManager.newSearchRequestFromQuery("sample query");
In this case we have created a new SearchRequest for the query 'sample query'. The SearchRequest will have reasonable defaults for running a basic search. However, there is two configuration options that we need to manually set. First, we need to set which scoring function to use when ranking documents. This is done via the addMatchingModel() method as shown below:
srq.addMatchingModel("Matching","BM25");
In this case we are using BM25, a classical model from the Best Match familty of document weighting models. Second, we need to specify in the SearchRequest that we want to use the post filter we enabled above named 'decorate':
srq.setControl("decorate", "on");
Finally, we issue the search:
queryingManager.runSearchRequest(srq);
Once finished, the results are stored in the SearchRequest as a ResultSet, which can be accessed via:
ResultSet results = srq.getResultSet();
The output of a search in Terrier is known as a ResultSet. The ResultSet contains five main pieces of information:
import java.io.File;
import java.io.FileReader;
import java.util.HashMap;
import org.terrier.indexing.Document;
import org.terrier.indexing.TaggedDocument;
import org.terrier.indexing.tokenisation.Tokeniser;
import org.terrier.matching.ResultSet;
import org.terrier.querying.Manager;
import org.terrier.querying.SearchRequest;
import org.terrier.realtime.memory.MemoryIndex;
import org.terrier.utility.ApplicationSetup;
import org.terrier.utility.Files;
public class IndexingAndRetrievalExample {
public static void main(String[] args) throws Exception {
// Directory containing files to index
String aDirectoryToIndex = "/my/directory/containing/files/";
// Configure Terrier
ApplicationSetup.setProperty("indexer.meta.forward.keys", "docno");
ApplicationSetup.setProperty("indexer.meta.forward.keylens", "30");
// Create a new Index
MemoryIndex memIndex = new MemoryIndex();
// For each file
for (String filename : new File(aDirectoryToIndex).list() ) {
String fullPath = aDirectoryToIndex+filename;
// Convert it to a Terrier Document
Document document = new TaggedDocument(Files.openFileReader(fullPath), new HashMap(), Tokeniser.getTokeniser());
// Add a meaningful identifier
document.getAllProperties().put("docno", filename);
// index it
memIndex.indexDocument(document);
}
// Enable the decorate enhancement
ApplicationSetup.setProperty("querying.postfilters.order", "org.terrier.querying.SimpleDecorate");
ApplicationSetup.setProperty("querying.postfilters.controls", "decorate:org.terrier.querying.SimpleDecorate");
// Create a new manager run queries
Manager queryingManager = new Manager(memIndex);
// Create a search request
SearchRequest srq = queryingManager.newSearchRequestFromQuery("search for document");
// Specify the model to use when searching
srq.addMatchingModel("Matching","BM25");
// Turn on decoration for this search request
srq.setControl("decorate", "on");
// Run the search
queryingManager.runSearchRequest(srq);
// Get the result set
ResultSet results = srq.getResultSet();
// Print the results
System.out.println(results.getExactResultSize()+" documents were scored");
System.out.println("The top "+results.getResultSize()+" of those documents were returned");
System.out.println("Document Ranking");
for (int i =0; i< results.getResultSize(); i++) {
int docid = results.getDocids()[i];
double score = results.getScores()[i];
System.out.println(" Rank "+i+": "+docid+" "+results.getMetaItem("docno", docid)+" "+score);
}
}
}
Webpage: http://terrier.org
Contact: School of Computing Science
Copyright (C) 2004-2016 University of Glasgow. All Rights Reserved.