org.terrier.utility.io

Class HadoopPlugin

java.lang.Object

org.terrier.utility.io.HadoopPlugin

All Implemented Interfaces:

ApplicationSetup.TerrierApplicationPlugin

public class HadoopPlugin
extends Object
implements ApplicationSetup.TerrierApplicationPlugin

This class provides the main glue between Terrier and Hadoop. It has several main roles:

1. Configure Terrier such that the Hadoop file systems can be accessed by Terrier.
2. Provide a means to access the Hadoop map-reduce cluster, using Hadoop on Demand (HOD) if necessary.

Configuring Terrier to access HDFS

Terrier can access a Hadoop Distributed File System (HDFS), allowing collections and indices to be placed there. To do so, ensure that your Hadoop conf/ is on your CLASSPATH, and that the Hadoop plugin is loaded by Terrier, by setting terrier.plugins=org.terrier.utility.io.HadoopPlugin in your terrier.properties file.

Since:

2.2

Author:

Craig Macdonald

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	config configuration used by this plugin
protected org.apache.hadoop.fs.FileSystem	hadoopFS distributed file system used by this plugin
protected static org.slf4j.Logger	logger The logger used
protected static org.apache.hadoop.conf.Configuration	singletonConfiguration main configuration object to use for Hadoop access
protected static HadoopPlugin	singletonHadoopPlugin instance of this class - it is a singleton

Constructor Summary

Constructors

Constructor and Description
HadoopPlugin() Constructs a new plugin

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.conf.Configuration	getConfiguration() Returns the Hadoop configuration underlying this plugin instance
static org.apache.hadoop.fs.FileSystem	getDefaultFileSystem() What is the default file system according to Hadoop
static String	getDefaultFileSystemPrefix() What is the String prefix of the default file system according to Hadoop
static URI	getDefaultFileSystemURI() What is the URI of the default file system according to Hadoop
static org.apache.hadoop.conf.Configuration	getGlobalConfiguration() Obtain the global Hadoop configuration in use by the plugin
void	initialise() Initialises the Plugin, by connecting to the distributed file system
static void	setGlobalConfiguration(org.apache.hadoop.conf.Configuration _config) Update the global Hadoop configuration in use by the plugin

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

singletonHadoopPlugin

protected static HadoopPlugin singletonHadoopPlugin

instance of this class - it is a singleton

singletonConfiguration

protected static org.apache.hadoop.conf.Configuration singletonConfiguration

main configuration object to use for Hadoop access

logger

protected static final org.slf4j.Logger logger

The logger used

config

protected org.apache.hadoop.conf.Configuration config

configuration used by this plugin

hadoopFS

protected org.apache.hadoop.fs.FileSystem hadoopFS

distributed file system used by this plugin

Constructor Detail

HadoopPlugin

public HadoopPlugin()

Constructs a new plugin

Method Detail

setGlobalConfiguration

public static void setGlobalConfiguration(org.apache.hadoop.conf.Configuration _config)

Update the global Hadoop configuration in use by the plugin

getGlobalConfiguration

public static org.apache.hadoop.conf.Configuration getGlobalConfiguration()

Obtain the global Hadoop configuration in use by the plugin

getDefaultFileSystemPrefix

public static String getDefaultFileSystemPrefix()

What is the String prefix of the default file system according to Hadoop

getDefaultFileSystemURI

public static URI getDefaultFileSystemURI()

What is the URI of the default file system according to Hadoop

getDefaultFileSystem

public static org.apache.hadoop.fs.FileSystem getDefaultFileSystem()
                                                            throws IOException

What is the default file system according to Hadoop

Throws:

IOException

initialise

public void initialise()
                throws Exception

Initialises the Plugin, by connecting to the distributed file system

Specified by:

initialise in interface ApplicationSetup.TerrierApplicationPlugin

Throws:

Exception

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

Returns the Hadoop configuration underlying this plugin instance