org.terrier.utility.io

Class HadoopUtility

java.lang.Object

org.terrier.utility.io.HadoopUtility

public class HadoopUtility
extends Object

Utility class for the setting up and configuring of Terrier MapReduce jobs. General scheme for a Hadoop Job JobFactory jf = HadoopUtility.getJobFactory("TerrierJob"); JobConf jc = jf.newJob(); HadoopUtility.makeTerrierJob(jc); &47;&47; populate jc &47;&47; if an index is needed in the MR job: HadoopUtility.toHConfiguration(index, jc); Running rj = JobClient.runJob(jc); HadoopUtility.finishTerrierJob(jc); During a MR job, the configure method should call HadoopUtility.loadTerrierJob(jc); To obtain an index, Index index = HadoopUtility.fromHConfiguration(jc);

Since:

2.2.

Author:

Craig Macdonald

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HadoopUtility.MapBase<K1,V1,K2,V2> Abstract class that provides default configure and close methods for a Mapper.
static class	HadoopUtility.MapReduceBase<K1,V1,K2,V2,K3,V3> Handy base class for MapReduce jobs.
static class	HadoopUtility.ReduceBase<K1,V1,K2,V2> Abstract class that provides default configure and close methods for a Reducer.

Field Summary

Fields

Modifier and Type	Field and Description
protected static String[]	checkSystemProperties
protected static String	HADOOP_TMP_PATH
protected static org.slf4j.Logger	logger
protected static Random	random

Constructor Summary

Constructors

Constructor and Description
HadoopUtility()

Method Summary

Methods

Modifier and Type	Method and Description
protected static void	deleteJobApplicationSetup(org.apache.hadoop.mapred.JobConf jobConf)
protected static org.apache.hadoop.fs.Path	findCacheFileByFragment(org.apache.hadoop.mapred.JobConf jc, String name)
protected static String[]	findJarFiles(String[] classPathLines)
static void	finishTerrierJob(org.apache.hadoop.mapred.JobConf jobConf) Call this after the MapReduce job specified by jobConf has completed, to clean up any leftover files
static IndexOnDisk	fromHConfiguration(org.apache.hadoop.conf.Configuration c) Get an Index saved to the specifified Hadoop configuration by toHConfiguration()
static boolean	isMap(org.apache.hadoop.mapred.JobConf jc) Utility method to detect if a task is a Map task or not
protected static void	loadApplicationSetup(org.apache.hadoop.mapred.JobConf jobConf)
static void	loadTerrierJob(org.apache.hadoop.mapred.JobConf jobConf) When the current ApplicationSetup has been saved to the JobConf, by makeTerrierJob(), use this method during the MR job to properly initialise Terrier.
protected static org.apache.hadoop.fs.Path	makeTemporaryFile(org.apache.hadoop.mapred.JobConf jobConf, String filename)
static void	makeTerrierJob(org.apache.hadoop.mapred.JobConf jobConf) Saves the current ApplicationSetup to the specified JobConf.
protected static void	removeClassPathFromJob(org.apache.hadoop.mapred.JobConf jobConf)
protected static void	saveApplicationSetupToJob(org.apache.hadoop.mapred.JobConf jobConf, boolean getFreshProperties)
protected static void	saveClassPathToJob(org.apache.hadoop.mapred.JobConf jobConf)
static boolean	setJobOutputCompression(org.apache.hadoop.mapred.JobConf conf) Utility method to set JobOutputCompression if possible.
static boolean	setMapOutputCompression(org.apache.hadoop.mapred.JobConf conf) Utility method to set MapOutputCompression if possible.
protected static boolean	startsWithAny(String source, String[] checks) Returns true if source contains any of the Strings held in checks.
static void	toHConfiguration(Index i, org.apache.hadoop.conf.Configuration c) Puts the specified index onto the given Hadoop configuration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

HADOOP_TMP_PATH

protected static final String HADOOP_TMP_PATH

checkSystemProperties

protected static final String[] checkSystemProperties

random

protected static final Random random

Constructor Detail

HadoopUtility

public HadoopUtility()

Method Detail

isMap

public static final boolean isMap(org.apache.hadoop.mapred.JobConf jc)

Utility method to detect if a task is a Map task or not

setMapOutputCompression

public static boolean setMapOutputCompression(org.apache.hadoop.mapred.JobConf conf)

Utility method to set MapOutputCompression if possible. In general, I find that MapOutputCompression fails for local job trackers, so this code checks the job tracker location first.

Parameters:

conf - JobConf of job.

Returns:

true if MapOutputCompression was set.

setJobOutputCompression

public static boolean setJobOutputCompression(org.apache.hadoop.mapred.JobConf conf)

Utility method to set JobOutputCompression if possible. In general, I find that JobOutputCompression fails for local job trackers, so this code checks the job tracker location first.

Parameters:

conf - JobConf of job.

Returns:

true if JobOutputCompression was set.

makeTerrierJob

public static void makeTerrierJob(org.apache.hadoop.mapred.JobConf jobConf)
                           throws IOException

Saves the current ApplicationSetup to the specified JobConf. After the JobConf job has run, use finishTerrierJob() to delete any leftover files

Throws:

IOException

loadTerrierJob

public static void loadTerrierJob(org.apache.hadoop.mapred.JobConf jobConf)
                           throws IOException

When the current ApplicationSetup has been saved to the JobConf, by makeTerrierJob(), use this method during the MR job to properly initialise Terrier.

Throws:

IOException

finishTerrierJob

public static void finishTerrierJob(org.apache.hadoop.mapred.JobConf jobConf)
                             throws IOException

Call this after the MapReduce job specified by jobConf has completed, to clean up any leftover files

Throws:

IOException

removeClassPathFromJob

protected static void removeClassPathFromJob(org.apache.hadoop.mapred.JobConf jobConf)
                                      throws IOException

Throws:

IOException

saveClassPathToJob

protected static void saveClassPathToJob(org.apache.hadoop.mapred.JobConf jobConf)
                                  throws IOException

Throws:

IOException

findJarFiles

protected static String[] findJarFiles(String[] classPathLines)

makeTemporaryFile

protected static org.apache.hadoop.fs.Path makeTemporaryFile(org.apache.hadoop.mapred.JobConf jobConf,
                                          String filename)
                                                      throws IOException

Throws:

IOException

deleteJobApplicationSetup

protected static void deleteJobApplicationSetup(org.apache.hadoop.mapred.JobConf jobConf)
                                         throws IOException

Throws:

IOException

saveApplicationSetupToJob

protected static void saveApplicationSetupToJob(org.apache.hadoop.mapred.JobConf jobConf,
                             boolean getFreshProperties)
                                         throws Exception

Throws:

Exception

findCacheFileByFragment

protected static org.apache.hadoop.fs.Path findCacheFileByFragment(org.apache.hadoop.mapred.JobConf jc,
                                                String name)
                                                            throws IOException

Throws:

IOException

loadApplicationSetup

protected static void loadApplicationSetup(org.apache.hadoop.mapred.JobConf jobConf)
                                    throws IOException

Throws:

IOException

fromHConfiguration

public static IndexOnDisk fromHConfiguration(org.apache.hadoop.conf.Configuration c)

Get an Index saved to the specifified Hadoop configuration by toHConfiguration()

toHConfiguration

public static void toHConfiguration(Index i,
                    org.apache.hadoop.conf.Configuration c)

Puts the specified index onto the given Hadoop configuration

startsWithAny

protected static boolean startsWithAny(String source,
                    String[] checks)

Returns true if source contains any of the Strings held in checks. Case insensitive.

Parameters:

source - String to check

checks - Strings to check for

Returns:

true if source starts with one of checks, false otherwise.