p2pMapReduce.mapreduceModule
Class Job

java.lang.Object
  extended by p2pMapReduce.mapreduceModule.JobContextImpl
      extended by p2pMapReduce.mapreduceModule.Job
All Implemented Interfaces:
java.io.Externalizable, java.io.Serializable, JobContext, MRJobConfig

public class Job
extends JobContextImpl
implements JobContext, java.io.Externalizable

The job submitter's view of the Job.

It allows the user to configure the job, submit it, control its execution, and query the state. The set methods only work until the job is submitted, afterwards they will throw an IllegalStateException.

Normally the user creates the application, describes various facets of the job via Job and then submits the job and monitor its progress.

Here is an example on how to submit a job:

     // Create a new Job
     Job job = new Job(new Configuration());
     job.setJarByClass(MyJob.class);
     
     // Specify various job-specific parameters     
     job.setJobName("myjob");
     
     job.setInputPath(new Path("in"));
     job.setOutputPath(new Path("out"));
     
     job.setMapperClass(MyJob.MyMapper.class);
     job.setReducerClass(MyJob.MyReducer.class);

     // Submit the job, then poll for progress until the job is complete
     job.waitForCompletion(true);
 

Note: must implement Externalizable to provide custom serialization method to manage super type data load and restore.

See Also:
Serialized Form

Nested Class Summary
static class Job.JobState
           
static class Job.TaskStatusFilter
           
 
Field Summary
static java.lang.String COMPLETION_POLL_INTERVAL_KEY
          Key in mapred-*.xml that sets completionPollInvervalMillis
static java.lang.String OUTPUT_FILTER
           
static java.lang.String PROGRESS_MONITOR_POLL_INTERVAL_KEY
          Key in mapred-*.xml that sets progMonitorPollIntervalMillis
static java.lang.String SUBMIT_REPLICATION
           
static java.lang.String USED_GENERIC_PARSER
           
 
Fields inherited from class p2pMapReduce.mapreduceModule.JobContextImpl
conf
 
Fields inherited from interface p2pMapReduce.mapreduceModule.MRJobConfig
CACHE_ARCHIVES, CACHE_ARCHIVES_SIZES, CACHE_ARCHIVES_TIMESTAMPS, CACHE_ARCHIVES_VISIBILITIES, CACHE_FILE_TIMESTAMPS, CACHE_FILE_VISIBILITIES, CACHE_FILES, CACHE_FILES_SIZES, CACHE_LOCALARCHIVES, CACHE_LOCALFILES, CACHE_SYMLINK, CLASSPATH_ARCHIVES, CLASSPATH_FILES, COMBINE_CLASS_ATTR, COMPLETED_MAPS_FOR_REDUCE_SLOWSTART, END_NOTIFICATION_RETRIE_INTERVAL, END_NOTIFICATION_RETRIES, END_NOTIFICATION_URL, GROUP_COMPARATOR_CLASS, HISTORY_LOCATION, ID, INPUT_FORMAT_CLASS_ATTR, INPUTSPLIT_CLASS_ATTR, INPUTSPLIT_SIZE, IO_SORT_FACTOR, IO_SORT_MB, JAR, JAR_UNPACK_PATTERN, JOB_ACL_MODIFY_JOB, JOB_ACL_VIEW_JOB, JOB_CANCEL_DELEGATION_TOKEN, JOB_DIR_ATTR, JOB_INPUT_DIR_ATTR, JOB_INPUTSPLIT_DIR_ATTR, JOB_JAR_DIR_ATTR, JOB_JOBTRACKER_ID, JOB_LOCAL_DIR, JOB_NAME, JOB_NAMENODES, JOB_OUTPUT_DIR_ATTR, JVM_NUMTASKS_TORUN, KEY_COMPARATOR, MAP_CLASS_ATTR, MAP_COMBINE_MIN_SPILLS, MAP_DEBUG_SCRIPT, MAP_ENV, MAP_FAILURES_MAX_PERCENT, MAP_INPUT_FILE, MAP_INPUT_PATH, MAP_INPUT_START, MAP_JAVA_OPTS, MAP_LOG_LEVEL, MAP_MAX_ATTEMPTS, MAP_MEMORY_MB, MAP_MEMORY_PHYSICAL_MB, MAP_OUTPUT_COMPRESS, MAP_OUTPUT_COMPRESS_CODEC, MAP_OUTPUT_DIR_ATTR, MAP_OUTPUT_KEY_CLASS, MAP_OUTPUT_KEY_FIELD_SEPERATOR, MAP_OUTPUT_KEY_VALUE_SEPARATOR_ATTR, MAP_OUTPUT_VALUE_CLASS, MAP_SKIP_INCR_PROC_COUNT, MAP_SKIP_MAX_RECORDS, MAP_SORT_SPILL_PERCENT, MAP_SPECULATIVE, MAP_ULIMIT, MAX_TASK_FAILURES_PER_TRACKER, NUM_MAP_PROFILES, NUM_MAPS, NUM_REDUCE_PROFILES, NUM_REDUCES, OUTPUT_FORMAT_CLASS_ATTR, OUTPUT_KEY_CLASS, OUTPUT_VALUE_CLASS, PARTITIONER_CLASS_ATTR, PRESERVE_FAILED_TASK_FILES, PRESERVE_FILES_PATTERN, PRIORITY, QUEUE_NAME, RECORDS_BEFORE_PROGRESS, REDUCE_CLASS_ATTR, REDUCE_DEBUG_SCRIPT, REDUCE_ENV, REDUCE_FAILURES_MAXPERCENT, REDUCE_INPUT_BUFFER_PERCENT, REDUCE_INPUT_DIR_ATTR, REDUCE_JAVA_OPTS, REDUCE_LOG_LEVEL, REDUCE_MARKRESET_BUFFER_PERCENT, REDUCE_MARKRESET_BUFFER_SIZE, REDUCE_MAX_ATTEMPTS, REDUCE_MEMORY_MB, REDUCE_MEMORY_PHYSICAL_MB, REDUCE_MEMORY_TOTAL_BYTES, REDUCE_MEMTOMEM_ENABLED, REDUCE_MEMTOMEM_THRESHOLD, REDUCE_MERGE_INMEM_THRESHOLD, REDUCE_OUTPUT_DIR_ATTR, REDUCE_SKIP_INCR_PROC_COUNT, REDUCE_SKIP_MAXGROUPS, REDUCE_SPECULATIVE, REDUCE_ULIMIT, SETUP_CLEANUP_NEEDED, SHUFFLE_CONNECT_TIMEOUT, SHUFFLE_FETCH_FAILURES, SHUFFLE_INPUT_BUFFER_PERCENT, SHUFFLE_MERGE_EPRCENT, SHUFFLE_NOTIFY_READERROR, SHUFFLE_PARALLEL_COPIES, SHUFFLE_READ_TIMEOUT, SKIP_OUTDIR, SKIP_RECORDS, SKIP_START_ATTEMPTS, SPECULATIVE_SLOWNODE_THRESHOLD, SPECULATIVE_SLOWTASK_THRESHOLD, SPECULATIVECAP, SPLIT_FILE, TASK_ATTEMPT_ID, TASK_DEBUGOUT_LINES, TASK_ID, TASK_ISMAP, TASK_OUTPUT_DIR, TASK_PARTITION, TASK_PROFILE, TASK_PROFILE_PARAMS, TASK_TEMP_DIR, TASK_TIMEOUT, TASK_USERLOG_LIMIT, USER_LOG_RETAIN_HOURS, USER_NAME, WORKING_DIR
 
Constructor Summary
Job()
           
Job(Configuration conf, JobID jobID)
           
 
Method Summary
 float cleanupProgress()
          Get the progress of the job's cleanup-tasks, as a float between 0.0 and 1.0.
static int getCompletionPollInterval(Configuration conf)
          The interval at which waitForCompletion() should check.
 long getFinishTime()
          Get finish time of the job.
 java.lang.String getHistoryUrl()
           
 java.lang.String getJobFile()
          Get the path of the submitted job configuration.
 java.lang.String getJobName()
          The user-specified job name.
 JobStatus.State getJobState()
          Returns the current state of the Job.
static int getProgressPollInterval(Configuration conf)
          The interval at which monitorAndPrintJob() prints status
 java.lang.String getSchedulingInfo()
          Get scheduling info of the job.
 long getStartTime()
          Get start time of the job.
 JobStatus getStatus()
           
static Job.TaskStatusFilter getTaskOutputFilter(Configuration conf)
          Get the task output filter.
 java.lang.String getTrackingURL()
          Get the URL where some job progress information will be displayed.
 boolean isComplete()
          Check if the job is finished or not.
 boolean isRetired()
           
 boolean isSuccessful()
          Check if the job completed successfully.
 void killJob()
          Kill the running job.
 float mapProgress()
          Get the progress of the job's map-tasks, as a float between 0.0 and 1.0.
 void readExternal(java.io.ObjectInput in)
           
 float reduceProgress()
          Get the progress of the job's reduce-tasks, as a float between 0.0 and 1.0.
 void setCancelDelegationTokenUponJobCompletion(boolean value)
          Sets the flag that will allow the JobTracker to cancel the HDFS delegation tokens upon job completion.
 void setCombinerClass(java.lang.Class<? extends Reducer> cls)
          Set the combiner class for the job.
 void setGroupingComparatorClass(java.lang.Class<? extends RawComparator> cls)
          Define the comparator that controls which keys are grouped together for a single call to Reducer#reduce(Object, Iterable, org.apache.hadoop.mapreduce.Reducer.Context)
 void setInputFormatClass(java.lang.Class<? extends InputFormat> cls)
          Set the InputFormat for the job.
 void setJar(java.lang.String jar)
          Set the job jar
 void setJarByClass(java.lang.Class<?> cls)
          Set the Jar by finding where a given class came from.
 void setJobName(java.lang.String name)
          Set the user-specified job name.
 void setJobSetupCleanupNeeded(boolean needed)
          Specify whether job-setup and job-cleanup is needed for the job
 void setJobState(JobStatus.State state)
           
 void setMapOutputKeyClass(java.lang.Class<?> theClass)
          Set the key class for the map output data.
 void setMapOutputValueClass(java.lang.Class<?> theClass)
          Set the value class for the map output data.
 void setMapperClass(java.lang.Class<? extends Mapper> cls)
          Set the Mapper for the job.
 void setMapSpeculativeExecution(boolean speculativeExecution)
          Turn speculative execution on or off for this job for map tasks.
 void setMaxMapAttempts(int n)
          Expert: Set the number of maximum attempts that will be made to run a map task.
 void setMaxReduceAttempts(int n)
          Expert: Set the number of maximum attempts that will be made to run a reduce task.
 void setNumReduceTasks(int tasks)
          Set the number of reduce tasks for the job.
 void setOutputFormatClass(java.lang.Class<? extends OutputFormat> cls)
          Set the OutputFormat for the job.
 void setOutputKeyClass(java.lang.Class<?> theClass)
          Set the key class for the job output data.
 void setOutputValueClass(java.lang.Class<?> theClass)
          Set the value class for job outputs.
 void setPartitionerClass(java.lang.Class<? extends Partitioner> cls)
          Set the Partitioner for the job.
 void setProfileEnabled(boolean newValue)
          Set whether the system should collect profiler information for some of the tasks in this job?
 void setProfileParams(java.lang.String value)
          Set the profiler configuration arguments.
 void setProfileTaskRange(boolean isMap, java.lang.String newValue)
          Set the ranges of maps or reduces to profile. setProfileEnabled(true) must also be called.
 void setReducerClass(java.lang.Class<? extends Reducer> cls)
          Set the Reducer for the job.
 void setReduceSpeculativeExecution(boolean speculativeExecution)
          Turn speculative execution on or off for this job for reduce tasks.
 void setSortComparatorClass(java.lang.Class<? extends RawComparator> cls)
          Define the comparator that controls how the keys are sorted before they are passed to the Reducer.
 void setSpeculativeExecution(boolean speculativeExecution)
          Turn speculative execution on or off for this job.
static void setTaskOutputFilter(Configuration conf, Job.TaskStatusFilter newValue)
          Modify the Configuration to set the task output filter.
 float setupProgress()
          Get the progress of the job's setup-tasks, as a float between 0.0 and 1.0.
 void setUser(java.lang.String user)
          Set the reported username for this job.
 void setWorkingDirectory(Path dir)
          Set the current working directory for the default file system.
 java.lang.String toString()
          Dump stats to screen.
 boolean waitForCompletion(boolean verbose)
          Submit the job to the cluster and wait for it to finish.
 void writeExternal(java.io.ObjectOutput out)
           
 
Methods inherited from class p2pMapReduce.mapreduceModule.JobContextImpl
getCombinerClass, getConfiguration, getGroupingComparator, getInputFormatClass, getJar, getJobID, getJobSetupCleanupNeeded, getMapOutputKeyClass, getMapOutputValueClass, getMapperClass, getMaxMapAttempts, getMaxReduceAttempts, getNumReduceTasks, getOutputFormatClass, getOutputKeyClass, getOutputValueClass, getPartitionerClass, getProfileEnabled, getProfileParams, getProfileTaskRange, getReducerClass, getSortComparator, getUser, getWorkingDirectory, setJobID
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 
Methods inherited from interface p2pMapReduce.mapreduceModule.JobContext
getCombinerClass, getConfiguration, getInputFormatClass, getJar, getJobID, getJobSetupCleanupNeeded, getMapOutputKeyClass, getMapOutputValueClass, getMapperClass, getMaxMapAttempts, getMaxReduceAttempts, getNumReduceTasks, getOutputFormatClass, getOutputKeyClass, getOutputValueClass, getPartitionerClass, getProfileEnabled, getProfileParams, getProfileTaskRange, getReducerClass, getSortComparator, getUser, getWorkingDirectory
 

Field Detail

OUTPUT_FILTER

public static final java.lang.String OUTPUT_FILTER
See Also:
Constant Field Values

COMPLETION_POLL_INTERVAL_KEY

public static final java.lang.String COMPLETION_POLL_INTERVAL_KEY
Key in mapred-*.xml that sets completionPollInvervalMillis

See Also:
Constant Field Values

PROGRESS_MONITOR_POLL_INTERVAL_KEY

public static final java.lang.String PROGRESS_MONITOR_POLL_INTERVAL_KEY
Key in mapred-*.xml that sets progMonitorPollIntervalMillis

See Also:
Constant Field Values

USED_GENERIC_PARSER

public static final java.lang.String USED_GENERIC_PARSER
See Also:
Constant Field Values

SUBMIT_REPLICATION

public static final java.lang.String SUBMIT_REPLICATION
See Also:
Constant Field Values
Constructor Detail

Job

public Job()

Job

public Job(Configuration conf,
           JobID jobID)
Method Detail

readExternal

public void readExternal(java.io.ObjectInput in)
                  throws java.io.IOException,
                         java.lang.ClassNotFoundException
Specified by:
readExternal in interface java.io.Externalizable
Throws:
java.io.IOException
java.lang.ClassNotFoundException

writeExternal

public void writeExternal(java.io.ObjectOutput out)
                   throws java.io.IOException
Specified by:
writeExternal in interface java.io.Externalizable
Throws:
java.io.IOException

getStatus

public JobStatus getStatus()

getJobState

public JobStatus.State getJobState()
Returns the current state of the Job.

Returns:
JobStatus#State
Throws:
java.io.IOException
java.lang.InterruptedException

setJobState

public void setJobState(JobStatus.State state)

getTrackingURL

public java.lang.String getTrackingURL()
Get the URL where some job progress information will be displayed.

Returns:
the URL where some job progress information will be displayed.

getJobFile

public java.lang.String getJobFile()
Get the path of the submitted job configuration.

Returns:
the path of the submitted job configuration.

getStartTime

public long getStartTime()
Get start time of the job.

Returns:
the start time of the job

getFinishTime

public long getFinishTime()
                   throws java.io.IOException,
                          java.lang.InterruptedException
Get finish time of the job.

Returns:
the finish time of the job
Throws:
java.io.IOException
java.lang.InterruptedException

getSchedulingInfo

public java.lang.String getSchedulingInfo()
Get scheduling info of the job.

Returns:
the scheduling info of the job

getJobName

public java.lang.String getJobName()
The user-specified job name.

Specified by:
getJobName in interface JobContext
Overrides:
getJobName in class JobContextImpl
Returns:
the job's name, defaulting to "".

getHistoryUrl

public java.lang.String getHistoryUrl()
                               throws java.io.IOException,
                                      java.lang.InterruptedException
Throws:
java.io.IOException
java.lang.InterruptedException

isRetired

public boolean isRetired()
                  throws java.io.IOException,
                         java.lang.InterruptedException
Throws:
java.io.IOException
java.lang.InterruptedException

toString

public java.lang.String toString()
Dump stats to screen.

Overrides:
toString in class java.lang.Object

mapProgress

public float mapProgress()
                  throws java.io.IOException,
                         java.lang.InterruptedException
Get the progress of the job's map-tasks, as a float between 0.0 and 1.0. When all map tasks have completed, the function returns 1.0.

Returns:
the progress of the job's map-tasks.
Throws:
java.io.IOException
java.lang.InterruptedException

reduceProgress

public float reduceProgress()
                     throws java.io.IOException,
                            java.lang.InterruptedException
Get the progress of the job's reduce-tasks, as a float between 0.0 and 1.0. When all reduce tasks have completed, the function returns 1.0.

Returns:
the progress of the job's reduce-tasks.
Throws:
java.io.IOException
java.lang.InterruptedException

cleanupProgress

public float cleanupProgress()
                      throws java.io.IOException,
                             java.lang.InterruptedException
Get the progress of the job's cleanup-tasks, as a float between 0.0 and 1.0. When all cleanup tasks have completed, the function returns 1.0.

Returns:
the progress of the job's cleanup-tasks.
Throws:
java.io.IOException
java.lang.InterruptedException

setupProgress

public float setupProgress()
                    throws java.io.IOException,
                           java.lang.InterruptedException
Get the progress of the job's setup-tasks, as a float between 0.0 and 1.0. When all setup tasks have completed, the function returns 1.0.

Returns:
the progress of the job's setup-tasks.
Throws:
java.io.IOException
java.lang.InterruptedException

isComplete

public boolean isComplete()
                   throws java.io.IOException,
                          java.lang.InterruptedException
Check if the job is finished or not. This is a non-blocking call.

Returns:
true if the job is complete, else false.
Throws:
java.io.IOException
java.lang.InterruptedException

isSuccessful

public boolean isSuccessful()
                     throws java.io.IOException,
                            java.lang.InterruptedException
Check if the job completed successfully.

Returns:
true if the job succeeded, else false.
Throws:
java.io.IOException
java.lang.InterruptedException

killJob

public void killJob()
             throws java.io.IOException,
                    java.lang.InterruptedException
Kill the running job. Blocks until all job tasks have been killed as well. If the job is no longer running, it simply returns.

Throws:
java.io.IOException
java.lang.InterruptedException

setNumReduceTasks

public void setNumReduceTasks(int tasks)
                       throws java.lang.IllegalStateException
Set the number of reduce tasks for the job.

Parameters:
tasks - the number of reduce tasks
Throws:
java.lang.IllegalStateException - if the job is submitted

setWorkingDirectory

public void setWorkingDirectory(Path dir)
Set the current working directory for the default file system.

Parameters:
dir - the new current working directory.
Throws:
java.lang.IllegalStateException - if the job is submitted

setInputFormatClass

public void setInputFormatClass(java.lang.Class<? extends InputFormat> cls)
                         throws java.lang.IllegalStateException
Set the InputFormat for the job.

Parameters:
cls - the InputFormat to use
Throws:
java.lang.IllegalStateException - if the job is submitted

setOutputFormatClass

public void setOutputFormatClass(java.lang.Class<? extends OutputFormat> cls)
                          throws java.lang.IllegalStateException
Set the OutputFormat for the job.

Parameters:
cls - the OutputFormat to use
Throws:
java.lang.IllegalStateException - if the job is submitted

setMapperClass

public void setMapperClass(java.lang.Class<? extends Mapper> cls)
                    throws java.lang.IllegalStateException
Set the Mapper for the job.

Parameters:
cls - the Mapper to use
Throws:
java.lang.IllegalStateException - if the job is submitted

setPartitionerClass

public void setPartitionerClass(java.lang.Class<? extends Partitioner> cls)
                         throws java.lang.IllegalStateException
Set the Partitioner for the job.

Parameters:
cls - the Partitioner to use
Throws:
java.lang.IllegalStateException - if the job is submitted

setCombinerClass

public void setCombinerClass(java.lang.Class<? extends Reducer> cls)
                      throws java.lang.IllegalStateException
Set the combiner class for the job.

Parameters:
cls - the combiner to use
Throws:
java.lang.IllegalStateException - if the job is submitted

setReducerClass

public void setReducerClass(java.lang.Class<? extends Reducer> cls)
                     throws java.lang.IllegalStateException
Set the Reducer for the job.

Parameters:
cls - the Reducer to use
Throws:
java.lang.IllegalStateException - if the job is submitted

setJarByClass

public void setJarByClass(java.lang.Class<?> cls)
Set the Jar by finding where a given class came from.

Parameters:
cls - the example class

setJar

public void setJar(java.lang.String jar)
Set the job jar


setUser

public void setUser(java.lang.String user)
Set the reported username for this job.

Parameters:
user - the username for this job.

setMapOutputKeyClass

public void setMapOutputKeyClass(java.lang.Class<?> theClass)
                          throws java.lang.IllegalStateException
Set the key class for the map output data. This allows the user to specify the map output key class to be different than the final output value class.

Parameters:
theClass - the map output key class.
Throws:
java.lang.IllegalStateException - if the job is submitted

setMapOutputValueClass

public void setMapOutputValueClass(java.lang.Class<?> theClass)
                            throws java.lang.IllegalStateException
Set the value class for the map output data. This allows the user to specify the map output value class to be different than the final output value class.

Parameters:
theClass - the map output value class.
Throws:
java.lang.IllegalStateException - if the job is submitted

setOutputKeyClass

public void setOutputKeyClass(java.lang.Class<?> theClass)
                       throws java.lang.IllegalStateException
Set the key class for the job output data.

Parameters:
theClass - the key class for the job output data.
Throws:
java.lang.IllegalStateException - if the job is submitted

setOutputValueClass

public void setOutputValueClass(java.lang.Class<?> theClass)
                         throws java.lang.IllegalStateException
Set the value class for job outputs.

Parameters:
theClass - the value class for job outputs.
Throws:
java.lang.IllegalStateException - if the job is submitted

setSortComparatorClass

public void setSortComparatorClass(java.lang.Class<? extends RawComparator> cls)
                            throws java.lang.IllegalStateException
Define the comparator that controls how the keys are sorted before they are passed to the Reducer.

Parameters:
cls - the raw comparator
Throws:
java.lang.IllegalStateException - if the job is submitted

setGroupingComparatorClass

public void setGroupingComparatorClass(java.lang.Class<? extends RawComparator> cls)
                                throws java.lang.IllegalStateException
Define the comparator that controls which keys are grouped together for a single call to Reducer#reduce(Object, Iterable, org.apache.hadoop.mapreduce.Reducer.Context)

Parameters:
cls - the raw comparator to use
Throws:
java.lang.IllegalStateException - if the job is submitted

setJobName

public void setJobName(java.lang.String name)
                throws java.lang.IllegalStateException
Set the user-specified job name.

Parameters:
name - the job's new name.
Throws:
java.lang.IllegalStateException - if the job is submitted

setSpeculativeExecution

public void setSpeculativeExecution(boolean speculativeExecution)
Turn speculative execution on or off for this job.

Parameters:
speculativeExecution - true if speculative execution should be turned on, else false.

setMapSpeculativeExecution

public void setMapSpeculativeExecution(boolean speculativeExecution)
Turn speculative execution on or off for this job for map tasks.

Parameters:
speculativeExecution - true if speculative execution should be turned on for map tasks, else false.

setReduceSpeculativeExecution

public void setReduceSpeculativeExecution(boolean speculativeExecution)
Turn speculative execution on or off for this job for reduce tasks.

Parameters:
speculativeExecution - true if speculative execution should be turned on for reduce tasks, else false.

setJobSetupCleanupNeeded

public void setJobSetupCleanupNeeded(boolean needed)
Specify whether job-setup and job-cleanup is needed for the job

Parameters:
needed - If true, job-setup and job-cleanup will be considered from OutputCommitter else ignored.

setMaxMapAttempts

public void setMaxMapAttempts(int n)
Expert: Set the number of maximum attempts that will be made to run a map task.

Parameters:
n - the number of attempts per map task.

setMaxReduceAttempts

public void setMaxReduceAttempts(int n)
Expert: Set the number of maximum attempts that will be made to run a reduce task.

Parameters:
n - the number of attempts per reduce task.

setProfileEnabled

public void setProfileEnabled(boolean newValue)
Set whether the system should collect profiler information for some of the tasks in this job? The information is stored in the user log directory.

Parameters:
newValue - true means it should be gathered

setProfileParams

public void setProfileParams(java.lang.String value)
Set the profiler configuration arguments. If the string contains a '%s' it will be replaced with the name of the profiling output file when the task runs. This value is passed to the task child JVM on the command line.

Parameters:
value - the configuration string

setProfileTaskRange

public void setProfileTaskRange(boolean isMap,
                                java.lang.String newValue)
Set the ranges of maps or reduces to profile. setProfileEnabled(true) must also be called.

Parameters:
newValue - a set of integer ranges of the map ids

setCancelDelegationTokenUponJobCompletion

public void setCancelDelegationTokenUponJobCompletion(boolean value)
Sets the flag that will allow the JobTracker to cancel the HDFS delegation tokens upon job completion. Defaults to true.


waitForCompletion

public boolean waitForCompletion(boolean verbose)
                          throws java.io.IOException,
                                 java.lang.InterruptedException,
                                 java.lang.ClassNotFoundException
Submit the job to the cluster and wait for it to finish.

Parameters:
verbose - print the progress to the user
Returns:
true if the job succeeded
Throws:
java.io.IOException - thrown if the communication with the JobTracker is lost
java.lang.InterruptedException
java.lang.ClassNotFoundException

getProgressPollInterval

public static int getProgressPollInterval(Configuration conf)
The interval at which monitorAndPrintJob() prints status


getCompletionPollInterval

public static int getCompletionPollInterval(Configuration conf)
The interval at which waitForCompletion() should check.


getTaskOutputFilter

public static Job.TaskStatusFilter getTaskOutputFilter(Configuration conf)
Get the task output filter.

Parameters:
conf - the configuration.
Returns:
the filter level.

setTaskOutputFilter

public static void setTaskOutputFilter(Configuration conf,
                                       Job.TaskStatusFilter newValue)
Modify the Configuration to set the task output filter.

Parameters:
conf - the Configuration to modify.
newValue - the value to set.