|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectp2pMapReduce.mapreduceModule.JobContextImpl
p2pMapReduce.mapreduceModule.Job
public class Job
The job submitter's view of the Job.
It allows the user to configure the job, submit it, control its execution, and query the state. The set methods only work until the job is submitted, afterwards they will throw an IllegalStateException.
Normally the user creates the application, describes various facets of the
job via Job
and then submits the job and monitor its progress.
Here is an example on how to submit a job:
Note: must implement// Create a new Job Job job = new Job(new Configuration()); job.setJarByClass(MyJob.class); // Specify various job-specific parameters job.setJobName("myjob"); job.setInputPath(new Path("in")); job.setOutputPath(new Path("out")); job.setMapperClass(MyJob.MyMapper.class); job.setReducerClass(MyJob.MyReducer.class); // Submit the job, then poll for progress until the job is complete job.waitForCompletion(true);
Externalizable
to provide custom serialization method
to manage super type data load and restore.
Nested Class Summary | |
---|---|
static class |
Job.JobState
|
static class |
Job.TaskStatusFilter
|
Field Summary | |
---|---|
static java.lang.String |
COMPLETION_POLL_INTERVAL_KEY
Key in mapred-*.xml that sets completionPollInvervalMillis |
static java.lang.String |
OUTPUT_FILTER
|
static java.lang.String |
PROGRESS_MONITOR_POLL_INTERVAL_KEY
Key in mapred-*.xml that sets progMonitorPollIntervalMillis |
static java.lang.String |
SUBMIT_REPLICATION
|
static java.lang.String |
USED_GENERIC_PARSER
|
Fields inherited from class p2pMapReduce.mapreduceModule.JobContextImpl |
---|
conf |
Constructor Summary | |
---|---|
Job()
|
|
Job(Configuration conf,
JobID jobID)
|
Method Summary | |
---|---|
float |
cleanupProgress()
Get the progress of the job's cleanup-tasks, as a float between 0.0 and 1.0. |
static int |
getCompletionPollInterval(Configuration conf)
The interval at which waitForCompletion() should check. |
long |
getFinishTime()
Get finish time of the job. |
java.lang.String |
getHistoryUrl()
|
java.lang.String |
getJobFile()
Get the path of the submitted job configuration. |
java.lang.String |
getJobName()
The user-specified job name. |
JobStatus.State |
getJobState()
Returns the current state of the Job. |
static int |
getProgressPollInterval(Configuration conf)
The interval at which monitorAndPrintJob() prints status |
java.lang.String |
getSchedulingInfo()
Get scheduling info of the job. |
long |
getStartTime()
Get start time of the job. |
JobStatus |
getStatus()
|
static Job.TaskStatusFilter |
getTaskOutputFilter(Configuration conf)
Get the task output filter. |
java.lang.String |
getTrackingURL()
Get the URL where some job progress information will be displayed. |
boolean |
isComplete()
Check if the job is finished or not. |
boolean |
isRetired()
|
boolean |
isSuccessful()
Check if the job completed successfully. |
void |
killJob()
Kill the running job. |
float |
mapProgress()
Get the progress of the job's map-tasks, as a float between 0.0 and 1.0. |
void |
readExternal(java.io.ObjectInput in)
|
float |
reduceProgress()
Get the progress of the job's reduce-tasks, as a float between 0.0 and 1.0. |
void |
setCancelDelegationTokenUponJobCompletion(boolean value)
Sets the flag that will allow the JobTracker to cancel the HDFS delegation tokens upon job completion. |
void |
setCombinerClass(java.lang.Class<? extends Reducer> cls)
Set the combiner class for the job. |
void |
setGroupingComparatorClass(java.lang.Class<? extends RawComparator> cls)
Define the comparator that controls which keys are grouped together for a single call to Reducer#reduce(Object, Iterable,
org.apache.hadoop.mapreduce.Reducer.Context) |
void |
setInputFormatClass(java.lang.Class<? extends InputFormat> cls)
Set the InputFormat for the job. |
void |
setJar(java.lang.String jar)
Set the job jar |
void |
setJarByClass(java.lang.Class<?> cls)
Set the Jar by finding where a given class came from. |
void |
setJobName(java.lang.String name)
Set the user-specified job name. |
void |
setJobSetupCleanupNeeded(boolean needed)
Specify whether job-setup and job-cleanup is needed for the job |
void |
setJobState(JobStatus.State state)
|
void |
setMapOutputKeyClass(java.lang.Class<?> theClass)
Set the key class for the map output data. |
void |
setMapOutputValueClass(java.lang.Class<?> theClass)
Set the value class for the map output data. |
void |
setMapperClass(java.lang.Class<? extends Mapper> cls)
Set the Mapper for the job. |
void |
setMapSpeculativeExecution(boolean speculativeExecution)
Turn speculative execution on or off for this job for map tasks. |
void |
setMaxMapAttempts(int n)
Expert: Set the number of maximum attempts that will be made to run a map task. |
void |
setMaxReduceAttempts(int n)
Expert: Set the number of maximum attempts that will be made to run a reduce task. |
void |
setNumReduceTasks(int tasks)
Set the number of reduce tasks for the job. |
void |
setOutputFormatClass(java.lang.Class<? extends OutputFormat> cls)
Set the OutputFormat for the job. |
void |
setOutputKeyClass(java.lang.Class<?> theClass)
Set the key class for the job output data. |
void |
setOutputValueClass(java.lang.Class<?> theClass)
Set the value class for job outputs. |
void |
setPartitionerClass(java.lang.Class<? extends Partitioner> cls)
Set the Partitioner for the job. |
void |
setProfileEnabled(boolean newValue)
Set whether the system should collect profiler information for some of the tasks in this job? |
void |
setProfileParams(java.lang.String value)
Set the profiler configuration arguments. |
void |
setProfileTaskRange(boolean isMap,
java.lang.String newValue)
Set the ranges of maps or reduces to profile. setProfileEnabled(true) must also be called. |
void |
setReducerClass(java.lang.Class<? extends Reducer> cls)
Set the Reducer for the job. |
void |
setReduceSpeculativeExecution(boolean speculativeExecution)
Turn speculative execution on or off for this job for reduce tasks. |
void |
setSortComparatorClass(java.lang.Class<? extends RawComparator> cls)
Define the comparator that controls how the keys are sorted before they are passed to the Reducer . |
void |
setSpeculativeExecution(boolean speculativeExecution)
Turn speculative execution on or off for this job. |
static void |
setTaskOutputFilter(Configuration conf,
Job.TaskStatusFilter newValue)
Modify the Configuration to set the task output filter. |
float |
setupProgress()
Get the progress of the job's setup-tasks, as a float between 0.0 and 1.0. |
void |
setUser(java.lang.String user)
Set the reported username for this job. |
void |
setWorkingDirectory(Path dir)
Set the current working directory for the default file system. |
java.lang.String |
toString()
Dump stats to screen. |
boolean |
waitForCompletion(boolean verbose)
Submit the job to the cluster and wait for it to finish. |
void |
writeExternal(java.io.ObjectOutput out)
|
Methods inherited from class p2pMapReduce.mapreduceModule.JobContextImpl |
---|
getCombinerClass, getConfiguration, getGroupingComparator, getInputFormatClass, getJar, getJobID, getJobSetupCleanupNeeded, getMapOutputKeyClass, getMapOutputValueClass, getMapperClass, getMaxMapAttempts, getMaxReduceAttempts, getNumReduceTasks, getOutputFormatClass, getOutputKeyClass, getOutputValueClass, getPartitionerClass, getProfileEnabled, getProfileParams, getProfileTaskRange, getReducerClass, getSortComparator, getUser, getWorkingDirectory, setJobID |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait |
Methods inherited from interface p2pMapReduce.mapreduceModule.JobContext |
---|
getCombinerClass, getConfiguration, getInputFormatClass, getJar, getJobID, getJobSetupCleanupNeeded, getMapOutputKeyClass, getMapOutputValueClass, getMapperClass, getMaxMapAttempts, getMaxReduceAttempts, getNumReduceTasks, getOutputFormatClass, getOutputKeyClass, getOutputValueClass, getPartitionerClass, getProfileEnabled, getProfileParams, getProfileTaskRange, getReducerClass, getSortComparator, getUser, getWorkingDirectory |
Field Detail |
---|
public static final java.lang.String OUTPUT_FILTER
public static final java.lang.String COMPLETION_POLL_INTERVAL_KEY
public static final java.lang.String PROGRESS_MONITOR_POLL_INTERVAL_KEY
public static final java.lang.String USED_GENERIC_PARSER
public static final java.lang.String SUBMIT_REPLICATION
Constructor Detail |
---|
public Job()
public Job(Configuration conf, JobID jobID)
Method Detail |
---|
public void readExternal(java.io.ObjectInput in) throws java.io.IOException, java.lang.ClassNotFoundException
readExternal
in interface java.io.Externalizable
java.io.IOException
java.lang.ClassNotFoundException
public void writeExternal(java.io.ObjectOutput out) throws java.io.IOException
writeExternal
in interface java.io.Externalizable
java.io.IOException
public JobStatus getStatus()
public JobStatus.State getJobState()
java.io.IOException
java.lang.InterruptedException
public void setJobState(JobStatus.State state)
public java.lang.String getTrackingURL()
public java.lang.String getJobFile()
public long getStartTime()
public long getFinishTime() throws java.io.IOException, java.lang.InterruptedException
java.io.IOException
java.lang.InterruptedException
public java.lang.String getSchedulingInfo()
public java.lang.String getJobName()
getJobName
in interface JobContext
getJobName
in class JobContextImpl
public java.lang.String getHistoryUrl() throws java.io.IOException, java.lang.InterruptedException
java.io.IOException
java.lang.InterruptedException
public boolean isRetired() throws java.io.IOException, java.lang.InterruptedException
java.io.IOException
java.lang.InterruptedException
public java.lang.String toString()
toString
in class java.lang.Object
public float mapProgress() throws java.io.IOException, java.lang.InterruptedException
java.io.IOException
java.lang.InterruptedException
public float reduceProgress() throws java.io.IOException, java.lang.InterruptedException
java.io.IOException
java.lang.InterruptedException
public float cleanupProgress() throws java.io.IOException, java.lang.InterruptedException
java.io.IOException
java.lang.InterruptedException
public float setupProgress() throws java.io.IOException, java.lang.InterruptedException
java.io.IOException
java.lang.InterruptedException
public boolean isComplete() throws java.io.IOException, java.lang.InterruptedException
true
if the job is complete, else false
.
java.io.IOException
java.lang.InterruptedException
public boolean isSuccessful() throws java.io.IOException, java.lang.InterruptedException
true
if the job succeeded, else false
.
java.io.IOException
java.lang.InterruptedException
public void killJob() throws java.io.IOException, java.lang.InterruptedException
java.io.IOException
java.lang.InterruptedException
public void setNumReduceTasks(int tasks) throws java.lang.IllegalStateException
tasks
- the number of reduce tasks
java.lang.IllegalStateException
- if the job is submittedpublic void setWorkingDirectory(Path dir)
dir
- the new current working directory.
java.lang.IllegalStateException
- if the job is submittedpublic void setInputFormatClass(java.lang.Class<? extends InputFormat> cls) throws java.lang.IllegalStateException
InputFormat
for the job.
cls
- the InputFormat
to use
java.lang.IllegalStateException
- if the job is submittedpublic void setOutputFormatClass(java.lang.Class<? extends OutputFormat> cls) throws java.lang.IllegalStateException
OutputFormat
for the job.
cls
- the OutputFormat
to use
java.lang.IllegalStateException
- if the job is submittedpublic void setMapperClass(java.lang.Class<? extends Mapper> cls) throws java.lang.IllegalStateException
Mapper
for the job.
cls
- the Mapper
to use
java.lang.IllegalStateException
- if the job is submittedpublic void setPartitionerClass(java.lang.Class<? extends Partitioner> cls) throws java.lang.IllegalStateException
Partitioner
for the job.
cls
- the Partitioner
to use
java.lang.IllegalStateException
- if the job is submittedpublic void setCombinerClass(java.lang.Class<? extends Reducer> cls) throws java.lang.IllegalStateException
cls
- the combiner to use
java.lang.IllegalStateException
- if the job is submittedpublic void setReducerClass(java.lang.Class<? extends Reducer> cls) throws java.lang.IllegalStateException
Reducer
for the job.
cls
- the Reducer
to use
java.lang.IllegalStateException
- if the job is submittedpublic void setJarByClass(java.lang.Class<?> cls)
cls
- the example classpublic void setJar(java.lang.String jar)
public void setUser(java.lang.String user)
user
- the username for this job.public void setMapOutputKeyClass(java.lang.Class<?> theClass) throws java.lang.IllegalStateException
theClass
- the map output key class.
java.lang.IllegalStateException
- if the job is submittedpublic void setMapOutputValueClass(java.lang.Class<?> theClass) throws java.lang.IllegalStateException
theClass
- the map output value class.
java.lang.IllegalStateException
- if the job is submittedpublic void setOutputKeyClass(java.lang.Class<?> theClass) throws java.lang.IllegalStateException
theClass
- the key class for the job output data.
java.lang.IllegalStateException
- if the job is submittedpublic void setOutputValueClass(java.lang.Class<?> theClass) throws java.lang.IllegalStateException
theClass
- the value class for job outputs.
java.lang.IllegalStateException
- if the job is submittedpublic void setSortComparatorClass(java.lang.Class<? extends RawComparator> cls) throws java.lang.IllegalStateException
Reducer
.
cls
- the raw comparator
java.lang.IllegalStateException
- if the job is submittedpublic void setGroupingComparatorClass(java.lang.Class<? extends RawComparator> cls) throws java.lang.IllegalStateException
Reducer#reduce(Object, Iterable,
org.apache.hadoop.mapreduce.Reducer.Context)
cls
- the raw comparator to use
java.lang.IllegalStateException
- if the job is submittedpublic void setJobName(java.lang.String name) throws java.lang.IllegalStateException
name
- the job's new name.
java.lang.IllegalStateException
- if the job is submittedpublic void setSpeculativeExecution(boolean speculativeExecution)
speculativeExecution
- true
if speculative execution
should be turned on, else false
.public void setMapSpeculativeExecution(boolean speculativeExecution)
speculativeExecution
- true
if speculative execution
should be turned on for map tasks,
else false
.public void setReduceSpeculativeExecution(boolean speculativeExecution)
speculativeExecution
- true
if speculative execution
should be turned on for reduce tasks,
else false
.public void setJobSetupCleanupNeeded(boolean needed)
needed
- If true
, job-setup and job-cleanup will be
considered from OutputCommitter
else ignored.public void setMaxMapAttempts(int n)
n
- the number of attempts per map task.public void setMaxReduceAttempts(int n)
n
- the number of attempts per reduce task.public void setProfileEnabled(boolean newValue)
newValue
- true means it should be gatheredpublic void setProfileParams(java.lang.String value)
value
- the configuration stringpublic void setProfileTaskRange(boolean isMap, java.lang.String newValue)
newValue
- a set of integer ranges of the map idspublic void setCancelDelegationTokenUponJobCompletion(boolean value)
public boolean waitForCompletion(boolean verbose) throws java.io.IOException, java.lang.InterruptedException, java.lang.ClassNotFoundException
verbose
- print the progress to the user
java.io.IOException
- thrown if the communication with the
JobTracker
is lost
java.lang.InterruptedException
java.lang.ClassNotFoundException
public static int getProgressPollInterval(Configuration conf)
public static int getCompletionPollInterval(Configuration conf)
public static Job.TaskStatusFilter getTaskOutputFilter(Configuration conf)
conf
- the configuration.
public static void setTaskOutputFilter(Configuration conf, Job.TaskStatusFilter newValue)
conf
- the Configuration to modify.newValue
- the value to set.
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |