|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectp2pMapReduce.mapreduceModule.conf.Configuration
p2pMapReduce.mapreduceModule.JobConf
public class JobConf
A map/reduce job configuration.
JobConf
is the primary interface for a user to describe a
map-reduce job to the Hadoop framework for execution. The framework tries to
faithfully execute the job as-is described by JobConf
, however:
setNumReduceTasks(int)
), some parameters interact subtly
rest of the framework and/or job-configuration and is relatively more
complex for the user to control finely (e.g. setNumMapTasks(int)
).
JobConf
typically specifies the Mapper
, combiner
(if any), Partitioner
, Reducer
, InputFormat
and
OutputFormat
implementations to be used etc.
Optionally JobConf
is used to specify other advanced facets
of the job such as Comparator
s to be used, files to be put in
the DistributedCache
, whether or not intermediate and/or job outputs
are to be compressed (and how), debugability via user-provided scripts
( setMapDebugScript(String)
/setReduceDebugScript(String)
),
for doing post-processing on task logs, task's stdout, stderr, syslog.
and etc.
Here is an example on how to configure a job via JobConf
:
// Create a new JobConf JobConf job = new JobConf(new Configuration(), MyJob.class); // Specify various job-specific parameters job.setJobName("myjob"); FileInputFormat.setInputPaths(job, new Path("in")); FileOutputFormat.setOutputPath(job, new Path("out")); job.setMapperClass(MyJob.MyMapper.class); job.setCombinerClass(MyJob.MyReducer.class); job.setReducerClass(MyJob.MyReducer.class); job.setInputFormat(SequenceFileInputFormat.class); job.setOutputFormat(SequenceFileOutputFormat.class);
JobClient
,
ClusterStatus
,
Tool
,
DistributedCache
,
Serialized FormNested Class Summary |
---|
Nested classes/interfaces inherited from class p2pMapReduce.mapreduceModule.conf.Configuration |
---|
Configuration.IntegerRanges |
Field Summary | |
---|---|
static java.util.logging.Level |
DEFAULT_LOG_LEVEL
Default logging level for map/reduce tasks. |
static java.lang.String |
DEFAULT_MAPRED_TASK_JAVA_OPTS
|
static java.lang.String |
DEFAULT_QUEUE_NAME
Name of the queue to which jobs will be submitted, if no queue name is mentioned. |
static long |
DISABLED_MEMORY_LIMIT
A value which if set for memory related configuration options, indicates that the options are turned off. |
static java.lang.String |
MAPRED_LOCAL_DIR_PROPERTY
Property name for the configuration property mapreduce.cluster.local.dir |
static java.lang.String |
MAPRED_MAP_TASK_ENV
Configuration key to set the maximum virutal memory available to the map tasks. |
static java.lang.String |
MAPRED_MAP_TASK_JAVA_OPTS
Configuration key to set the java command line options for the map tasks. |
static java.lang.String |
MAPRED_MAP_TASK_LOG_LEVEL
Configuration key to set the logging Level for the map task. |
static java.lang.String |
MAPRED_MAP_TASK_ULIMIT
Configuration key to set the maximum virutal memory available to the map tasks (in kilo-bytes). |
static java.lang.String |
MAPRED_REDUCE_TASK_ENV
Configuration key to set the maximum virutal memory available to the reduce tasks. |
static java.lang.String |
MAPRED_REDUCE_TASK_JAVA_OPTS
Configuration key to set the java command line options for the reduce tasks. |
static java.lang.String |
MAPRED_REDUCE_TASK_LOG_LEVEL
Configuration key to set the logging Level for the reduce task. |
static java.lang.String |
MAPRED_REDUCE_TASK_ULIMIT
Configuration key to set the maximum virutal memory available to the reduce tasks (in kilo-bytes). |
static java.lang.String |
MAPRED_TASK_DEFAULT_MAXVMEM_PROPERTY
Deprecated. |
static java.lang.String |
MAPRED_TASK_ENV
Deprecated. Use MAPRED_MAP_TASK_ENV or
MAPRED_REDUCE_TASK_ENV |
static java.lang.String |
MAPRED_TASK_JAVA_OPTS
Deprecated. Use MAPRED_MAP_TASK_JAVA_OPTS or
MAPRED_REDUCE_TASK_JAVA_OPTS |
static java.lang.String |
MAPRED_TASK_MAXPMEM_PROPERTY
Deprecated. |
static java.lang.String |
MAPRED_TASK_MAXVMEM_PROPERTY
Deprecated. Use MAPRED_JOB_MAP_MEMORY_MB_PROPERTY and
MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY |
static java.lang.String |
MAPRED_TASK_ULIMIT
Deprecated. Use MAPRED_MAP_TASK_ULIMIT or
MAPRED_REDUCE_TASK_ULIMIT |
static java.util.regex.Pattern |
UNPACK_JAR_PATTERN_DEFAULT
Pattern for the default unpacking behavior for job jars |
static java.lang.String |
UPPER_LIMIT_ON_TASK_VMEM_PROPERTY
Deprecated. |
Constructor Summary | |
---|---|
JobConf()
Construct a map/reduce job configuration. |
|
JobConf(boolean loadDefaults)
A new map/reduce configuration where the behavior of reading from the default resources can be turned off. |
|
JobConf(java.lang.Class exampleClass)
Construct a map/reduce job configuration. |
|
JobConf(Configuration conf)
Construct a map/reduce job configuration. |
|
JobConf(Configuration conf,
java.lang.Class exampleClass)
Construct a map/reduce job configuration. |
|
JobConf(Path config)
Construct a map/reduce configuration. |
|
JobConf(java.lang.String config)
Construct a map/reduce configuration. |
Method Summary | |
---|---|
java.lang.Class<? extends Reducer> |
getCombinerClass()
Get the user-defined combiner class used to combine map-outputs before being sent to the reducers. |
InputFormat |
getInputFormat()
Get the InputFormat implementation for the map-reduce job,
defaults to TextInputFormat if not specified explicity. |
java.lang.String |
getJar()
Get the user jar for the map-reduce job. |
java.util.regex.Pattern |
getJarUnpackPattern()
Get the pattern for jar contents to unpack on the tasktracker |
java.lang.String |
getJobEndNotificationURI()
Get the uri to be invoked in-order to send a notification after the job has completed (success/failure). |
java.lang.String |
getJobLocalDir()
Get job-specific shared directory for use as scratch space When a job starts, a shared directory is created at location
${mapreduce.cluster.local.dir}/taskTracker/$user/jobcache/$jobid/work/ . |
java.lang.String |
getJobName()
Get the user-specified job name. |
boolean |
getKeepFailedTaskFiles()
Should the temporary files for failed tasks be kept? |
java.lang.String |
getKeepTaskFilesPattern()
Get the regular expression that is matched against the task names to see if we need to keep the files. |
java.lang.String[] |
getLocalDirs()
|
java.lang.String |
getMapDebugScript()
Get the map task's debug script. |
java.lang.Class<?> |
getMapOutputKeyClass()
Get the key class for the map output data. |
java.lang.Class<?> |
getMapOutputValueClass()
Get the value class for the map output data. |
java.lang.Class<? extends Mapper> |
getMapperClass()
Get the Mapper class for the job. |
boolean |
getMapSpeculativeExecution()
Should speculative execution be used for this job for map tasks? |
int |
getMaxMapAttempts()
Get the configured number of maximum attempts that will be made to run a map task, as specified by the mapreduce.map.maxattempts
property. |
int |
getMaxMapTaskFailuresPercent()
Get the maximum percentage of map tasks that can fail without the job being aborted. |
int |
getMaxReduceAttempts()
Get the configured number of maximum attempts that will be made to run a reduce task, as specified by the mapreduce.reduce.maxattempts
property. |
int |
getMaxReduceTaskFailuresPercent()
Get the maximum percentage of reduce tasks that can fail without the job being aborted. |
int |
getMaxTaskFailuresPerTracker()
Expert: Get the maximum no. of failures of a given job per tasktracker. |
long |
getMemoryForMapTask()
Get memory required to run a map task of the job, in MB. |
long |
getMemoryForReduceTask()
Get memory required to run a reduce task of the job, in MB. |
int |
getNumMapTasks()
Get configured the number of reduce tasks for this job. |
int |
getNumReduceTasks()
Get configured the number of reduce tasks for this job. |
int |
getNumTasksToExecutePerJvm()
Get the number of tasks that a spawned JVM should execute |
OutputCommitter |
getOutputCommitter()
Get the OutputCommitter implementation for the map-reduce job,
defaults to FileOutputCommitter if not specified explicitly. |
OutputFormat |
getOutputFormat()
Get the OutputFormat implementation for the map-reduce job,
defaults to TextOutputFormat if not specified explicity. |
java.lang.Class<?> |
getOutputKeyClass()
Get the key class for the job output data. |
RawComparator |
getOutputKeyComparator()
Get the RawComparator comparator used to compare keys. |
java.lang.Class<?> |
getOutputValueClass()
Get the value class for job outputs. |
RawComparator |
getOutputValueGroupingComparator()
Get the user defined WritableComparable comparator for
grouping keys of inputs to the reduce. |
java.lang.Class<? extends Partitioner> |
getPartitionerClass()
Get the Partitioner used to partition Mapper -outputs
to be sent to the Reducer s. |
boolean |
getProfileEnabled()
Get whether the task profiling is enabled. |
java.lang.String |
getProfileParams()
Get the profiler configuration arguments. |
Configuration.IntegerRanges |
getProfileTaskRange(boolean isMap)
Get the range of maps or reduces to profile. |
java.lang.String |
getQueueName()
Return the name of the queue to which this job is submitted. |
java.lang.String |
getReduceDebugScript()
Get the reduce task's debug Script |
java.lang.Class<? extends Reducer> |
getReducerClass()
Get the Reducer class for the job. |
boolean |
getReduceSpeculativeExecution()
Should speculative execution be used for this job for reduce tasks? |
java.lang.String |
getSessionId()
Deprecated. |
boolean |
getSpeculativeExecution()
Should speculative execution be used for this job? |
java.lang.String |
getUser()
Get the reported username for this job. |
Path |
getWorkingDirectory()
Get the current working directory for the default file system. |
static long |
normalizeMemoryConfigValue(long val)
Normalize the negative values in configuration |
void |
setCombinerClass(java.lang.Class<? extends Reducer> theClass)
Set the user-defined combiner class used to combine map-outputs before being sent to the reducers. |
void |
setCompressMapOutput(boolean compress)
Should the map outputs be compressed before transfer? |
void |
setInputFormat(java.lang.Class<? extends InputFormat> theClass)
Set the InputFormat implementation for the map-reduce job. |
void |
setJar(java.lang.String jar)
Set the user jar for the map-reduce job. |
void |
setJarByClass(java.lang.Class cls)
Set the job's jar file by finding an example class location. |
void |
setJobEndNotificationURI(java.lang.String uri)
Set the uri to be invoked in-order to send a notification after the job has completed (success/failure). |
void |
setJobName(java.lang.String name)
Set the user-specified job name. |
void |
setKeepFailedTaskFiles(boolean keep)
Set whether the framework should keep the intermediate files for failed tasks. |
void |
setKeepTaskFilesPattern(java.lang.String pattern)
Set a regular expression for task names that should be kept. |
void |
setMapDebugScript(java.lang.String mDbgScript)
Set the debug script to run when the map tasks fail. |
void |
setMapOutputKeyClass(java.lang.Class<?> theClass)
Set the key class for the map output data. |
void |
setMapOutputValueClass(java.lang.Class<?> theClass)
Set the value class for the map output data. |
void |
setMapperClass(java.lang.Class<? extends Mapper> theClass)
Set the Mapper class for the job. |
void |
setMapSpeculativeExecution(boolean speculativeExecution)
Turn speculative execution on or off for this job for map tasks. |
void |
setMaxMapAttempts(int n)
Expert: Set the number of maximum attempts that will be made to run a map task. |
void |
setMaxMapTaskFailuresPercent(int percent)
Expert: Set the maximum percentage of map tasks that can fail without the job being aborted. |
void |
setMaxReduceAttempts(int n)
Expert: Set the number of maximum attempts that will be made to run a reduce task. |
void |
setMaxReduceTaskFailuresPercent(int percent)
Set the maximum percentage of reduce tasks that can fail without the job being aborted. |
void |
setMaxTaskFailuresPerTracker(int noFailures)
Set the maximum no. of failures of a given job per tasktracker. |
void |
setMemoryForMapTask(long mem)
|
void |
setMemoryForReduceTask(long mem)
|
void |
setNumMapTasks(int n)
Set the number of map tasks for this job. |
void |
setNumReduceTasks(int n)
Set the requisite number of reduce tasks for this job. |
void |
setNumTasksToExecutePerJvm(int numTasks)
Sets the number of tasks that a spawned task JVM should run before it exits |
void |
setOutputCommitter(java.lang.Class<? extends OutputCommitter> theClass)
Set the OutputCommitter implementation for the map-reduce job. |
void |
setOutputFormat(java.lang.Class<? extends OutputFormat> theClass)
Set the OutputFormat implementation for the map-reduce job. |
void |
setOutputKeyClass(java.lang.Class<?> theClass)
Set the key class for the job output data. |
void |
setOutputKeyComparatorClass(java.lang.Class<? extends RawComparator> theClass)
Set the RawComparator comparator used to compare keys. |
void |
setOutputValueClass(java.lang.Class<?> theClass)
Set the value class for job outputs. |
void |
setOutputValueGroupingComparator(java.lang.Class<? extends RawComparator> theClass)
Set the user defined RawComparator comparator for
grouping keys in the input to the reduce. |
void |
setPartitionerClass(java.lang.Class<? extends Partitioner> theClass)
Set the Partitioner class used to partition
Mapper -outputs to be sent to the Reducer s. |
void |
setProfileEnabled(boolean newValue)
Set whether the system should collect profiler information for some of the tasks in this job? |
void |
setProfileParams(java.lang.String value)
Set the profiler configuration arguments. |
void |
setProfileTaskRange(boolean isMap,
java.lang.String newValue)
Set the ranges of maps or reduces to profile. setProfileEnabled(true) must also be called. |
void |
setQueueName(java.lang.String queueName)
Set the name of the queue to which this job should be submitted. |
void |
setReduceDebugScript(java.lang.String rDbgScript)
Set the debug script to run when the reduce tasks fail. |
void |
setReducerClass(java.lang.Class<? extends Reducer> theClass)
Set the Reducer class for the job. |
void |
setReduceSpeculativeExecution(boolean speculativeExecution)
Turn speculative execution on or off for this job for reduce tasks. |
void |
setSessionId(java.lang.String sessionId)
Deprecated. |
void |
setSpeculativeExecution(boolean speculativeExecution)
Turn speculative execution on or off for this job. |
void |
setUser(java.lang.String user)
Set the reported username for this job. |
void |
setWorkingDirectory(Path dir)
Set the current working directory for the default file system. |
Methods inherited from class p2pMapReduce.mapreduceModule.conf.Configuration |
---|
addDefaultResource, addDeprecation, addDeprecation, addResource, addResource, addResource, addResource, clear, get, get, getBoolean, getClass, getClass, getClassByName, getClasses, getClassLoader, getConfResourceAsInputStream, getConfResourceAsReader, getEnum, getFile, getFloat, getInstances, getInt, getLong, getPattern, getProps, getRange, getRaw, getResource, getStringCollection, getStrings, getStrings, getTrimmedStringCollection, getTrimmedStrings, getTrimmedStrings, iterator, readExternal, readFields, reloadConfiguration, set, setBoolean, setBooleanIfUnset, setClass, setClassLoader, setEnum, setFloat, setIfUnset, setInt, setLong, setPattern, setQuietMode, setStrings, size, toString, write, writeExternal |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait |
Field Detail |
---|
@Deprecated public static final java.lang.String MAPRED_TASK_MAXVMEM_PROPERTY
MAPRED_JOB_MAP_MEMORY_MB_PROPERTY
and
MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY
@Deprecated public static final java.lang.String UPPER_LIMIT_ON_TASK_VMEM_PROPERTY
@Deprecated public static final java.lang.String MAPRED_TASK_DEFAULT_MAXVMEM_PROPERTY
@Deprecated public static final java.lang.String MAPRED_TASK_MAXPMEM_PROPERTY
public static final long DISABLED_MEMORY_LIMIT
public static final java.lang.String MAPRED_LOCAL_DIR_PROPERTY
public static final java.lang.String DEFAULT_QUEUE_NAME
public static final java.util.regex.Pattern UNPACK_JAR_PATTERN_DEFAULT
@Deprecated public static final java.lang.String MAPRED_TASK_JAVA_OPTS
MAPRED_MAP_TASK_JAVA_OPTS
or
MAPRED_REDUCE_TASK_JAVA_OPTS
MAPRED_TASK_ULIMIT
can be used to
control the maximum virtual memory of the child processes.
The configuration variable MAPRED_TASK_ENV
can be used to pass
other environment variables to the child processes.
public static final java.lang.String MAPRED_MAP_TASK_JAVA_OPTS
MAPRED_MAP_TASK_ULIMIT
can be used to
control the maximum virtual memory of the map processes.
The configuration variable MAPRED_MAP_TASK_ENV
can be used to pass
other environment variables to the map processes.
public static final java.lang.String MAPRED_REDUCE_TASK_JAVA_OPTS
MAPRED_REDUCE_TASK_ULIMIT
can be used
to control the maximum virtual memory of the reduce processes.
The configuration variable MAPRED_REDUCE_TASK_ENV
can be used to
pass process environment variables to the reduce processes.
public static final java.lang.String DEFAULT_MAPRED_TASK_JAVA_OPTS
@Deprecated public static final java.lang.String MAPRED_TASK_ULIMIT
MAPRED_MAP_TASK_ULIMIT
or
MAPRED_REDUCE_TASK_ULIMIT
MAPRED_TASK_JAVA_OPTS
, else the VM might not start.
public static final java.lang.String MAPRED_MAP_TASK_ULIMIT
MAPRED_MAP_TASK_JAVA_OPTS
, else the VM might not start.
public static final java.lang.String MAPRED_REDUCE_TASK_ULIMIT
MAPRED_REDUCE_TASK_JAVA_OPTS
, else the VM might not start.
@Deprecated public static final java.lang.String MAPRED_TASK_ENV
MAPRED_MAP_TASK_ENV
or
MAPRED_REDUCE_TASK_ENV
k1=v1,k2=v2
. Further it can
reference existing environment variables via $key
.
Example:
public static final java.lang.String MAPRED_MAP_TASK_ENV
k1=v1,k2=v2
. Further it can
reference existing environment variables via $key
.
Example:
public static final java.lang.String MAPRED_REDUCE_TASK_ENV
k1=v1,k2=v2
. Further it can
reference existing environment variables via $key
.
Example:
public static final java.lang.String MAPRED_MAP_TASK_LOG_LEVEL
Level
for the map task.
The allowed logging levels are:
OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
public static final java.lang.String MAPRED_REDUCE_TASK_LOG_LEVEL
Level
for the reduce task.
The allowed logging levels are:
OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
public static final java.util.logging.Level DEFAULT_LOG_LEVEL
Constructor Detail |
---|
public JobConf()
public JobConf(java.lang.Class exampleClass)
exampleClass
- a class whose containing jar is used as the job's jar.public JobConf(Configuration conf)
conf
- a Configuration whose settings will be inherited.public JobConf(Configuration conf, java.lang.Class exampleClass)
conf
- a Configuration whose settings will be inherited.exampleClass
- a class whose containing jar is used as the job's jar.public JobConf(java.lang.String config)
config
- a Configuration-format XML job description file.public JobConf(Path config)
config
- a Configuration-format XML job description file.public JobConf(boolean loadDefaults)
loadDefaults
is false, the new instance
will not load resources from the default files.
loadDefaults
- specifies whether to load from the default filesMethod Detail |
---|
public java.lang.String getJar()
public void setJar(java.lang.String jar)
jar
- the user jar for the map-reduce job.public java.util.regex.Pattern getJarUnpackPattern()
public void setJarByClass(java.lang.Class cls)
cls
- the example class.public java.lang.String[] getLocalDirs() throws java.io.IOException
java.io.IOException
public java.lang.String getUser()
public void setUser(java.lang.String user)
user
- the username for this job.public void setKeepFailedTaskFiles(boolean keep)
keep
- true
if framework should keep the intermediate files
for failed tasks, false
otherwise.public boolean getKeepFailedTaskFiles()
public void setKeepTaskFilesPattern(java.lang.String pattern)
pattern
- the java.util.regex.Pattern to match against the
task names.public java.lang.String getKeepTaskFilesPattern()
public void setWorkingDirectory(Path dir)
dir
- the new current working directory.public Path getWorkingDirectory()
public void setNumTasksToExecutePerJvm(int numTasks)
numTasks
- the number of tasks to execute; defaults to 1;
-1 signifies no limitpublic int getNumTasksToExecutePerJvm()
public InputFormat getInputFormat()
InputFormat
implementation for the map-reduce job,
defaults to TextInputFormat
if not specified explicity.
InputFormat
implementation for the map-reduce job.public void setInputFormat(java.lang.Class<? extends InputFormat> theClass)
InputFormat
implementation for the map-reduce job.
theClass
- the InputFormat
implementation for the map-reduce
job.public OutputFormat getOutputFormat()
OutputFormat
implementation for the map-reduce job,
defaults to TextOutputFormat
if not specified explicity.
OutputFormat
implementation for the map-reduce job.public OutputCommitter getOutputCommitter()
OutputCommitter
implementation for the map-reduce job,
defaults to FileOutputCommitter
if not specified explicitly.
OutputCommitter
implementation for the map-reduce job.public void setOutputCommitter(java.lang.Class<? extends OutputCommitter> theClass)
OutputCommitter
implementation for the map-reduce job.
theClass
- the OutputCommitter
implementation for the map-reduce
job.public void setOutputFormat(java.lang.Class<? extends OutputFormat> theClass)
OutputFormat
implementation for the map-reduce job.
theClass
- the OutputFormat
implementation for the map-reduce
job.public void setCompressMapOutput(boolean compress)
compress
- should the map outputs be compressed?public java.lang.Class<?> getMapOutputKeyClass()
public void setMapOutputKeyClass(java.lang.Class<?> theClass)
theClass
- the map output key class.public java.lang.Class<?> getMapOutputValueClass()
public void setMapOutputValueClass(java.lang.Class<?> theClass)
theClass
- the map output value class.public java.lang.Class<?> getOutputKeyClass()
public void setOutputKeyClass(java.lang.Class<?> theClass)
theClass
- the key class for the job output data.public RawComparator getOutputKeyComparator()
RawComparator
comparator used to compare keys.
RawComparator
comparator used to compare keys.public void setOutputKeyComparatorClass(java.lang.Class<? extends RawComparator> theClass)
RawComparator
comparator used to compare keys.
theClass
- the RawComparator
comparator used to
compare keys.setOutputValueGroupingComparator(Class)
public RawComparator getOutputValueGroupingComparator()
WritableComparable
comparator for
grouping keys of inputs to the reduce.
for details.
public void setOutputValueGroupingComparator(java.lang.Class<? extends RawComparator> theClass)
RawComparator
comparator for
grouping keys in the input to the reduce.
This comparator should be provided if the equivalence rules for keys
for sorting the intermediates are different from those for grouping keys
before each call to
Reducer#reduce(Object, java.util.Iterator, OutputCollector, Reporter)
.
For key-value pairs (K1,V1) and (K2,V2), the values (V1, V2) are passed in a single call to the reduce function if K1 and K2 compare as equal.
Since setOutputKeyComparatorClass(Class)
can be used to control
how keys are sorted, this can be used in conjunction to simulate
secondary sort on values.
Note: This is not a guarantee of the reduce sort being stable in any sense. (In any case, with the order of available map-outputs to the reduce being non-deterministic, it wouldn't make that much sense.)
theClass
- the comparator class to be used for grouping keys.
It should implement RawComparator
.setOutputKeyComparatorClass(Class)
public java.lang.Class<?> getOutputValueClass()
public void setOutputValueClass(java.lang.Class<?> theClass)
theClass
- the value class for job outputs.public java.lang.Class<? extends Mapper> getMapperClass()
Mapper
class for the job.
Mapper
class for the job.public void setMapperClass(java.lang.Class<? extends Mapper> theClass)
Mapper
class for the job.
theClass
- the Mapper
class for the job.public java.lang.Class<? extends Partitioner> getPartitionerClass()
Partitioner
used to partition Mapper
-outputs
to be sent to the Reducer
s.
Partitioner
used to partition map-outputs.public void setPartitionerClass(java.lang.Class<? extends Partitioner> theClass)
Partitioner
class used to partition
Mapper
-outputs to be sent to the Reducer
s.
theClass
- the Partitioner
used to partition map-outputs.public java.lang.Class<? extends Reducer> getReducerClass()
Reducer
class for the job.
Reducer
class for the job.public void setReducerClass(java.lang.Class<? extends Reducer> theClass)
Reducer
class for the job.
theClass
- the Reducer
class for the job.public java.lang.Class<? extends Reducer> getCombinerClass()
Reducer
for the job i.e. getReducerClass()
.
public void setCombinerClass(java.lang.Class<? extends Reducer> theClass)
The combiner is an application-specified aggregation operation, which
can help cut down the amount of data transferred between the
Mapper
and the Reducer
, leading to better performance.
The framework may invoke the combiner 0, 1, or multiple times, in both the mapper and reducer tasks. In general, the combiner is called as the sort/merge result is written to disk. The combiner must:
Typically the combiner is same as the Reducer
for the
job i.e. setReducerClass(Class)
.
theClass
- the user-defined combiner class used to combine
map-outputs.public boolean getSpeculativeExecution()
true
.
true
if speculative execution be used for this job,
false
otherwise.public void setSpeculativeExecution(boolean speculativeExecution)
speculativeExecution
- true
if speculative execution
should be turned on, else false
.public boolean getMapSpeculativeExecution()
true
.
true
if speculative execution be
used for this job for map tasks,
false
otherwise.public void setMapSpeculativeExecution(boolean speculativeExecution)
speculativeExecution
- true
if speculative execution
should be turned on for map tasks,
else false
.public boolean getReduceSpeculativeExecution()
true
.
true
if speculative execution be used
for reduce tasks for this job,
false
otherwise.public void setReduceSpeculativeExecution(boolean speculativeExecution)
speculativeExecution
- true
if speculative execution
should be turned on for reduce tasks,
else false
.public int getNumMapTasks()
1
.
public void setNumMapTasks(int n)
Note: This is only a hint to the framework. The actual
number of spawned map tasks depends on the number of InputSplit
s
generated by the job's InputFormat#getSplits(JobConf, int)
.
A custom InputFormat
is typically used to accurately control
the number of map tasks for the job.
The number of maps is usually driven by the total size of the inputs i.e. total number of blocks of the input files.
The right level of parallelism for maps seems to be around 10-100 maps per-node, although it has been set up to 300 or so for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute.
The default behavior of file-based InputFormat
s is to split the
input into logical InputSplit
s based on the total size, in
bytes, of input files. However, the FileSystem
blocksize of the
input files is treated as an upper bound for input splits. A lower bound
on the split size can be set via
mapreduce.input.fileinputformat.split.minsize.
Thus, if you expect 10TB of input data and have a blocksize of 128MB,
you'll end up with 82,000 maps, unless setNumMapTasks(int)
is
used to set it even higher.
n
- the number of map tasks for this job.InputFormat#getSplits(JobConf, int)
,
FileInputFormat
,
FileSystem#getDefaultBlockSize()
,
FileStatus#getBlockSize()
public int getNumReduceTasks()
1
.
public void setNumReduceTasks(int n)
The right number of reduces seems to be 0.95
or
1.75
multiplied by (<no. of nodes> *
mapreduce.tasktracker.reduce.tasks.maximum).
With 0.95
all of the reduces can launch immediately and
start transfering map outputs as the maps finish. With 1.75
the faster nodes will finish their first round of reduces and launch a
second wave of reduces doing a much better job of load balancing.
Increasing the number of reduces increases the framework overhead, but increases load balancing and lowers the cost of failures.
The scaling factors above are slightly less than whole numbers to reserve a few reduce slots in the framework for speculative-tasks, failures etc.
It is legal to set the number of reduce-tasks to zero
.
In this case the output of the map-tasks directly go to distributed
file-system, to the path set by
FileOutputFormat#setOutputPath(JobConf, Path)
. Also, the
framework doesn't sort the map-outputs before writing it out to HDFS.
n
- the number of reduce tasks for this job.public int getMaxMapAttempts()
mapreduce.map.maxattempts
property. If this property is not already set, the default is 4 attempts.
public void setMaxMapAttempts(int n)
n
- the number of attempts per map task.public int getMaxReduceAttempts()
mapreduce.reduce.maxattempts
property. If this property is not already set, the default is 4 attempts.
public void setMaxReduceAttempts(int n)
n
- the number of attempts per reduce task.public java.lang.String getJobName()
public void setJobName(java.lang.String name)
name
- the job's new name.@Deprecated public java.lang.String getSessionId()
@Deprecated public void setSessionId(java.lang.String sessionId)
sessionId
- the new session id.public void setMaxTaskFailuresPerTracker(int noFailures)
noFailures
, the
tasktracker is blacklisted for this job.
noFailures
- maximum no. of failures of a given job per tasktracker.public int getMaxTaskFailuresPerTracker()
public int getMaxMapTaskFailuresPercent()
getMaxMapAttempts()
attempts before being declared as failed.
Defaults to zero
, i.e. any failed map-task results in
the job being declared as JobStatus#FAILED
.
public void setMaxMapTaskFailuresPercent(int percent)
getMaxMapAttempts()
attempts
before being declared as failed.
percent
- the maximum percentage of map tasks that can fail without
the job being aborted.public int getMaxReduceTaskFailuresPercent()
getMaxReduceAttempts()
attempts before being declared as failed.
Defaults to zero
, i.e. any failed reduce-task results
in the job being declared as JobStatus#FAILED
.
public void setMaxReduceTaskFailuresPercent(int percent)
getMaxReduceAttempts()
attempts before being declared as failed.
percent
- the maximum percentage of reduce tasks that can fail without
the job being aborted.public boolean getProfileEnabled()
public void setProfileEnabled(boolean newValue)
newValue
- true means it should be gatheredpublic java.lang.String getProfileParams()
public void setProfileParams(java.lang.String value)
value
- the configuration stringpublic Configuration.IntegerRanges getProfileTaskRange(boolean isMap)
isMap
- is the task a map?
public void setProfileTaskRange(boolean isMap, java.lang.String newValue)
newValue
- a set of integer ranges of the map idspublic void setMapDebugScript(java.lang.String mDbgScript)
The debug script can aid debugging of failed map tasks. The script is given task's stdout, stderr, syslog, jobconf files as arguments.
The debug command, run on the node where the map failed, is:
$script $stdout $stderr $syslog $jobconf.
The script file is distributed through DistributedCache
APIs. The script needs to be symlinked.
Here is an example on how to submit a script
job.setMapDebugScript("./myscript"); DistributedCache.createSymlink(job); DistributedCache.addCacheFile("/debug/scripts/myscript#myscript");
mDbgScript
- the script namepublic java.lang.String getMapDebugScript()
setMapDebugScript(String)
public void setReduceDebugScript(java.lang.String rDbgScript)
The debug script can aid debugging of failed reduce tasks. The script is given task's stdout, stderr, syslog, jobconf files as arguments.
The debug command, run on the node where the map failed, is:
$script $stdout $stderr $syslog $jobconf.
The script file is distributed through DistributedCache
APIs. The script file needs to be symlinked
Here is an example on how to submit a script
job.setReduceDebugScript("./myscript"); DistributedCache.createSymlink(job); DistributedCache.addCacheFile("/debug/scripts/myscript#myscript");
rDbgScript
- the script namepublic java.lang.String getReduceDebugScript()
setReduceDebugScript(String)
public java.lang.String getJobEndNotificationURI()
null
if it hasn't
been set.setJobEndNotificationURI(String)
public void setJobEndNotificationURI(java.lang.String uri)
The uri can contain 2 special parameters: $jobId and $jobStatus. Those, if present, are replaced by the job's identifier and completion-status respectively.
This is typically used by application-writers to implement chaining of Map-Reduce jobs in an asynchronous manner.
uri
- the job end notification uriJobStatus
,
Job Completion and Chainingpublic java.lang.String getJobLocalDir()
When a job starts, a shared directory is created at location
${mapreduce.cluster.local.dir}/taskTracker/$user/jobcache/$jobid/work/
.
This directory is exposed to the users through
mapreduce.job.local.dir
.
So, the tasks can use this space
as scratch space and share files among them.
public long getMemoryForMapTask()
DISABLED_MEMORY_LIMIT
.
For backward compatibility, if the job configuration sets the
key MAPRED_TASK_MAXVMEM_PROPERTY
to a value different
from DISABLED_MEMORY_LIMIT
, that value will be used
after converting it from bytes to MB.
DISABLED_MEMORY_LIMIT
if unset.public void setMemoryForMapTask(long mem)
public long getMemoryForReduceTask()
DISABLED_MEMORY_LIMIT
.
For backward compatibility, if the job configuration sets the
key MAPRED_TASK_MAXVMEM_PROPERTY
to a value different
from DISABLED_MEMORY_LIMIT
, that value will be used
after converting it from bytes to MB.
DISABLED_MEMORY_LIMIT
if unset.public void setMemoryForReduceTask(long mem)
public java.lang.String getQueueName()
public void setQueueName(java.lang.String queueName)
queueName
- Name of the queuepublic static long normalizeMemoryConfigValue(long val)
val
-
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |