org.apache.nutch.fetcher
Class Fetcher
java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.nutch.fetcher.Fetcher
- All Implemented Interfaces:
- org.apache.hadoop.conf.Configurable, org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.mapred.MapRunnable<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>, org.apache.hadoop.util.Tool
public class Fetcher
- extends org.apache.hadoop.conf.Configured
- implements org.apache.hadoop.util.Tool, org.apache.hadoop.mapred.MapRunnable<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>
The fetcher. Most of the work is done by plugins.
Constructor Summary |
Fetcher()
|
Fetcher(org.apache.hadoop.conf.Configuration conf)
|
Method Summary |
void |
close()
|
void |
configure(org.apache.hadoop.mapred.JobConf job)
|
void |
fetch(org.apache.hadoop.fs.Path segment,
int threads)
|
static boolean |
isParsing(org.apache.hadoop.conf.Configuration conf)
|
static boolean |
isStoringContent(org.apache.hadoop.conf.Configuration conf)
|
static void |
main(String[] args)
Run the fetcher. |
void |
run(org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable> input,
org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,NutchWritable> output,
org.apache.hadoop.mapred.Reporter reporter)
|
int |
run(String[] args)
|
Methods inherited from class org.apache.hadoop.conf.Configured |
getConf, setConf |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Methods inherited from interface org.apache.hadoop.conf.Configurable |
getConf, setConf |
LOG
public static final org.apache.commons.logging.Log LOG
PERM_REFRESH_TIME
public static final int PERM_REFRESH_TIME
- See Also:
- Constant Field Values
CONTENT_REDIR
public static final String CONTENT_REDIR
- See Also:
- Constant Field Values
PROTOCOL_REDIR
public static final String PROTOCOL_REDIR
- See Also:
- Constant Field Values
Fetcher
public Fetcher()
Fetcher
public Fetcher(org.apache.hadoop.conf.Configuration conf)
configure
public void configure(org.apache.hadoop.mapred.JobConf job)
- Specified by:
configure
in interface org.apache.hadoop.mapred.JobConfigurable
close
public void close()
isParsing
public static boolean isParsing(org.apache.hadoop.conf.Configuration conf)
isStoringContent
public static boolean isStoringContent(org.apache.hadoop.conf.Configuration conf)
run
public void run(org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable> input,
org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,NutchWritable> output,
org.apache.hadoop.mapred.Reporter reporter)
throws IOException
- Specified by:
run
in interface org.apache.hadoop.mapred.MapRunnable<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>
- Throws:
IOException
fetch
public void fetch(org.apache.hadoop.fs.Path segment,
int threads)
throws IOException
- Throws:
IOException
main
public static void main(String[] args)
throws Exception
- Run the fetcher.
- Throws:
Exception
run
public int run(String[] args)
throws Exception
- Specified by:
run
in interface org.apache.hadoop.util.Tool
- Throws:
Exception
Copyright © 2006 The Apache Software Foundation