Difference between revisions of "Hadoop: Contoh Program Sederhana"

Latest revision as of 16:58, 9 November 2015

Sumber:

Source Code

Contoh source code WordCount.java untuk menghitung jumlah masing-masing kata dari sebuah input set.

cd ~
vi WordCount.java

package org.myorg;

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
	
public class WordCount {
	
   public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
     private final static IntWritable one = new IntWritable(1);
     private Text word = new Text();

     public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
       String line = value.toString();
       StringTokenizer tokenizer = new StringTokenizer(line);
       while (tokenizer.hasMoreTokens()) {
         word.set(tokenizer.nextToken());
         output.collect(word, one);
       }
     }
   }

   public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
     public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
       int sum = 0;
       while (values.hasNext()) {
         sum += values.next().get();
       }
       output.collect(key, new IntWritable(sum));
     }
   }

   public static void main(String[] args) throws Exception {
     JobConf conf = new JobConf(WordCount.class);
     conf.setJobName("wordcount"); 
	
     conf.setOutputKeyClass(Text.class);
     conf.setOutputValueClass(IntWritable.class);

     conf.setMapperClass(Map.class);
     conf.setCombinerClass(Reduce.class);
     conf.setReducerClass(Reduce.class);

     conf.setInputFormat(TextInputFormat.class);
     conf.setOutputFormat(TextOutputFormat.class);

     FileInputFormat.setInputPaths(conf, new Path(args[0]));
     FileOutputFormat.setOutputPath(conf, new Path(args[1]));

     JobClient.runJob(conf);
   }
}

Compile

Asumsinya HADOOP_HOME adalah root instalasi dan HADOOP_VERSION adalah versi Hadoop yang di install, compile WordCount.java dan buat jar:

export HADOOP_HOME=/usr/local/hadoop/share/hadoop/common
export HADOOP_VERSION=2.7.1

mkdir wordcount_classes
javac -classpath ${HADOOP_HOME}/hadoop-common-${HADOOP_VERSION}.jar -d wordcount_classes WordCount.java
jar -cvf /usr/joe/wordcount.jar -C wordcount_classes/ .

Penggunaan

Asumsi

/usr/joe/wordcount/input - input directory di HDFS
/usr/joe/wordcount/output - output directory di HDFS

Sample text-files as input:

bin/hadoop dfs -ls /usr/joe/wordcount/input/
/usr/joe/wordcount/input/file01
/usr/joe/wordcount/input/file02

bin/hadoop dfs -cat /usr/joe/wordcount/input/file01
Hello World Bye World

bin/hadoop dfs -cat /usr/joe/wordcount/input/file02
Hello Hadoop Goodbye Hadoop

Jalankan aplikasi

bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output

Output:

bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2

Difference between revisions of "Hadoop: Contoh Program Sederhana"

Latest revision as of 16:58, 9 November 2015

Contents

Source Code

Compile

Penggunaan

Referensi

Navigation menu

Search

@@ Line 1: / Line 1: @@
-Sumber: http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197
+Sumber:
+* http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197
+* https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html
-Hadoop: Writing and Running Your First Project
-MapReduce on small datasets can be run easily and without much coding or fiddling — provided you know what to do. Here's how.
-In the first part of this series on Apache Hadoop, I explained how MapReduce works at a conceptual level. In this installment, the second of three, I show how to write code that runs on Hadoop — starting with a MapReduce program in Java.
-Development Environment
-To get started, we need Java (Oracle JDK 6 is required), Git, Maven, and Hadoop itself. Download the latest stable release of Apache Hadoop (1.0.4) from the releases page, then extract it to a suitable place. On my laptop:
- % tar zxf hadoop-1.0.4.tar.gz
- % export HADOOP_HOME=$(pwd)/hadoop-1.0.4
- % $HADOOP_HOME/bin/hadoop version
- Hadoop 1.0.4
-In another directory, checkout the Git repository that accompanies this article:
- % git clone git://github.com/tomwhite/hadoop-drdobbs.git
- % cd hadoop-drdobbs
- % mvn install
-The repository contains a small amount of sample data for testing:
- % cat data/*.tsv
- dobbs	2007	20	18	15
- dobbs	2008	22	20	12
- doctor	2007	545525	366136	57313
- doctor	2008	668666	446034	72694
-The file contains a few lines from the Google Books Ngram Dataset, which I mentioned  in the first part of the series. To recap, the first line says that the word "dobbs" in books from 2007 occurred 20 times overall, and these occurrences were found on 18 pages in 15 books.
-==Java MapReduce==
-Let's write the MapReduce job to find the total count for each word. We start with the map function, which is represented in Java by an instance of org.apache.hadoop.mapreduce.Mapper. The first thing we need to decide about our mapper is the types of the input key-value pairs and the output key-value pairs. The declaration of the Mapper class is:
- public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>
+==Source Code==
-Because we are processing text, we use TextInputFormat, which determines the input types for us as LongWritable and Text (both found in the org.apache.hadoop.io package). These Writable types are wrappers around standard Java types (long and String, in this case)  optimized for efficiency of serialization. Authors of MapReduce programs can use the Writable types without worrying about serialization. The only time that you might be exposed to serialization is when writing a custom Writable type. In such cases, it is usually better to use a serialization library, such as Avro.
+Contoh source code WordCount.java untuk menghitung jumlah masing-masing kata dari sebuah input set.
-Going back to the input type, TextInputFormat presents the input to our mapper as (LongWritable, Text) pairs, like this:
+ cd ~
+ vi WordCount.java
- (0,  "dobbs	2007	20	18	15")
- (20, "dobbs	2008	22	20	12")
- (40, "doctor	2007	545525	366136	57313")
- (72, "doctor	2008	668666	446034	72694")
-The key is the offset within the file, and the value is the content of the line. It is the job of the mapper to extract the word and the number of occurrences, and ignore everything else. Therefore, its output is (word, count) pairs, of type (Text, LongWritable). The signature of the mapper looks like this:
- public class ProjectionMapper extends Mapper<LongWritable, Text, Text, LongWritable>
+  package org.myorg;
-All that remains is for us to write the implementation of the map() method. The source for the whole mapper class appears in Listing One (ProjectionMapper.java).
-Listing One: ProjectionMapper.java.
-  package com.tom_e_white.drdobbs.mapreduce;
- import org.apache.hadoop.io.LongWritable;
-  import org.apache.hadoop.io.Text;
- import org.apache.hadoop.mapreduce.Mapper;
   import java.io.IOException;
+ import java.util.*;
-  public class ProjectionMapper extends Mapper<LongWritable, Text, Text, LongWritable> {
-   private Text word = new Text();
+ import org.apache.hadoop.fs.Path;
-   private LongWritable count = new LongWritable();
+ import org.apache.hadoop.conf.*;
+ import org.apache.hadoop.io.*;
-   @Override
+ import org.apache.hadoop.mapred.*;
-   protected void map(LongWritable key, Text value, Context context)
+ import org.apache.hadoop.util.*;
-       throws IOException, InterruptedException {
-     // value is tab separated values: word, year, occurrences, #books, #pages
+  public class WordCount {
-     // we project out (word, occurrences) so we can sum over all years
-     String[] split = value.toString().split("\t+");
+    public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
-     word.set(split[0]);
+      private final static IntWritable one = new IntWritable(1);
-     if (split.length > 2) {
+      private Text word = new Text();
-       try {
-         count.set(Long.parseLong(split[2]));
+      public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
-         context.write(word, count);
+        String line = value.toString();
-       } catch (NumberFormatException e) {
+        StringTokenizer tokenizer = new StringTokenizer(line);
-         // cannot parse - ignore
+        while (tokenizer.hasMoreTokens()) {
-       }
+          word.set(tokenizer.nextToken());
-     }
+          output.collect(word, one);
-   }
+        }
+      }
+    }
+    public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
+      public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
+        int sum = 0;
+        while (values.hasNext()) {
+          sum += values.next().get();
+        }
+        output.collect(key, new IntWritable(sum));
+      }
+    }
+    public static void main(String[] args) throws Exception {
+      JobConf conf = new JobConf(WordCount.class);
+      conf.setJobName("wordcount");
+      conf.setOutputKeyClass(Text.class);
+      conf.setOutputValueClass(IntWritable.class);
+      conf.setMapperClass(Map.class);
+      conf.setCombinerClass(Reduce.class);
+      conf.setReducerClass(Reduce.class);
+      conf.setInputFormat(TextInputFormat.class);
+      conf.setOutputFormat(TextOutputFormat.class);
+      FileInputFormat.setInputPaths(conf, new Path(args[0]));
+      FileOutputFormat.setOutputPath(conf, new Path(args[1]));
+      JobClient.runJob(conf);
+    }
   }
-There are a few things to note about this code. First, there are two instance variables, word and count, which are used to store the map output key and value. The map() method is called once per input record, so it pays to avoid unnecessary object creation. The body of map() is straightforward: It splits the tab-separated input line into fields, and uses the first field as the word, and the third as the count. The map output is written using the write method in Context. For simplicity, this code ignores lines with an occurrence field that is not a number, but there are other actions you could take, such as incrementing a MapReduce counter to track how many lines it affects (see the getCounter() method on Context for details).
+==Compile==
-Running through our tiny dataset, the map output looks like this:
+Asumsinya HADOOP_HOME adalah root instalasi dan HADOOP_VERSION adalah versi Hadoop yang di install, compile WordCount.java dan buat jar:
-  ("dobbs", 20)
+  export HADOOP_HOME=/usr/local/hadoop/share/hadoop/common
-  ("dobbs", 22)
+  export HADOOP_VERSION=2.7.1
- ("doctor", 545525)
- ("doctor", 668666)
-As I discussed in the first part of the series, Hadoop transforms the map output so that the values are brought together for a given key, in a process called the shuffle. In our abstract representation, the input to the reduce step looks like this:
+ mkdir wordcount_classes
+ javac -classpath ${HADOOP_HOME}/hadoop-common-${HADOOP_VERSION}.jar -d wordcount_classes WordCount.java
+ jar -cvf /usr/joe/wordcount.jar -C wordcount_classes/ .
- ("dobbs", [20, 22])
+==Penggunaan==
- ("doctor", [545525, 668666])
-All our reduce implementation has to do is sum the counts. We need an implementation of org.apache.hadoop.mapreduce.Reducer with the following signature:
+Asumsi
- public class LongSumReducer extends Reducer<
-        Text, LongWritable, Text, LongWritable>
-We could write the class ourselves, but we don't need to because Hadoop comes with an implementation, shown in Listing Two (LongSumReducer.java).
-Listing Two LongSumReducer.java (code from Apache Hadoop project).
- package org.apache.hadoop.mapreduce.lib.reduce;
- import java.io.IOException;
- import org.apache.hadoop.io.LongWritable;
- import org.apache.hadoop.mapreduce.Reducer;
- public class LongSumReducer<KEY> extends Reducer<KEY, LongWritable,
-                                                  KEY,LongWritable> {
-   private LongWritable result = new LongWritable();
-   public void reduce(KEY key, Iterable<LongWritable> values,
-                      Context context) throws IOException, InterruptedException {
-     long sum = 0;
-     for (LongWritable val : values) {
-       sum += val.get();
-     }
-     result.set(sum);
-     context.write(key, result);
-   }
- }
-Notice that the reduce() method signature is different from map() because it has an iterator over the values, rather than a single value. This reflects the grouping that the framework performs on the values for a key. In LongSumReducer, the implementation is very simple: It sums the values, then writes the total out using the same key as the input.
+ /usr/joe/wordcount/input - input directory di HDFS
+ /usr/joe/wordcount/output - output directory di HDFS
-The output of the reducer will be:
+Sample text-files as input:
-  ("dobbs", 42)
+  bin/hadoop dfs -ls /usr/joe/wordcount/input/
-  ("doctor", 1214191)
+  /usr/joe/wordcount/input/file01
+ /usr/joe/wordcount/input/file02
+ bin/hadoop dfs -cat /usr/joe/wordcount/input/file01
+ Hello World Bye World
+ bin/hadoop dfs -cat /usr/joe/wordcount/input/file02
+ Hello Hadoop Goodbye Hadoop
+Jalankan aplikasi
+ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output
+Output:
+ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
+ Bye 1
+ Goodbye 1
+ Hadoop 2
+ Hello 2
+ World 2
@@ Line 157: / Line 117: @@
 * http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197
+* http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197?pgno=2
+* https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html