Hadoop: Menjalankan MapReduce Job
Sumber: http://www.bogotobogo.com/Hadoop/BigData_hadoop_Running_MapReduce_Job.php
MapReduce Preparation
Before we jump into the programming our MapReduce, we may need to talk about the preparation steps that are commonly taken. Because MapReduce is usually operating on a huge data, we need to consider those steps before we actually do the MapReduce.
The underlying structure of the HDFS filesystem is very different from our normal file systems. The block sizes are quite a bit larger, and the actual block size for our clusters dependent on the cluster configuration as shown in the picture below: 64, 128, or 256 MB. So, we may need to have blocks with customized partitioned. MapRPrep.png
Picture source : Hadoop MapReduce Fundamentals.
Another consideration is where we're going to retrieve our data from in order to perform the MapReduce operations or the parallel processing on it. Though we'll work with the core Hadoop filesystem, we may execute MapReduce algorithms against information stored on different locations such as native filesystem, cloud storage such as Amazon S3 buckets, or Windows Azure blobs.
Another considration is the output of the MapReduce job results are immutable. So, our output is a one-time output, and when a new output is generated, we have a new file name for it.
The last consideration in preparing for MapReduce is about the logic that we'll be writing, and it should fit our situation that we're trying to address. We'll be writing logic in some programming language, library, or tools to map our data to, and then reduce it, and then we have some output.
Note also that we'll be working with key-value pairs, so regardless of the format of the data coming in, we want to output key-value pairs.
Hadoop shell commands
Before performing MapReduce jobs, we should be familiar with some of the Hadoop shell commands. Please visit List of Apache Hadoop hdfs commands.
Running a MapReduce Job
Now it's time to run our first Hadoop MapReduce job. We will use one of the examples that come with Hadoop package.
hduser@k:~$ cd /usr/local/hadoop
hduser@k:/usr/local/hadoop$ ls bin include libexec logs README.txt share etc lib LICENSE.txt NOTICE.txt sbin
hduser@k:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar pi 2 5 Number of Maps = 2 Samples per Map = 5 14/07/14 01:28:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Wrote input for Map #0 Wrote input for Map #1 Starting Job 14/07/14 01:28:07 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 14/07/14 01:28:07 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 14/07/14 01:28:07 INFO input.FileInputFormat: Total input paths to process : 2 14/07/14 01:28:07 INFO mapreduce.JobSubmitter: number of splits:2 14/07/14 01:28:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1228885165_0001 ... File Input Format Counters Bytes Read=236 File Output Format Counters Bytes Written=97 Job Finished in 6.072 seconds Estimated value of Pi is 3.60000000000000000000 hduser@k:/usr/local/hadoop$
Hadoop FileSystem (HDFS)
Files are stored in the Hadoop Distributed File System (HDFS). Suppose we're going to store a file called data.txt in HDFS.
This file is 160 megabytes. When a file is loaded into HDFS, it's split into chunks which are called blocks. The default size of each block is 64 megabytes. Each block is given a unique name, which is blk, an underscore, and a large number. In our case, the first block is 64 megabytes. The second block is 64 megabytes. The third block is the remaining 32 megabytes, to make up our 160 megabyte file.
HDFS_Cloud.png
As the file is uploaded to HDFS, each block will get stored on one node in the cluster. There's a Daemon running on each of the machines in the cluster, and it is called the DataNode. Now, we need to know which blocks make up the original file. And that's handled by a separate machine, running the Daemon called the NameNode. The information stored on the NameNode is known as the Metadata.
NoSQL.png
HDFS Commands
While Hadoop is running, let's create hdfsTest.txt in our home directory:
hduser@k:~$ echo "hdfs test" > hdfsTest.txt
Then, we want to create Home Directory in HDFS :
hduser@ubuntu:~$ hadoop fs -mkdir -p /user/hduser
We can copy file hdfsTest.txt from local disk to the user's directory in HDFS:
hduser@ubuntu:~$ hadoop fs -copyFromLocal hdfsTest.txt hdfsTest.txt
We could have used put instead of copyFromLocal:
hduser@ubuntu:~$ hadoop fs -put hdfsTest.txt
Get a directory listing of the user's home directory in HDFS:
hduser@k:~$ hadoop fs -ls Found 1 items -rw-r--r-- 1 hduser supergroup 5 2014-07-14 01:49 hdfsTest.txt
If we want to display the contents of the HDFS file /user/hduser/hdfsTest.txt:
hduser@ubuntu:~$ hadoop fs -cat /user/hduser/hdfsTest.txt
copy that file to the local disk from HDFS, named as hdfsTest2.txt :
hduser@k:~$ hadoop fs -copyToLocal /user/hduser/hdfsTest.txt hdfsTest2.txt
hduser@k:~$ ls hdfsTest2.txt hdfsTest.txt
To delete the file from Hadoop HDFS:
hduser@k:~$ hadoop fs -rm hdfsTest.txt
hduser@k:~$ hadoop fs -ls hduser@k:~$
Hadoop Setup for Development
HadoopSetup.png
Picture source : Hadoop MapReduce Fundamentals.
Throughout my tutorials on Hadoop Echo Systems, I used:
Hadoop Binaries - Local (Linux), Cloudera's Demo VM, and AWS for Cloud. Data Storage - Local (HDFS Pseudo-distributed, single-node) and Cloud. MapReduce - Both Local and Cloud.
Ways to MapReduce
Java is the most common language to use, but other languages can be used: WayToMapReduces.png
Picture source : Hadoop MapReduce Fundamentals.