Hadoop: Perintah Shell
Sumber: http://www.bogotobogo.com/Hadoop/BigData_Hadoop_fs_commands_list.php
HDFS shell commands
Hadoop Distributed File System (HDFS) is designed to reliably store very large files across machines in a large cluster.
Hadoop file system (fs) shell commands are used to perform various file operations such as copying a file, viewing the contents of the file, changing ownership of files, changing permissions, creating directories etc.
In this tutorial, we'll see basic Hadoop HDFS operations managed through shell commands which are useful to manage files on HDFS clusters.
hadoop fs -ls
To list the contents of a directory:
$ hadoop fs -ls /user Found 8 items drwxr-xr-x - cloudera cloudera 0 2015-03-28 23:43 /user/cloudera drwxr-xr-x - hdfs supergroup 0 2015-03-14 20:11 /user/hdfs drwxr-xr-x - mapred hadoop 0 2015-03-15 14:08 /user/history drwxrwxrwx - hive hive 0 2014-12-18 04:33 /user/hive drwxrwxr-x - hue hue 0 2015-03-21 15:34 /user/hue drwxrwxrwx - oozie oozie 0 2014-12-18 04:34 /user/oozie drwxr-xr-x - sample sample 0 2015-03-14 22:05 /user/sample drwxr-xr-x - spark spark 0 2014-12-18 04:34 /user/spark
upload: hadoop fs -cat
To see contents of a file:
$ hadoop fs -cat /user/hdfs/dir1/myfile This is myfile
hadoop fs -mkdir
To create a directory in HDFS:
$ hadoop fs -mkdir /user/hdfs/dir1 mkdir: Permission denied: user=cloudera, access=WRITE, inode="/user/hdfs":hdfs:supergroup:drwxr-xr-x
$ sudo su hdfs bash-4.1$ hadoop fs -mkdir /user/hdfs/dir1
upload: hadoop fs -put
To copy src files from local file system to the Hadoop data file system:
$ hadoop fs -put <local_src> <HDFS_dest_Path>
bash-4.1$ whoami hdfs
bash-4.1$ hadoop fs -put myfile /user/hdfs/dir1
download: hadoop fs -get
To copy files from the Hadoop data file system to the local file system:
$ hadoop fs -get /user/hdfs/dir1/myfile .
copy: hadoop fs -cp
To copy a file from source to destination:
bash-4.1$ hadoop fs -cp /user/hdfs/dir1/myfile /user/hdfs/dir2
bash-4.1$ hadoop fs -ls /user/hdfs/dir2/myfile -rw-r--r-- 1 hdfs supergroup 7 2015-03-29 11:43 /user/hdfs/dir2/myfile
copyFromLocal: hadoop fs -copyFromLocal
To copy a file from Local file system to HDFS
hadoop fs -copyFromLocal <localsrc> URI
bash-4.1$ pwd /home/cloudera/workspace/temp
bash-4.1$ ls myfile myfile2
bash-4.1$ hadoop fs -copyFromLocal myfile2 /user/hdfs/dir2/
bash-4.1$ hadoop fs -cat /user/hdfs/dir2/myfile2 This is myfile2
copyToLocal: hadoop fs -copyToLocal
To copy a file to Local file system from HDFS
$ pwd /home/cloudera/workspace/temp
$ ls myfile myfile2
$ hadoop fs -copyToLocal /user/hdfs/dir1/myfile myfile3
$ ls myfile myfile2 myfile3
hadoop fs -rm
To remove a file from HDFS
bash-4.1$ hadoop fs -ls /user/hdfs/dir2 Found 2 items -rw-r--r-- 1 hdfs supergroup 7 2015-03-29 11:43 /user/hdfs/dir2/myfile -rw-r--r-- 1 hdfs supergroup 16 2015-03-29 11:52 /user/hdfs/dir2/myfile2
bash-4.1$ hadoop fs -rm /user/hdfs/dir2/myfile2
bash-4.1$ hadoop fs -ls /user/hdfs/dir2 -rw-r--r-- 1 hdfs supergroup 7 2015-03-29 11:43 /user/hdfs/dir2/myfile
directory remove: hadoop fs -rm -r
To remove a directory from HDFS
bash-4.1$ hadoop fs -ls /user/hdfs/ Found 3 items drwxr-xr-x - hdfs supergroup 0 2015-03-28 14:08 /user/hdfs/.Trash drwxr-xr-x - hdfs supergroup 0 2015-03-29 10:55 /user/hdfs/dir1 drwxr-xr-x - hdfs supergroup 0 2015-03-29 12:12 /user/hdfs/dir2
bash-4.1$ hadoop fs -rm -r /user/hdfs/dir2
bash-4.1$ hadoop fs -ls /user/hdfs/ drwxr-xr-x - hdfs supergroup 0 2015-03-28 14:08 /user/hdfs/.Trash drwxr-xr-x - hdfs supergroup 0 2015-03-29 10:55 /user/hdfs/dir1