Difference between revisions of "Hadoop: Perintah Shell"
Onnowpurbo (talk | contribs) (New page: Sumber: http://www.bogotobogo.com/Hadoop/BigData_Hadoop_fs_commands_list.php ==HDFS shell commands== Hadoop Distributed File System (HDFS) is designed to reliably store very large file...) |
Onnowpurbo (talk | contribs) |
||
| (One intermediate revision by the same user not shown) | |||
| Line 22: | Line 22: | ||
To list the contents of a directory: | To list the contents of a directory: | ||
| − | $ hadoop fs -ls /user | + | $ hadoop fs -ls /user |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| + | Found 1 items | ||
| + | drwxr-xr-x - hduser supergroup 0 2015-11-09 16:35 /user/hduser | ||
| Line 41: | Line 32: | ||
To see contents of a file: | To see contents of a file: | ||
| − | $ hadoop fs -cat /user/hdfs/dir1/myfile | + | $ hadoop fs -cat /user/hdfs/dir1/myfile |
| − | This is myfile | + | This is myfile |
| − | |||
| − | |||
| − | |||
| − | |||
==hadoop fs -mkdir== | ==hadoop fs -mkdir== | ||
| Line 52: | Line 39: | ||
To create a directory in HDFS: | To create a directory in HDFS: | ||
| − | $ hadoop fs -mkdir /user/hdfs/dir1 | + | $ hadoop fs -mkdir /user/hdfs/dir1 |
| − | mkdir: Permission denied: user=cloudera, access=WRITE, inode="/user/hdfs":hdfs:supergroup:drwxr-xr-x | + | mkdir: Permission denied: user=cloudera, access=WRITE, inode="/user/hdfs":hdfs:supergroup:drwxr-xr-x |
| − | $ sudo su hdfs | + | $ sudo su hdfs |
| − | bash-4.1$ hadoop fs -mkdir /user/hdfs/dir1 | + | bash-4.1$ hadoop fs -mkdir /user/hdfs/dir1 |
| Line 66: | Line 53: | ||
To copy src files from local file system to the Hadoop data file system: | To copy src files from local file system to the Hadoop data file system: | ||
| − | $ hadoop fs -put <local_src> <HDFS_dest_Path> | + | $ hadoop fs -put <local_src> <HDFS_dest_Path> |
| − | bash-4.1$ whoami | + | bash-4.1$ whoami |
| − | hdfs | + | hdfs |
| − | bash-4.1$ hadoop fs -put myfile /user/hdfs/dir1 | + | bash-4.1$ hadoop fs -put myfile /user/hdfs/dir1 |
| Line 77: | Line 64: | ||
| − | download: hadoop fs -get | + | ==download: hadoop fs -get== |
To copy files from the Hadoop data file system to the local file system: | To copy files from the Hadoop data file system to the local file system: | ||
| − | $ hadoop fs -get /user/hdfs/dir1/myfile . | + | $ hadoop fs -get /user/hdfs/dir1/myfile . |
| Line 91: | Line 78: | ||
To copy a file from source to destination: | To copy a file from source to destination: | ||
| − | bash-4.1$ hadoop fs -cp /user/hdfs/dir1/myfile /user/hdfs/dir2 | + | bash-4.1$ hadoop fs -cp /user/hdfs/dir1/myfile /user/hdfs/dir2 |
| − | bash-4.1$ hadoop fs -ls /user/hdfs/dir2/myfile | + | bash-4.1$ hadoop fs -ls /user/hdfs/dir2/myfile |
| − | -rw-r--r-- 1 hdfs supergroup 7 2015-03-29 11:43 /user/hdfs/dir2/myfile | + | -rw-r--r-- 1 hdfs supergroup 7 2015-03-29 11:43 /user/hdfs/dir2/myfile |
| Line 106: | Line 93: | ||
hadoop fs -copyFromLocal <localsrc> URI | hadoop fs -copyFromLocal <localsrc> URI | ||
| − | bash-4.1$ pwd | + | bash-4.1$ pwd |
| − | /home/cloudera/workspace/temp | + | /home/cloudera/workspace/temp |
| − | bash-4.1$ ls | + | bash-4.1$ ls |
| − | myfile myfile2 | + | myfile myfile2 |
| − | bash-4.1$ hadoop fs -copyFromLocal myfile2 /user/hdfs/dir2/ | + | bash-4.1$ hadoop fs -copyFromLocal myfile2 /user/hdfs/dir2/ |
| − | bash-4.1$ hadoop fs -cat /user/hdfs/dir2/myfile2 | + | bash-4.1$ hadoop fs -cat /user/hdfs/dir2/myfile2 |
| − | This is myfile2 | + | This is myfile2 |
| Line 125: | Line 112: | ||
To copy a file to Local file system from HDFS | To copy a file to Local file system from HDFS | ||
| − | $ pwd | + | $ pwd |
| − | /home/cloudera/workspace/temp | + | /home/cloudera/workspace/temp |
| − | $ ls | + | $ ls |
| − | myfile myfile2 | + | myfile myfile2 |
| − | $ hadoop fs -copyToLocal /user/hdfs/dir1/myfile myfile3 | + | $ hadoop fs -copyToLocal /user/hdfs/dir1/myfile myfile3 |
| − | $ ls | + | $ ls |
| − | myfile myfile2 myfile3 | + | myfile myfile2 myfile3 |
Latest revision as of 17:25, 9 November 2015
Sumber: http://www.bogotobogo.com/Hadoop/BigData_Hadoop_fs_commands_list.php
HDFS shell commands
Hadoop Distributed File System (HDFS) is designed to reliably store very large files across machines in a large cluster.
Hadoop file system (fs) shell commands are used to perform various file operations such as copying a file, viewing the contents of the file, changing ownership of files, changing permissions, creating directories etc.
In this tutorial, we'll see basic Hadoop HDFS operations managed through shell commands which are useful to manage files on HDFS clusters.
hadoop fs -ls
To list the contents of a directory:
$ hadoop fs -ls /user
Found 1 items drwxr-xr-x - hduser supergroup 0 2015-11-09 16:35 /user/hduser
upload: hadoop fs -cat
To see contents of a file:
$ hadoop fs -cat /user/hdfs/dir1/myfile This is myfile
hadoop fs -mkdir
To create a directory in HDFS:
$ hadoop fs -mkdir /user/hdfs/dir1 mkdir: Permission denied: user=cloudera, access=WRITE, inode="/user/hdfs":hdfs:supergroup:drwxr-xr-x
$ sudo su hdfs bash-4.1$ hadoop fs -mkdir /user/hdfs/dir1
upload: hadoop fs -put
To copy src files from local file system to the Hadoop data file system:
$ hadoop fs -put <local_src> <HDFS_dest_Path>
bash-4.1$ whoami hdfs
bash-4.1$ hadoop fs -put myfile /user/hdfs/dir1
download: hadoop fs -get
To copy files from the Hadoop data file system to the local file system:
$ hadoop fs -get /user/hdfs/dir1/myfile .
copy: hadoop fs -cp
To copy a file from source to destination:
bash-4.1$ hadoop fs -cp /user/hdfs/dir1/myfile /user/hdfs/dir2
bash-4.1$ hadoop fs -ls /user/hdfs/dir2/myfile -rw-r--r-- 1 hdfs supergroup 7 2015-03-29 11:43 /user/hdfs/dir2/myfile
copyFromLocal: hadoop fs -copyFromLocal
To copy a file from Local file system to HDFS
hadoop fs -copyFromLocal <localsrc> URI
bash-4.1$ pwd /home/cloudera/workspace/temp
bash-4.1$ ls myfile myfile2
bash-4.1$ hadoop fs -copyFromLocal myfile2 /user/hdfs/dir2/
bash-4.1$ hadoop fs -cat /user/hdfs/dir2/myfile2 This is myfile2
copyToLocal: hadoop fs -copyToLocal
To copy a file to Local file system from HDFS
$ pwd /home/cloudera/workspace/temp
$ ls myfile myfile2
$ hadoop fs -copyToLocal /user/hdfs/dir1/myfile myfile3
$ ls myfile myfile2 myfile3
hadoop fs -rm
To remove a file from HDFS
bash-4.1$ hadoop fs -ls /user/hdfs/dir2 Found 2 items -rw-r--r-- 1 hdfs supergroup 7 2015-03-29 11:43 /user/hdfs/dir2/myfile -rw-r--r-- 1 hdfs supergroup 16 2015-03-29 11:52 /user/hdfs/dir2/myfile2
bash-4.1$ hadoop fs -rm /user/hdfs/dir2/myfile2
bash-4.1$ hadoop fs -ls /user/hdfs/dir2 -rw-r--r-- 1 hdfs supergroup 7 2015-03-29 11:43 /user/hdfs/dir2/myfile
directory remove: hadoop fs -rm -r
To remove a directory from HDFS
bash-4.1$ hadoop fs -ls /user/hdfs/ Found 3 items drwxr-xr-x - hdfs supergroup 0 2015-03-28 14:08 /user/hdfs/.Trash drwxr-xr-x - hdfs supergroup 0 2015-03-29 10:55 /user/hdfs/dir1 drwxr-xr-x - hdfs supergroup 0 2015-03-29 12:12 /user/hdfs/dir2
bash-4.1$ hadoop fs -rm -r /user/hdfs/dir2
bash-4.1$ hadoop fs -ls /user/hdfs/ drwxr-xr-x - hdfs supergroup 0 2015-03-28 14:08 /user/hdfs/.Trash drwxr-xr-x - hdfs supergroup 0 2015-03-29 10:55 /user/hdfs/dir1