hdfs的命令
hdfs的命令
9.5、hdfs的命令行使用
Usage: hdfs dfs -ls [-R] <args>
Options:
- The -R option will return stat recursively through the directory structure.
For a file returns stat on the file with the following format:
permissions number_of_replicas userid groupid filesize modification_date modification_time filename
For a directory it returns list of its direct children as in Unix. A directory is listed as:
permissions userid groupid modification_date modification_time dirname
Example:
- hdfs dfs -ls /user/hadoop/file1
Exit Code:
Returns 0 on success and -1 on error.
Usage: hdfs dfs -lsr <args>
Recursive version of ls.
Note: This command is deprecated. Instead use hdfs dfs -ls -R
Usage: hdfs dfs -mkdir [-p] <paths>
Takes path uri's as argument and creates directories.
Options:
- The -p option behavior is much like Unix mkdir -p, creating parent directories along the path.
Example:
- hdfs dfs -mkdir /user/hadoop/dir1 /user/hadoop/dir2
- hdfs dfs -mkdir hdfs://nn1.example.com/user/hadoop/dir hdfs://nn2.example.com/user/hadoop/dir
Exit Code:
Returns 0 on success and -1 on error.
Usage: hdfs dfs -moveFromLocal <localsrc> <dst>
Similar to put command, except that the source localsrc is deleted after it's copied.
Usage: hdfs dfs -moveToLocal [-crc] <src> <dst>
Displays a "Not implemented yet" message.
Usage: hdfs dfs -mv URI [URI ...] <dest>
Moves files from source to destination. This command allows multiple sources as well in which case the destination needs to be a directory. Moving files across file systems is not permitted.
Example:
- hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2
- hdfs dfs -mv hdfs://nn.example.com/file1 hdfs://nn.example.com/file2 hdfs://nn.example.com/file3 hdfs://nn.example.com/dir1
Exit Code:
Returns 0 on success and -1 on error.
Usage: hdfs dfs -put <localsrc> ... <dst>
Copy single src, or multiple srcs from local file system to the destination file system. Also reads input from stdin and writes to destination file system.
- hdfs dfs -put localfile /user/hadoop/hadoopfile
- hdfs dfs -put localfile1 localfile2 /user/hadoop/hadoopdir
- hdfs dfs -put localfile hdfs://nn.example.com/hadoop/hadoopfile
- hdfs dfs -put - hdfs://nn.example.com/hadoop/hadoopfile Reads the input from stdin.
Exit Code:
Returns 0 on success and -1 on error.
Usage: hdfs dfs -appendToFile <localsrc> ... <dst>
追加一个或者多个文件到hdfs指定文件中.也可以从命令行读取输入.
- hdfs dfs -appendToFile localfile /user/hadoop/hadoopfile
- hdfs dfs -appendToFile localfile1 localfile2 /user/hadoop/hadoopfile
- hdfs dfs -appendToFile localfile hdfs://nn.example.com/hadoop/hadoopfile
- hdfs dfs -appendToFile - hdfs://nn.example.com/hadoop/hadoopfile Reads the input from stdin.
Exit Code:
Returns 0 on success and 1 on error.
Usage: hdfs dfs -cat URI [URI ...]
查看内容.
Example:
- hdfs dfs -cat hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2
- hdfs dfs -cat file:///file3 /user/hadoop/file4
Exit Code:
Returns 0 on success and -1 on error.
Usage: hdfs dfs -cp [-f] [-p | -p[topax]] URI [URI ...] <dest>
复制文件(夹)，可以覆盖，可以保留原有权限信息
Options:
- The -f option will overwrite the destination if it already exists.
- The -p option will preserve file attributes [topx] (timestamps, ownership, permission, ACL, XAttr). If -p is specified with no arg, then preserves timestamps, ownership, permission. If -pa is specified, then preserves permission also because ACL is a super-set of permission. Determination of whether raw namespace extended attributes are preserved is independent of the -p flag.
Example:
- hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2
- hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir
Exit Code:
Returns 0 on success and -1 on error.
Usage: hdfs dfs -rm [-f] [-r|-R] [-skipTrash] URI [URI ...]
Delete files specified as args.
Options:
- The -f option will not display a diagnostic message or modify the exit status to reflect an error if the file does not exist.
- The -R option deletes the directory and any content under it recursively.
- The -r option is equivalent to -R.
- The -skipTrash option will bypass trash, if enabled, and delete the specified file(s) immediately. This can be useful when it is necessary to delete files from an over-quota directory.
Example:
- hdfs dfs -rm hdfs://nn.example.com/file /user/hadoop/emptydir
Exit Code:
Returns 0 on success and -1 on error.
Usage: hdfs dfs -rmr [-skipTrash] URI [URI ...]
Recursive version of delete.
Note: This command is deprecated. Instead use hdfs dfs -rm -r
Usage: hdfs dfs -chmod [-R] <MODE[,MODE]... | OCTALMODE> URI [URI ...]
修改权限.
Options
- The -R option will make the change recursively through the directory structure.
Usage: hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ]
修改所有者.
Options
- The -R option will make the change recursively through the directory structure.
expunge
Usage: hdfs dfs -expunge
清空回收站.
9.6、hdfs的高级使用命令
1、HDFS文件限额配置
hdfs文件的限额配置允许我们以文件大小或者文件个数来限制我们在某个目录下上传的文件数量或者文件内容总量，以便达到我们类似百度网盘网盘等限制每个用户允许上传的最大的文件的量
1、数量限额
hdfs dfs -mkdir -p /user/root/lisi #创建hdfs文件夹
hdfs dfsadmin -setQuota 2 lisi # 给该文件夹下面设置最多上传两个文件，上传文件，发现只能上传一个文件
hdfs dfsadmin -clrQuota /user/root/lisi # 清除文件数量限制
2、空间大小限额
hdfs dfsadmin -setSpaceQuota 4k /user/root/lisi # 限制空间大小4KB
hdfs dfs -put /export/softwares/zookeeper-3.4.5-cdh5.14.0.tar.gz /user/root/lisi
#上传超过4Kb的文件大小上去提示文件超过限额
hdfs dfsadmin -clrSpaceQuota /user/root/lisi #清除空间限额
hdfs dfs -put /export/softwares/zookeeper-3.4.5-cdh5.14.0.tar.gz /user/root/lisi
#重新上传成功
查看hdfs文件限额数量
hdfs dfs -count -q -h /user/root/lisi
2、hdfs的安全模式
安全模式是HDFS所处的一种特殊状态，在这种状态下，文件系统只接受读数据请求，而不接受删除、修改等变更请求。在NameNode主节点启动时，HDFS首先进入安全模式，DataNode在启动的时候会向namenode汇报可用的block等状态，当整个系统达到安全标准时，HDFS自动离开安全模式。如果HDFS出于安全模式下，则文件block不能进行任何的副本复制操作，因此达到最小的副本数量要求是基于datanode启动时的状态来判定的，启动时不会再做任何复制（从而达到最小副本数量要求），hdfs集群刚启动的时候，默认30S钟的时间是出于安全期的，只有过了30S之后，集群脱离了安全期，然后才可以对集群进行操作
hdfs dfsadmin -safemode