文档章节

hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息

 张欢19933
发布于 2017/06/26 17:07
字数 1753
阅读 208
收藏 0

在HDFS中,提供了fsck命令,用于检查HDFS上文件和目录的健康状态、获取文件的block信息和位置信息等。

fsck命令必须由HDFS超级用户来执行,普通用户无权限。

[hadoop@dev ~]$ hdfs fsck
Usage: DFSck  [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]]
          start checking from this path
        -move   move corrupted files to /lost+found
        -delete delete corrupted files
        -files  print out files being checked
        -openforwrite   print out files opened for write
        -includeSnapshots       include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it
        -list-corruptfileblocks print out list of missing blocks and files they belong to
        -blocks print out block report
        -locations      print out locations for every block
        -racks  print out network topology for data-node locations
  • -move: 移动损坏的文件到/lost+found目录下
  • -delete: 删除损坏的文件
  • -files: 输出正在被检测的文件
  • -openforwrite: 输出检测中的正在被写的文件
  • -includeSnapshots: 检测的文件包括系统snapShot快照目录下的
  • -list-corruptfileblocks: 输出损坏的块及其所属的文件
  • -blocks: 输出block的详细报告
  • -locations: 输出block的位置信息
  • -racks: 输出block的网络拓扑结构信息
  • -storagepolicies: 输出block的存储策略信息
  • -blockId: 输出指定blockId所属块的状况,位置等信息

查看文件中损坏的块(-list-corruptfileblocks)

[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -list-corruptfileblocks
The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/' has 0 CORRUPT files

将损坏的文件移动至/lost+found目录(-move)

[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 -move
FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 at Thu Aug 13 09:36:35 CST 2015
.Status: HEALTHY
 Total size:    13497058 B
 Total dirs:    0
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 13497058 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2
 Average block replication:     2.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          15
 Number of racks:               1
FSCK ended at Thu Aug 13 09:36:35 CST 2015 in 1 milliseconds
 
 
The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168' is HEALTHY

删除损坏的文件(-delete)

[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 -delete
FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 at Thu Aug 13 09:37:58 CST 2015
.Status: HEALTHY
 Total size:    13497058 B
 Total dirs:    0
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 13497058 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2
 Average block replication:     2.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          15
 Number of racks:               1
FSCK ended at Thu Aug 13 09:37:58 CST 2015 in 1 milliseconds
 
 
The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168' is HEALTHY

检查并列出所有文件状态(-files)

[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -files
FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ at Thu Aug 13 09:39:38 CST 2015
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/ dir
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/_SUCCESS 0 bytes, 0 block(s):  OK
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00000 13583807 bytes, 1 block(s):  OK
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00001 13577427 bytes, 1 block(s):  OK
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00002 13588601 bytes, 1 block(s):  OK
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00003 13479213 bytes, 1 block(s):  OK
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00004 13497012 bytes, 1 block(s):  OK
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00005 13557451 bytes, 1 block(s):  OK
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00006 13580267 bytes, 1 block(s):  OK
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00007 13486035 bytes, 1 block(s):  OK
/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00008 13481498 bytes, 1 block(s):  OK
...

检查并打印正在被打开执行写操作的文件(-openforwrite)

[hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -openforwrite
FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ at Thu Aug 13 09:41:28 CST 2015
....................................................................................................
....................................................................................................
.Status: HEALTHY
 Total size:    2704782548 B
 Total dirs:    1
 Total files:   201
 Total symlinks:                0
 Total blocks (validated):      200 (avg. block size 13523912 B)
 Minimally replicated blocks:   200 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2
 Average block replication:     2.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          15
 Number of racks:               1
FSCK ended at Thu Aug 13 09:41:28 CST 2015 in 10 milliseconds
 
The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/' is HEALTHY

打印文件的Block报告(-blocks)

需要和-files一起使用。

[hadoop@dev ~]$  hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks
FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015
/logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s):  OK
0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2
1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2
2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2
3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2
4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2
5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2
6. BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728 repl=2
7. BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728 repl=2
8. BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728 repl=2
9. BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728 repl=2
10. BP-1034052771-172.16.212.130-1405595752491:blk_1075893003_2152402 len=134217728 repl=2
11. BP-1034052771-172.16.212.130-1405595752491:blk_1075893004_2152403 len=134217728 repl=2
12. BP-1034052771-172.16.212.130-1405595752491:blk_1075893005_2152404 len=134217728 repl=2
13. BP-1034052771-172.16.212.130-1405595752491:blk_1075893006_2152405 len=134217728 repl=2
14. BP-1034052771-172.16.212.130-1405595752491:blk_1075893007_2152406 len=134217728 repl=2
...

其中,/logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): 表示文件的总大小和block数;

0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2
1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2
2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2

前面的0. 1. 2.代表该文件的block索引,56的文件块,就从0-55;

BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381表示block id;

len=134217728 表示该文件块大小;

repl=2 表示该文件块副本数;

打印文件块的位置信息(-locations)

需要和-files -blocks一起使用。

[hadoop@dev ~]$  hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks -locations
FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015
/logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s):  OK
0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2 [172.16.212.139:50010, 172.16.212.135:50010]
1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2 [172.16.212.140:50010, 172.16.212.133:50010]
2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2 [172.16.212.136:50010, 172.16.212.141:50010]
3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.135:50010]
4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2 [172.16.212.142:50010, 172.16.212.139:50010]
5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.139:50010]
6. BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728 repl=2 [172.16.212.141:50010, 172.16.212.135:50010]
7. BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728 repl=2 [172.16.212.144:50010, 172.16.212.142:50010]
8. BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.138:50010]
9. BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728 repl=2 [172.16.212.140:50010, 172.16.212.134:50010]

和打印出的文件块信息相比,多了一个文件块的位置信息:[172.16.212.139:50010, 172.16.212.135:50010]

打印文件块位置所在的机架信息(-racks)

[hadoop@dev ~]$  hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks -locations -racks
FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015
/logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s):  OK
0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2 [/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010]
1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2 [/default-rack/172.16.212.140:50010, /default-rack/172.16.212.133:50010]
2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2 [/default-rack/172.16.212.136:50010, /default-rack/172.16.212.141:50010]
3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2 [/default-rack/172.16.212.133:50010, /default-rack/172.16.212.135:50010]
4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2 [/default-rack/172.16.212.142:50010, /default-rack/172.16.212.139:50010]
5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2 [/default-rack/172.16.212.133:50010, /default-rack/172.16.212.139:50010]

和前面打印出的信息相比,多了机架信息:[/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010]

本文转载自:http://lxw1234.com/archives/2015/08/452.htm

共有 人打赏支持
粉丝 38
博文 509
码字总数 237880
作品 0
海淀
私信 提问
HDFS应用场景、原理、基本架构

一、HDFS是什么 源自于Google的GFS论文 发表于2003年10月 HDFS是GFS克隆版  Hadoop Distributed File System 易于扩展的分布式文件系统 运行在大量普通廉价机器上,提供容错机制 为大量用户...

小红牛
2018/05/24
0
0
HDFS入门笔记------架构以及应用介绍

引言—HDFS的重要性: Hadoop的定义:适合大数据的分布式存储与计算的一个平台,其中大数据的分布式存储就是由HDFS来完成的,因此掌握好HDFS的相关概念与应用非常重要! 本篇博客将从以下几个...

a2011480169
2016/06/14
0
0
HBase 表修复在线方式和离线方式

一、在线修复 1.1 使用检查命令 $ ./bin/hbase hbck 该命令可完整修复 HBase 元数据信息;存在有错误信息会进行输出; 也可以通过如下命令查看详细信息: $ ./bin/hbase hbck -details 1.2 ...

Ryan-瑞恩
2018/09/24
0
0
Hadoop 2.x HDFS新特性

Hadoop 2.x HDFS新特性 HDFS联邦 HDFS HA(要用到zookeeper等,留在后面再讲) HDFS快照 回顾: HDFS两层模型 Namespace: 包括目录、文件和块。它支持所有命名空间相关的文件操作,如创建、...

张超
2015/01/12
0
0
技术解析:HDFS应用场景、原理和基本架构

  【IT168 技术】HDFS是什么? 易于扩展的分布式文件系统,运行在大量普通廉价机器上,提供容错机制为大量用户提供性能不错的文件存取服务。   一、HDFS是什么?   源自于Google的GFS论文...

云栖社区
2018/05/25
0
0

没有更多内容

加载失败,请刷新页面

加载更多

《大数据白皮书(2018年)》发布(解读版+完整版PPT)

数据观微信小编获悉,为更好促进大数据与实体经济融合,研判技术发展路径,总结管理痛点、描绘发展趋势,总结行业应用渗透路径,4月18日,在“2018大数据产业峰会”上,中国信息通信研究院(...

了凡川
6分钟前
0
0
从小白到月薪上万,一份完整的大数据路线分析出自我成长书单

大数据原理与实践 大数据分三大部分,包括:大数据基础、技术原理和创新实践。 大数据基础部分主要介绍大数据的基本概念、技术架构和大数据的应用场景; 第二部分大数据技术原理主要介绍大数...

董黎明
17分钟前
0
0
斗鱼直播确定赴美IPO 此前融资额已达70亿元

据最新消息,斗鱼直播高层人士向新京报证实,斗鱼直播确定赴美IPO(首次公开募股),此前融资额已达70亿元。 此前,多家媒体报道,由国内知名直播平台斗鱼(Douyu)已秘密提交赴美IPO文件。 ...

ThinkSNS官方帐号
19分钟前
0
0
虎牙直播在微服务改造方面的实践和总结

相比文字和图片,直播提供了人与人之间更丰富的沟通形式,其对平台稳定性的考验很大,那么倡导“以技术驱动娱乐”的虎牙直播(以下简称“虎牙”)是如何在技术上赋能娱乐,本文将为您介绍虎牙...

阿里云云栖社区
22分钟前
0
0
采用SpringBoot整合Mybatis框架插入数据时报错及解决

这两天做SpringBoot整合Mybatis项目的时候,插入时报错: 3:45:59.936 DEBUG [http-nio-8080-exec-8] o.s.w.s.m.m.a.ExceptionHandlerExceptionResolver 133 - Resolving exception from ha......

aiChuang
28分钟前
0
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部