zookeeper报错“no space left on device”
博客专区 > 刀锋 的博客 > 博客详情
zookeeper报错“no space left on device”
刀锋 发表于3周前
zookeeper报错“no space left on device”
  • 发表于 3周前
  • 阅读 15
  • 收藏 0
  • 点赞 0
  • 评论 0

移动开发云端新模式探索实践 >>>   

问题描述

线上环境使用的是cdh搭建的集群,今天zookeeper报错“no space left on device”,经查为zookeeper的datadir的默认目录(/var/lib/zookeeper)空间所剩无几导致,重新调整了数据的存储目录,再次启动zookeeper,结果报如下错误:

2018-05-07 11:25:07,425 INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration from: /var/run/cloudera-scm-agent/process/604-zookeeper-server/zoo.cfg
2018-05-07 11:25:07,446 INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig: Defaulting to majority quorums
2018-05-07 11:25:07,453 INFO org.apache.zookeeper.server.DatadirCleanupManager: autopurge.snapRetainCount set to 5
2018-05-07 11:25:07,453 INFO org.apache.zookeeper.server.DatadirCleanupManager: autopurge.purgeInterval set to 24
2018-05-07 11:25:07,454 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task started.
2018-05-07 11:25:07,460 ERROR org.apache.zookeeper.server.DatadirCleanupManager: Error occured while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing snap directory /data/var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:122)
	at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
	at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
2018-05-07 11:25:07,474 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task completed.
2018-05-07 11:25:07,483 INFO org.apache.zookeeper.server.quorum.QuorumPeerMain: Starting quorum peer
2018-05-07 11:25:07,497 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181
2018-05-07 11:25:07,512 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing snap directory /data/var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:122)
	at org.apache.zookeeper.server.quorum.QuorumPeer.<init>(QuorumPeer.java:490)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:138)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

解决过程

经日志可得,原因为目录  /data/var/lib/zookeeper/version-2 无法自动创建,根据提示可以通过设置属性 zookeeper.datadir.autocreate=true 来解决,或者手动创建该目录并修改所属用户和用户组,处理细节具体如下:

mkdir /data/var/lib/zookeeper/version-2
chown zookeeper:zookeeper /data/var/lib/zookeeper/version-2

然后重启zookeeper,启动成功。但是,接下来启动hdfs失败,错误日志如下:

FATAL org.apache.hadoop.ha.ZKFailoverController: Unable to start failover controller. Parent znode does not exist.

参考:zookeeper hadoop-ha znode丢失导致hadoop不能failover的解决办法,执行如下命令:

/usr/local/hadoop/bin/hdfs zkfc -formatZK # 重新进入hadoop namenode服务器

然后,重启hdfs,问题解决。

 

  • 打赏
  • 点赞
  • 收藏
  • 分享
共有 人打赏支持
粉丝 2
博文 18
码字总数 288073
评论 (0)
×
刀锋
如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!
* 金额(元)
¥1 ¥5 ¥10 ¥20 其他金额
打赏人
留言
* 支付类型
微信扫码支付
打赏金额:
已支付成功
打赏金额: