文档章节

ceph osd盘挂掉,无法修复

加油2018
 加油2018
发布于 2014/12/24 10:57
字数 1686
阅读 1657
收藏 0
点赞 0
评论 4

由于网络更换ip段,导致ceph启动时monitor进程无法启动,解决了这个问题后,ceph能够启动,但一块osd盘坏掉了,错误日志如下:

2014-12-24 10:53:30.353262 7f3fbbd78800  0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578), process ceph-osd, pid 9794
2014-12-24 10:53:30.359829 7f3fbbd78800  0 filestore(/var/lib/ceph/osd/ceph-4) backend xfs (magic 0x58465342)
2014-12-24 10:53:30.536660 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: FIEMAP ioctl is supported and appears to work
2014-12-24 10:53:30.536683 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-12-24 10:53:30.537600 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: syscall(SYS_syncfs, fd) fully supported
2014-12-24 10:53:30.537684 7f3fbbd78800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_feature: extsize is disabled by conf
2014-12-24 10:53:30.537684 7f3fbbd78800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_feature: extsize is disabled by conf
2014-12-24 10:53:30.592079 7f3fbbd78800  0 filestore(/var/lib/ceph/osd/ceph-4) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-12-24 10:53:40.394856 7f3fbbd78800 -1 os/FileStore.cc: In function 'int FileStore::_check_replay_guard(int, const SequencerPosition&)' thread 7f3fbbd78800 time 2014-12-24 10:53:40.391498
os/FileStore.cc: 2181: FAILED assert(!m_filestore_fail_eio || r != -5)

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: (FileStore::_check_replay_guard(int, SequencerPosition const&)+0x2d3) [0x730633]
 2: (FileStore::_check_replay_guard(coll_t, ghobject_t, SequencerPosition const&)+0x148) [0x742918]
 3: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x58a4) [0x75e7a4]
 4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x763894]
 5: (JournalingObjectStore::journal_replay(unsigned long)+0x47c) [0x77578c]
 6: (FileStore::mount()+0x2baa) [0x751d4a]
 7: (OSD::do_convertfs(ObjectStore*)+0x1a) [0x64e75a]
 8: (main()+0x33e4) [0x613044]
 9: (__libc_start_main()+0xfd) [0x31e6a1ed1d]
 10: /usr/bin/ceph-osd() [0x60ec19]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -21> 2014-12-24 10:53:30.348526 7f3fbbd78800  5 asok(0x5278230) register_command perfcounters_dump hook 0x5200010
   -20> 2014-12-24 10:53:30.348569 7f3fbbd78800  5 asok(0x5278230) register_command 1 hook 0x5200010
   -19> 2014-12-24 10:53:30.348579 7f3fbbd78800  5 asok(0x5278230) register_command perf dump hook 0x5200010
   -18> 2014-12-24 10:53:30.348590 7f3fbbd78800  5 asok(0x5278230) register_command perfcounters_schema hook 0x5200010
   -17> 2014-12-24 10:53:30.348600 7f3fbbd78800  5 asok(0x5278230) register_command 2 hook 0x5200010
   -19> 2014-12-24 10:53:30.348579 7f3fbbd78800  5 asok(0x5278230) register_command perf dump hook 0x5200010
   -18> 2014-12-24 10:53:30.348590 7f3fbbd78800  5 asok(0x5278230) register_command perfcounters_schema hook 0x5200010
   -17> 2014-12-24 10:53:30.348600 7f3fbbd78800  5 asok(0x5278230) register_command 2 hook 0x5200010
   -16> 2014-12-24 10:53:30.348609 7f3fbbd78800  5 asok(0x5278230) register_command perf schema hook 0x5200010

   -15> 2014-12-24 10:53:30.348618 7f3fbbd78800  5 asok(0x5278230) register_command config show hook 0x5200010
asok(0x5278230) register_command config show hook 0x5200010
   -14> 2014-12-24 10:53:30.348629 7f3fbbd78800  5    -14> 2014-12-24 10:53:30.348629 7f3fbbd78800  5 asok(0x5278230) register_command config set hook 0x5200010
   -13> 2014-12-24 10:53:30.348638 7f3fbbd78800  5 asok(0x5278230) register_command config get hook 0x5200010asok(0x5278230) register_command config get hook 0x5200010
   -12> 2014-12-24 10:53:30.348647 7f3fbbd78800  5 asok(0x5278230) register_command config diff hook 0x5200010asok(0x5278230) register_command config diff hook 0x5200010
   -11> 2014-12-24 10:53:30.348657 7f3fbbd78800  5 asok(0x5278230) register_command log flush hook 0x5200010asok(0x5278230) register_command log flush hook 0x5200010
   -10> 2014-12-24 10:53:30.348666 7f3fbbd78800  5 asok(0x5278230) register_command log dump hook 0x5200010asok(0x5278230) register_command log dump hook 0x5200010
    -9> 2014-12-24 10:53:30.348675 7f3fbbd78800  5 asok(0x5278230) register_command log reopen hook 0x5200010
    -8> 2014-12-24 10:53:30.353262 7f3fbbd78800  0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578), process ceph-osd, pid 9794
    -7> 2014-12-24 10:53:30.355214 7f3fbbd78800  1 finished global_init_daemonize
    -6> 2014-12-24 10:53:30.359829 7f3fbbd78800  0 filestore(/var/lib/ceph/osd/ceph-4) backend xfs (magic 0x58465342)
    -5> 2014-12-24 10:53:30.536660 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: FIEMAP ioctl is supported and appears to work
    -4> 2014-12-24 10:53:30.536683 7f3fbbd78800  0     -5> 2014-12-24 10:53:30.536660 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: FIEMAP ioctl is supported and appears to work
    -4> 2014-12-24 10:53:30.536683 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
    -3> 2014-12-24 10:53:30.537600 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/o    -3> 2014-12-24 10:53:30.537600 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: syscall(SYS_syncfs, fd) fully supported
    -2> 2014-12-24 10:53:30.537684 7f3fbbd78800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_feature: extsize is disabled by conf

    -1> 2014-12-24 10:53:30.592079 7f3fbbd78800  0 filestore(/var/lib/ceph/osd/ceph-4) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled

     0> 2014-12-24 10:53:40.394856 7f3fbbd78800 -1 os/FileStore.cc: In function 'int FileStore::_check_replay_guard(int, const SequencerPosition&)' thread 7f3fbbd78800 time 2014-12-24 10:53:40.391498
os/FileStore.cc: 2181: FAILED assert(!m_filestore_fail_eio || r != -5)

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: (FileStore::_check_replay_guard(int, SequencerPosition const&)+0x2d3) [0x730633]
 2: (FileStore::_check_replay_guard(coll_t, ghobject_t, SequencerPosition const&)+0x148) [0x742918]
 3: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x58a4) [0x75e7a4]
 4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x763894]
 5: (JournalingObjectStore::journal_replay(unsigned long)+0x47c) [0x77578c]
 6: (FileStore::mount()+0x2baa) [0x751d4a]
 7: (OSD::do_convertfs(ObjectStore*)+0x1a) [0x64e75a]
 8: (main()+0x33e4) [0x613044]
 9: (__libc_start_main()+0xfd) [0x31e6a1ed1d]
 10: /usr/bin/ceph-osd() [0x60ec19]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 noneos/FileStore.cc: In function 'int FileStore::_check_replay_guard(int, const SequencerPosition&)' thread 7f3fbbd78800 time 2014-12-24 10:53:40.391498
os/FileStore.cc: 2181: FAILED assert(!m_filestore_fail_eio || r != -5)

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: (FileStore::_check_replay_guard(int, SequencerPosition const&)+0x2d3) [0x730633]
 2: (FileStore::_check_replay_guard(coll_t, ghobject_t, SequencerPosition const&)+0x148) [0x742918]
 3: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x58a4) [0x75e7a4]
 4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x763894]
 5: (JournalingObjectStore::journal_replay(unsigned long)+0x47c) [0x77578c]
 6: (FileStore::mount()+0x2baa) [0x751d4a]
 7: (OSD::do_convertfs(ObjectStore*)+0x1a) [0x64e75a]
 8: (main()+0x33e4) [0x613044]
 9: (__libc_start_main()+0xfd) [0x31e6a1ed1d]
 10: /usr/bin/ceph-osd() [0x60ec19]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   1/ 5 mds_balancer
   1/ 5 mds_balancer
   1/ 5 mds_locker   1/ 5 mds_locker
   1/ 5 mds_log   1/ 5 mds_log
   1/ 5 mds_log_expire

   1/ 5 mds_migrator   1/ 5 mds_migrator
   0/ 0 buffer

   0/ 0 timer

   0/ 1 filer

   0/ 1 striper

   0/ 1 objecter

   0/ 5 rados

   0/ 0 rbd

   0/ 5 rbd_replay

   0/ 0 journaler

   0/ 5 objectcacher
   0/ 5 client   0/ 5 client
   0/ 0 osd   0/ 0 osd
   0/ 0 optracker   0/ 0 optracker
   0/ 0 objclass   0/ 0 objclass
   0/ 0 filestore   0/ 0 filestore
   1/ 3 keyvaluestore
   1/ 3 keyvaluestore
   0/ 0 journal

   0/ 0 ms

   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp

   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   1/ 5 crypto
   0/ 0 finisher

   0/ 0 heartbeatmap

   0/ 0 perfcounter

   1/ 5 rgw

   1/10 civetweb

   1/ 5 javaclient

   0/ 0 asok

   0/ 0 throttle

   0/ 0 refs

  -2/-2 (syslog threshold)

  -1/-1 (stderr threshold)

  max_recent     10000

  max_new         1000

  log_file /var/log/ceph/ceph-osd.4.log

--- end dump of recent events ---
2014-12-24 10:53:40.400275 7f3fbbd78800 -1 *** Caught signal (Aborted) **
 in thread 7f3fbbd78800

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: /usr/bin/ceph-osd() [0xa02425]
 2: /lib64/libpthread.so.0() [0x31e6e0f710]
 3: (gsignal()+0x35) [0x31e6a32925]
 4: (abort()+0x175) [0x31e6a34105]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x31f2abea7d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2014-12-24 10:53:40.400275 7f3fbbd78800 -1 *** Caught signal (Aborted) **
 in thread 7f3fbbd78800

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: /usr/bin/ceph-osd() [0xa02425]
 2: /lib64/libpthread.so.0() [0x31e6e0f710]
 3: (gsignal()+0x35) [0x31e6a32925]
 4: (abort()+0x175) [0x31e6a34105]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x31f2abea7d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   1/ 5 mds_balancer

   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   1/ 5 mds_migrator
   0/ 0 buffer

   0/ 0 timer
   0/ 1 filer

   0/ 1 striper

   0/ 1 objecter

   0/ 5 rados

   0/ 0 rbd

   0/ 5 rbd_replay

   0/ 0 journaler

   0/ 5 objectcacher

   0/ 5 client

   0/ 0 osd

   0/ 0 optracker

   0/ 0 objclass

   0/ 0 filestore

   1/ 3 keyvaluestore

   0/ 0 journal

   0/ 0 ms

   0/ 0 mon

   0/ 0 monc

   0/ 0 paxos
   0/ 0 tp   0/ 0 tp
   0/ 0 auth   0/ 0 auth
   1/ 5 crypto   1/ 5 crypto
   0/ 0 finisher   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter   0/ 0 heartbeatmap
   0/ 0 perfcounter
   1/ 5 rgw

   1/10 civetweb

   1/ 5 javaclient

   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.4.log
--- end dump of recent events ---



暂时无法解决,有遇到过类似问题的朋友吗?

© 著作权归作者所有

共有 人打赏支持
加油2018
粉丝 150
博文 184
码字总数 243077
作品 0
海淀
架构师
加载中

评论(4)

haiquan517
haiquan517
首先查看monmap表, 如果都改过来了, 尝试重建journallog
加油2018
加油2018

引用来自“oscfox”的评论

这还真没遇到过,不过更换ip段不需要stop整个集群,重新push config 然后再重启吗?
公司搬家,所有服务器都分配了新的ip。stop了,修改配置后start,遇到了上面的问题。
Yason_Luo
Yason_Luo
这还真没遇到过,不过更换ip段不需要stop整个集群,重新push config 然后再重启吗?
叶秀兰
叶秀兰
这个去讨论区问比较靠谱吧,比较多人看到,可以讨论。@oscfox
ip改变引起的ceph monitor异常及osd盘崩溃的总结

公司搬家,所有服务器的ip改变。对ceph服务器配置好ip后启动,发现monitor进程启动失败,monitor进程总是试图绑定到以前的ip地址,那当然不可能成功了。开始以为服务器的ip设置有问题,在改变...

壬癸甲乙 ⋅ 2014/12/25 ⋅ 6

Openstack之Ceph集群操作

Oepnstack之CEPH系列是根据Ceph Cookbook整理的笔记,分为以下几个部分: 1. 《Ceph简介》 2. 《Ceph集群操作》 3. 《Ceph块设备管理与Openstack配置》 4. 《深入Ceph》 5. 《ceph优化与性能...

hiubuntu ⋅ 2016/12/01 ⋅ 0

CEPH 可靠性的计算方法分析

原文出处:RaySun的博客 在开始正文之前,首先要感谢UnitedStack工程师朱荣泽对这篇博文的大力帮助和悉心指教。本文主要针对UnitedStack公司在巴黎峰会上对Ceph可靠性的计算方法(https://ww...

红薯 ⋅ 2015/01/25 ⋅ 0

开源实践分享:Ceph bluestore部署实践

Ceph bluestore部署 首先为大家分享Ceph bluestore具体该如何部署,使用环境如下 • 单节点 • CentOS 7.4 • 使用 ceph-deploy 进行部署 准备 centos 7.4 系统的机器,并配置好 ssh 的无密码...

OpenInfra ⋅ 05/24 ⋅ 0

从传统运维到云运维演进历程之软件定义存储(六)完结

回到最初的Ceph运维工程师的问题,本系列讲述的是传统运维向新一代云运维转型之软件定义存储部分的转型,运维是企业业务系统从规划、设计、实施、交付到运维的最后一个步骤,也是重要的步骤。...

Devin ⋅ 2016/12/20 ⋅ 0

ceph分布式存储实战(3)——ceph内部组件

一、RADOS(Reliable Autonomic Distributed Object Storage) 1、rados的意思是 可靠、自动、分布式对象存储(可靠的自主分布式对象存储) 2、rados的作用 1、rados是ceph集群的基础,是cep...

TtrToby ⋅ 01/05 ⋅ 0

ceph基于pglog的一致性协议

分布式存储系统通常采用多副本的方式来保证系统的可靠性,而多副本之间如何保证数据的一致性就是系统的核心。ceph号称统一存储,其核心RADOS既支持多副本,也支持纠删码。本文主要分析ceph的...

linuxhunter ⋅ 2016/05/23 ⋅ 0

ceph集群报错:HEALTH_ERR 1 pgs inconsistent; 1 scrub errors

报错信息如下: [root@ceph-6-11 ~]# ceph health detail HEALTH_ERR 1 pgs inconsistent; 1 scrub errors; pg 2.37c is active+clean+inconsistent, acting [75,6,35] 1 scrub errors 报错信......

kjh2007abc ⋅ 2017/07/24 ⋅ 0

CEPH新手拜门帖,ceph写入数据时经常osd宕机,不知何故

@壬癸甲乙 你好,想跟你请教个问题: 从您的博客中看到,您对ceph有较深入的了解,所以诚心请教,如果打扰到您工作,请见谅 我刚刚接触ceph不久,在4台linux环境下部署了osd,每台linux_ser...

纳尼亚索 ⋅ 2016/04/08 ⋅ 0

Openstack之CEPH简介

Oepnstack之CEPH系列是根据Ceph Cookbook整理的笔记,分为以下几个部分: 1. 《Ceph简介》 2. 《Ceph集群操作》 3. 《Ceph块设备管理与Openstack配置》 4. 《深入Ceph》 5. 《ceph优化与性能...

hiubuntu ⋅ 2016/11/30 ⋅ 0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

mysql in action / alter table

change character set ALTER SCHEMA `employees` DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_general_ci ;ALTER TABLE `employees`.`t2` CHARACTER SET = utf8mb4 , COLLAT......

qwfys ⋅ 今天 ⋅ 0

Java 开发者不容错过的 12 种高效工具

Java 开发者常常都会想办法如何更快地编写 Java 代码,让编程变得更加轻松。目前,市面上涌现出越来越多的高效编程工具。所以,以下总结了一系列工具列表,其中包含了大多数开发人员已经使用...

jason_kiss ⋅ 昨天 ⋅ 0

Linux下php访问远程ms sqlserver

1、安装freetds(略,安装在/opt/local/freetds 下) 2、cd /path/to/php-5.6.36/ 进入PHP源码目录 3、cd ext/mssql进入MSSQL模块源码目录 4、/opt/php/bin/phpize生成编译配置文件 5、 . ./...

wangxuwei ⋅ 昨天 ⋅ 0

如何成为技术专家

文章来源于 -- 时间的朋友 拥有良好的心态。首先要有空杯心态,用欣赏的眼光发现并学习别人的长处,包括但不限于工具的使用,工作方法,解决问题以及规划未来的能力等。向别人学习的同时要注...

长安一梦 ⋅ 昨天 ⋅ 0

Linux vmstat命令实战详解

vmstat命令是最常见的Linux/Unix监控工具,可以展现给定时间间隔的服务器的状态值,包括服务器的CPU使用率,内存使用,虚拟内存交换情况,IO读写情况。这个命令是我查看Linux/Unix最喜爱的命令...

刘祖鹏 ⋅ 昨天 ⋅ 0

MySQL

查看表相关命令 - 查看表结构    desc 表名- 查看生成表的SQL    show create table 表名- 查看索引    show index from  表名 使用索引和不使用索引 由于索引是专门用于加...

stars永恒 ⋅ 昨天 ⋅ 0

easyui学习笔记

EasyUI常用控件禁用方法 combobox $("#id").combobox({ disabled: true }); ----- $("#id").combobox({ disabled: false}); validatebox $("#id").attr("readonly", true); ----- $("#id").r......

miaojiangmin ⋅ 昨天 ⋅ 0

金山WPS发布了Linux WPS Office

导读 近日,金山WPS发布了Linux WPS Office中文社区版新版本,支持大部分主流Linux系统,功能更加完善,兼容性、稳定性大幅度提升。本次更新WPS将首次在Linux提供专业办公文件云存储服务,实...

问题终结者 ⋅ 昨天 ⋅ 0

springboot2输出metrics到influxdb

序 本文主要研究一下如何将springboot2的metrics输出到influxdb maven <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-bo......

go4it ⋅ 昨天 ⋅ 0

微信小程序 - 选择图片显示操作菜单

之前我分享过选择图片这个文章,但是我在实际开发测试使用中发现一个问题在使用 wx.chooseImage 选择照片显示出第一格是拍照,后面是相册里的图片。这种实现之前说过了,效果如下。 但是你从...

hello_hp ⋅ 昨天 ⋅ 0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

返回顶部
顶部