文档章节

keepalived+ceph rbd配置nfs的高可用

加油2018
 加油2018
发布于 2014/12/31 11:12
字数 1204
阅读 388
收藏 1

原理比较简单,利用ceph的rbd导出nfs,然后配置keepalived,在不同主机映射的rbd块间进行vip漂移。

1. 创建并映射rbd块设备

试验机器为ceph中的osd1和osd2主机,ip分别为192.168.32.3、192.168.32.4,vip为192.168.32.5。

先创建一个rbd块设备,然后在两台机器上导出此相同的块:

[root@osd1 keepalived]# rbd ls test
test.img
[root@osd1 keepalived]# rbd showmapped
id pool image    snap device    
0  test test.img -    /dev/rbd0 
[root@osd2 keepalived]# rbd showmapped
id pool image    snap device    
0  test test.img -    /dev/rbd0

然后将/dev/rbd0格式化后进行挂载,如/mnt目录。

2. 配置keepalived

http://www.keepalived.org/download.html下载最新的keepalived 1.2.15版本,安装比较简单,直接按照INSTALL的说明进行默认安装即可。为了方便管理keepalived服务,需进行以下操作:

cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
mkdir /etc/keepalived
cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/sbin/keepalived /usr/sbin/
chkconfig --add keepalived
chkconfig keepalived on
chkconfig --list keepalived
配置/etc/keepalived.conf。osd1的配置如下:
[root@osd1 keepalived]# cat keepalived.conf
global_defs
{
    notification_email
    {
    }
    router_id osd1
}

vrrp_instance VI_1 {
    state MASTER
    interface em1
    virtual_router_id 100
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.32.5/24
    }
}
osd2的配置如下:
[root@osd2 keepalived]# cat keepalived.conf 
global_defs
{
    notification_email
    {
    #    admin@example.com
    }
    #notification_email_from admin@example.com
    #smtp_server 127.0.0.1
    #stmp_connect_timeout 30
    router_id osd2
}

vrrp_instance VI_1 {
    state BACKUP
    interface em1
    virtual_router_id 100
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    notify_master "/etc/keepalived/ChangeToMaster.sh"
    notify_backup "/etc/keepalived/ChangeToBackup.sh"
    virtual_ipaddress {
        192.168.32.5/24
    }
}
在osd2写了两个控制脚本,用于osd2的keepalived状态改变时执行。ChangeToMaster.sh:
[root@osd2 keepalived]# cat ChangeToMaster.sh 
#!/bin/bash
service nfs start
ssh lm "umount -f /mnt"
ssh lm "mount -t nfs 192.168.32.5:/mnt /mnt"
ChangeToBackup.sh:
[root@osd2 keepalived]# cat ChangeToBackup.sh 
#!/bin/bash
ssh lm "umount -f /mnt"
ssh osd1 "service nfs stop"
ssh osd1 "umount /mnt"
ssh osd1 "rbd unmap /dev/rbd0"
ssh osd1 "rbd map test/test.img"
ssh osd1 "mount /dev/rbd0 /mnt"
ssh osd1 "service nfs start"
ssh lm "mount -t nfs 192.168.32.5:/mnt /mnt"
service nfs stop
umount /mnt
rbd unmap /dev/rbd0
rbd map test/test.img
mount /dev/rbd0

3. 配置nfs

在ceph的一个节点利用rbd map一个块设备,然后格式化并挂载在一个目录,如/mnt。在此节点上安装nfs的rpm包:

yum -y install nfs-utils

设置挂载目录:

cat /etc/exports 
/mnt 192.168.101.157(rw,async,no_subtree_check,no_root_squash)
/mnt 192.168.108.4(rw,async,no_subtree_check,no_root_squash)

启动并导出:

service nfs start
chkconfig nfs on
exportfs -r

客户端查看一下:

showmount -e mon0
Export list for mon0:
/mnt 192.168.108.4,192.168.101.157
然后挂载:
mount -t nfs mon0:/mnt /mnt

需要注意的是,NFS默认是用UDP协议,如果网络不稳定,换成TCP协议即可:

mount -t nfs mon0:/mnt /mnt -o proto=tcp -o nolock

4. 测试

关闭osd1的网卡em1查看结果:

[root@osd1 keepalived]# ifdown em1
[root@osd1 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether c8:1f:66:de:5e:65 brd ff:ff:ff:ff:ff:ff
查看osd2的网卡:
[root@osd2 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether c8:1f:66:f7:61:5d brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.4/24 brd 192.168.32.255 scope global em1
       valid_lft forever preferred_lft forever
    inet 192.168.32.5/24 scope global secondary em1
       valid_lft forever preferred_lft forever
    inet6 fe80::ca1f:66ff:fef7:615d/64 scope link 
       valid_lft forever preferred_lft forever
vip已经漂移到osd2,查看客户端挂载情况:
[root@lm /]# df -hT
Filesystem        Type   Size  Used Avail Use% Mounted on
/dev/sda1         ext4   454G   79G  353G  19% /
tmpfs             tmpfs  1.7G  4.6M  1.7G   1% /dev/shm
192.168.32.5:/mnt nfs    100G   21G   80G  21% /mnt
打开osd1的网卡em1:
[root@osd1 keepalived]# ifup em1
Determining if ip address 192.168.32.3 is already in use for device em1...
[root@osd1 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether c8:1f:66:de:5e:65 brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.3/24 brd 192.168.32.255 scope global em1
       valid_lft forever preferred_lft forever
    inet 192.168.32.5/24 scope global secondary em1
       valid_lft forever preferred_lft forever
    inet6 fe80::ca1f:66ff:fede:5e65/64 scope link 
       valid_lft forever preferred_lft forever
osd2的em1:
[root@osd2 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether c8:1f:66:f7:61:5d brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.4/24 brd 192.168.32.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::ca1f:66ff:fef7:615d/64 scope link 
       valid_lft forever preferred_lft forever
现在的客户端:
[root@lm /]# df -hT
Filesystem        Type   Size  Used Avail Use% Mounted on
/dev/sda1         ext4   454G   79G  353G  19% /
tmpfs             tmpfs  1.7G  4.6M  1.7G   1% /dev/shm
192.168.32.5:/mnt nfs    100G   21G   80G  21% /mnt
[root@lm /]# ls /mnt
31.txt  a.txt  b.txt  c.txt  etc  linux-3.17.4  linux-3.17.4.tar  m2.txt  test.img  test.img2

也可以为ceph rbd的iscsi配置vip,后面会进行记录。

© 著作权归作者所有

共有 人打赏支持
加油2018
粉丝 150
博文 276
码字总数 243077
作品 0
海淀
架构师
加载中

评论(1)

刘鹏鹏
性能怎么样,比如上传速度?
基于rbd+iscsi+corosync+pacemaker高可用nfs

一、软件安装 1.1一键安装包地址 1.2配置ntp(略) 1.3配置hosts(略) 1.4配置双机互信(略) 二、配置corosync 2.1 corosync配置文件(两台) # mv /etc/corosync/corosync.conf.example /etc/co...

安静美男子
2015/07/06
0
2
Heartbeat+MySQL+NFS 实现高可用(HA)的MySQL集群

目录 一、环境准备 二、拓扑准备 三、安装与配置heartbrat 四、配置NFS服务 五、安装与配置mysql 六、crm配置资源 一、环境准备 1.操作系统 redhat 5.4 i386 位系统 2.软件环境 mysql-5.5.20...

feilong0663
07/03
0
0
Kubernetes持久化Ceph存储

一、依然简介 Kubernetes支持的卷类型详见:https://kubernetes.io/docs/concepts/storage/volumes/ Kubernetes使用Persistent Volume和Persistent Volume Claim两种API资源来管理存储。 Pers......

心远何方
08/10
0
0
开源PaaS | Rainbondv3.5.1全面支持高可用部署

Rainbond(云帮)是一款以应用为中心的开源PaaS,深度整合基于Kubernetes的容器管理、Service Mesh微服务架构最佳实践、多类型CI/CD应用构建与交付、多数据中心资源管理等技术,为用户提供云...

好雨云帮
04/23
0
3
swarm 部署高可用harbor

harbor官方下载地址: https://github.com/vmware/harbor/releases 配置参考: https://github.com/vmware/harbor/blob/master/docs/installationguide.md 此处使用离线下载安装包: wget h......

Wilson9527
06/14
0
0

没有更多内容

加载失败,请刷新页面

加载更多

Spring Boot 使用parent方式引用时 获取值属性方式默认@

采用官网提供的 引入依赖时,application.properties中的值变量,无法通过${}获取,这是由于Spring Boot 已经将字符方式调整为 <resource.delimiter>@</resource.delimiter>。故如果在,工程中...

翱翔fly
15分钟前
1
0
HTTPS证书验证流程及SSL证书生成步骤【附nginx开启https配置】

------------------------------------------------------------ HTTPS证书验证流程(极简化版) 1.客户端向服务端请求证书(server.crt) 2.服务端下发证书(server.crt) 3.客户端用预制的...

xiaomin0322
16分钟前
1
0
预编译sql语句和参数化sql能有效防止——sql注入攻击——

预编译和参数化处理sql原理类似: 参数化处理sql 是将参数进行了字符串化处理,把参数中的单引号,井号#,等都做了转义处理,进而在执行sql语句时不会被当做sql中的定界符和指定符来执行。 ...

fang_faye
16分钟前
1
0
springboot最全配置解释

​​​​​​​# ----------------------------------------# 核心属性# ----------------------------------------# 文件编码banner.charset= UTF-8# 文件位置banner.locati...

废柴大叔
18分钟前
72
1
(大致了解下)从单机到2000万QPS: 知乎Redis平台发展与演进之路

导读:知乎存储平台团队基于开源Redis 组件打造的知乎 Redis 平台,经过不断的研发迭代,目前已经形成了一整套完整自动化运维服务体系,提供很多强大的功能。本文作者是是该系统的负责人,文...

Ocean_K
25分钟前
2
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部