文档章节

基于Docker Swarm+Shipyard实现企业级Docker私有云平台——Docker Swarm集群创建与可视化管理

atompi
 atompi
发布于 2017/02/18 17:09
字数 1749
阅读 455
收藏 4

节点信息:

manager:192.168.220.194

worker1:192.168.220.173

worker2:192.168.220.197

 

技术栈:

Docker 1.12

Shipyard

swarm

etcd

SwarmOps

rethinkdb

docker-proxy

 

所需资源包:

quick start:

1、安装docker 1.12 (所有节点,内核限制最低centos7)

设置docker1.12 yum源:

cat > /etc/yum.repos.d/docker.repo <<-EOF

[dockerrepo]

name=Docker Repository

baseurl=https://yum.dockerproject.org/repo/experimental/centos/7/

enabled=1

gpgcheck=1

gpgkey=https://yum.dockerproject.org/gpg

EOF

 

开放端口(manager节点):

firewall-cmd --zone=public --add-port=2375/tcp --permanent

firewall-cmd --zone=public --add-port=2377/tcp --permanent

firewall-cmd --zone=public --add-port=3375/tcp --permanent

firewall-cmd --zone=public --add-port=4001/tcp --permanent

firewall-cmd --zone=public --add-port=7001/tcp --permanent

firewall-cmd --zone=public --add-port=8080/tcp --permanent

firewall-cmd --zone=public --add-port=10130/tcp --permanent

firewall-cmd --reload

 

开放端口(worker节点):

firewall-cmd --zone=public --add-port=2375/tcp --permanent

firewall-cmd --zone=public --add-port=2377/tcp --permanent

firewall-cmd --zone=public --add-port=3375/tcp --permanent

firewall-cmd --reload

 

安装docker1.12:

yum -y install docker-engine

 

设置开机自启动:

systemctl enable docker

开启docker服务:

systemctl start docker

 

设置加速器(国内源):

curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://e80d24c6.m.daocloud.io

重启docker服务:

systemctl restart docker

 

初始化swarm集群:

docker swarm init

输出信息如下:

Swarm initialized: current node (9r76p77projw3olxbmzmdfura) is now a manager.

 

To add a worker to this swarm, run the following command:

 

docker swarm join \

--token SWMTKN-1-1m8h511662w4s79nus0az57fucoypykm199b6lc88hr6mu7zeb-1b7ks1s15o2d5xssllqq8u4no \

192.168.220.143:2377

 

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

 

worker节点加入swarm集群:

docker swarm join --token SWMTKN-1-1m8h511662w4s79nus0az57fucoypykm199b6lc88hr6mu7zeb-1b7ks1s15o2d5xssllqq8u4no 192.168.220.143:2377

输出信息如下:

This node joined a swarm as a worker.

 

查看swarm节点:

docker node ls

输出信息如下:

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS

9r76p77projw3olxbmzmdfura * DockerH1 Ready Active Leader

athdt83y3sxw6oo8sp7tnnmh3 DockerH1 Ready Active

 

查看master节点token:

1、添加manager节点:

docker swarm join-token manager

2、添加worker节点:

docker swarm join-token worker

输出信息如下:

To add a manager to this swarm, run the following command:

 

docker swarm join \

--token SWMTKN-1-1m8h511662w4s79nus0az57fucoypykm199b6lc88hr6mu7zeb-3knrcvyaj6uz3df7jkm13w0iu \

192.168.220.143:2377

 

 

 

To add a worker to this swarm, run the following command:

 

docker swarm join \

--token SWMTKN-1-1m8h511662w4s79nus0az57fucoypykm199b6lc88hr6mu7zeb-1b7ks1s15o2d5xssllqq8u4no \

192.168.220.143:2377

 

节点退出swarm集群:

docker swarm leave

 

2、安装Shipyard集群管理服务:

导入启动文件:

docker.sh

 

文件内容:

【docker.sh】

--------------------------------------------------------------------------------------------------------

#!/bin/bash

 

if [ "$1" != "" ] && [ "$1" = "-h" ]; then

echo "Shipyard Deploy uses the following environment variables:"

echo " ACTION: this is the action to use (deploy, upgrade, node, remove)"

echo " DISCOVERY: discovery system used by Swarm (only if using 'node' action)"

echo " IMAGE: this overrides the default Shipyard image"

echo " PREFIX: prefix for container names"

echo " SHIPYARD_ARGS: these are passed to the Shipyard controller container as controller args"

echo " TLS_CERT_PATH: path to certs to enable TLS for Shipyard"

echo " PORT: specify the listen port for the controller (default: 8080)"

echo " IP: specify the address at which the controller or node will be available (default: eth0 ip)"

echo " PROXY_PORT: port to run docker proxy (default: 2375)"

exit 1

fi

 

if [ -z "`which docker`" ]; then

echo "You must have the Docker CLI installed on your \$PATH"

echo " See http://docs.docker.com for details"

exit 1

fi

 

ACTION=${ACTION:-deploy}

IMAGE=${IMAGE:-shipyard/shipyard:latest}

PREFIX=${PREFIX:-shipyard}

SHIPYARD_ARGS=${SHIPYARD_ARGS:-""}

TLS_CERT_PATH=${TLS_CERT_PATH:-}

CERT_PATH="/etc/shipyard"

PROXY_PORT=${PROXY_PORT:-2375}

SWARM_PORT=3375

SHIPYARD_PROTOCOL=http

SHIPYARD_PORT=${PORT:-8080}

SHIPYARD_IP=${IP}

DISCOVERY_BACKEND=etcd

DISCOVERY_PORT=4001

DISCOVERY_PEER_PORT=7001

ENABLE_TLS=0

CERT_FINGERPRINT=""

LOCAL_CA_CERT=""

LOCAL_SSL_CERT=""

LOCAL_SSL_KEY=""

LOCAL_SSL_CLIENT_CERT=""

LOCAL_SSL_CLIENT_KEY=""

SSL_CA_CERT=""

SSL_CERT=""

SSL_KEY=""

SSL_CLIENT_CERT=""

SSL_CLIENT_KEY=""

 

show_cert_help() {

echo "To use TLS in Shipyard, you must have existing certificates."

echo "The certs must be named ca.pem, server.pem, server-key.pem, cert.pem and key.pem"

echo "If you need to generate certificates, see https://github.com/ehazlett/certm for examples."

}

 

check_certs() {

if [ -z "$TLS_CERT_PATH" ]; then

return

fi

 

if [ ! -e $TLS_CERT_PATH ]; then

echo "Error: unable to find certificates in $TLS_CERT_PATH"

show_cert_help

exit 1

fi

 

if [ "$PROXY_PORT" = "2375" ]; then

PROXY_PORT=2376

fi

SWARM_PORT=3376

SHIPYARD_PROTOCOL=https

LOCAL_SSL_CA_CERT="$TLS_CERT_PATH/ca.pem"

LOCAL_SSL_CERT="$TLS_CERT_PATH/server.pem"

LOCAL_SSL_KEY="$TLS_CERT_PATH/server-key.pem"

LOCAL_SSL_CLIENT_CERT="$TLS_CERT_PATH/cert.pem"

LOCAL_SSL_CLIENT_KEY="$TLS_CERT_PATH/key.pem"

SSL_CA_CERT="$CERT_PATH/ca.pem"

SSL_CERT="$CERT_PATH/server.pem"

SSL_KEY="$CERT_PATH/server-key.pem"

SSL_CLIENT_CERT="$CERT_PATH/cert.pem"

SSL_CLIENT_KEY="$CERT_PATH/key.pem"

CERT_FINGERPRINT=$(openssl x509 -noout -in $LOCAL_SSL_CERT -fingerprint -sha256 | awk -F= '{print $2;}')

 

if [ ! -e $LOCAL_SSL_CA_CERT ] || [ ! -e $LOCAL_SSL_CERT ] || [ ! -e $LOCAL_SSL_KEY ] || [ ! -e $LOCAL_SSL_CLIENT_CERT ] || [ ! -e $LOCAL_SSL_CLIENT_KEY ]; then

echo "Error: unable to find certificates"

show_cert_help

exit 1

fi

 

ENABLE_TLS=1

}

 

# container functions

start_certs() {

ID=$(docker run \

-ti \

-d \

--restart=always \

--name $PREFIX-certs \

-v $CERT_PATH \

alpine \

sh)

if [ $ENABLE_TLS = 1 ]; then

docker cp $LOCAL_SSL_CA_CERT $PREFIX-certs:$SSL_CA_CERT

docker cp $LOCAL_SSL_CERT $PREFIX-certs:$SSL_CERT

docker cp $LOCAL_SSL_KEY $PREFIX-certs:$SSL_KEY

docker cp $LOCAL_SSL_CLIENT_CERT $PREFIX-certs:$SSL_CLIENT_CERT

docker cp $LOCAL_SSL_CLIENT_KEY $PREFIX-certs:$SSL_CLIENT_KEY

fi

}

 

remove_certs() {

docker rm -fv $PREFIX-certs > /dev/null 2>&1

}

 

get_ip() {

if [ -z "$SHIPYARD_IP" ]; then

SHIPYARD_IP=`docker run --rm --net=host alpine ip route get 8.8.8.8 | awk '{ print $7; }'`

fi

}

 

start_discovery() {

get_ip

 

ID=$(docker run \

-ti \

-d \

-p 4001:4001 \

-p 7001:7001 \

--restart=always \

--name $PREFIX-discovery \

microbox/etcd:latest -addr $SHIPYARD_IP:$DISCOVERY_PORT -peer-addr $SHIPYARD_IP:$DISCOVERY_PEER_PORT)

}

 

remove_discovery() {

docker rm -fv $PREFIX-discovery > /dev/null 2>&1

}

 

start_rethinkdb() {

ID=$(docker run \

-ti \

-d \

--restart=always \

--name $PREFIX-rethinkdb \

rethinkdb)

}

 

remove_rethinkdb() {

docker rm -fv $PREFIX-rethinkdb > /dev/null 2>&1

}

 

start_proxy() {

TLS_OPTS=""

if [ $ENABLE_TLS = 1 ]; then

TLS_OPTS="-e SSL_CA=$SSL_CA_CERT -e SSL_CERT=$SSL_CERT -e SSL_KEY=$SSL_KEY -e SSL_SKIP_VERIFY=1"

fi

# Note: we add SSL_SKIP_VERIFY=1 to skip verification of the client

# certificate in the proxy image. this will pass it to swarm that

# does verify. this helps with performance and avoids certificate issues

# when running through the proxy. ultimately if the cert is invalid

# swarm will fail to return.

ID=$(docker run \

-ti \

-d \

-p $PROXY_PORT:$PROXY_PORT \

--hostname=$HOSTNAME \

--restart=always \

--name $PREFIX-proxy \

-v /var/run/docker.sock:/var/run/docker.sock \

-e PORT=$PROXY_PORT \

--volumes-from=$PREFIX-certs $TLS_OPTS\

shipyard/docker-proxy:latest)

}

 

remove_proxy() {

docker rm -fv $PREFIX-proxy > /dev/null 2>&1

}

 

start_swarm_manager() {

get_ip

 

TLS_OPTS=""

if [ $ENABLE_TLS = 1 ]; then

TLS_OPTS="--tlsverify --tlscacert=$SSL_CA_CERT --tlscert=$SSL_CERT --tlskey=$SSL_KEY"

fi

 

EXTRA_RUN_OPTS=""

 

if [ -z "$DISCOVERY" ]; then

DISCOVERY="$DISCOVERY_BACKEND://discovery:$DISCOVERY_PORT"

EXTRA_RUN_OPTS="--link $PREFIX-discovery:discovery"

fi

ID=$(docker run \

-ti \

-d \

--restart=always \

--name $PREFIX-swarm-manager \

--volumes-from=$PREFIX-certs $EXTRA_RUN_OPTS \

swarm:latest \

m --replication --addr $SHIPYARD_IP:$SWARM_PORT --host tcp://0.0.0.0:$SWARM_PORT $TLS_OPTS $DISCOVERY)

}

 

remove_swarm_manager() {

docker rm -fv $PREFIX-swarm-manager > /dev/null 2>&1

}

 

start_swarm_agent() {

get_ip

 

if [ -z "$DISCOVERY" ]; then

DISCOVERY="$DISCOVERY_BACKEND://discovery:$DISCOVERY_PORT"

EXTRA_RUN_OPTS="--link $PREFIX-discovery:discovery"

fi

ID=$(docker run \

-ti \

-d \

--restart=always \

--name $PREFIX-swarm-agent $EXTRA_RUN_OPTS \

swarm:latest \

j --addr $SHIPYARD_IP:$PROXY_PORT $DISCOVERY)

}

 

remove_swarm_agent() {

docker rm -fv $PREFIX-swarm-agent > /dev/null 2>&1

}

 

start_controller() {

#-v $CERT_PATH:/etc/docker:ro \

TLS_OPTS=""

if [ $ENABLE_TLS = 1 ]; then

TLS_OPTS="--tls-ca-cert $SSL_CA_CERT --tls-cert=$SSL_CERT --tls-key=$SSL_KEY --shipyard-tls-ca-cert=$SSL_CA_CERT --shipyard-tls-cert=$SSL_CERT --shipyard-tls-key=$SSL_KEY"

fi

 

ID=$(docker run \

-ti \

-d \

--restart=always \

--name $PREFIX-controller \

--link $PREFIX-rethinkdb:rethinkdb \

--link $PREFIX-swarm-manager:swarm \

-p $SHIPYARD_PORT:$SHIPYARD_PORT \

--volumes-from=$PREFIX-certs \

$IMAGE \

--debug \

server \

--listen :$SHIPYARD_PORT \

-d tcp://swarm:$SWARM_PORT $TLS_OPTS $SHIPYARD_ARGS)

}

 

wait_for_available() {

set +e

IP=$1

PORT=$2

echo Waiting for Shipyard on $IP:$PORT

 

docker pull ehazlett/curl > /dev/null 2>&1

 

TLS_OPTS=""

if [ $ENABLE_TLS = 1 ]; then

TLS_OPTS="-k"

fi

 

until $(docker run --rm ehazlett/curl --output /dev/null --connect-timeout 1 --silent --head --fail $TLS_OPTS $SHIPYARD_PROTOCOL://$IP:$PORT/ > /dev/null 2>&1); do

printf '.'

sleep 1

done

printf '\n'

}

 

remove_controller() {

docker rm -fv $PREFIX-controller > /dev/null 2>&1

}

 

if [ "$ACTION" = "deploy" ]; then

set -e

 

check_certs

 

get_ip

 

echo "Deploying Shipyard"

echo " -> Starting Database"

start_rethinkdb

echo " -> Starting Discovery"

start_discovery

echo " -> Starting Cert Volume"

start_certs

echo " -> Starting Proxy"

start_proxy

echo " -> Starting Swarm Manager"

start_swarm_manager

echo " -> Starting Swarm Agent"

start_swarm_agent

echo " -> Starting Controller"

start_controller

 

wait_for_available $SHIPYARD_IP $SHIPYARD_PORT

 

echo "Shipyard available at $SHIPYARD_PROTOCOL://$SHIPYARD_IP:$SHIPYARD_PORT"

if [ $ENABLE_TLS = 1 ] && [ ! -z "$CERT_FINGERPRINT" ]; then

echo "SSL SHA-256 Fingerprint: $CERT_FINGERPRINT"

fi

echo "Username: admin Password: shipyard"

 

elif [ "$ACTION" = "node" ]; then

set -e

 

if [ -z "$DISCOVERY" ]; then

echo "You must set the DISCOVERY environment variable"

echo "with the discovery system used with Swarm"

exit 1

fi

 

check_certs

 

echo "Adding Node"

echo " -> Starting Cert Volume"

start_certs

echo " -> Starting Proxy"

start_proxy

echo " -> Starting Swarm Manager"

start_swarm_manager $DISCOVERY

echo " -> Starting Swarm Agent"

start_swarm_agent

 

echo "Node added to Swarm: $SHIPYARD_IP"

elif [ "$ACTION" = "upgrade" ]; then

set -e

 

check_certs

 

get_ip

 

echo "Upgrading Shipyard"

echo " -> Pulling $IMAGE"

docker pull $IMAGE

 

echo " -> Upgrading Controller"

remove_controller

start_controller

 

wait_for_available $SHIPYARD_IP $SHIPYARD_PORT

 

echo "Shipyard controller updated"

 

elif [ "$ACTION" = "remove" ]; then

# ignore errors

set +e

 

echo "Removing Shipyard"

echo " -> Removing Database"

remove_rethinkdb

echo " -> Removing Discovery"

remove_discovery

echo " -> Removing Cert Volume"

remove_certs

echo " -> Removing Proxy"

remove_proxy

echo " -> Removing Swarm Agent"

remove_swarm_agent

echo " -> Removing Swarm Manager"

remove_swarm_manager

echo " -> Removing Controller"

remove_controller

 

echo "Done"

else

echo "Unknown action $ACTION"

exit 1

fi

 

--------------------------------------------------------------------------------------------------------

 

加载镜像(manager节点)

docker load < shipyard.tar.gz

docker load < shipyard-docker-proxy.tar.gz

docker load < swarm.tar.gz

docker load < microbox-etcd.tar.gz

docker load < rethinkdb.tar.gz

docker load < alpine.tar.gz

 

运行以下命令:

bash docker.sh

 

加载镜像(worker节点)

docker load < shipyard-docker-proxy.tar.gz

docker load < swarm.tar.gz

docker load < alpine.tar.gz

 

运行以下命令:

export ACTION=node DISCOVERY=etcd://<manager host ip>:4001 && bash docker.sh

 

查看所有节点swarm-manager日志:

docker logs -f shipyard-swarm-manager

显示如下则表示服务启动成功:

若出现如下错误:

解决方案:(所有节点)

停止docker服务,删除/etc/docker/key.json,启动docker服务

systemctl stop docker.service (所有节点同时进行,即都在删除key.json文件前执行)

rm -rf /etc/docker/key.json (所有节点同时进行,即都在停止docker服务后删除key.json)

systemctl start docker.service (所有节点同时进行,即都在删除key.json文件后执行)

 

浏览器访问:

http://<manager host ip>:8080

节点信息:

 

3、安装SwarmOps:

加载SwarmOps镜像:

docker load < swarmops.tar.gz

启动SwarmOps容器:

docker run -tdi --name swarmops --net=host --restart=always swarmops:0.0.2

浏览器访问:

http://<manager host ip>:10130

可以看到第一步创建的swarm集群

有兴趣的童鞋可以回复我获取文档中的pachages

© 著作权归作者所有

atompi

atompi

粉丝 21
博文 5
码字总数 7555
作品 0
深圳
运维
私信 提问
编排管理成容器云关键 Kubernetes(K8s)和Swarm对比分析

容器管理和编排将成云计算主战场 从云化的诱因说起。中国云计算实践八年多,市场认知逐渐提升,锐意创新企业对云的期待已经不是资源弹性、成本优势那么简单,业务的灵活性和稳定性才是直接目...

店家小二
2018/12/16
0
0
Docker是什么?它会颠覆企业云市场吗?

Docker到底是什么?很多人问,因为过去一年,几乎全球所有的主流IaaS公有云提供商,都宣布提供Docker(容器)服务。Docker公司的市值,如今已经超过20亿美元,而且还在继续极速飙升。 笔者在...

huangbo929
2016/04/05
0
0
阿里云容器服务新增支持Kubernetes编排系统,性能重大提升

场景研读 2017-11-29 10:33:53 浏览90 评论0 深度学习 架构 docker 性能 公有云 云服务器 镜像 集群 微服务 容器 弹性伸缩 负载均衡分布式存储 企业应用 kubernetes 摘要: 作为容器编排系统...

姬子玉
2017/11/30
0
0
细说Mammut大数据系统测试环境Docker迁移之路

欢迎访问网易云社区,了解更多网易技术产品运营经验。 前言 最近几个月花了比较多精力在项目的测试环境Docker迁移上,从最初的docker“门外汉”到现在组里的同学(大部分测试及少数的开发)都...

网易云
2018/11/21
0
0
企业为什么要使用基于Docker的PaaS/CaaS平台

前言 早在十多年前国内外的一些大厂就开始投入研发和使用容器技术,比如Google,对他们来说,使用容器能够充分利用计算资源节省硬件成本。而这几年——众所周知——真正把容器技术发扬光大的...

精灵云
2017/04/07
0
0

没有更多内容

加载失败,请刷新页面

加载更多

linux负载均衡总结性说明 四层负载和七层负载有什么区别

这篇文章主要为大家详细介绍了linux负载均衡的相关资料,什么是负载均衡?四层负载和七层负载有什么区别?具有一定的参考价值,感兴趣的小伙伴们可以参考一下 在常规运维工作中,经常会运用到...

天子剑毅
21分钟前
2
0
mysql in与or效率比较

在网上一直看到的是or和in的效率没啥区别,一直也感觉是这样,前几天刚好在看《mysql数据库开发的36条军规》的文章,里面提到了or和in的效率问题,文中提到or的效率为O(n),而in的效率为O(l...

whatwhowhy
21分钟前
2
0
使用docker 基于pxc镜像搭建mysql高可用集群

前置条件 docker已安装: 第一步:拉取镜像 docker pull percona/percona-xtradb-cluster:5.7.21 第二步:复制重命名镜像(可选) docker tag percona/percona-xtradb-cluster:5.7.21 pxc 第...

小海bug
27分钟前
4
0
windows安装nginx负载均衡

第一步:下载安装nginx 地址:http://nginx.org/en/docs/windows.html 下载完成,比如放在C盘根目录下: cd c:\ unzip nginx-1.15.3.zip //解压文件 cd nginx-1.15.3 //进入目录 start ngin...

你好夜故事
29分钟前
4
0
Jenkins CLI,助你轻松管理 Jenkins

本文首发于:Jenkins 中文社区 作者:Donghui Wang Jenkins CLI,简称 jcli,一个使用 Golang 开发的开源的 Jenkins 命令行工具。 它可以帮忙你轻松地管理 Jenkins。 无论你是 Jenkins 插件开...

Jenkins中文社区
31分钟前
3
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部