文档章节

将Kubernetes安装到Docker容器里面

openthings
 openthings
发布于 03/16 10:48
字数 1657
阅读 143
收藏 1
点赞 0
评论 0

将Kubernetes安装到Docker容器里面,使用DinD(Docker in Docker)完成。

kubeadm-dind-cluster Build Status

A Kubernetes multi-node cluster for developer of Kubernetes and projects that extend Kubernetes. Based on kubeadm and DIND (Docker in Docker).

Supports both local workflows and workflows utilizing powerful remote machines/cloud instances for building Kubernetes, starting test clusters and running e2e tests.

If you're an application developer, you may be better off with Minikube because it's more mature and less dependent on the local environment, but if you're feeling adventurous you may give kubeadm-dind-cluster a try, too. In particular you can run kubeadm-dind-cluster in CI environment such as Travis without having issues with nested virtualization.

Requirements

Docker 1.12+ is recommended. If you're not using one of the preconfigured scripts (see below) and not building from source, it's better to have kubectl executable in your path matching the version of k8s binaries you're using (i.e. for example better don't use kubectl 1.8.x with hyperkube 1.7.x).

kubeadm-dind-cluster supports k8s versions 1.7.x (tested with 1.7.12), 1.8.x (tested with 1.8.6) and 1.9.x (tested with 1.9.1).

As of now, running kubeadm-dind-cluster on Docker with btrfs storage driver is not supported.

The problems include inability to properly clean up DIND volumes due to a docker bug which is not really fixed and, more importantly, a kubelet problem. If you want to run kubeadm-dind-cluster on btrfs anyway, set RUN_ON_BTRFS_ANYWAY environment variable to a non-empty value.

By default kubeadm-dind-cluster uses dockerized builds, so no Go installation is necessary even if you're building Kubernetes from source. If you want you can overridde this behavior by setting KUBEADM_DIND_LOCAL to a non-empty value in config.sh.

Mac OS X considerations

Ensure to have md5sha1sum installed. If not existing can be installed via brew install md5sha1sum.

When building Kubernetes from source on Mac OS X, it should be possible to build kubectl locally, i.e. make WHAT=cmd/kubectl must work.

Using preconfigured scripts

kubeadm-dind-cluster currently provides preconfigured scripts for Kubernetes 1.7, 1.8 and 1.9. This may be convenient for use with projects that extend or use Kubernetes. For example, you can start Kubernetes 1.8 like this:

$ wget https://cdn.rawgit.com/Mirantis/kubeadm-dind-cluster/master/fixed/dind-cluster-v1.8.sh
$ chmod +x dind-cluster-v1.8.sh

$ # start the cluster
$ ./dind-cluster-v1.8.sh up

$ # add kubectl directory to PATH
$ export PATH="$HOME/.kubeadm-dind-cluster:$PATH"

$ kubectl get nodes
NAME          STATUS    AGE       VERSION
kube-master   Ready     6m        v1.8.6
kube-node-1   Ready     5m        v1.8.6
kube-node-2   Ready     5m        v1.8.6

$ # k8s dashboard available at http://localhost:8080/ui

$ # restart the cluster, this should happen much quicker than initial startup
$ ./dind-cluster-v1.8.sh up

$ # stop the cluster
$ ./dind-cluster-v1.8.sh down

$ # remove DIND containers and volumes
$ ./dind-cluster-v1.8.sh clean

Replace 1.8 with 1.7 or 1.9 to use other Kubernetes versions. Important note: you need to do ./dind-cluster....sh clean when you switch between Kubernetes versions (but no need to do this between rebuilds if you use BUILD_HYPERKUBE=y like described below).

Using with Kubernetes source

$ git clone git@github.com:Mirantis/kubeadm-dind-cluster.git ~/dind

$ cd ~/work/kubernetes/src/k8s.io/kubernetes

$ export BUILD_KUBEADM=y
$ export BUILD_HYPERKUBE=y

$ # build binaries+images and start the cluster
$ ~/dind/dind-cluster.sh up

$ kubectl get nodes
NAME          STATUS         AGE
kube-master   Ready,master   1m
kube-node-1   Ready          34s
kube-node-2   Ready          34s

$ # k8s dashboard available at http://localhost:8080/ui

$ # run conformance tests
$ ~/dind/dind-cluster.sh e2e

$ # restart the cluster rebuilding
$ ~/dind/dind-cluster.sh up

$ # run particular e2e test based on substring
$ ~/dind/dind-cluster.sh e2e "existing RC"

$ # shut down the cluster
$ ~/dind/dind-cluster.sh down

The first dind/dind-cluster.sh up invocation can be slow because it needs to build the base image and Kubernetes binaries. Subsequent invocations are much faster.

IPv6 Mode (experimental)

To run Kubernetes in IPv6 only mode, set the environment variable IP_MODE to "ipv6". There are additional customizations that you can make for IPv6, to set the prefix used for DNS64, subnet prefix to use for DinD, and the service subnet CIDR (among other settings - see dind-cluster.sh):

export EMBBEDDED_CONFIG=y
export DNS64_PREFIX=fd00:77:64:ff9b::
export DIND_SUBNET=fd00:77::
export SERVICE_CIDR=fd00:77:30::/110

As of November 28th, there are two IPv6 Kuberentes PRs in-flight. One is for Kubenet and one for E2E tests (neither is required for IPv6 use). You can cherry pick these PRs, if desired, and then set the BUILD_HYPERKUBE and BUILD_KUBEADM flags, to include the changes in a local Kubernetes repo.

PR #56245 Updates kubenet CNI template for v0.3.1"
PR #52748 "Add brackets around IPv6 addrs in e2e test IP:port endpoints"

git fetch origin pull/56245/head:pr56245
git log --abbrev-commit pr56245 --oneline --abbrev-commit -n 1 | cut -f 1 -d" "

git fetch origin pull/52748/head:pr52748
git log --abbrev-commit pr52748 --oneline --abbrev-commit -n 1 | cut -f 1 -d" "

Note: If you run into a kube-proxy crash during an attempt to modify conntrack settings, you'll need to patch that is mentioned in this issue:

https://github.com/Mirantis/kubeadm-dind-cluster/issues/50

Configuration

You may edit config.sh to override default settings. See comments in the file for more info. In particular, you can specify CNI plugin to use via CNI_PLUGIN variable (bridge, flannel, calico, weave).

Remote Docker / GCE

It's possible to build Kubernetes on a remote machine running Docker. kubeadm-dind-cluster can consume binaries directly from the build data container without copying them back to developer's machine. An example utilizing GCE instance is provided in gce-setup.sh. You may try running it using source (.) so that docker-machine shell environment is preserved, e.g.

. gce-setup.sh

The example is based on sample commands from build/README.md in Kubernetes source.

When using a remote machine, you need to use ssh port forwarding to forward KUBE_RSYNC_PORT and APISERVER_PORT you choose.

Dumping cluster state

In case of CI environment such as Travis CI or Circle CI, it's often desirable to get detailed cluster state for a failed job. Moreover, in case of e.g. Travis CI there's no way to store the artefacts without using an external service such as Amazon S3. Because of this, kubeadm-dind-cluster supports dumping cluster state as a text block that can be later split into individual files. For cases where there are limits on the log size (e.g. 4 Mb log limit in Travis CI) it's also possible to dump the lzma-compressed text block using base64 encoding.

The following commands can be used to work with cluster state dumps:

  • ./dind-cluster.sh dump dumps the cluster state as a text block
  • ./dind-cluster.sh dump64 dumps the cluster state as a base64 blob
  • ./dind-cluster.sh split-dump splits the text block into individual files using @@@ filename @@@ markers which are generated by dump. The output is stored in cluster-dump/ subdirectory of the current directory.
  • ./dind-cluster.sh split-dump64 splits the base64 blob into separate files. The blob has start and end markers so it can be extracted automatically from a build job log. The output is stored in cluster-dump/ subdirectory of the current directory.

All of the above commands work with 'fixed' scripts, too. kubeadm-dind-cluster's own Travis CI jobs dump base64 blobs in case of failure. Such blocks can be then extracted directly from the output of travis command line utility, e.g.

travis logs NNN.N | ./dind-cluster.sh split-dump64

The following information is currently stored in the dump:

  • status and logs for the following systemd units on each DIND node, if the exist: kubelet.service, dindnet.service, criproxy.service and dockershim.service (the latter two are used by CRI Proxy)
  • ps auxww, docker ps -a, ip a and ip r output for each DIND node
  • the logs of all the containers of each pod in the cluster
  • the output of kubectl get all --all-namespaces -o wide, kubectl describe all --all-namespaces and kubectl get nodes -o wide

Motivation

hack/local-up-cluster.sh is widely used for k8s development. It has a couple of serious issues though. First of all, it only supports single node clusters, which means that it's hard to use it to work on e.g. scheduler-related issues and e2e tests that require several nodes can't be run. Another problem is that it has little resemblance to real clusters.

There's also k8s vagrant provider, but it's quite slow. Besides, cluster/ directory in k8s source is now considered deprecated.

Another widely suggested solution for development clusters is minikube, but currently it's not very well suited for development of Kubernetes itself. Besides, it's currently only supports single node, too, unless used with additional DIND layer like nkube.

kubernetes-dind-cluster is very nice & useful but uses a custom method of cluster setup (same as 2nd problem with local-up-cluster).

There's also sometimes a need to use a powerful remote machine or a cloud instance to build and test Kubernetes. Having Docker as the only requirement for such machine would be nice. Builds and unit tests are already covered by jbeda's work on dockerized builds, but being able to quickly start remote test clusters and run e2e tests is also important.

kubeadm-dind-cluster uses kubeadm to create a cluster consisting of docker containers instead of VMs. That's somewhat of a compromise but allows one to (re)start clusters quickly which is quite important when making changes to k8s source.

Moreover, some projects that extend Kubernetes such as Virtlet need a way to start kubernetes cluster quickly in CI environment without involving nested virtulization. Current kubeadm-dind-cluster version provides means to do this without the need to build Kubernetes locally.

Additional notes

At the moment, all non-serial [Conformance] e2e tests pass for clusters created by kubeadm-dind-cluster. [Serial]...[Conformance] tests currently have some issues. You may still try running them though:

$ dind/dind-cluster.sh e2e-serial

Related work

  • kubeadm-dind-cluster was initially derived from kubernetes-dind-cluster, although as of now the code was completely rewritten. kubernetes-dind-cluster is somewhat faster but uses less standard way of k8s deployment. It also doesn't include support for consuming binaries from remote dockerized builds.
  • kubeadm-ci-dind, kubeadm-ci-packager and kubeadm-ci-tester. These projects are similar to kubeadm-dind-cluster but are intended primarily for CI. They include packaging step which is too slow for the purpose of having convenient k8s "playground". kubeadm-dind-cluster uses Docker images from kubeadm-ci-dind.
  • nkube starts Kubernetes-in-Kubernetes clusters.

本文转载自:https://github.com/openthings/kubeadm-dind-cluster

共有 人打赏支持
openthings
粉丝 241
博文 877
码字总数 459499
作品 1
东城
架构师
如何在VMware vSphere上安装Kubernetes来运行Docker

安装谷歌Kubernetes不需要VMware vSphere或者任何其他的虚拟机控制器。在虚拟机上安装十分方便,所以在配置和测试环境的时候也强烈推荐安装。 先不管虚拟机或者物理机,安装Linux OS是必须的...

Caicloud ⋅ 2016/06/20 ⋅ 0

将项目迁移到Kubernetes上的5个诀窍

我们将在本文中提供5个诀窍帮你将项目迁移到Kubernetes上,这些诀窍来源于过去12个月中OpenFaas社区的经验。下文的内容与Kubernetes 1.8兼容,并且已经应用于OpenFaaS - Serverless Functio...

Docker ⋅ 04/27 ⋅ 0

IBM基于Kubernetes的容器云全解析

容器云最主要的功能是以应用为中心,帮助用户把所有的应用以容器的形式在分布式里面跑起来,最后把应用以服务的形式呈现给用户。容器云里有两个关键点,一是容器编排,二是资源调度。 容器编...

刘光亚 ⋅ 2017/02/09 ⋅ 0

我们为什么选择Kubernetes

【编者的话】这篇文章介绍了作者在选择容器编排工具过程中的一些总结和经验,各个编排工具的特性比较,以及自己团队为什么选择Kubernetes。 几个月前,我开始调研Docker容器的编排工具,例如...

奔跑的阿飞 ⋅ 2017/09/07 ⋅ 0

从 0 开始了解 Docker

Docker 自开源以来受到了各大公司的广泛关注,或许现在互联网公司的运维体系不承载在 Docker(或 Pouch 等)之上都不好意思说自己的互联网公司。 本文会简单介绍下 Docker 的基础概念,入门级...

Docker ⋅ 04/18 ⋅ 0

化敌为友,Docker 宣布拥抱 Kubernetes

在 DockerCon EU 2017 大会上,Docker 官方宣布支持 Kubernetes。过去几年,Kubernetes 得到了包括Google、Huawei、Microsoft、IBM、AWS、Rancher、Redhat、CoreOS 等企业的一致认可。前几日...

达尔文 ⋅ 2017/10/20 ⋅ 9

Docker的那些事(Kubernetes+docker)

采用VMware的虚拟机虚拟化三台Centos7的系统,作为我们安装Docker的环境系统; 提示出现的问题 重新安装的过程中选择相应的网卡即可:安装的过程选择相应的网卡即可 选择相应的网卡 1.根据官...

owensliu ⋅ 02/28 ⋅ 0

Kubernetes结合Docker的优势

你应该会看到很多关于Kubernetes和Docker的消息,以至于你可能想知道哪个更好。 事实上,没有“更好”一说,因为它们所解决的问题是不一样的。Docker就像飞机,而Kubernetes就像飞机场。就像...

m2l0zgssvc7r69efdtj ⋅ 05/30 ⋅ 0

Docker集群的两种模式K8s和swarm究竟有何异同?未来企业应用哪个会更加广泛?

1、出生不同,Google根据其在Linux上容器管理经验,改造到docker管理上,就是kubernetes。他的在许多方面表现良好。最重要的是构造于Google多年的宝贵经验只上。kubernetes并不是为了docker写...

51aa ⋅ 2017/06/29 ⋅ 0

Containerd给Kubernetes带来了更多容器运行时的可选方案

容器运行时是在某个节点上执行容器并且管理容器镜像的软件。如今,最广为人知的容器运行时是Docker,但是,生态系统内也有一些别的容器运行时,比如rkt,containerd和Ixd。Docker是目前为止在...

Docker ⋅ 02/20 ⋅ 0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

windows profesional 2017 build problem

.net framework .... https://stackoverflow.com/questions/43330915/could-not-load-file-or-assembly-microsoft-build-frameworkvs-2017...

机油战士 ⋅ 41分钟前 ⋅ 0

python3中报错的解决方法(长期更新)

1、ImportError: No module named ‘DjangoUeditor’ 出错原因:安装DjangoUeditor库适用于python2,需要下载适用python3的 下载地址:https://github.com/twz915/DjangoUeditor3 2、python3......

xiaoge2016 ⋅ 46分钟前 ⋅ 0

数据结构与算法之双向链表

一、双向链表 1.双向链表的结点结构 typedef struct DualNode{ ElemType data; struct DualNode *prior; // 前驱结点 struct DualNode *next; // 后继结点}DualNode, *DuL...

aibinxiao ⋅ 今天 ⋅ 0

五大最核心的大数据技术

大数据技术有5个核心部分,数据采集、数据存储、数据清洗、数据挖掘、数据可视化。关于这5个部分,有哪些核心技术?这些技术有哪些潜在价值?看完今天的文章就知道了。 大数据学习群:7165810...

董黎明 ⋅ 今天 ⋅ 0

PhpStorm 头部注释、类注释和函数注释的设置

首先,PhpStorm中文件、类、函数等注释的设置在:setting-》Editor-》FIle and Code Template-》Includes下设置即可,其中方法的默认是这样的: /**${PARAM_DOC}#if (${TYPE_HINT} != "v...

nsns ⋅ 今天 ⋅ 0

spring.net AOP

http://www.springframework.net/doc-latest/reference/html/aop-quickstart.html https://www.cnblogs.com/wujy/archive/2013/04/06/3003120.html...

whoisliang ⋅ 今天 ⋅ 0

【HAVENT原创】创建 Dockerfile 生成新的镜像,并发布到 DockerHub

注意:Win7 与 Win10 的版本存在差异,Win7 版本使用 Docker Quickstart Terminal 进入控制台,Win10下面直接用管理员权限打开控制台或者 PowerShell 即可;另外 Win7 下面只能访问 C盘,/ap...

HAVENT ⋅ 今天 ⋅ 0

pom.xml出现web.xml is missing ...解决方案

提示信息应该能看懂。也就是缺少了web.xml文件,<failOnMissingWebXml>被设置成true了。 搜索了一下,Stack Overflow上的答案解决了问题,分享一下。 目前被顶次数最多的回答原文如下: This...

源哥L ⋅ 今天 ⋅ 0

js时间戳与日期格式之间相互转换

1. 将时间戳转换成日期格式 1 2 3 4 5 6 7 8 9 10 11 12 13 14 // 简单的一句代码 var date = new Date(时间戳); //获取一个时间对象 /** 1. 下面是获取时间日期的方法,需要什么样的格式自己...

Jack088 ⋅ 今天 ⋅ 0

web添加log4j

添加xml配置log4j.properties # Global logging configuration---root日志设置#log4j.rootLogger=info,dailyRollingFile,stdoutlog4j.rootLogger=debug,stdout,dailyRollingFile---......

黄柳淞 ⋅ 今天 ⋅ 0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

返回顶部
顶部