Kubernetes Jenkins动态创建Slave

2019/04/10 10:10
阅读数 88

[TOC]

0、前言

首先,我们考虑个问题,为何需要jenkins slave?其实在生产环境中,如果用单master,除非你单机器的配置特别高并且构建次数不多情况下,可以不考虑使用slave,但是,在构建次数上百次并且jenkins master运行在kubernetes环境中,借助kubernetes的灵活性,强烈推荐使用slave,master负责自动创建Slave Pod,然后将任务推送给Slave Pod,任务执行完毕后,Slave Pod会自动被回收/销毁。

创建slave流程图:

1、Jenkins部署

jenkins交付进kubernetes

1.准备镜像文件

$ docker pull jenkins/jenkins:2.204.1
$ docker tag a3f949e5ebfd harbor.od.com/infra/jenkins:v2.204.1
$ docker push harbor.od.com/infra/jenkins:v2.204.1

2.资源配置文件

  • rbac
$ vi /data/k8s-yaml/jenkins_slave/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: infra
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: infra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: infra

Note:Jenkins会自动创建slave pod,所以需要给jenkins绑定Kubernetes的RBAC权限

  • Deployment
$ vi /data/k8s-yaml/jenkins_slave/dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: jenkins
  namespace: infra
  labels: 
    name: jenkins
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: jenkins
  template:
    metadata:
      labels: 
        app: jenkins 
        name: jenkins
    spec:
      serviceAccount: jenkins
      volumes:
      - name: data
        nfs: 
          server: hdss7-200.host.com
          path: /data/nfs-volume/jenkins_home
      containers:
      - name: jenkins
        image: harbor.od.com/infra/jenkins:v2.204.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        env:
        - name: JAVA_OPTS
          value: "-Xms1G -Xmx1G -XX:PermSize=512m -XX:MaxPermSize=1024m -Duser.timezone=Asia/Shanghai"
        - name: TRY_UPGRADE_IF_NO_MARKER
          value: "true"
        volumeMounts:
        - name: data
          mountPath: /var/jenkins_home
      imagePullSecrets:
      - name: harbor
      securityContext: 
        runAsUser: 0
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600
  • Service
$ vi /data/k8s-yaml/jenkins_slave/svc.yaml
kind: Service
apiVersion: v1
metadata: 
  name: jenkins
  namespace: infra
spec:
  ports:
  - name: web
    port: 80
    targetPort: 8080
    protocol: TCP
  - name: agent
    port: 50000
    targetPort: 50000
    protocol: TCP
  selector:
    app: jenkins
  • Ingress
$ vi /data/k8s-yaml/jenkins_slave/ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: jenkins
  namespace: infra
spec:
  rules:
  - host: jenkins.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: jenkins
          servicePort: 80

Note:这里的Service对外暴露了80以及50000端口,80作为Jenkins Server Web端口,50000位创建的Jenkins Salve和Master建立通信连接的默认端口,如果该端口不暴露的话,Slave就无法和Master建立连接。

3.应用资源配置清单

$ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/rbac.yaml
$ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/dp.yaml
$ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/svc.yaml
$ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/ingress.yaml

2、配置jenkins动态slave

初始化jenkins后,需要安装kubernetes 插件。

1.安装插件完成之后,点击 Manage Jenkins —> Configure System —> (拖到最下方)Add a new cloud —> 选择 Kubernetes,然后填写 Kubernetes 和 Jenkins 配置信息

填写kubernetes集群内部访问地址:https://kubernetes.default.svc.cluster.local,点击Test Connection,如果出现 Connection test successful 的提示信息证明 Jenkins 已经可以和 Kubernetes 系统正常通信了,然后下方的 Jenkins URL 地址:http://jenkins.infra.svc.cluster.local:80

为何我这里使用k8s svc的名称使用方式,为了方便通过DNS记录能够解析成该Service的Cluster IP。

2.创建Pipeline动态构建测试 创建一个类型为Pipeline类型Job命名为test-slave,然后在Pipeline脚本填写下面一个简单的测试脚本

def label = "jenkins-slave-${UUID.randomUUID().toString()}"
podTemplate(label: label, cloud: 'kubernetes') {
    node(label) {
        stage('Run shell') {
            sh 'sleep 10s'
            sh 'echo hello world.'
        }
    }
}

3.点击构建流水线

可以看到在jenkins的namespaces下自动创建了对应的agent pod 相当于就是一个jenkins 的node 当任务执行完成这个pod会自动退出这个pod默认会去pull一个jenkins/jnlp-slave:x.xx-xx-alpine的镜像

[root@hdss7-21 ~]# kubectl get pods -n infra
NAME                                                             READY   STATUS    RESTARTS   AGE
jenkins-77b9c47874-qjgfd                                         1/1     Running   1          13h
jenkins-slave-c07daa7b-31ef-41ea-825e-05c9c721edad-sb7h6-lpgwv   1/1     Running   0          18s

3、dubbo服务构建

我们在构建dubbo服务的时候,需要编译dubbo后制作镜像并推送到harbor,这时候就要用到maven和docker命令,所以需要自己去构建基础镜像,然后在pipeline里调用

注意:这里并不实现如何将dubbo服务交付到k8s,而是演示实现动态jenkins创建slave去构建dubbo服务

3.1、制作dubbo镜像底包

底包需要有通用性,所有的dubbo微服务制作镜像时都会引用这个底包来制作,该底包主要实现一些功能,例如jmx监控(jmx_javaagent),和dubbo微服务的启动脚本,以及依赖的jdk环境

1.准备镜像(jdk环境)

$ docker pull stanleyws/jre8:8u112
$ docker tag stanleyws/jre8:8u112 harbor.od.com/public/jre:8u112
$ docker push harbor.od.com/public/jre:8u112

2.自定义Dockerfile

  • /data/dockerfile/jre8/Dockerfile
FROM harbor.od.com/public/jre:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo 'Asia/Shanghai' >/etc/timezone
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/
WORKDIR /opt/project_dir
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]
  • config.yml(这是jmx_agent读取的配置文件)
---
rules:
  - pattern: '.*'
  • jmx_javaagent-0.3.1.jar
# 采集jvm监控数据的jar包
$ wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar
  • entrypoint.sh(不要忘了给该文件执行权限)
#!/bin/sh
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"
C_OPTS=${C_OPTS}
JAR_BALL=${JAR_BALL}
exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}

按照上面的java启动选项解释

-agentpath:<pathname>[=<options>]   // 加载java代理
-Duser.timezone:<timezone>          // 指定时区

C_OPTS=${C_OPTS}      // 额外的启动参数,默认为空,可在k8s资源配置清单中添加额外参数
JAR_BALL=${JAR_BALL}  // 启动的jar包名字,在k8s资源配置清单中指定

3.制作dubbo服务docker底包

$ ls -l
total 372
-rw-r--r-- 1 root root    405 Jan 16 15:26 Dockerfile
-rw-r--r-- 1 root root     41 Jan 16 15:28 config.yaml
-rwxr-xr-x 1 root root    234 Jan 16 15:37 entrypoint.sh
-rw-r--r-- 1 root root 367417 May 10  2018 jmx_prometheus_javaagent-0.3.1.jar

$ docker build . -t harbor.od.com/base/jre8:8u112
$ docker push harbor.od.com/base/jre8:8u112

3.2、制作slave基础镜像

我们现在有个dubbo项目,需要用到maven构建项目,然后通过Docker打包成镜像并推送到Harbor,所以需要用到两个镜像,Maven以及Docker镜像。

3.2.1、Maven镜像

该镜像主要用于构建java应用,这里就选择:maven:v3.3.9-jdk8

准备镜像文件(推送到本地仓库)

$ docker pull maven:3.3.9-jdk-8-alpine
$ docker tag dd9d4e1cd9db harbor.od.com/public/maven:v3.3.9-jdk8
$ docker push harbor.od.com/public/maven:v3.3.9-jdk8

3.2.2、Docker镜像

该镜像主要用于将dubbo项目打包成镜像并推送到harbor,但需要定制化一下镜像,需要将一台已经实现docker login 登录到harbor仓库所生成的配置文件,路径为:/root/.docker/config.json,与原始Docker镜像一起打包生成新的Docker镜像并推送到本地仓库。

1.准备镜像文件

$ docker pull docker:19.03
$ docker tag e036013d6d10 harbor.od.com/public/docker:v19.03
$ docker push harbor.od.com/public/docker:v19.03

2.Dockerfile

  • vim /data/dockerfile/docker/Dockerfile
FROM harbor.od.com/public/docker:v19.03
USER root
ADD config.json /root/.docker/config.json

3.将/root/.docker/config.json文件拷贝到Dockerfile目录下

{
	"auths": {
		"harbor.od.com": {
			"auth": "YWRtaW46SGFyYm9yMTIzNDU="
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.6 (linux)"
	}

需要通过该文件才能访问到harbor仓库

4.制作并推送镜像(推送到本地仓库)

$ docker build ./ -t harbor.od.com/public/docker:v19.03
$ docker push  harbor.od.com/public/docker:v19.03

3.3、添加git key

我们这里使用到了Jenkins的Git插件来拉取代码,所以需要先创建一堆密钥,然后将公钥添加到git仓库,再将私钥添加到jenkins凭证,如下:

3.4、创建dubbo流水线

1.添加参数化构建

  • The String Parameter:app_name

    Describe:项目名称,例如dubbo-service

  • String Parameter:image_name

    Describe:docker镜像名称,格式:<仓库名>/<镜像名> 例如:app/dubbo-demo-service

  • String Parameter:git_repo

    Describe:项目所在的git中央仓库的地址:如https://gitee.com/jasonminghao/dubbo-demo-service.git

  • String Parameter:git_ver

    Describe:项目所在git中央仓库对应项目的分支或者版本号,例如master分支:*/master,commit ID:903b4e6

  • String Parameter:image_ver

    Describe:镜像版本和git_ver一致即可,但是不要有任何的特殊符号

  • String Parameter:add_tag

    Describe:docker镜像标签的一部分,日期时间戳,例如:20200121_1734

  • String Parameter:target_dir

    Default:./target

    Describe:编译项目的目录,产生jar/war包所在的目录

  • String Parameter:mvn_cmd

    Default:mvn clean package -Dmaven.test.skip=true

    Describe:执行mvn编译所用的命令

  • Choice Parameter:base_image

    Choice Value: base/jre8:8u112

    Describe:项目使用的docker底包镜像

2.pipeline

podTemplate(cloud:'kubernetes',containers: [
    containerTemplate(
       name: 'maven', 
       image: 'harbor.od.com/public/maven:v3.3.9-jdk8', 
       ttyEnabled: true,
       command: 'cat'),
    containerTemplate(
       name: 'docker',
      ttyEnabled: true,
      image: 'harbor.od.com/public/docker:v19.03'),
    ],
    volumes: [
       nfsVolume(mountPath: '/root/.m2', readOnly: false, serverAddress: 'hdss7-200.host.com', serverPath: '/data/nfs-volume/maven_repo/'),
       hostPathVolume(hostPath: '/run/docker.sock', mountPath: '/run/docker.sock')
            ]
 ){
    node(POD_LABEL) {
        stage('Get a Maven project') {
        // 从git仓库拉取代码
        checkout([$class: 'GitSCM', branches: [[name: "${params.git_ver}"]], browser: [$class: 'GitLab', repoUrl: ''], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'git', name: 'git', refspec: 'refs/changes/*:refs/changes/*', url: "${params.git_repo}"]]])

            container('maven') {
                stage('Build a Maven project') {
                    // 执行maven构建
                    sh "${params.mvn_cmd}"
                }
            }
        }
        stage('Docker build') {
            container('docker') {
                stage('create dir') {
                    // /tmp目录创建一个临时用于构建镜像的工作目录,将jar包移动到该目录
                    sh "mkdir /tmp/${params.app_name}"
                    sh "cd ${params.target_dir} && mkdir /tmp/${params.app_name}/project_dir && mv *.jar /tmp/${params.app_name}/project_dir"
                }
                stage('docker build image') {
                        // 动态生成Dockerfile,构建镜像并推送到harbor
                        sh "cd /tmp/${params.app_name}/ && ls -lha"
                        sh """
                        echo "FROM harbor.od.com/${params.base_image}" >/tmp/${params.app_name}/Dockerfile
                        echo "ADD ./project_dir /opt/project_dir" >>/tmp/${params.app_name}/Dockerfile
                        """
                        // writeFile 在当前pipeline脚本中不生效(抛弃使用)
                        // writeFile file: "/tmp/${params.app_name}/Dockerfile", text: """FROM harbor.od.com/${params.base_image} ADD ./project_dir /opt/project_dir"""
                        sh "ls -lha /tmp/${params.app_name}"
                        sh "cat /tmp/${params.app_name}/Dockerfile"
                        sh "cd /tmp/${params.app_name}/ && pwd && docker build ./ -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} "                              
                    }
            }
        }
    }
}

podTemplate只适用于pipeline

3.5、执行流水线构建

1.填写对应参数

2.查看infra名称空间下的Pod

[root@hdss7-200 harbor]# kubectl get pods -n infra
NAME                                    READY   STATUS    RESTARTS   AGE
apollo-portal-57bc86966d-4tr6w          1/1     Running   8          37h
dubbo-demo-slave-16-trktm-8x2d7-bw5dr   3/3     Running   0          53s
dubbo-monitor-555c94f4b7-85plg          1/1     Running   32         7d14h
jenkins-75fbb46546-f5ltc                1/1     Running   6          18h

可以看到jenkins slave会以在jenkins创建的项目名为命名来创建pod

3.构建结果

原文出处:https://www.cnblogs.com/jasonminghao/p/12468407.html

展开阅读全文
打赏
0
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部