深度剖析Kubernetes动态准入控制之Admission Webhooks

原创
2017/12/19 00:16
阅读数 2.1W

Author: xidianwangtao@gmail.com

Admission Controll的最佳配置

这部分内容,请参考我的上一篇博文深度剖析Kubernetes动态准入控制之Initializers

External Admission Webhooks工作机制

External Admission Webhooks有什么用

我们什么时候需要用External Admission Webhooks呢?当集群管理员需要强制对某些请求或者所有请求都进行校验或者修改的时候,就可以考虑使用ValidatingAdmissionWebhook或MutatingAdmissionWebhook,二者区别如下:

  • MutatingAdmissionWebhook允许你在webhook中对object进行mutate修改;
  • ValidatingAdmissionWebhook(Kubernetes 1.9之前叫GenericAdmissionWebhook)不允许你在webhook中对Object进行mutate修改,只是返回validate结果为true or false;

如何启用External Admission Webhooks

  • 前面提到,需要在每个kube-apiserver实例(考虑到Kubernetes Master HA)中--admission-controll中添加ValidatingAdmissionWebhookMutatingAdmissionWebhook
  • 另外,还需要在每个kube-apiserver实例的--runtime-config中添加admissionregistration.k8s.io/v1alpha1。disable的时候,也记得要从这里删除。

MutatingAdmissionWebhook注意事项

  • beta in 1.9;
  • 需要注意,MutatingAdmissionWebhook是让匹配的webhooks串行执行的,因为每个webhook都可能会mutate object。
  • 同Initializers的使用类似,MutatingAdmissionWebhook是通过创建MutatingWebhookConfiguration来配置什么样的request会触发哪个webhook。
  • 注意apiserver调用webhook时一定是通过TLS认证的,所以MutatingWebhookConfiguration中一定要配置caBundle。

ValidatingAdmissionWebhook注意事项

  • alpha in 1.8,beta in 1.9;
  • 需要注意,ValidatingAdmissionWebhook是让匹配的webhooks并发执行的,因为每个webhook只会进行validate操作,而不会mutate object。
  • 同Initializers的使用类似,ValidatingAdmissionWebhook是通过创建ValidatingWebhookConfiguration来配置什么样的request会触发哪个webhook。
  • 注意apiserver调用webhook时一定是通过TLS认证的,所以ValidatingWebhookConfiguration中一定要配置caBundle。

Sample

我们以ValidatingAdmissionWebhook为例子,看看怎么玩。

  1. 部署ValidatingWebhook,比如我们就简单的以Pod部署一个webhook,并创建对应的Service:
apiVersion: v1
kind: Pod
metadata:
  labels:
    role: webhook
  name: webhook
spec:
  containers:
    - name: webhook
      image: example-webhook:latest
      imagePullPolicy: IfNotPresent
      ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  labels:
    role: webhook
  name: webhook
spec:
  ports:
    - port: 443
      targetPort: 8000
  selector:
    role: webhook

其中example-webhook为github的项目caesarxuchao/example-webhook-admission-controller,通过github repo根目录下的Dockerfile制作example-webhook镜像。

FROM golang:1.8

WORKDIR /go/src
RUN mkdir -p github.com/caesarxuchao/example-webhook-admission-controller
COPY . ./github.com/caesarxuchao/example-webhook-admission-controller
RUN go install github.com/caesarxuchao/example-webhook-admission-controller
CMD ["example-webhook-admission-controller","--alsologtostderr","-v=4","2>&1"]
  1. 创建ValidatingWebhookConfiguration Object,比如:
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingWebhookConfiguration
metadata:
  name: config1
externalAdmissionHooks:
  - name: podimage.k8s.io
    rules:
      - operations:
          - CREATE
        apiGroups:
          - ""
        apiVersions:
          - v1
        resources:
          - pods
    failurePolicy: Ignore
    clientConfig:
      caBundle: xxxx
      service:
        namespace: default
        name: webhook
- 注意failurePolicy可以为Ignore或者Fail,意味着如果和webhook通信出现问题导致调用失败,那么将根据failurePolicy进行抉择,是忽略失败(admit)还是认为准入失败(reject)。在Kubernetes 1.7中,该值只允许配置为Ignore。
- 对比initializerConfiguration,ValidatingWebhookConfiguration和MutatingWebhookConfiguration在rule的定义时,增加了operations field,在resources定义时候可以指定subresource,格式为resource/subresource。
  1. ExternalAdmissionHookConfiguration创建后,你需要等待几秒,然后通过通过Deployment或者直接创建Pod,这时创建Pod的请求就会被apiserver拦住,调用ValidatingAdmissionWebhook进行检查是否Admit通过。比如,上面的example-webhook是检查容器镜像是否以"gcr.io"为前缀的。

如何开发一个Externel Admission Webhook

admission controller实际上发送一个admissionReview请求给webhook server,然后webhook server从admissionReview.spec中获取object,oldobject,userInfo等信息,进行mutate或者validate后再发回一个admissionReview作为返回,其中返回的admissionReivew.status中有admission决定。

AdmissionReview

下面是AdmissionReview结构体的定义:

type AdmissionReview struct {
	metav1.TypeMeta `json:",inline"`
	// Request describes the attributes for the admission request.
	// +optional
	Request *AdmissionRequest `json:"request,omitempty" protobuf:"bytes,1,opt,name=request"`
	// Response describes the attributes for the admission response.
	// +optional
	Response *AdmissionResponse `json:"response,omitempty" protobuf:"bytes,2,opt,name=response"`
}

// AdmissionRequest describes the admission.Attributes for the admission request.
type AdmissionRequest struct {
	// UID is an identifier for the individual request/response. It allows us to distinguish instances of requests which are
	// otherwise identical (parallel requests, requests when earlier requests did not modify etc)
	// The UID is meant to track the round trip (request/response) between the KAS and the WebHook, not the user request.
	// It is suitable for correlating log entries between the webhook and apiserver, for either auditing or debugging.
	UID types.UID `json:"uid" protobuf:"bytes,1,opt,name=uid"`
	// Kind is the type of object being manipulated.  For example: Pod
	Kind metav1.GroupVersionKind `json:"kind" protobuf:"bytes,2,opt,name=kind"`
	// Resource is the name of the resource being requested.  This is not the kind.  For example: pods
	Resource metav1.GroupVersionResource `json:"resource" protobuf:"bytes,3,opt,name=resource"`
	// SubResource is the name of the subresource being requested.  This is a different resource, scoped to the parent
	// resource, but it may have a different kind. For instance, /pods has the resource "pods" and the kind "Pod", while
	// /pods/foo/status has the resource "pods", the sub resource "status", and the kind "Pod" (because status operates on
	// pods). The binding resource for a pod though may be /pods/foo/binding, which has resource "pods", subresource
	// "binding", and kind "Binding".
	// +optional
	SubResource string `json:"subResource,omitempty" protobuf:"bytes,4,opt,name=subResource"`
	// Name is the name of the object as presented in the request.  On a CREATE operation, the client may omit name and
	// rely on the server to generate the name.  If that is the case, this method will return the empty string.
	// +optional
	Name string `json:"name,omitempty" protobuf:"bytes,5,opt,name=name"`
	// Namespace is the namespace associated with the request (if any).
	// +optional
	Namespace string `json:"namespace,omitempty" protobuf:"bytes,6,opt,name=namespace"`
	// Operation is the operation being performed
	Operation Operation `json:"operation" protobuf:"bytes,7,opt,name=operation"`
	// UserInfo is information about the requesting user
	UserInfo authenticationv1.UserInfo `json:"userInfo" protobuf:"bytes,8,opt,name=userInfo"`
	// Object is the object from the incoming request prior to default values being applied
	// +optional
	Object runtime.RawExtension `json:"object,omitempty" protobuf:"bytes,9,opt,name=object"`
	// OldObject is the existing object. Only populated for UPDATE requests.
	// +optional
	OldObject runtime.RawExtension `json:"oldObject,omitempty" protobuf:"bytes,10,opt,name=oldObject"`
}

// AdmissionResponse describes an admission response.
type AdmissionResponse struct {
	// UID is an identifier for the individual request/response.
	// This should be copied over from the corresponding AdmissionRequest.
	UID types.UID `json:"uid" protobuf:"bytes,1,opt,name=uid"`

	// Allowed indicates whether or not the admission request was permitted.
	Allowed bool `json:"allowed" protobuf:"varint,2,opt,name=allowed"`

	// Result contains extra details into why an admission request was denied.
	// This field IS NOT consulted in any way if "Allowed" is "true".
	// +optional
	Result *metav1.Status `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`

	// The patch body. Currently we only support "JSONPatch" which implements RFC 6902.
	// +optional
	Patch []byte `json:"patch,omitempty" protobuf:"bytes,4,opt,name=patch"`

	// The type of Patch. Currently we only allow "JSONPatch".
	// +optional
	PatchType *PatchType `json:"patchType,omitempty" protobuf:"bytes,5,opt,name=patchType"`
}

example-webhook

下面是前面提到的example-webhook的main文件,可以借鉴它来开发自己的AdmissionWebhook。

// only allow pods to pull images from specific registry.
func admit(data []byte) *v1alpha1.AdmissionReviewStatus {
	ar := v1alpha1.AdmissionReview{}
	if err := json.Unmarshal(data, &ar); err != nil {
		glog.Error(err)
		return nil
	}
	// The externalAdmissionHookConfiguration registered via selfRegistration
	// asks the kube-apiserver only sends admission request regarding pods.
	podResource := metav1.GroupVersionResource{Group: "", Version: "v1", Resource: "pods"}
	if ar.Spec.Resource != podResource {
		glog.Errorf("expect resource to be %s", podResource)
		return nil
	}

	raw := ar.Spec.Object.Raw
	pod := v1.Pod{}
	if err := json.Unmarshal(raw, &pod); err != nil {
		glog.Error(err)
		return nil
	}
	reviewStatus := v1alpha1.AdmissionReviewStatus{}
	for _, container := range pod.Spec.Containers {
		// gcr.io is just an example.
		if !strings.Contains(container.Image, "gcr.io") {
			reviewStatus.Allowed = false
			reviewStatus.Result = &metav1.Status{
				Reason: "can only pull image from grc.io",
			}
			return &reviewStatus
		}
	}
	reviewStatus.Allowed = true
	return &reviewStatus
}

func serve(w http.ResponseWriter, r *http.Request) {
	var body []byte
	if r.Body != nil {
		if data, err := ioutil.ReadAll(r.Body); err == nil {
			body = data
		}
	}

	// verify the content type is accurate
	contentType := r.Header.Get("Content-Type")
	if contentType != "application/json" {
		glog.Errorf("contentType=%s, expect application/json", contentType)
		return
	}

	reviewStatus := admit(body)
	ar := v1alpha1.AdmissionReview{
		Status: *reviewStatus,
	}

	resp, err := json.Marshal(ar)
	if err != nil {
		glog.Error(err)
	}
	if _, err := w.Write(resp); err != nil {
		glog.Error(err)
	}
}

func main() {
	flag.Parse()
	http.HandleFunc("/", serve)
	clientset := getClient()
	server := &http.Server{
		Addr:      ":8000",
		TLSConfig: configTLS(clientset),
	}
	go selfRegistration(clientset, caCert)
	server.ListenAndServeTLS("", "")
}

思考

我们发现,MutatingAdmissionWebhook其实跟Initializers的目的都是一样的,都是对object做修改,只是实现的方式不同而已。那为什么两者要并存呢?深度剖析Kubernetes动态准入控制之Initializers中提到,Initializers具有如下缺陷:

  • 如果你部署的Initializers Controllers不能正常工作了或者性能很低,在高并发场景下会导致大量的相关对象停留在uninitialized状态,无法进行后续的调度。这可能会影响你的业务,比如你使用了HPA对相关Deployment对象进行弹性扩容,当负载上来的时候,你的Initializers Controllers不能正常工作了,会导致你的应用不能弹性伸缩,后果可想而知!

所以我个人的理解是,MutatingAdmissionWebhook是mutate object的一种比Initializers更优雅的、性能更好的实现方式。而且从Kubernetes 1.9发布的情况来看,MutatingAdmissionWebhook是Beta,而Initializers还停留在Alpha,并且前面提到的Kubernetes 1.9中官方推荐的--adminssion-control配置中推荐的是MutatingAdmissionWebhook,而不是Initializers。因此我觉得社区其实是想用MutatingAdmissionWebhook替代Initializers。

总结

到这里,我就解开了我的疑惑(为什么有了Initializers,还会有MutatingAdmissionWebhook),截止Kubernetes 1.9发布而言,我推荐在Admission Control这一层:

  • 使用MutatingAdmissionWebhook来修改object(比如注入sidecar),而不使用Initializers,因为它的实现方式导致性能低下及其他重大缺陷。
  • 使用ValidatingAdmissionWebhook来validate request。
展开阅读全文
打赏
2
3 收藏
分享
加载中
打赏
3 评论
3 收藏
2
分享
返回顶部
顶部