文档章节

深入分析Kubernetes Critical Pod(二)

WaltonWang
 WaltonWang
发布于 2018/07/12 23:54
字数 1149
阅读 117
收藏 0

深入分析Kubernetes Critical Pod(一)介绍了Scheduler对Critical Pod的处理逻辑,下面我们再看下Kubelet Eviction Manager对Critical Pod的处理逻辑是怎样的,以便我们了解Kubelet Evict Pod时对Critical Pod是否有保护措施,如果有,又是如何保护的。

Kubelet Eviction Manager Admit

kubelet在syncLoop中每个1s会循环调用syncLoopIteration,从config change channel | pleg channel | sync channel | houseKeeping channel | liveness manager's update channel中获取event,然后分别调用对应的event handler进行处理。

  • configCh: dispatch the pods for the config change to the appropriate handler callback for the event type
  • plegCh: update the runtime cache; sync pod
  • syncCh: sync all pods waiting for sync
  • houseKeepingCh: trigger cleanup of pods
  • liveness manager's update channel: sync pods that have failed or in which one or more containers have failed liveness checks

特别提一下,houseKeeping channel是每隔houseKeeping(10s)时间就会有event,然后执行HandlePodCleanups,执行以下清理操作:

  • Stop the workers for no-longer existing pods.(每个pod对应会有一个worker,也就是goruntine)
  • killing unwanted pods
  • removes the volumes of pods that should not be running and that have no containers running.
  • Remove any orphaned mirror pods.
  • Remove any cgroups in the hierarchy for pods that are no longer running.
pkg/kubelet/kubelet.go:1753

func (kl *Kubelet) syncLoopIteration(configCh <-chan kubetypes.PodUpdate, handler SyncHandler,
	syncCh <-chan time.Time, housekeepingCh <-chan time.Time, plegCh <-chan *pleg.PodLifecycleEvent) bool {
	select {
	case u, open := <-configCh:
		
		if !open {
			glog.Errorf("Update channel is closed. Exiting the sync loop.")
			return false
		}

		switch u.Op {
		case kubetypes.ADD:
			
			handler.HandlePodAdditions(u.Pods)
		...
		case kubetypes.RESTORE:
			glog.V(2).Infof("SyncLoop (RESTORE, %q): %q", u.Source, format.Pods(u.Pods))
			// These are pods restored from the checkpoint. Treat them as new
			// pods.
			handler.HandlePodAdditions(u.Pods)
		...
		}

		if u.Op != kubetypes.RESTORE {
			...
		}
	case e := <-plegCh:
		...
	case <-syncCh:
		...
	case update := <-kl.livenessManager.Updates():
		...
	case <-housekeepingCh:
		...
	}
	return true
}

syncLoopIteration中定义了当kubelet配置变更重启后的逻辑:kubelet会对正在running的Pods进行Admission处理,Admission的结果有可能会让该Pod被本节点拒绝。

HandlePodAdditions就是用来处理Kubelet ConficCh中的event的Handler。

// HandlePodAdditions is the callback in SyncHandler for pods being added from a config source.
func (kl *Kubelet) HandlePodAdditions(pods []*v1.Pod) {
	start := kl.clock.Now()
	sort.Sort(sliceutils.PodsByCreationTime(pods))
	for _, pod := range pods {
		...

		if !kl.podIsTerminated(pod) {
			...
			// Check if we can admit the pod; if not, reject it.
			if ok, reason, message := kl.canAdmitPod(activePods, pod); !ok {
				kl.rejectPod(pod, reason, message)
				continue
			}
		}
		...
	}
}

如果该Pod Status不是属于Terminated,就调用canAdmitPod对该Pod进行准入检查。如果准入检查结果表示该Pod被拒绝,那么就会将该Pod Phase设置为Failed。

pkg/kubelet/kubelet.go:1643

func (kl *Kubelet) canAdmitPod(pods []*v1.Pod, pod *v1.Pod) (bool, string, string) {
	// the kubelet will invoke each pod admit handler in sequence
	// if any handler rejects, the pod is rejected.
	// TODO: move out of disk check into a pod admitter
	// TODO: out of resource eviction should have a pod admitter call-out
	attrs := &lifecycle.PodAdmitAttributes{Pod: pod, OtherPods: pods}
	for _, podAdmitHandler := range kl.admitHandlers {
		if result := podAdmitHandler.Admit(attrs); !result.Admit {
			return false, result.Reason, result.Message
		}
	}

	return true, "", ""
}

canAdmitPod就会调用kubelet启动时注册的一系列admitHandlers对该Pod进行准入检查,其中就包括kubelet eviction manager对应的admitHandle。

pkg/kubelet/eviction/eviction_manager.go:123

// Admit rejects a pod if its not safe to admit for node stability.
func (m *managerImpl) Admit(attrs *lifecycle.PodAdmitAttributes) lifecycle.PodAdmitResult {
	m.RLock()
	defer m.RUnlock()
	if len(m.nodeConditions) == 0 {
		return lifecycle.PodAdmitResult{Admit: true}
	}
	
	if utilfeature.DefaultFeatureGate.Enabled(features.ExperimentalCriticalPodAnnotation) && kubelettypes.IsCriticalPod(attrs.Pod) {
		return lifecycle.PodAdmitResult{Admit: true}
	}

	if hasNodeCondition(m.nodeConditions, v1.NodeMemoryPressure) {
		notBestEffort := v1.PodQOSBestEffort != v1qos.GetPodQOS(attrs.Pod)
		if notBestEffort {
			return lifecycle.PodAdmitResult{Admit: true}
		}
	}

		return lifecycle.PodAdmitResult{
		Admit:   false,
		Reason:  reason,
		Message: fmt.Sprintf(message, m.nodeConditions),
	}
}

eviction manager的Admit的逻辑如下:

  • 如果该node的Conditions为空,则Admit成功;
  • 如果enable了ExperimentalCriticalPodAnnotation Feature Gate,并且该Pod是Critical Pod(Pod有Critical的Annotation,或者Pod的优先级不小于SystemCriticalPriority),则Admit成功;
    • SystemCriticalPriority的值为2 billion。
  • 如果该node的Condition为Memory Pressure,并且Pod QoS为非best-effort,则Admit成功;
  • 其他情况都表示Admit失败,即不允许该Pod在该node上Running。

Kubelet Eviction Manager SyncLoop

另外,在kubelet eviction manager的syncLoop中,也会对Critical Pod有特殊处理,代码如下。

pkg/kubelet/eviction/eviction_manager.go:226

// synchronize is the main control loop that enforces eviction thresholds.
// Returns the pod that was killed, or nil if no pod was killed.
func (m *managerImpl) synchronize(diskInfoProvider DiskInfoProvider, podFunc ActivePodsFunc) []*v1.Pod {
	...

	// we kill at most a single pod during each eviction interval
	for i := range activePods {
		pod := activePods[i]
		
		if utilfeature.DefaultFeatureGate.Enabled(features.ExperimentalCriticalPodAnnotation) &&
			kubelettypes.IsCriticalPod(pod) && kubepod.IsStaticPod(pod) {
			continue
		}
		...
		return []*v1.Pod{pod}
	}
	glog.Infof("eviction manager: unable to evict any pods from the node")
	return nil
}

当触发了kubelet evict pod时,如果该pod满足以下所有条件时,将不会被kubelet eviction manager kill掉。

  • 该Pod Status不是Terminated;
  • Enable ExperimentalCriticalPodAnnotation Feature Gate;
  • 该Pod是Critical Pod;
  • 该Pod时Static Pod;

总结

经过上面的分析,我们得到以下Kubelet Eviction Manager对Critical Pod处理的关键点:

  • kubelet重启后,eviction manager的Admit流程中对Critical Pod做如下特殊处理:如果enable了ExperimentalCriticalPodAnnotation Feature Gate,则允许该Critical Pod准入该node,无视该node的Condition。

  • 当触发了kubelet evict pod时,如果该Critical Pod满足以下所有条件时,将不会被kubelet eviction manager kill掉。

    • 该Pod Status不是Terminated;
    • Enable ExperimentalCriticalPodAnnotation Feature Gate;
    • 该Pod是Static Pod;

© 著作权归作者所有

共有 人打赏支持
WaltonWang
粉丝 210
博文 104
码字总数 220998
作品 0
深圳
程序员
私信 提问
深入分析Kubernetes Critical Pod(三)

本文介绍了Kubelet在Predicate Admit准入检查时对CriticalPod的资源抢占的原理,以及Priority Admission Controller对CriticalPod的PriorityClassName特殊处理。 深入分析Kubernetes Critic...

WaltonWang
2018/07/12
0
0
深入分析Kubernetes Critical Pod(四)

摘要:本文分析了DeamonSetController及PriorityClass Validate时,对CriticalPod的所做的特殊处理。 Daemonset Controller对CriticalPod的特殊处理 深入分析Kubernetes Critical Pod系列:深...

WaltonWang
2018/07/12
0
0
深入分析Kubernetes Critical Pod(一)

大家在Kubernetes集群中部署核心组件时,经常会用到Critical Pod,那么你知道Critical Pod到底有何特别吗?要完整的了解这一点,其实并不是那么简单,它关系到调度、Kubelet Eviction Manage...

WaltonWang
2018/07/12
0
0
深入分析Kubernetes DaemonSet Controller

Author: xidianwangtao@gmail.com Version: Kubernetes 1.13 摘要:DaemonSet是Kubernetes中用户最常用的对象之一,我们用它来部署Nodes上守护应用,比如日志组件、节点监控组件等。从用户的...

WaltonWang
03/10
0
0
深入分析Kubelet Bootstrap Checkpoint

Author: xidianwangtao@gmail.com , Version: Kubernetes 1.12 摘要:本文对Kubelet Bootstrap Checkpoint的使用方法、应用场景、工作机制及其代码工作流程进行了全面分析,目前仍处于Alpha...

WaltonWang
2018/11/24
0
0

没有更多内容

加载失败,请刷新页面

加载更多

js中原型模式

原型模式就是用克隆对象来创建对象的,在es6中提供了Object.create方法来克隆对象 在不支持该方法的浏览器中,可以使用以下代码: Object.create = Object.create || function (obj) { ...

莫西摩西
8分钟前
0
0
ORACLE插入insert带where条件

ORACLE插入带条件 insert into THIRD_PAYEE_ACCOUNT_DETAIL t (id, collection_mode, t.store_code) select Third_Payee_Account_Detail_Seq.Nextval,'006','A05K' from dual where not......

Cobbage
17分钟前
0
0
分享几张Java架构大牛总结的架构知识脑图

前言 系统架构师是一个既需要掌控整体又需要洞悉局部瓶颈并依据具体的业务场景给出解决方案的团队领导型人物。一个架构师得需要足够的想像力,能把各种目标需求进行不同维度的扩展,为目标客户...

Java-飞鱼
25分钟前
0
0
2019BATJ面试题详解:MyBatis+MySQL+Spring+Redis+多线程

这里为大家分享一些面试的一手资料,供大家迎接接下来的金三银四跳槽季 Spring Spring 概述 什么是spring? 使用Spring框架的好处是什么? Spring由哪些模块组成? 解释AOP模块 Spring配置文件...

别打我会飞
27分钟前
0
0
首场百度大脑开放日来袭 | 全新开放24项AI技术

活动当天,百度AI技术生态部总经理喻友平,就百度大脑平台与生态进行了全面的详解,同时展示了百度大脑开放平台Q1核心升级内容,包括语音技术、视觉技术、自然语言处理、知识图谱等通用AI能力...

PaddleWeekly
28分钟前
0
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部