8.metric-server and HPA
1. 监控架构概述
kubernetes监控指标大体可以分为两类:核心监控指标和自定义指标,核心监控指标是kubernetes内置稳定可靠监控指标,早期由heapster完成,现由metric-server实现;自定义指标用于实现核心指标的扩展,能够提供更丰富的指标支持,如应用状态指标,自定义指标需要通过Aggregator和k8s api集成,当前主流通过promethues实现。
监控指标用途:
- kubectl top 查看node和pod的cpu+内存使用情况
- kubernetes-dashbaord 控制台查看节点和pod资源监控
- Horizontal Pod Autoscaler 水平横向动态扩展
- Scheduler 调度器调度选择条件
2. metric-server架构和安装
2. 1 metric-server简介
Metrics Server is a cluster-wide aggregator of resource usage data. Resource metrics are used by components like kubectl top and the Horizontal Pod Autoscaler to scale workloads. To autoscale based upon a custom metric, you need to use the Prometheus Adapter Metric-server是一个集群级别的资源指标收集器,用于收集资源指标数据
- 提供基础资源如CPU、内存监控接口查询;
- 接口通过 Kubernetes aggregator注册到kube-apiserver中;
- 对外通过Metric API暴露给外部访问;
- 自定义指标使用需要借助Prometheus实现。
The Metrics API
- /node 获取所有节点的指标,指标名称为NodeMetrics
- /node/
特定节点指标 - /namespaces/{namespace}/pods 获取命名空间下的所有pod指标
- /namespaces/{namespace}/pods/{pod} 特定pod的指标,指标名称为PodMetrics
未来将能够支持指标聚合,如max最大值,min最小值,95th峰值,以及自定义时间窗口,如1h,1d,1w等。
2.2 metric-server架构
监控架构分两部分内容:核心监控(图白色部分)和自定义监控(图蓝色部分)
1、 核心监控实现
- 通过kubelet收集资源估算+使用估算
- metric-server负责数据收集,不负责数据存储
- metric-server对外暴露Metric API接口
- 核心监控指标客用户HPA,kubectl top,scheduler和dashboard
2、 自定义监控实现
- 自定义监控指标包括监控指标和服务指标
- 需要在每个node上部署一个agent上报至集群监控agent,如prometheus
- 集群监控agent收集数据后需要将监控指标+服务指标通过API adaptor转换为apiserver能够处理的接口
- HPA通过自定义指标实现更丰富的弹性扩展能力,需要通过HPA adaptor API做次转换。
2.3 metric-server部署
1、获取metric-server安装脚本
[root@node-1 ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml
2、修改metrics镜像
vi components.yaml
old:
image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
new:
image: registry.cn-shenzhen.aliyuncs.com/cookcodeblog/metrics-server:v0.3.7
kubectl apply -f components.yaml
3、创建metrics-server,检查metric-server部署的情况,查看metric-server的Pod已部署成功
kubectl apply -f components.yaml
metrics-server git:(master) kubectl get deployments metrics-server -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
metrics-server 1/1 1 1 9h
3. HPA水平横向动态扩展
3.1 HPA概述
The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics). Note that Horizontal Pod Autoscaling does not apply to objects that can’t be scaled, for example, DaemonSets.
HPA即Horizontal Pod Autoscaler,Pod水平横向动态扩展,即根据应用分配资源使用情况,动态增加或者减少Pod副本数量,以实现集群资源的扩容,其实现机制为:
- HPA需要依赖于监控组件,调用监控数据实现动态伸缩,如调用Metrics API接口
- HPA是二级的副本控制器,建立在Deployments,ReplicaSet,StatefulSets等副本控制器基础之上
- HPA根据获取资源指标不同支持两个版本:v1和v2alpha1
- HPA V1获取核心资源指标,如CPU和内存利用率,通过调用Metric-server API接口实现
- HPA V2获取自定义监控指标,通过Prometheus获取监控数据实现
- HPA根据资源API周期性调整副本数,检测周期horizontal-pod-autoscaler-sync-period定义的值,默认15s
3.2 HPA实现
如下开始延时HPA功能的实现,先创建一个Deployment副本控制器,然后再通过HPA定义资源度量策略,当CPU利用率超过requests分配的80%时即扩容。
1、创建Deployment副本控制器,其中resources部分可以确保获取资源限制。
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-apache
spec:
selector:
matchLabels:
run: php-apache
replicas: 1
template:
metadata:
labels:
run: php-apache
spec:
containers:
- name: php-apache
image: registry.cn-shanghai.aliyuncs.com/k8sgcrio_containers/hpa-example
ports:
- containerPort: 80
resources:
limits:
cpu: 500m
requests:
cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
name: php-apache
labels:
run: php-apache
spec:
ports:
- port: 80
selector:
run: php-apache
2、创建HPA控制器,基于CPU实现横向扩展,策略为至少2个Pod,最大5个,targetCPUUtilizationPercentage表示CPU实际使用率占requests百分比
kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
3、应用HPA规则并查看详情,由于策略需确保最小2个副本,Deployment默认不是2个副本,因此需要扩容,在详情日志中看到副本扩展至2个
➜ hpa git:(master) ✗ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0%/50% 1 10 1 22m
4、查看Deployment列表校验确认扩容情况,已达到HPA基础最小化策略
➜ hpa git:(master) ✗ kubectl get deployments php-apache --show-labels
NAME READY UP-TO-DATE AVAILABLE AGE LABELS
php-apache 1/1 1 1 24m <none>
NAME READY STATUS RESTARTS AGE
pod/nfs-client-provisioner-69bbdb7f8b-b6bbh 1/1 Running 0 89m
pod/php-apache-6b68bb854c-bh4hv 1/1 Running 0 24m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 30h
service/php-apache ClusterIP 10.99.85.122 <none> 80/TCP 24m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nfs-client-provisioner 1/1 1 1 5h53m
deployment.apps/php-apache 1/1 1 1 24m
NAME DESIRED CURRENT READY AGE
replicaset.apps/nfs-client-provisioner-69bbdb7f8b 1 1 1 89m
replicaset.apps/nfs-client-provisioner-6b558dcbb8 0 0 0 5h53m
replicaset.apps/php-apache-6b68bb854c 1 1 1 24m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/php-apache Deployment/php-apache 0%/50% 1 10 1 24m
5、我们利用siege来模拟压力测试,CPU利用率增高,会自动横向增加Pod来实现,下面开始通过CPU压测来演示Deployment的扩展
sudo apt install siege
siege -q -c 5 -t 1m "http://10.99.85.122:80"
ip 10.99.85.122来自“service/php-apache ClusterIP 10.99.85.122”
再次查看HPA的日志,提示已扩容,原因是cpu resource utilization (percentage of request) above target,即CPU资源利用率超过requests设置的百分比
➜ hpa git:(master) ✗ kubectl describe horizontalpodautoscalers.autoscaling php-apache
Name: php-apache
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Sat, 05 Dec 2020 16:35:49 +0800
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 0% (1m) / 50%
Min replicas: 1
Max replicas: 10
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooFewReplicas the desired replica count is less than the minimum replica count
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 24m (x5 over 25m) horizontal-pod-autoscaler did not receive metrics for any ready pods
Warning FailedComputeMetricsReplicas 24m (x5 over 25m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods
Normal SuccessfulRescale 22m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 21m horizontal-pod-autoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 20m horizontal-pod-autoscaler New size: 10; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 15m horizontal-pod-autoscaler New size: 1; reason: All metrics below target
查看副本的个数,确认扩容情况,已成功扩容至10个
6、停止CPU压测业务,HPA会自定缩减Pod的副本个数,直至满足条件
➜ hpa git:(master) ✗ kubectl describe horizontalpodautoscalers.autoscaling php-apache
Name: php-apache
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Sat, 05 Dec 2020 16:35:49 +0800
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 0% (1m) / 50%
Min replicas: 1
Max replicas: 10
Deployment pods: 10 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 1
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooFewReplicas the desired replica count is less than the minimum replica count
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 33m (x5 over 34m) horizontal-pod-autoscaler did not receive metrics for any ready pods
Warning FailedComputeMetricsReplicas 33m (x5 over 34m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods
Normal SuccessfulRescale 31m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 30m horizontal-pod-autoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 8m3s horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 7m3s horizontal-pod-autoscaler New size: 6; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 6m48s (x2 over 29m) horizontal-pod-autoscaler New size: 10; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 16s (x2 over 24m) horizontal-pod-autoscaler New size: 1; reason: All metrics below target
确认副本的个数,已缩减至最小数量1个
[root@node-1 ~]# kubectl get pods -l run=php-apache
NAME READY STATUS RESTARTS AGE
pod/php-apache-6b68bb854c-bh4hv 1/1 Running 0 36m
通过上面的例子可以知道,HPA可以基于metric-server提供的API监控数据实现水平动态弹性扩展的需求,从而可以根据业务CPU使用情况,动态水平横向扩展,保障业务的可用性。当前HPA V1扩展使用指标只能基于CPU分配使用率进行扩展,功能相对有限,更丰富的功能需要由HPA V2版来实现,其由不同的API来实现:
- metrics.k8s.io 资源指标API,通过metric-server提供,提供node和pod的cpu,内存资源查询;
- custom.metrics.k8s.io 自定义指标,通过adapter和kube-apiserver集成,如promethues;
- external.metrics.k8s.io 外部指标,和自定义指标类似,需要通过adapter和k8s集成。