8.metric-server and HPA

1. 监控架构概述

kubernetes监控指标大体可以分为两类:核心监控指标和自定义指标,核心监控指标是kubernetes内置稳定可靠监控指标,早期由heapster完成,现由metric-server实现;自定义指标用于实现核心指标的扩展,能够提供更丰富的指标支持,如应用状态指标,自定义指标需要通过Aggregator和k8s api集成,当前主流通过promethues实现。

监控指标用途:

  • kubectl top 查看node和pod的cpu+内存使用情况
  • kubernetes-dashbaord 控制台查看节点和pod资源监控
  • Horizontal Pod Autoscaler 水平横向动态扩展
  • Scheduler 调度器调度选择条件

2. metric-server架构和安装

2. 1 metric-server简介

Metrics Server is a cluster-wide aggregator of resource usage data. Resource metrics are used by components like kubectl top and the Horizontal Pod Autoscaler to scale workloads. To autoscale based upon a custom metric, you need to use the Prometheus Adapter Metric-server是一个集群级别的资源指标收集器,用于收集资源指标数据

  • 提供基础资源如CPU、内存监控接口查询;
  • 接口通过 Kubernetes aggregator注册到kube-apiserver中;
  • 对外通过Metric API暴露给外部访问;
  • 自定义指标使用需要借助Prometheus实现。

The Metrics API

  • /node 获取所有节点的指标,指标名称为NodeMetrics
  • /node/ 特定节点指标
  • /namespaces/{namespace}/pods 获取命名空间下的所有pod指标
  • /namespaces/{namespace}/pods/{pod} 特定pod的指标,指标名称为PodMetrics

未来将能够支持指标聚合,如max最大值,min最小值,95th峰值,以及自定义时间窗口,如1h,1d,1w等。

2.2 metric-server架构

img

监控架构分两部分内容:核心监控(图白色部分)和自定义监控(图蓝色部分)

1、 核心监控实现

  • 通过kubelet收集资源估算+使用估算
  • metric-server负责数据收集,不负责数据存储
  • metric-server对外暴露Metric API接口
  • 核心监控指标客用户HPA,kubectl top,scheduler和dashboard

2、 自定义监控实现

  • 自定义监控指标包括监控指标和服务指标
  • 需要在每个node上部署一个agent上报至集群监控agent,如prometheus
  • 集群监控agent收集数据后需要将监控指标+服务指标通过API adaptor转换为apiserver能够处理的接口
  • HPA通过自定义指标实现更丰富的弹性扩展能力,需要通过HPA adaptor API做次转换。

2.3 metric-server部署

1、获取metric-server安装脚本

[root@node-1 ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

2、修改metrics镜像

vi components.yaml
old:
 image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
new:
 image: registry.cn-shenzhen.aliyuncs.com/cookcodeblog/metrics-server:v0.3.7
  kubectl apply -f components.yaml

3、创建metrics-server,检查metric-server部署的情况,查看metric-server的Pod已部署成功

 kubectl apply -f components.yaml 

metrics-server git:(master) kubectl get deployments metrics-server -n kube-system  
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           9h

3. HPA水平横向动态扩展

3.1 HPA概述

The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics). Note that Horizontal Pod Autoscaling does not apply to objects that can’t be scaled, for example, DaemonSets.

HPA即Horizontal Pod Autoscaler,Pod水平横向动态扩展,即根据应用分配资源使用情况,动态增加或者减少Pod副本数量,以实现集群资源的扩容,其实现机制为:

img

  • HPA需要依赖于监控组件,调用监控数据实现动态伸缩,如调用Metrics API接口
  • HPA是二级的副本控制器,建立在Deployments,ReplicaSet,StatefulSets等副本控制器基础之上
  • HPA根据获取资源指标不同支持两个版本:v1和v2alpha1
  • HPA V1获取核心资源指标,如CPU和内存利用率,通过调用Metric-server API接口实现
  • HPA V2获取自定义监控指标,通过Prometheus获取监控数据实现
  • HPA根据资源API周期性调整副本数,检测周期horizontal-pod-autoscaler-sync-period定义的值,默认15s

3.2 HPA实现

如下开始延时HPA功能的实现,先创建一个Deployment副本控制器,然后再通过HPA定义资源度量策略,当CPU利用率超过requests分配的80%时即扩容。

1、创建Deployment副本控制器,其中resources部分可以确保获取资源限制。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
spec:
  selector:
    matchLabels:
      run: php-apache
  replicas: 1
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - name: php-apache
        image: registry.cn-shanghai.aliyuncs.com/k8sgcrio_containers/hpa-example
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 500m
          requests:
            cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
  name: php-apache
  labels:
    run: php-apache
spec:
  ports:
  - port: 80
  selector:
    run: php-apache

2、创建HPA控制器,基于CPU实现横向扩展,策略为至少2个Pod,最大5个,targetCPUUtilizationPercentage表示CPU实际使用率占requests百分比

kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10

3、应用HPA规则并查看详情,由于策略需确保最小2个副本,Deployment默认不是2个副本,因此需要扩容,在详情日志中看到副本扩展至2个

  hpa git:(master)  kubectl get hpa
NAME         REFERENCE               TARGETS                        MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%                         1         10        1          22m

4、查看Deployment列表校验确认扩容情况,已达到HPA基础最小化策略

  hpa git:(master)  kubectl get deployments php-apache  --show-labels  
NAME         READY   UP-TO-DATE   AVAILABLE   AGE   LABELS
php-apache   1/1     1            1           24m   <none>

NAME                                          READY   STATUS    RESTARTS   AGE
pod/nfs-client-provisioner-69bbdb7f8b-b6bbh   1/1     Running   0          89m
pod/php-apache-6b68bb854c-bh4hv               1/1     Running   0          24m

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   30h
service/php-apache   ClusterIP   10.99.85.122    <none>        80/TCP    24m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nfs-client-provisioner   1/1     1            1           5h53m
deployment.apps/php-apache               1/1     1            1           24m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/nfs-client-provisioner-69bbdb7f8b   1         1         1       89m
replicaset.apps/nfs-client-provisioner-6b558dcbb8   0         0         0       5h53m
replicaset.apps/php-apache-6b68bb854c               1         1         1       24m

NAME                                             REFERENCE               TARGETS                        MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/php-apache   Deployment/php-apache   0%/50%                         1         10        1          24m

5、我们利用siege来模拟压力测试,CPU利用率增高,会自动横向增加Pod来实现,下面开始通过CPU压测来演示Deployment的扩展

sudo apt install siege
siege -q -c 5 -t 1m "http://10.99.85.122:80" 

ip 10.99.85.122来自service/php-apache   ClusterIP   10.99.85.122
再次查看HPA的日志提示已扩容原因是cpu resource utilization (percentage of request) above target即CPU资源利用率超过requests设置的百分比
  hpa git:(master)  kubectl describe horizontalpodautoscalers.autoscaling  php-apache
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 05 Dec 2020 16:35:49 +0800
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (1m) / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 1 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  recommended size matches current size
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count is less than the minimum replica count
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetResourceMetric       24m (x5 over 25m)  horizontal-pod-autoscaler  did not receive metrics for any ready pods
  Warning  FailedComputeMetricsReplicas  24m (x5 over 25m)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods
  Normal   SuccessfulRescale             22m                horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             21m                horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             20m                horizontal-pod-autoscaler  New size: 10; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             15m                horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

查看副本的个数确认扩容情况已成功扩容至10个

image-20201205170644044

image-20201205170349581

6、停止CPU压测业务,HPA会自定缩减Pod的副本个数,直至满足条件

  hpa git:(master)  kubectl describe horizontalpodautoscalers.autoscaling  php-apache
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 05 Dec 2020 16:35:49 +0800
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (1m) / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       10 current / 1 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    SucceededRescale  the HPA controller was able to update the target scale to 1
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count is less than the minimum replica count
Events:
  Type     Reason                        Age                  From                       Message
  ----     ------                        ----                 ----                       -------
  Warning  FailedGetResourceMetric       33m (x5 over 34m)    horizontal-pod-autoscaler  did not receive metrics for any ready pods
  Warning  FailedComputeMetricsReplicas  33m (x5 over 34m)    horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods
  Normal   SuccessfulRescale             31m                  horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             30m                  horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             8m3s                 horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             7m3s                 horizontal-pod-autoscaler  New size: 6; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             6m48s (x2 over 29m)  horizontal-pod-autoscaler  New size: 10; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             16s (x2 over 24m)    horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

确认副本的个数,已缩减至最小数量1个

[root@node-1 ~]# kubectl get pods -l run=php-apache
NAME                        READY   STATUS    RESTARTS   AGE
pod/php-apache-6b68bb854c-bh4hv               1/1     Running   0          36m

通过上面的例子可以知道,HPA可以基于metric-server提供的API监控数据实现水平动态弹性扩展的需求,从而可以根据业务CPU使用情况,动态水平横向扩展,保障业务的可用性。当前HPA V1扩展使用指标只能基于CPU分配使用率进行扩展,功能相对有限,更丰富的功能需要由HPA V2版来实现,其由不同的API来实现:

  • metrics.k8s.io 资源指标API,通过metric-server提供,提供node和pod的cpu,内存资源查询;
  • custom.metrics.k8s.io 自定义指标,通过adapter和kube-apiserver集成,如promethues;
  • external.metrics.k8s.io 外部指标,和自定义指标类似,需要通过adapter和k8s集成。
这些信息有用吗?
Do you have any suggestions for improvement?

Thanks for your feedback!