8.metric-server and HPA

阅读大约需要 5 分钟

1. 监控架构概述

kubernetes监控指标大体可以分为两类：核心监控指标和自定义指标，核心监控指标是kubernetes内置稳定可靠监控指标，早期由heapster完成，现由metric-server实现；自定义指标用于实现核心指标的扩展，能够提供更丰富的指标支持，如应用状态指标，自定义指标需要通过Aggregator和k8s api集成，当前主流通过promethues实现。

监控指标用途：

kubectl top 查看node和pod的cpu+内存使用情况
kubernetes-dashbaord 控制台查看节点和pod资源监控
Horizontal Pod Autoscaler 水平横向动态扩展
Scheduler 调度器调度选择条件

2. metric-server架构和安装

2. 1 metric-server简介

Metrics Server is a cluster-wide aggregator of resource usage data. Resource metrics are used by components like kubectl top and the Horizontal Pod Autoscaler to scale workloads. To autoscale based upon a custom metric, you need to use the Prometheus Adapter Metric-server是一个集群级别的资源指标收集器，用于收集资源指标数据

提供基础资源如CPU、内存监控接口查询；
接口通过 Kubernetes aggregator注册到kube-apiserver中；
对外通过Metric API暴露给外部访问；
自定义指标使用需要借助Prometheus实现。

The Metrics API

/node 获取所有节点的指标，指标名称为NodeMetrics
/node/ 特定节点指标
/namespaces/{namespace}/pods 获取命名空间下的所有pod指标
/namespaces/{namespace}/pods/{pod} 特定pod的指标，指标名称为PodMetrics

未来将能够支持指标聚合，如max最大值，min最小值，95th峰值，以及自定义时间窗口，如1h，1d，1w等。

2.2 metric-server架构

监控架构分两部分内容：核心监控(图白色部分)和自定义监控（图蓝色部分）

1、核心监控实现

通过kubelet收集资源估算+使用估算
metric-server负责数据收集，不负责数据存储
metric-server对外暴露Metric API接口
核心监控指标客用户HPA，kubectl top，scheduler和dashboard

2、自定义监控实现

自定义监控指标包括监控指标和服务指标
需要在每个node上部署一个agent上报至集群监控agent，如prometheus
集群监控agent收集数据后需要将监控指标+服务指标通过API adaptor转换为apiserver能够处理的接口
HPA通过自定义指标实现更丰富的弹性扩展能力，需要通过HPA adaptor API做次转换。

2.3 metric-server部署

1、获取metric-server安装脚本

[root@node-1 ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

2、修改metrics镜像

vi components.yaml
old:
 image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
new:
 image: registry.cn-shenzhen.aliyuncs.com/cookcodeblog/metrics-server:v0.3.7
  kubectl apply -f components.yaml

3、创建metrics-server，检查metric-server部署的情况,查看metric-server的Pod已部署成功

 kubectl apply -f components.yaml 

metrics-server git:(master) kubectl get deployments metrics-server -n kube-system  
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           9h

3. HPA水平横向动态扩展

3.1 HPA概述

The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics). Note that Horizontal Pod Autoscaling does not apply to objects that can’t be scaled, for example, DaemonSets.

HPA即Horizontal Pod Autoscaler,Pod水平横向动态扩展，即根据应用分配资源使用情况，动态增加或者减少Pod副本数量，以实现集群资源的扩容，其实现机制为：

HPA需要依赖于监控组件，调用监控数据实现动态伸缩，如调用Metrics API接口
HPA是二级的副本控制器，建立在Deployments，ReplicaSet，StatefulSets等副本控制器基础之上
HPA根据获取资源指标不同支持两个版本：v1和v2alpha1
HPA V1获取核心资源指标，如CPU和内存利用率，通过调用Metric-server API接口实现
HPA V2获取自定义监控指标，通过Prometheus获取监控数据实现
HPA根据资源API周期性调整副本数，检测周期horizontal-pod-autoscaler-sync-period定义的值，默认15s

3.2 HPA实现

如下开始延时HPA功能的实现，先创建一个Deployment副本控制器，然后再通过HPA定义资源度量策略，当CPU利用率超过requests分配的80%时即扩容。

1、创建Deployment副本控制器,其中resources部分可以确保获取资源限制。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
spec:
  selector:
    matchLabels:
      run: php-apache
  replicas: 1
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - name: php-apache
        image: registry.cn-shanghai.aliyuncs.com/k8sgcrio_containers/hpa-example
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 500m
          requests:
            cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
  name: php-apache
  labels:
    run: php-apache
spec:
  ports:
  - port: 80
  selector:
    run: php-apache

2、创建HPA控制器，基于CPU实现横向扩展,策略为至少2个Pod，最大5个，targetCPUUtilizationPercentage表示CPU实际使用率占requests百分比

kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10

3、应用HPA规则并查看详情，由于策略需确保最小2个副本，Deployment默认不是2个副本，因此需要扩容，在详情日志中看到副本扩展至2个

➜  hpa git:(master) ✗ kubectl get hpa
NAME         REFERENCE               TARGETS                        MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%                         1         10        1          22m

4、查看Deployment列表校验确认扩容情况，已达到HPA基础最小化策略

➜  hpa git:(master) ✗ kubectl get deployments php-apache  --show-labels  
NAME         READY   UP-TO-DATE   AVAILABLE   AGE   LABELS
php-apache   1/1     1            1           24m   <none>

NAME                                          READY   STATUS    RESTARTS   AGE
pod/nfs-client-provisioner-69bbdb7f8b-b6bbh   1/1     Running   0          89m
pod/php-apache-6b68bb854c-bh4hv               1/1     Running   0          24m

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   30h
service/php-apache   ClusterIP   10.99.85.122    <none>        80/TCP    24m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nfs-client-provisioner   1/1     1            1           5h53m
deployment.apps/php-apache               1/1     1            1           24m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/nfs-client-provisioner-69bbdb7f8b   1         1         1       89m
replicaset.apps/nfs-client-provisioner-6b558dcbb8   0         0         0       5h53m
replicaset.apps/php-apache-6b68bb854c               1         1         1       24m

NAME                                             REFERENCE               TARGETS                        MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/php-apache   Deployment/php-apache   0%/50%                         1         10        1          24m

5、我们利用siege来模拟压力测试，CPU利用率增高，会自动横向增加Pod来实现，下面开始通过CPU压测来演示Deployment的扩展

sudo apt install siege
siege -q -c 5 -t 1m "http://10.99.85.122:80" 

ip 10.99.85.122来自“service/php-apache   ClusterIP   10.99.85.122”
再次查看HPA的日志，提示已扩容，原因是cpu resource utilization (percentage of request) above target，即CPU资源利用率超过requests设置的百分比
➜  hpa git:(master) ✗ kubectl describe horizontalpodautoscalers.autoscaling  php-apache
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 05 Dec 2020 16:35:49 +0800
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (1m) / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 1 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  recommended size matches current size
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count is less than the minimum replica count
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetResourceMetric       24m (x5 over 25m)  horizontal-pod-autoscaler  did not receive metrics for any ready pods
  Warning  FailedComputeMetricsReplicas  24m (x5 over 25m)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods
  Normal   SuccessfulRescale             22m                horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             21m                horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             20m                horizontal-pod-autoscaler  New size: 10; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             15m                horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

查看副本的个数，确认扩容情况，已成功扩容至10个

6、停止CPU压测业务，HPA会自定缩减Pod的副本个数，直至满足条件

➜  hpa git:(master) ✗ kubectl describe horizontalpodautoscalers.autoscaling  php-apache
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 05 Dec 2020 16:35:49 +0800
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (1m) / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       10 current / 1 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    SucceededRescale  the HPA controller was able to update the target scale to 1
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count is less than the minimum replica count
Events:
  Type     Reason                        Age                  From                       Message
  ----     ------                        ----                 ----                       -------
  Warning  FailedGetResourceMetric       33m (x5 over 34m)    horizontal-pod-autoscaler  did not receive metrics for any ready pods
  Warning  FailedComputeMetricsReplicas  33m (x5 over 34m)    horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods
  Normal   SuccessfulRescale             31m                  horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             30m                  horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             8m3s                 horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             7m3s                 horizontal-pod-autoscaler  New size: 6; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             6m48s (x2 over 29m)  horizontal-pod-autoscaler  New size: 10; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             16s (x2 over 24m)    horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

确认副本的个数，已缩减至最小数量1个

[root@node-1 ~]# kubectl get pods -l run=php-apache
NAME                        READY   STATUS    RESTARTS   AGE
pod/php-apache-6b68bb854c-bh4hv               1/1     Running   0          36m

通过上面的例子可以知道，HPA可以基于metric-server提供的API监控数据实现水平动态弹性扩展的需求，从而可以根据业务CPU使用情况，动态水平横向扩展，保障业务的可用性。当前HPA V1扩展使用指标只能基于CPU分配使用率进行扩展，功能相对有限，更丰富的功能需要由HPA V2版来实现，其由不同的API来实现：

metrics.k8s.io 资源指标API，通过metric-server提供，提供node和pod的cpu，内存资源查询；
custom.metrics.k8s.io 自定义指标，通过adapter和kube-apiserver集成，如promethues；
external.metrics.k8s.io 外部指标，和自定义指标类似，需要通过adapter和k8s集成。

这些信息有用吗？

Do you have any suggestions for improvement?

Thanks for your feedback!