Kubernetes: Autoscaling using Prometheus as a external metrics provider
5 min read | by Jordi Prats
Using an external metrics provider (Kubernetes 1.10+) we can use an HorizontalPodAutoscaler to automatically scale applications using any metric collected by Prometheus. Let's take a look on how to configure it
To test this out we'll need to have prometheus and prometheus-adapter installed, to do so we can use it's community-supported helm chart. First we'll need to add it's repository:
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update Then we'll have to install both charts:
$ helm install prometheus prometheus-community/prometheus --namespace prometheus $ helm install prometheus-adapter prometheus-community/prometheus-adapter --namespace prometheus We are going to use the prometheus-adapter to be able to present prometheus metrics to the cluster in a way it can understand and therefore suitable for autoscaling using an HorizontalPodAutoscaler
Let's suppose we have an application that exposes metrics using /metrics, first we'll need to let prometheus know this endpoint can be scrapped using the following annotations on the Pod template:
apiVersion: apps/v1 kind: Deployment metadata: name: demo-prometheus-enabled-app spec: selector: matchLabels: app: metricsdemo template: metadata: annotations: prometheus.io/path: "/metrics" prometheus.io/scrape: "true" prometheus.io/port: "8000" labels: app: metricsdemo spec: containers: - name: metricsdemo image: "jordiprats/demoappprometrics:1.0" Prometheus once it sees these annotations, it will start scrapping it's metrics. Once we have the metrics available on prometheus, they might not be entirely suitable to use it together with the HorizontalPodAutoscaler. We can transform them using some rules, to do so we'll have to update the prometheus-adapter values to include the rules we need using the rules.custom option.
For example, let's assume the application publishes a metricsdemo_requests that we want to average within a 2 minutes window and then calculate it's rate. To do so we would need to create the following rule:
rules: custom: - seriesQuery: "metricsdemo_requests" resources: overrides: kubernetes_namespace: resource: metricsdemo kubernetes_pod_name: resource: pod name: matches: "^(.*)" as: "\u0024{1}_avg" metricsQuery: "sum(rate(<<.Series>>[2m])) by (<<.GroupBy>>)" Once we push this config to Kubernetes, it will take a while for it to become available. We can check when it's available using the following command:
$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | python -m json.tool { "kind": "APIResourceList", "apiVersion": "v1", "groupVersion": "custom.metrics.k8s.io/v1beta1", "resources": [ { "name": "pods/metricsdemo_requests_avg", "singularName": "", "namespaced": true, "kind": "MetricValueList", "verbs": [ "get" ] }, { "name": "namespaces/metricsdemo_requests_avg", "singularName": "", "namespaced": false, "kind": "MetricValueList", "verbs": [ "get" ] } ] } Once it's available, we can start creating HorizontalPodAutoscaler objects using it, for example:
apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: demo-prometheus-enabled-app labels: {{- include "metricsdemo.labels" . | nindent 4 }} spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: demo-prometheus-enabled-app minReplicas: 3 maxReplicas: 30 metrics: - type: Pods pods: metricName: metricsdemo_requests_avg targetAverageValue: 10 Once we apply it, it will control the number of replicas of the Deployment so we won't be able to use kubectl scale:
$ kubectl describe hpa metricsdemo -n metricsdemo Name: metricsdemo Namespace: metricsdemo (...) Reference: Deployment/metricsdemo Metrics: ( current / target ) "metricsdemo_requests_avg" on pods: 0 / 10 Min replicas: 3 Max replicas: 30 Deployment pods: 3 current / 3 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ReadyForNewScale recommended size matches current size ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric metricsdemo_requests_avg ScalingLimited True TooFewReplicas the desired replica count is less than the minimum replica count Events: <none> As soon as the metrics reaches the configured threshold, the HPA will kicks in changing the number of replicas of the Deployment (please notice the Deployment pods line)
$ kubectl describe hpa -n metricsdemo Name: metricsdemo Namespace: metricsdemo (...) Reference: Deployment/metricsdemo Metrics: ( current / target ) "metricsdemo_requests_avg" on pods: 794791m / 10 Min replicas: 3 Max replicas: 30 Deployment pods: 3 current / 6 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 6 ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric metricsdemo_requests_avg ScalingLimited True ScaleUpLimit the desired replica count is increasing faster than the maximum scale rate (...) $ kubectl describe hpa -n metricsdemo Name: metricsdemo Namespace: metricsdemo (...) Reference: Deployment/metricsdemo Metrics: ( current / target ) "metricsdemo_requests_avg" on pods: 794791m / 10 Min replicas: 3 Max replicas: 30 Deployment pods: 6 current / 6 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale False FailedUpdateScale the HPA controller was unable to update the target scale: Operation cannot be fulfilled on deployments.apps "metricsdemo": the object has been modified; please apply your changes to the latest version and try again ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric metricsdemo_requests_avg ScalingLimited True ScaleUpLimit the desired replica count is increasing faster than the maximum scale rate (...) $ kubectl describe hpa -n metricsdemo Name: metricsdemo Namespace: metricsdemo (...) Reference: Deployment/metricsdemo Metrics: ( current / target ) "metricsdemo_requests_avg" on pods: 0 / 10 Min replicas: 3 Max replicas: 30 Deployment pods: 24 current / 30 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 30 ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric metricsdemo_requests_avg ScalingLimited True TooManyReplicas the desired replica count is more than the maximum replica count Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 11m horizontal-pod-autoscaler New size: 3; reason: Current number of replicas below Spec.MinReplicas (...) We can achieve similar results using datadog as an external metrics provider
Posted on 05/04/2022