Autoscaling using datadog as a external metrics provider

kubernetes hpa datadog external metrics

3 min read | by Jordi Prats

Since Kubernetes v1.2 we can autoscale an application based on metrics like CPU provided by the metrics-server. As of Kubernetes v1.6, it is possible to autoscale off of custom metrics and later on, starting Kubernetes v1.10, we can autoscale using any metric from outside the cluster, like the ones collected by datadog

To be able to configure an HPA object to use datadog's metrics we will have to enable the metrics provider. If we are using it's helm chart to deploy the Cluster Agent we will have to set the following values:

datadog:  appKey: abcd...  apiKey: efgh... clusterAgent:  enabled: true  metricsProvider:  enabled: true

Once the Cluster Agent have been upgraded and the metrics provider is available:

$ kubectl get pods -n datadog NAME READY STATUS RESTARTS AGE datadog-2pc4x 3/3 Running 0 7m13s datadog-cluster-agent-6cc7cf55d6-6946r 1/1 Running 1 120m datadog-kube-state-metrics-699964c777-wrkwz 1/1 Running 0 100m datadog-xk79r 3/3 Running 0 6m49s $ kubectl get apiservice | grep datadog v1beta1.external.metrics.k8s.io datadog/datadog-cluster-agent-metrics-api True 100m

We can now start pushing HPA objects with references to external metrics, for example:

apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata:  name: ampa spec:  scaleTargetRef:  apiVersion: apps/v1  kind: Deployment  name: ampa  minReplicas: 2  maxReplicas: 22  metrics:  - type: External  external:  metricName: kubernetes.kubelet.cpu.usage  metricSelector:  matchLabels:  cluster_name: example  targetAverageValue: 50

The important setting that controls how the HPA behaves are:

  • metricName: Defines the metrics we are going to use to autoscale, it can be any metric that the Cluster Agent retrieves, for example kubernetes_state.deployment.replicas_available or kubernetes.kubelet.cpu.usage
  • metricsSelector.matchLabels: We can specify a set of tags we are going to use to filter the metric
  • targetValue or targetAverageValue: We are setting the desired value of the metric that will be used to scale up and down the objecy defined under spec.scaleTargetRef. If we use targetValue, it's values is the absolute value. For targetAverageValue we are defining the individual value for each replica (so it divides the value by the number of replicas)

We can achieve similar results using prometheus as an external metrics provider

Update: To use the final autoscaling/v2 version there are some minor changes:

apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata:  name: ampa spec:  minReplicas: 2  maxReplicas: 22  scaleTargetRef:  apiVersion: apps/v1  kind: Deployment  name: spin-fiat  metrics:  - type: External  external:  metric:  name: kubernetes.kubelet.cpu.usage  selector:  matchLabels:  cluster_name: example  target:  averageValue: "50"  type: AverageValue

We can use kubectl convert to adjust the existing manifests.

Posted on 10/01/2022

Categories