Kubernetes: How to identify when and how an object has been modified

3 min read | by Jordi Prats

In the same way we can use git blame to identify who and when has modified a specific line, with kubectl blame we'll be able to do the same to Kubernetes objects

It is a krew plugin so installing it is as easy as follows:

$ kubectl krew install blame 

Once installed we just need to specify the object we want to see:

$ kubectl blame hpa ampa-core  apiVersion: autoscaling/v1  kind: HorizontalPodAutoscaler  metadata: Go-http-client (Update 18 hours ago) annotations: kube-controller-manager (Update 18 hours ago) autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2022-05-12T23:14:33Z","reason":"ScaleDownStabilized","message":"recent  recommendations were higher than current one, applying the highest recent recommendation"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2022-05-13T05:18:05Z","reason":"ValidMetricFound","message":"the  HPA was able to successfully calculate a replica count from cpu resource utilization  (percentage of request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2022-05-13T05:18:05Z","reason":"DesiredWithinRange","message":"the  desired count is within the acceptable range"}]' kube-controller-manager (Update 18 hours ago) autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":2,"currentAverageValue":"22m"}}]' Go-http-client (Update 18 hours ago) helm.sh/hook: post-install,post-upgrade Go-http-client (Update 18 hours ago) helm.sh/hook-weight: "100"  creationTimestamp: "2022-05-12T23:14:30Z"  name: ampa-core  namespace: pet2cattle  resourceVersion: "138072472"  uid: 4bc1d4b0-4e8d-4ecb-b7df-6951c7db6b04  spec: Go-http-client (Update 18 hours ago) maxReplicas: 2 Go-http-client (Update 18 hours ago) minReplicas: 2  scaleTargetRef: Go-http-client (Update 18 hours ago) apiVersion: apps/v1 Go-http-client (Update 18 hours ago) kind: Deployment Go-http-client (Update 18 hours ago) name: ampa-core Go-http-client (Update 18 hours ago) targetCPUUtilizationPercentage: 70  status: kube-controller-manager (Update 18 hours ago) currentCPUUtilizationPercentage: 2 kube-controller-manager (Update 18 hours ago) currentReplicas: 2 kube-controller-manager (Update 18 hours ago) desiredReplicas: 2 

For example, if we edit the object with kubectl edit:

$ kubectl edit hpa ampa-core horizontalpodautoscaler.autoscaling/ampa-core edited 

And repeat the kubectl krew command we'll be able to see the difference:

$ kubectl blame hpa ampa-core  apiVersion: autoscaling/v1  kind: HorizontalPodAutoscaler  metadata: Go-http-client (Update 18 hours ago) annotations: kube-controller-manager (Update 18 hours ago) autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2022-05-12T23:14:33Z","reason":"ScaleDownStabilized","message":"recent  recommendations were higher than current one, applying the highest recent recommendation"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2022-05-13T05:18:05Z","reason":"ValidMetricFound","message":"the  HPA was able to successfully calculate a replica count from cpu resource utilization  (percentage of request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2022-05-13T05:18:05Z","reason":"DesiredWithinRange","message":"the  desired count is within the acceptable range"}]' kube-controller-manager (Update 18 hours ago) autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":2,"currentAverageValue":"21m"}}]' Go-http-client (Update 18 hours ago) helm.sh/hook: post-install,post-upgrade Go-http-client (Update 18 hours ago) helm.sh/hook-weight: "100"  creationTimestamp: "2022-05-12T23:14:30Z"  name: ampa-core  namespace: pet2cattle  resourceVersion: "138072984"  uid: 4bc1d4b0-4e8d-4ecb-b7df-6951c7db6b04  spec: Go-http-client (Update 18 hours ago) maxReplicas: 2 kubectl-edit (Update 5 seconds ago) minReplicas: 1  scaleTargetRef: Go-http-client (Update 18 hours ago) apiVersion: apps/v1 Go-http-client (Update 18 hours ago) kind: Deployment Go-http-client (Update 18 hours ago) name: ampa-core Go-http-client (Update 18 hours ago) targetCPUUtilizationPercentage: 70  status: kube-controller-manager (Update 18 hours ago) currentCPUUtilizationPercentage: 2 kube-controller-manager (Update 18 hours ago) currentReplicas: 2 kube-controller-manager (Update 18 hours ago) desiredReplicas: 2 

It uses the .metadata.manageFields to highlight the changes:

$ kubectl get hpa spin-orca -o yaml apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata:  managedFields:  - apiVersion: autoscaling/v1  fieldsType: FieldsV1  fieldsV1:  f:metadata:  f:annotations:  .: {}  f:helm.sh/hook: {}  f:helm.sh/hook-weight: {}  f:spec:  f:maxReplicas: {}  f:scaleTargetRef:  f:apiVersion: {}  f:kind: {}  f:name: {}  f:targetCPUUtilizationPercentage: {}  manager: Go-http-client  operation: Update  time: "2022-05-12T23:14:30Z"  - apiVersion: autoscaling/v1  fieldsType: FieldsV1  fieldsV1:  f:spec:  f:minReplicas: {}  manager: kubectl-edit  operation: Update  time: "2022-05-13T05:21:38Z"  - apiVersion: autoscaling/v1  fieldsType: FieldsV1  fieldsV1:  f:metadata:  f:annotations:  f:autoscaling.alpha.kubernetes.io/conditions: {}  f:autoscaling.alpha.kubernetes.io/current-metrics: {}  f:status:  f:currentCPUUtilizationPercentage: {}  f:currentReplicas: {}  f:desiredReplicas: {}  f:lastScaleTime: {}  manager: kube-controller-manager  operation: Update  time: "2022-05-13T05:23:05Z"  name: ampa-core  namespace: pet2cattle  resourceVersion: "138075242"  uid: 4bc1d4b0-4e8d-4ecb-b7df-6951c7db6b04 (...) 

Posted on 13/05/2022