How to debug a crossplane provider
4 min read | by Jordi Prats
To be able to setup a crossplane provider there are some pieces that need to be aligned to be able to use it. For example, if we want to setup the AWS provider using an IAM Role for ServiceAccount. If something is missaligned, we might end up with an error while creating resources that doesn't really clarify what's the actual error:
$ kubectl describe bucket.s3.aws.crossplane.io/test-bucket Name: test-bucket Namespace: Labels: <none> Annotations: crossplane.io/external-name: pet2cattle-demo API Version: s3.aws.crossplane.io/v1beta1 Kind: Bucket Metadata: (...) Spec: (...) Provider Config Ref: Name: aws-provider Status: At Provider: Arn: Conditions: Last Transition Time: 2022-02-22T21:43:23Z Message: observe failed: failed to query Bucket: api error MovedPermanently: Moved Permanently Reason: ReconcileError Status: False Type: Synced Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning CannotObserveExternalResource 7s (x6 over 36s) managed/bucket.s3.aws.crossplane.io failed to query Bucket: api error MovedPermanently: Moved Permanently Assuming we have the following configuration for the AWS provider:
apiVersion: pkg.crossplane.io/v1alpha1 kind: ControllerConfig metadata: name: aws-config spec: podSecurityContext: fsGroup: 2000 --- apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: provider-aws spec: package: crossplane/provider-aws:v0.24.1 controllerConfigRef: name: aws-config --- apiVersion: aws.crossplane.io/v1beta1 kind: ProviderConfig metadata: name: aws-provider spec: credentials: source: InjectedIdentity What we can do is run a shell on the AWS provider pod to check for the processes' command-line arguments:
$ kubectl get pods NAME READY STATUS RESTARTS AGE crossplane-c7b774b95-t8t6n 1/1 Running 0 6h31m crossplane-rbac-manager-6cff7fbf67-nz7bb 1/1 Running 0 6h31m provider-aws-f78664a342f1-575cccfd-g629p 1/1 Running 0 8m21s $ kubectl exec -it provider-aws-f78664a342f1-575cccfd-g629p -- sh / $ ps PID USER TIME COMMAND 1 2000 1:14 crossplane-aws-provider 17 2000 0:00 sh 24 2000 0:00 ps / $ crossplane-aws-provider --help usage: crossplane-aws-provider [<flags>] AWS support for Crossplane. Flags: --help Show context-sensitive help (also try --help-long and --help-man). -d, --debug Run with debug logging. -s, --sync=1h Sync interval controls how often all resources will be double checked for drift. --poll=1m Poll interval controls how often an individual resource should be checked for drift. -l, --leader-election Use leader election for the conroller manager. / $ Since the provider's deployment is managed by the crossplane controller we won't be able to patch it using neither kubectl edit nor kubectl patch. Instead, we can use the ControllerConfig. We can use spec.args to add a -debug flag:
apiVersion: pkg.crossplane.io/v1alpha1 kind: ControllerConfig metadata: name: aws-config spec: podSecurityContext: fsGroup: 2000 args: - '--debug' If we check for the container's logs we'll be able to see more output but it might not be enough to understand the problem:
$ kubectl logs provider-aws-f78664a342f1-68999bf77b-xg7pt -f (...) 2022-02-22T22:26:19.457Z DEBUG provider-aws Reconciling {"controller": "managed/bucket.s3.aws.crossplane.io", "request": "/test-bucket"} 2022-02-22T22:26:19.565Z DEBUG provider-aws Cannot observe external resource {"controller": "managed/bucket.s3.aws.crossplane.io", "request": "/test-bucket", "uid": "2e5c0379-ce37-4546-89f1-16a833808ecc", "version": "145001855", "external-name": "pet2cattle-demo", "error": "failed to query Bucket: api error BadRequest: Bad Request", "errorVerbose": "api error BadRequest: Bad Request\nfailed to query Bucket\ngithub.com/crossplane/provider-aws/pkg/clients.Wrap\n\t/home/runner/work/provider-aws/provider-aws/pkg/clients/aws.go:976\ngithub.com/crossplane/provider-aws/pkg/controller/s3.(*external).Observe\n\t/home/runner/work/provider-aws/provider-aws/pkg/controller/s3/bucket.go:108\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\t/home/runner/work/provider-aws/provider-aws/vendor/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed/reconciler.go:681\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.17.6/x64/src/runtime/asm_amd64.s:1581"} 2022-02-22T22:26:19.568Z DEBUG controller-runtime.manager.events Warning {"object": {"kind":"Bucket","name":"test-bucket","uid":"2e5c0379-ce37-4546-89f1-16a833808ecc","apiVersion":"s3.aws.crossplane.io/v1beta1","resourceVersion":"145001855"}, "reason": "CannotObserveExternalResource", "message": "failed to query Bucket: api error BadRequest: Bad Request"} (...) We can also set spec.securityContext.runAsUser to 0 to be able to install the AWS cli on the container to get the actual error. That would look like this:
apiVersion: pkg.crossplane.io/v1alpha1 kind: ControllerConfig metadata: name: aws-config spec: podSecurityContext: fsGroup: 2000 args: - '--debug' securityContext: runAsUser: 0 Once the provider's pod have been refreshed we can spawn a shell on it to install awscli as follows:
$ kubectl exec -it provider-aws-f78664a342f1-5bb5d9f9d8-ch2kv -- sh / # apk --no-cache add python3 py3-pip; pip3 install --upgrade pip; pip3 install --no-cache-dir awscli; aws s3 ls (...) An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity This error might better point us on the right direction to fix whatever is causing it
Posted on 23/02/2022