Enable access logs for an Ingress using ALB

AWS Kubernetes Ingress ALB access logs

3 min read | by Jordi Prats

When configuring the ALB using an Ingress object we can enable storing access logs to an S3 bucket in the same way we can do it for any ALB using terraform

To do so we need to use the following annotation alb.ingress.kubernetes.io/load-balancer-attributes. Using this annotations we'll need to specify the actual S3 bucket an the prefix we want to use. For the following example we are going to use:

  • A bucket named: access-log-bucket
  • Logs prefix: demo-app

So it's annotation will look as follows:

alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=access-log-bucket,access_logs.s3.prefix=demo-app

Once applied, we can get an error message (depends on the S3 bucket policy) complaining about S3 permissions (Access Denied for bucket Please check S3bucket permission):

$ kubectl describe ingress demo-web Name: demo-web Namespace: demo-flask Address: internal-k8s-demoflask-1a00b666b7-1000851519.us-west-2.elb.amazonaws.com Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) Rules:  Host Path Backends  ---- ---- --------  demo-app.pet2cattle.com   / ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)  / demo-app:9000 (10.12.102.98:9000,10.12.105.61:9000) Annotations: alb.ingress.kubernetes.io/actions.ssl-redirect:  {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}  alb.ingress.kubernetes.io/group.name: demo-app  alb.ingress.kubernetes.io/listen-ports: [{"HTTP":80},{"HTTPS":443}]  alb.ingress.kubernetes.io/load-balancer-attributes:  access_logs.s3.enabled=true,access_logs.s3.bucket=access-log-bucket,access_logs.s3.prefix=demo-app  alb.ingress.kubernetes.io/scheme: internal  alb.ingress.kubernetes.io/target-type: ip  kubernetes.io/ingress.class: alb  meta.helm.sh/release-name: demo-app  meta.helm.sh/release-namespace: demo-flask Events:  Type Reason Age From Message  ---- ------ ---- ---- -------  Warning FailedDeployModel 23m ingress Failed deploy model due to InvalidConfigurationRequest: Access Denied for bucket: access-log-bucket. Please check S3bucket permission  status code: 400, request id: 40996c42-e02c-471c-a468-bc01ef8ab5af

We'll need to add a new statement to the bucket's policy:

(...) "Statement": [ { "Effect": "Allow", "Principal": {  "AWS": "arn:aws:iam::elb-account-id:root"  }, "Action": "s3:PutObject",  "Resource": "arn:aws:s3:::bucket-name/prefix/AWSLogs/your-aws-account-id/*"  }

We need to set the following values to it's appropriate value

  • elb-account-id: We'll need to check AWS's documentation for enabling access logs on Application Load Balancers for a table to identify the correct account number for our AZ. For example, for us-west-2 it's going to be 797873946194
  • bucket-name: As state previously, for this example we are using access-log-bucket
  • your-aws-account-id: Let's assume our account number is: 123456789876

We'll need to update the bucket's policy to include at least the following statement:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": {  "AWS": "arn:aws:iam::797873946194:root"  }, "Action": "s3:*",  "Resource": "arn:aws:s3:::access-log-bucket/demo-app/AWSLogs/123456789876/*"  }, { (...)

Once applied, the Ingress is going to be able to reconcile, so eventually we will get the following event:

$ kubectl describe ingress demo-web Name: demo-web Namespace: demo-flask Address: internal-k8s-demoflask-1a00b666b7-1000851519.us-west-2.elb.amazonaws.com Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) Rules:  Host Path Backends  ---- ---- --------  demo-app.pet2cattle.com   / ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)  / demo-app:9000 (10.12.102.98:9000,10.12.105.61:9000) Annotations: alb.ingress.kubernetes.io/actions.ssl-redirect:  {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}  alb.ingress.kubernetes.io/group.name: demo-app  alb.ingress.kubernetes.io/listen-ports: [{"HTTP":80},{"HTTPS":443}]  alb.ingress.kubernetes.io/load-balancer-attributes:  access_logs.s3.enabled=true,access_logs.s3.bucket=access-log-bucket,access_logs.s3.prefix=demo-app  alb.ingress.kubernetes.io/scheme: internal  alb.ingress.kubernetes.io/target-type: ip  kubernetes.io/ingress.class: alb  meta.helm.sh/release-name: demo-app  meta.helm.sh/release-namespace: demo-flask Events: (...)  Normal SuccessfullyReconciled 15s (x3 over 7d1h) ingress Successfully reconciled

As soon as the ALB is allowd to push data into the S3 bucket, we are going to be able to see how the prefix we have defined gets created:

$ aws s3 ls s3://access-log-bucket  PRE demo-app/

Inside this folder we'll find several files that will match our setup:

$ aws s3 ls s3://sdlc-spinnaker-pet2cattle/demo-app/ --recursive 2022-06-28 01:12:42 104 demo-app/AWSLogs/123456789876/ELBAccessLogTestFile 2022-06-28 01:15:03 2306 demo-app/AWSLogs/123456789876/elasticloadbalancing/us-west-2/2022/06/27/123456789876_elasticloadbalancing_us-west-2_app.k8s-demoflask-1a00b666b7.ef7dfbd0eea82b38_20220628T0115Z_10.12.104.169_w36ca3td.log.gz 2022-06-28 01:15:03 123606 demo-app/AWSLogs/123456789876/elasticloadbalancing/us-west-2/2022/06/27/123456789876_elasticloadbalancing_us-west-2_app.k8s-demoflask-1a00b666b7.ef7dfbd0eea82b38_20220628T0115Z_10.12.106.222_da5mavql.log.gz (...)

Posted on 28/06/2022

Categories