I recently updated a deployed helm chart’s ingress definition to add an SSL redirect on port 80 and was surprised to find that the operation also resulted in the removal of several listener rules from my ALB and a failure to update the SSL redirect.
I pulled the logs from the AWS LoadBalancer Controller pods and saw this error:
> kubectl logs aws-load-balancer-controller-7566dd7d54-xfbtl -n kube-system
{"level":"error","ts":1632874919.362528,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"$alb_name.group001","namespace":"","error":"AccessDenied: User: arn:aws:sts::*:assumed-role/aws-loadbalancer-controller-irsa-role/$long_number is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:us-west-2:*:listener-rule/app/k8s-$alb_name-648749f01f/2b2d6134687eabc4/def8ef7e1a866fad/bb6224aad0aac065\n\tstatus code: 403, request id: df9edb23-9f2b-4f67-ba31-93adb6bb526b"}
This issue was caused by manual changes to the ALB that is being managed by the AWS LoadBalancer Controller service. Removing the manual changes allowed the AWS Load Balancer controller to add all of the correct rules and it worked as expected.
Leave a Reply