Category: devops
-
AWS RDS Aurora Blue/Green Deployment
Performing a blue/green deployment with RDS is a great way to validate changes and cutover while keeping the same endpoint and disrupting applications as little as possible. There are some important deployment considerations that you have to know before you create your cluster. Most of the time, teams find out about this after running a…
-
Terraform Module to Build an AMI using CodeBuild and Packer
I’ve created a terraform module that will allow you to build any AMI with a Packer configuration in a Git repository, using CodeBuild, and place the AMI ID in an SSM Parameter for use by other modules. I’ve created this module so that any AMI that I build can easily be deployed and built on…
-
Connect to GitHub from AWS CodeBuild
In order to build and deliver software in AWS using Github as the source repository, you have to setup a connection to authorize AWS to use Github. The best way is to use a GitHub App connection. AWS has some great documentation on how to do this. First, you need access to a GitHub account…
-
Packer Build: Timeout waiting for SSH
Back to basics. Building an AMI from the official Amazon Linux 2023 base AMI should be as easy as it gets. Packer 1.9.4 on Mac installed with brew. A simple build script. Nothing complicated. 80% of the time it would fail with this error: Timeout waiting for SSH. Further investigation showed that the AWS Session…
-
IMDSv2 Calls w/ Older Boto3
I had a recent problem where I needed to disable IMDSv1 on an older application that could not be upgraded. The boto3 version was 1.8.9 and would require significant changes to support IMDSv2. Instead of upgrading, I was able to set the AWS config and credentials using puppet, using the EC2 instance profile role. This…
-
Accessing Docker as non-root user on Ubuntu 20.04
After installing docker on Ubuntu 20.04, in order to run docker commands against the docker daemon, you must add the user to the docker group. If a permissions denied state persists, try logging out and back in again before running additional commands. The most common error message is: If you are using the ubuntu user,…
-
Using minikube instead of Docker Desktop on Mac OS
With the recent license changes on Docker Desktop, many users have switched to minikube on Mac. minikube is simple to setup and run, and also gives you kubernetes. Everyone should be using kubernetes. This assumes the user is using brew for package management and that Docker Desktop has been removed. After this installation, docker commands…
-
AccessDenied: User: is not authorized to perform: elasticloadbalancing:AddTags on resource
I recently updated a deployed helm chart’s ingress definition to add an SSL redirect on port 80 and was surprised to find that the operation also resulted in the removal of several listener rules from my ALB and a failure to update the SSL redirect. I pulled the logs from the AWS LoadBalancer Controller pods…
-
Drain and Replace EKS Worker Nodes
Unliked managed node groups, EKS worker node groups have to be recycled outside EKS after updating the AMI reference. I recently migrated all of my hosted sites from ECS to EKS and am using terragrunt/terraform for all infrastructure as code. I then upgraded the AMI that I used for the worker node groups and had…
-
Bitnami WordPress Helm Chart Fails Liveness / Readiness Probes
The update to wordpress 5.8 has resulted in all helm deployments failing due to liveness and readiness probe failures. The problem seems to be that if the wordpressScheme is set to https, the liveness/readiness probes use http and fail to stabilize the container. I submitted a PR that was merged to fix this so that…