SSH in a for loop is a solution…

I just read an article by Jay Valentine on LinkedIn where he talks about Puppet and how they were not profitable, and also noted that Chef is not, and has never been, profitable. That got me to thinking, why are IT professionals investing in these technologies (time, knowledge, effort…).

As an IT pro, it’s tempting to become a “fan boy” — someone who learns something difficult to use, and then because so much has been invested (time, effort, knowledge), it benefits the IT pro to evangelize the tool or software to make it more relevant (and thus make the IT pro’s skills more valuable and relevant).

This happens to me all the time, Linux, cfengine, puppet, ruby, etc… With little regard for objective analysis of what would work best. I had switched to puppet, from cfengine, when I heard Redhat had adopted Puppet. That was long ago, and they have since switched to Ansible — time to focus more on containers and, when necessary, Ansible. (Although I will continue to support my clients in whatever technology they desire, like any good consultant.)

While this is not a complete waste and is, most of the time, a very good thing, since it will enable quick execution on projects with known skills and tools, it is not ideal in the long run. The reason for this is that all of these projects and tools become very complicated over time. Take puppet or chef — they do require a significant amount of knowledge to effectively deploy. Even worse, they change rapidly. A system deployed one year could require a major re-write (of the manifest/recipe) the following year, if it were upgraded. Many deployments of these configuration management tools go for years without major updates because the effort in upgrading large numbers of services, servers, and configurations is incredible.

This is a huge amount of technical debt. I’d now venture to say that the more time you must spend deploying a configuration management solution, the more technical debt you will incur, unless you do have a very focused plan to upgrade frequently, and maintain a dedicated “puppet/chef/xxxx” IT pro.

I recall reading and/or hearing the famous Luke Kanies (of Puppetlabs) quote where he says, “ssh in a for loop is not a solution”… This has always bothered me, and I couldn’t quantify the reason very well, but it’s similar to the basic text processing argument in old school linux circles — text output is universal. Any app, tool, utility, can process text. Once you move to binary or other output, you lose the ability to universally process the output. It may be more efficient to process it in other manners, but it’s no longer universal.

“SSH in a for loop” is universal.

Standalone puppet with hiera 5 error…

With puppet moving more and more away from supporting a standalone model, it’s somewhat difficult to get puppet standalone working. I recently got bit by a hiera update that caused my puppet standalone deployments to stop interacting with hiera the way that I had deployed it.

Affected versions:

  • puppet 4.10.10
  • hiera 3.4.3

The error that I was receiving was similar to the following — note that this example cites an error with the ec2tagfacts module, which I have modified to work with puppet 4.*:

Error: Evaluation Error: Error while evaluating a Function Call, Lookup of key 'ec2tagfacts::aws_access_key_id' failed: DataBinding 'hiera': v5 hiera.yaml is only to be used inside an environment or a module and cannot be given to the global hiera at $path_to/puppet/manifests/site.pp:12:3 on node $this_node

The new way of managing hiera (via puppet server) is to contain hiera within each environment and module. This does not work with [the way I use] puppet standalone because of the way you have to reference the hiera configuration. I need to try putting puppet in the default locations and try that at some point.

I was able to resolve the issue by downgrading hiera to version 3.1.1. I am testing with other versions. Updates to follow.

Adding Global Environment Variables to Jenkins via puppet…

When using Jenkins in any environment, it’s useful to have variables related to that environment available to Jenkins jobs. I recently worked on a project where I used puppet to deploy global environment variables to Jenkins for use with AWS commands — typically to execute the awscli, one must have knowledge of the region, account, and other items.

In order to make global environment variables available to Jenkins, we can create an init.groovy.d directory in $JENKINS_HOME, as part of the Jenkins puppet profile, ie:

class profile::jenkins::install () {
  file { '/var/lib/jenkins/init.groovy.d':
    ensure => directory,
    owner  => jenkins,
    group  => jenkins,
    mode   => '0755',

We then need to create the puppet template (epp) that we will deploy to this location, as a groovy script:

import jenkins.model.Jenkins
import hudson.slaves.EnvironmentVariablesNodeProperty
import hudson.slaves.NodeProperty

def instance             = Jenkins.instance
def environment_property = new EnvironmentVariablesNodeProperty();

for (property in environment_property) {
  property.envVars.put("AWS_VARIABLE1", "<%= @ec2_tag_variable1 -%>")
  property.envVars.put("AWS_VARIABLE2", "<%= @ec2_tag_variable2 -%>")
  property.envVars.put("AWS_VARIABLE3", "<%= @ec2_tag_variable3 -%>")


Note that in this instance, I am using the ec2tagfacts puppet module that allows me to use EC2 tags as facts in puppet. I will later move to dynamic fact enumeration using a script with facter.

The next step is to add another file resource to the Jenkins puppet profile to place the groovy script in the proper location and restart the Jenkins Service:

class profile::jenkins::install () {
  file { '/var/lib/jenkins/init.groovy.d/aws-variables.groovy':
    ensure  => present,
    mode    => '0755',
    owner   => jenkins,
    group   => jenkins,
    notify  => Service['jenkins'],
    content => template('jenkins/aws-variables.groovy.epp'),

Now when puppet next runs, this will deploy the groovy script and restart Jenkins to take effect.

Note that these environment variables are not viewable under System Information under Manage Jenkins, but are only available inside each Jenkins job, ie inside a shell build section:

#!/bin/bash -x

echo "${AWS_VARIABLE1}"

Retrieving puppet facts from AWS System Manager

AWS System Manager makes it easy to store and retrieve parameters for use across servers, services, and applications in AWS. One great benefit is storing secrets for use, as needed. I recently needed to retrieve some parameters to place in a configuration file via puppet and wrote a short script to retrieve these values as facts.

Create a script like the following in /etc/facter/facts.d, make it executable.


aws configure set region us-east-1
application_username=$(aws ssm get-parameter --name application_username | egrep "Value" | awk -F\" '{print $4}')
application_password=$(aws ssm get-parameter --name application_password --with-decryption | egrep "Value" | awk -F\" '{print $4}')

echo "application_username=${application_username}"
echo "application_password=${application_password}"

exit 0;

Note that this assumes the username is not an encrypted secret, while the password is.

This can be tested with the following:

# facter -p application_username
# facter -p application_password

These facts can then be used in templates, like the following:

# config.cfg.erb
connection_string = <%= @application_username %>:<%= @application_password %>

Running Apache 2 under Ubuntu 16.04 on Docker

I recently wanted to setup a new Ubuntu 16.04 host running Apache under Docker for some AWS ECS/Fargate testing I was doing and encountered the following error:

docker run -p 8085:80 aws-ecr-hello-world:v0.5
[Thu Mar 15 00:11:31.074011 2018] [core:warn] [pid 1] AH00111: Config variable ${APACHE_LOCK_DIR} is not defined
[Thu Mar 15 00:11:31.074576 2018] [core:warn] [pid 1] AH00111: Config variable ${APACHE_PID_FILE} is not defined
AH00526: Syntax error on line 74 of /etc/apache2/apache2.conf:
Invalid Mutex directory in argument file:${APACHE_LOCK_DIR}

This is a typical Ubuntu problem where the /etc/apache2/envvars file needs to be sourced before apache2 can start properly. To figure out which ones needed to be added, I commented out the CMD to start apache and instead entered a command to print out the contents of the envvars file. I also added a sed command to print out line 74 of the apache2.conf file so I could further troubleshoot what was happening there.

# Dockerfile
CMD ["cat", "/etc/apache2/envvars"]
CMD ["sed", "-n", "74p", "/etc/apache2/apache2.conf"]

This output showed that I had to add a few environment variables to the Dockerfile, and verify that they exist when I run the container:

# Dockerfile
ENV APACHE_LOG_DIR   /var/log/apache2
ENV APACHE_PID_FILE  /var/run/apache2/
ENV APACHE_RUN_DIR   /var/run/apache2
ENV APACHE_LOCK_DIR  /var/lock/apache2
ENV APACHE_LOG_DIR   /var/log/apache2


I also verified that the directories would exist to prevent any issues there:

# Dockerfile

After I finished that, I rebuilt the image and was able to run the container without issues.

The full Dockerfile is:

FROM ubuntu:16.04

# Install dependencies
RUN apt-get update -y
RUN apt-get install -y apache2

# Install apache and write hello world message
RUN echo "Hello World!" > /var/www/index.html

# Configure apache
RUN a2enmod rewrite
RUN chown -R www-data:www-data /var/www

ENV APACHE_LOG_DIR   /var/log/apache2
ENV APACHE_PID_FILE  /var/run/apache2/
ENV APACHE_RUN_DIR   /var/run/apache2
ENV APACHE_LOCK_DIR  /var/lock/apache2
ENV APACHE_LOG_DIR   /var/log/apache2



# CMD ["sed", "-n", "74p", "/etc/apache2/apache2.conf"]
# CMD ["cat", "/etc/apache2/envvars"]
 CMD ["/usr/sbin/apache2", "-D",  "FOREGROUND"]

Build the container image:

> docker build -t aws-ecr-hello-world:v0.9.1 .
Sending build context to Docker daemon   2.56kB
Step 1/18 : FROM ubuntu:16.04
 ---> f975c5035748
Step 2/18 : RUN apt-get update -y
 ---> Using cache
 ---> 1716ac62d2f6
Step 3/18 : RUN apt-get install -y apache2
 ---> Using cache
 ---> b03c08c103b5
Step 4/18 : RUN echo "Hello World!" > /var/www/index.html
 ---> Using cache
 ---> a8352375b937
Step 5/18 : RUN a2enmod rewrite
 ---> Using cache
 ---> 313f2e8046ec
Step 6/18 : RUN chown -R www-data:www-data /var/www
 ---> Using cache
 ---> c2e7512d4fe8
Step 7/18 : ENV APACHE_RUN_USER  www-data
 ---> Using cache
 ---> 2054c48681ae
Step 8/18 : ENV APACHE_RUN_GROUP www-data
 ---> Using cache
 ---> 493b20667534
Step 9/18 : ENV APACHE_LOG_DIR   /var/log/apache2
 ---> Using cache
 ---> 8c5029eb8e83
Step 10/18 : ENV APACHE_PID_FILE  /var/run/apache2/
 ---> Using cache
 ---> 701ddcccf335
Step 11/18 : ENV APACHE_RUN_DIR   /var/run/apache2
 ---> Using cache
 ---> 6700b8a02ca0
Step 12/18 : ENV APACHE_LOCK_DIR  /var/lock/apache2
 ---> Using cache
 ---> ac692e86caf7
Step 13/18 : ENV APACHE_LOG_DIR   /var/log/apache2
 ---> Using cache
 ---> 660af37232bc
Step 14/18 : RUN mkdir -p $APACHE_RUN_DIR
 ---> Running in 02978786f1b5
Removing intermediate container 02978786f1b5
 ---> 3e5ef0c00431
Step 15/18 : RUN mkdir -p $APACHE_LOCK_DIR
 ---> Running in 68408f3091c1
Removing intermediate container 68408f3091c1
 ---> 90efa3a2f9bc
Step 16/18 : RUN mkdir -p $APACHE_LOG_DIR
 ---> Running in f1ee7e4d5a4b
Removing intermediate container f1ee7e4d5a4b
 ---> 9fb6a50c6792
Step 17/18 : EXPOSE 80
 ---> Running in f3fd904326e4
Removing intermediate container f3fd904326e4
 ---> b4ba8575620d
Step 18/18 : CMD ["/usr/sbin/apache2", "-D",  "FOREGROUND"]
 ---> Running in a3cba653d7b3
Removing intermediate container a3cba653d7b3
 ---> 0bfa187abf69
Successfully built 0bfa187abf69
Successfully tagged aws-ecr-hello-world:v0.9.1

Run the container:

> docker run -p 8085:80 aws-ecr-hello-world:v0.9.1
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using Set the 'ServerName' directive globally to suppress this message

UPDATE: it looks like James Turnbull had already solved this problem here.

The Phoenix Project

I recently stumbled upon a novel that talks about managing IT Operations. “The Phoenix Project”, by Gene Kim, Kevin Behr, and George Spafford. Wow, what a great read. This book accurately describes many of my experiences in IT with many different companies.

This book has some exceptional concepts around optimizing interaction between Development, IT Operations, and the customer. The book pushes the reader (and fictional characters) to visualize development and IT Operations as a factory floor, using the same terminology to analyze dis-function. Each station includes: machine, man, method, measure. These components can be used to optimize work flow, find bottlenecks, and improve overall efficiency of the factory or deveopment/IT Operations space.

The Three Ways

The First Way: Left to right workflow from development to IT Operations to the Customer. Small batch sizes and intervals of work. Reduce WIP or inventory of tasks.

The Second Way: Constant feedback from right to left at all stages.

The Third Way: Creating a culture that fosters continual experimentation (risk) and understanding that repetition and practice is the prerequisite to mastery.

The Four Types of Work

Business projects: Business initiatives, tracked and PM’d.

Internal IT projects: Infrastructure or IT projects.

Changes: Scheduled updates, releases, etc.. — configuration management.

Unplanned work or recovery work: Production issues, unplanned incidents or problems that disrupt the above 3 types of work.

As a DevOps consultant, I can immediately use these concepts to improve the value that I provide to each client by working with these concepts as a guide.

I highly recommend this book.

Puppet deprecation in stdlib module…

As part of the long upgrade to become fully compatible with puppet 4 and drop puppet 3 support — version 4.13+ of the stdlib module introduced some breaking changes for other modules that I use. I recently upgrade some individual modules using the ‘puppet module upgrade’ method.

Upon upgrading, I received the following message:

Error: Evaluation Error: Error while evaluating a Function Call, undefined method `function_deprecation' ...

The solution, for now, until the modules that I use are upgraded to work with the newer version of stdlib, is to downgrade to version 4.12 of puppetlabs-stdlib.

Before downgrading, check to see if there are other modules which require a version greater than 4.12, ie:

> for file in $(find modules/ -type f -name metadata.json); do echo -n ${file}; echo -n ":  "; egrep -i stdlib ${file} || echo ""; done
modules//apt/metadata.json:      {"name":"puppetlabs/stdlib","version_requirement":">= 4.5.0 < 5.0.0"}
modules//archive/metadata.json:        "name": "puppetlabs/stdlib",
modules//concat/metadata.json:      {"name":"puppetlabs/stdlib","version_requirement":">= 4.2.0 < 5.0.0"}
modules//ec2tagfacts/metadata.json:      {"name":"puppetlabs/stdlib","version_requirement":">= 3.2.0 < 5.0.0"},
modules//epel/metadata.json:      {"name":"puppetlabs/stdlib","version_requirement":">= 3.0.0"}
modules//java/metadata.json:      {"name":"puppetlabs/stdlib","version_requirement":">= 2.4.0 < 5.0.0"}
modules//jenkins/metadata.json:      {"name":"puppetlabs/stdlib","version_requirement":">= 4.6.0 < 5.0.0"},
modules//lvm/metadata.json:      {"name":"puppetlabs/stdlib","version_requirement":">=4.1.0 < 5.0.0"}
modules//mysql/metadata.json:      {"name":"puppetlabs/stdlib","version_requirement":">= 3.2.0 < 5.0.0"},
modules//python/metadata.json:      {"name":"puppetlabs/stdlib","version_requirement":">= 4.6.0 < 6.0.0"},
modules//rvm/metadata.json:      {"name":"puppetlabs/stdlib","version_requirement":">=4.2.0"},

The following modules require a version of puppetlabs-stdlib greater than 4.12:

  • apt – 4.5+
  • concat – 4.2.0+
  • jenkins- 4.6.0+
  • python4.6.0+
  • rvm – 4.2.0+

Sting with the apt module, the apt module is version 4.1.0. According to the puppetlabs-apt change log page, they recommend staying on version 2.3.0 unless you’re ready for the latest puppet 4 changes (they actually say any version 2 release but we’ll let that go after figuring it out…), so we need to downgrade that module first:

> puppet module upgrade puppetlabs-apt --version 2.3.0 --modulepath=modules/
Notice: Preparing to upgrade 'puppetlabs-apt' ...
Notice: Found 'puppetlabs-apt' (v2.4.0) in .../puppet/modules ...
Notice: Downloading from ...
Error: Could not upgrade module 'puppetlabs-apt' (v2.4.0 -> v2.3.0)
  Downgrading is not allowed.

Ok, so you can’t use the upgrade command to downgrade modules, so I need to be a little more heavy handed. I removed the apt module manually:

> rm -rf modules/apt/

Next, install version 2.3.0 of puppetlabs-apt:

> puppet module install puppetlabs-apt --version 2.3.0 --modulepath=modules/
Notice: Preparing to install into .../puppet/modules ...
Notice: Downloading from ...
Notice: Installing -- do not interrupt ...
└─┬ puppetlabs-apt (v2.3.0)
  └── puppetlabs-stdlib (v4.19.0)

Now that I’m deep into this problem, I see great value in using something like r10k or Code Manager (which uses r10k) to manage these modules and dependencies.

Another potentially useful tip – I removed the puppetlabs-stdlib module and ran a module list command, and it then told me which modules were dependent upon the missing module, and which versions:

> puppet module list --modulepath=modules/
Warning: Missing dependency 'puppetlabs-stdlib':
  'puppetlabs-apt' (v2.3.0) requires 'puppetlabs-stdlib' (>= 4.5.0 < 5.0.0)
  'puppet-archive' (v2.0.0) requires 'puppetlabs-stdlib' (>= 4.13.0 < 5.0.0)
  'puppetlabs-concat' (v2.2.0) requires 'puppetlabs-stdlib' (>= 4.2.0 < 5.0.0)
  'bryana-ec2tagfacts' (v0.2.0) requires 'puppetlabs-stdlib' (>= 3.2.0 < 5.0.0)
  'stahnma-epel' (v1.2.2) requires 'puppetlabs-stdlib' (>= 3.0.0)
  'puppetlabs-java' (v1.5.0) requires 'puppetlabs-stdlib' (>= 2.4.0 < 5.0.0)
  'rtyler-jenkins' (v1.7.0) requires 'puppetlabs-stdlib' (>= 4.6.0 < 5.0.0)
  'puppetlabs-lvm' (v0.7.0) requires 'puppetlabs-stdlib' (>=4.1.0 < 5.0.0)
  'puppetlabs-mysql' (v3.10.0) requires 'puppetlabs-stdlib' (>= 3.2.0 < 5.0.0)
  'stankevich-python' (v1.14.2) requires 'puppetlabs-stdlib' (>= 4.6.0 < 6.0.0)
  'maestrodev-rvm' (v1.13.1) requires 'puppetlabs-stdlib' (>=4.2.0)

Moving on to using r10k..

gem install r10k

I then created a Puppetfile using my modules and am using r10k to manage them.

Note: converting to using r10k took around 20 minutes — if you’re not using r10k (or Code Manager), it’s time to start.

docker run docs confusing – which port?

I was reviewing the docker docs today in an attempt to get things working on OSX and ran into a conflict when starting a new container running nginx. The docs say to run the following command:

> docker run -d -p 80:80 --name webserver nginx

That seems pretty straight forward, so I ran the command:

> docker run -d -p 80:80 --name webserver nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
94ed0c431eb5: Pull complete
9406c100a1c3: Pull complete
aa74daafd50c: Pull complete
Digest: sha256:788fa27763db6d69ad3444e8ba72f947df9e7e163bad7c1f5614f8fd27a311c3
Status: Downloaded newer image for nginx:latest
docker: Error response from daemon: driver failed programming external connectivity on endpoint webserver (4fa9e84616ebb7c7aa86a345a4caa01910d8fdcb00b71d0bd8daf6b9443a0a7a): Error starting userland proxy: Bind for unexpected error (Failure EADDRINUSE).

After getting this error, I see that I have something listening on port 80 already, that’s no problem. The logical thing to do would be to refer to the docs real quick to figure out which side of the colon mapped to the port on the container and which side mapped to the port on my OSX host. I could not find an explanation for this in the docker docs. I did a bit of searching around as well, with no luck. I don’t see this documented.

The next logical step, or more likely the first, was to just try them both and see which one worked. The left side is the docker host and the right side is the container, ie:

> docker run  -d -p 3060:80 --name webserver3060 nginx

I was then able to successful query the service on port 3060 by using curl or using my browser at http://localhost:3060/.

Throttling Requests with the Ruby aws-sdk

A common problem of late is throttling requests when using the ruby aws-sdk gem to access AWS services. Handling these exceptions is fairly trivial with a while loop like the following:

retry_count   = 0 
retry_success = 0 

while retry_success == 0
  retry_success = 1

    # enter code to interact with AWS here

  rescue Aws::APIGateway::Errors::TooManyRequestsException => tmre

  # note that different AWS services have different exceptions
  # for this type of response, be sure to check your error output

    sleep_time = ( 2 ** retry_count )
    retry_success = 0 
    sleep sleep_time
    retry_count = retry_count + 1 


Note that there are different exceptions for different services that might indicate a throttling scenario so be sure to check the output received or the documentation around which exception to handle. Also note that additional exceptions should be handled around bad requests, missing, duplicate, unavailable, or mal-formed objects.

Latest Amazon EC2 AMI Supports Puppet 3.7.4

Good news! After quite some time without a supported puppet and ruby combination from the EC2 yum repositories, the latest AMI has support for puppet 3.7.4.

This will make deploying puppet environments easier and not require use of the gem and the development packages requirement to compile it.