I logged in this morning and one of my clients had an issue with a Passenger app that was not responding and nginx was returning 403s for requests to the app.
I dug into the nginx error.log and it showed that it could not find the PassengerWatchdog file that it would use to start the app which resulted in a static HTML page request in the document root.
[alert] xxxxx#0: Unable to start the Phusion Passenger watchdog because its executable (/usr/lib/phusion-passenger/agents/PassengerWatchdog) does not exist. This probably means that your Phusion Passenger installation is broken or incomplete, or that your 'passenger_root' directive is set to the wrong value. Please reinstall Phusion Passenger or fix your 'passenger_root' directive, whichever is applicable. (-1: Unknown error)
Turns out, one of the devs had cleared the gemsets for the rvm profile which removed the passenger gem that we had compiled nginx for.
I was able to verify this by comparing the output of rvm gem list on a working host and this host.
To fix this, I had to install the passenger nginx module, the PCRE module, run ldconfig, and restart nginx.
Install the PCRE module first:
wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.20.tar.gz tar xzvf pcre-8.20.tar.gz cd pcre-8.20 ./configure && make && make install
Recompile nginx with the passenger module:
Once that is complete, without error, run ldconfig and restart nginx.