Monitor resque workers via upstart

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. Resque works firing up a set number of workers than will coninuously look for job to be processed and will execute them following the rules defined when the workers are launched. You can have a single worker process any queue or having a single worker processing a single queue or even fire up more workers that will look forward to a single queue. Read more about resque.

A very important topic is to be sure that the workers are always available and that any time your application is deployed also the workers are restarted. This is crucial cause the worker load an instance of your application and will then process the jobs against this application instance. If you deploy a new version of your application but you do not restart your workers the jobs will be run on an outdated code version and this is clearly something you do not want!

I have solved both the issues using upstart to monitor the workers deamon and a rake task that kill all the workers any time the application is deployed. Basically the idea is that any time a worker is killed upstart will restart it with the most up to date application version.

Here is a sample of the upstart script I use to monitor the workers for my application:

#!upstart

description "Workers [resque]"

start on (local-filesystems and net-device-up IFACE=eth0)
stop on shutdown

respawn
respawn limit 99 5
console none

script
   su -c "source '/home/myuser/.rvm/scripts/rvm';rvm 1.9.2; cd /myapp/; RAILS_ENV={env} bundle exec rake resque:work QUEUE=* RESQUE_NAMESPACE={namespace} >> ~/resque_workers.log 2>&1" my_user
end script

If you are not familiar with upstart you can read more here.

What this script does is  launching the workers in a defined rails environment using a passed namespace ( I use namespace to ensure that I can safely use the same Redis engine for different applications/environments ). You can also note that the script execute the command from the application directory using an unpriviledged user. This trick was due to the fact that I run the application and the workers using a specific user ( ‘my_user’ in the example) and not as root as upstart attempt to do. The rest of the script load in the shell rvm with ruby 1.9.2 before executing the script.

Once this conf script has started upstart will monitor the processes and will restart it something gets killed or does not respond.

Great! I know need a task to be runned via capistrano when a new deployment gets completed. Killing the worker will invoke upstart to restart them with the lastest deployed codebase.

require 'resque/tasks'

task "resque:setup" => :environment

namespace :queue do
  task :restart_workers => :environment do
    pids = Array.new    
    Resque.workers.each do |worker|
      pids << worker.to_s.split(/:/).second
    end    
    if pids.size > 0
      system("kill -QUIT #{pids.join(' ')}")
    end
  end
end

Invoking the task will kill any workers that is registered with the Resque instance running in your application. Be sure to run the task just before to restart your application. In this way you are sure that the task will attempt to kill the correct pids.

$: rake queue:restart_workers

Et voila! Any feed bask is welcome.