Using Monit to monitor delayed_job in Rails 3

I am developing an application using RubyonRails3 . My application need to send alert notification , reminder via sms , email to hundreds of users. To make my application responsive I have to do these tasks in the background process. After exploring many alternatives I see that delayed_job fits my needs.  Delayed_job run a separate process with my rails app.

Delayed Job workers generally have a lifecycle that is equivalent to an application deployment.Because of this, their memory consumption grows over time and may eventually have high swapusage, causing workers to become unresponsive so  I  decide to use a monitoring tool Monit to watch  jobs. Monit can restart them when their memory usage hits a certain point. The good thing I like from monit is when a pre defined condition is matched, Monit can alert my for example by email by telling us what happen so I can take action accordingly.

I am using Ruby 1.9.2, rvm 1.8.6 , rails 3.07 on Ubuntu 10.10.

my application is located at : /var/www/rails3

/var/www/rails/.rvmrc file contains the following content:

rvm use 1.9.2@dyrm --create

Include the following line in you /var/www/rails3/Gemfile

gem "delayed_job"

Then to install delayed_job gem, in your terminal type:

/var/www/rails3$    bundle install

then

/var/www/rails3$ rails g delayed_job
/var/www/rails3$ rake db:migrate

#inserting job in my delayed_job
def self.alert notification
    notification.users.each do |user|
      Delayed::Job.enqueue(EmailJob.new(user, message)) unless user.email.blank?
      Delayed::Job.enqueue(SmsJob.new(user, message)) unless user.phone_number.blank?
    end
  end

My Email job:

  class EmailJob
  def initialize user, message
    @user = user
    @message = message
  end

  def perform
    UserMailer.notify_members(@user, @message).deliver
  end
end

  class SmsJob
  def initialize user, message
    @user = user
    @message = message
  end

  def perform
    Sms.send do |sms|
      sms.to = @user.phone_number
      sms.body = @message
    end
  end
end

To be able to puts your job into delayed_job, you job objects need to response to perform method. because perform method does not require any parameter so I need to pass through the constructor .

Installing Monit

In your terminal :

sudo apt-get install monit

configure monit to start up as daemon by editing the file /etc/default/monit with:

startup=1

Then start the daemon with:

sudo /etc/init.d/monit start

In my case I need to edit file

Some time the monit daemon does not start, however the output from terminal console

telling us the service  already started. I find the command:

sudo monit summary

is very useful to debug this. After all edit the file /etc/monit/monitrc

set alert channa@instedd.org

to set the alert for monit to alert to.

set httpd port 2812 and
#    use address localhost  # only accept connection from localhost
     allow localhost        # allow localhost to connect to the server and
     allow admin:monit      # require user 'admin' with password 'monit'
#    allow @monit           # allow users of group 'monit' to connect (rw)
#    allow @users readonly  # allow users of group 'users' to connect readonly

Use web user interface to monitor my process(on port 2812). Monit comes bundled with a web server using http basic auth.
allow admin:monit to login to web ui with user equal to admin and password monit

include /etc/monit/conf.d/*

I include all my monit script located in side folder: /etc/monit/conf.d

Create a file called notification inside /etc/monit/conf.d. It's name is not important at all.

check process delayed_job
      with pidfile /var/www/rails/tmp/pids/delayed_job.pid
      stop program  = "/usr/bin/env RAILS_ENV=production  BUNDLE_GEMFILE=/var/www/rails/Gemfile /home/chenseng/.rvm/bin/bundle-ruby-1.9.2-p290@dyrm exec /var/www/rails/script/delayed_job stop"
      start program = "/usr/bin/env RAILS_ENV=production  BUNDLE_GEMFILE=/var/www/rails/Gemfile /home/my_user/.rvm/bin/bundle-ruby-1.9.2-p290@dyrm exec /var/www/rails/script/delayed_job start" 

      if totalmem > 60 MB for 3 cycles then restart
      if cpu usage > 50% for 3 cycles then restart

Here we use rvm wrapper to run delayed_job from a monit process. rvm wrapper give us ability to run any ruby script in context of it's gemset without cd to it's own directory.

Create wrapper

   /var/www/rails$ rvm wrapper 1.9.2@dyrm "" bundle

this will create /home/my_user/.rvm/bin/bundle-ruby-1.9.2-p290@dyrm executable.

Debug

Before making sure that monit can start my delayed_job process correctly, I try to run delayed_job with my rvm wrapper independently.
In terminal cd to your home directory:
/home/my_user$ /usr/bin/env RAILS_ENV=production BUNDLE_GEMFILE=/var/www/rails/Gemfile /home/my_user/.rvm/bin/bundle-ruby-1.9.2-p290@dyrm exec /var/www/rails/script/delayed_job start

check the output if nothing go wrong then it is time to restart my monit:

  /home/my_app$ sudo monit stop all
  /home/my_app$ sudo /etc/init.d/monit/restart
  /home/my_app$ sudo monit start all
  /home/my_app$ sudo monit summary

It is quite helpful to check if my monit config files are correctly written with the correct permission. To do this
sudo monit -c /etc/monit/monitrc
sudo monit -c /etc/monit/conf.d/notification

Web interface:

http://localhost:2832