Delayed Job workers generally have a lifecycle that is equivalent to an application deployment.Because of this, their memory consumption grows over time and may eventually have high swapusage, causing workers to become unresponsive so I decide to use a monitoring tool Monit to watch jobs. Monit can restart them when their memory usage hits a certain point. The good thing I like from monit is when a pre defined condition is matched, Monit can alert my for example by email by telling us what happen so I can take action accordingly.
I am using Ruby 1.9.2, rvm 1.8.6 , rails 3.07 on Ubuntu 10.10.
my application is located at : /var/www/rails3
/var/www/rails/.rvmrc file contains the following content:
rvm use 1.9.2@dyrm --create
Include the following line in you /var/www/rails3/Gemfile
gem "delayed_job"
Then to install delayed_job gem, in your terminal type:
/var/www/rails3$ bundle install
then
/var/www/rails3$ rails g delayed_job
/var/www/rails3$ rake db:migrate
#inserting job in my delayed_job
def self.alert notification
notification.users.each do |user|
Delayed::Job.enqueue(EmailJob.new(user, message)) unless user.email.blank?
Delayed::Job.enqueue(SmsJob.new(user, message)) unless user.phone_number.blank?
end
end
My Email job:
class EmailJob
def initialize user, message
@user = user
@message = message
end
def perform
UserMailer.notify_members(@user, @message).deliver
end
end
class SmsJob
def initialize user, message
@user = user
@message = message
end
def perform
Sms.send do |sms|
sms.to = @user.phone_number
sms.body = @message
end
end
end
To be able to puts your job into delayed_job, you job objects need to response to perform method. because perform method does not require any parameter so I need to pass through the constructor .
Installing Monit
In your terminal :
sudo apt-get install monit
configure monit to start up as daemon by editing the file /etc/default/monit with:
startup=1
Then start the daemon with:
sudo /etc/init.d/monit start
In my case I need to edit file
Some time the monit daemon does not start, however the output from terminal console
telling us the service already started. I find the command:
sudo monit summary
is very useful to debug this. After all edit the file /etc/monit/monitrc
set alert channa@instedd.org
to set the alert for monit to alert to.
set httpd port 2812 and
# use address localhost # only accept connection from localhost
allow localhost # allow localhost to connect to the server and
allow admin:monit # require user 'admin' with password 'monit'
# allow @monit # allow users of group 'monit' to connect (rw)
# allow @users readonly # allow users of group 'users' to connect readonly
Use web user interface to monitor my process(on port 2812). Monit comes bundled with a web server using http basic auth.
allow admin:monit to login to web ui with user equal to admin and password monit
include /etc/monit/conf.d/*
I include all my monit script located in side folder: /etc/monit/conf.d
Create a file called notification inside /etc/monit/conf.d. It's name is not important at all.
check process delayed_job
with pidfile /var/www/rails/tmp/pids/delayed_job.pid
stop program = "/usr/bin/env RAILS_ENV=production BUNDLE_GEMFILE=/var/www/rails/Gemfile /home/chenseng/.rvm/bin/bundle-ruby-1.9.2-p290@dyrm exec /var/www/rails/script/delayed_job stop"
start program = "/usr/bin/env RAILS_ENV=production BUNDLE_GEMFILE=/var/www/rails/Gemfile /home/my_user/.rvm/bin/bundle-ruby-1.9.2-p290@dyrm exec /var/www/rails/script/delayed_job start"
if totalmem > 60 MB for 3 cycles then restart
if cpu usage > 50% for 3 cycles then restart
Here we use rvm wrapper to run delayed_job from a monit process. rvm wrapper give us ability to run any ruby script in context of it's gemset without cd to it's own directory.
Create wrapper
/var/www/rails$ rvm wrapper 1.9.2@dyrm "" bundle
this will create /home/my_user/.rvm/bin/bundle-ruby-1.9.2-p290@dyrm executable.
Debug
Before making sure that monit can start my delayed_job process correctly, I try to run delayed_job with my rvm wrapper independently.
In terminal cd to your home directory:
/home/my_user$ /usr/bin/env RAILS_ENV=production BUNDLE_GEMFILE=/var/www/rails/Gemfile /home/my_user/.rvm/bin/bundle-ruby-1.9.2-p290@dyrm exec /var/www/rails/script/delayed_job start
check the output if nothing go wrong then it is time to restart my monit:
/home/my_app$ sudo monit stop all
/home/my_app$ sudo /etc/init.d/monit/restart
/home/my_app$ sudo monit start all
/home/my_app$ sudo monit summary
It is quite helpful to check if my monit config files are correctly written with the correct permission. To do this
sudo monit -c /etc/monit/monitrc
sudo monit -c /etc/monit/conf.d/notification
Web interface:
http://localhost:2832
