(#7398) Use DelayedJob for background processing.

Delayed_Job is a gem that implements robust background processing for Ruby
tasks.  Now we have vendored it, we can start to use it to process report
import in the background.

This has two advantages: one, we can import in parallel, since we can have
more than one worker in the background working on the YAML transformation and
database updating.

Two, we now return to the report submitter in short order: all we have to do
is spool the YAML to disk, and register the background job, and we are assured
that we will eventually get that into the database.

This eliminates a lot of the master => dashboard submission pipeline delay, so
that the master gets back to serving clients immediately, rather than having
to wait until the report is entirely ingested.

Reviewed-By: Daniel Pittman <[email protected]>
Reviewed-By: Matt Robinson <[email protected]>

README.markdown

@@ -105,7 +105,12 @@ Installation
 
 2.  Create a `config/database.yml` file to specify Puppet Dashboard's database configuration. Please see the `config/database.yml.example` file for further details about database configurations and environments. These files paths are relative to the path of the Puppet Dashboard software containing this `README.markdown` file.
 
-3.  Setup a MySQL database server, create a user and database for use with the Puppet Dashboard by either:
+3.  Configure MySQL maximum packet size, to permit larger rows in the database.  Puppet Dashboard can send up to 17MB of data in a single row, although it is extraordinarily rare that it will.  You should configure your server `my.cnf` to increase the limit to at least 24MB (32MB or more recommended), and restart MySQL for this to take effect.  (See [the MySQL documentation][http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_max_allowed_packet] for more details, and how to up the limit without restarting.)
+
+           # Allowing 32MB ensures our 17MB row with plenty of spare room
+           max_allowed_packet = 32M
+
+4.  Setup a MySQL database server, create a user and database for use with the Puppet Dashboard by either:
 
     1.  Using a `rake` task to create just the database from settings in the `config/database.yml` file. You must `cd` into the directory with the Puppet Dashboard software containing this `README.markdown` file before running these commands:
 
@@ -117,7 +122,7 @@ Installation
             CREATE USER 'dashboard'@'localhost' IDENTIFIED BY 'my_password';
             GRANT ALL PRIVILEGES ON dashboard.* TO 'dashboard'@'localhost';
 
-4.  Populate the database with the tables for the Puppet Dashboard.
+5.  Populate the database with the tables for the Puppet Dashboard.
 
     1.  For typical use with the `production` environment:
 
@@ -343,6 +348,13 @@ Third-party tools that can help secure a Puppet Dashboard include:
 
 4.  HTTPS (SSL) Encryption is supported when running Dashboard under Apache and Passenger. The example configuration in `ext/passenger/dashboard-vhost.conf` includes a commented-out vhost configured to use SSL. You may need to change the Apache directives SSLCertificateFile, SSLCertificateKeyFile, SSLCACertificateFile, and SSLCARevocationFile to the paths of the files created by the `cert` rake tasks. (See `Generating certs and connecting to the puppet master` for how to create these files)
 
+Background Processing
+---------------------
+
+The Puppet Dashboard performs a number of tasks, such as report import, that can consume significant system resources.  To ensure that performance remains snappy under load, we use the `delayed_job` background processing system to manage these tasks without tying up a web front end thread.
+
+The Puppet Dashboard code will automatically spawn a manager and worker in the background, which will run these tasks without tying up the resources of the web server.  This will share the same credentials and access as the web front-end, so should introduce no additional security risk.
+
 Performance
 -----------
 

app/models/report.rb

@@ -96,6 +96,10 @@ def self.create_from_yaml(report_yaml)
     report
   end
 
+  class << self
+    handle_asynchronously :create_from_yaml
+  end
+
   def assign_to_node
     self.node = Node.find_or_create_by_name(self.host)
   end

config/environment.rb

@@ -17,6 +17,7 @@
   config.gem 'sass'
   config.gem 'will_paginate'
   config.gem 'maruku'
+  config.gem 'daemons', :version => '1.0.10'
 
   # Change this to adjust log rotation. Logger.new(log_file, number_of_logs, max_log_size).
   config.logger = Logger.new("#{RAILS_ROOT}/log/#{RAILS_ENV}.log", 50, 10.megabytes)

config/initializers/delayed_job.rb

@@ -0,0 +1,24 @@
+DELAYED_JOB_PID_PATH = "#{Rails.root}/tmp/pids/delayed_job.pid"
+
+Delayed::Worker.destroy_failed_jobs = false
+Delayed::Worker.max_attempts = 3
+
+def start_delayed_job
+  Thread.new do
+    `#{Rails.root}/script/delayed_job -p dashboard -m start`
+  end
+end
+
+def process_is_dead?
+  begin
+    pid = File.read(DELAYED_JOB_PID_PATH).strip
+    Process.kill(0, pid.to_i)
+    false
+  rescue
+    true
+  end
+end
+
+if !File.exist?(DELAYED_JOB_PID_PATH) && process_is_dead?
+  start_delayed_job
+end

db/migrate/20110614234202_create_delayed_jobs.rb

@@ -0,0 +1,31 @@
+class CreateDelayedJobs < ActiveRecord::Migration
+  def self.up
+    create_table :delayed_jobs, :force => true do |table|
+      # Allows some jobs to jump to the front of the queue
+      table.integer  :priority, :default => 0
+      # Provides for retries, but still fail eventually.
+      table.integer  :attempts, :default => 0
+      # YAML-encoded string of the object that will do work
+      table.text     :handler,  :limit => 16.megabytes
+      # reason for last failure (See Note below)
+      table.text     :last_error
+      # When to run. Could be Time.zone.now for immediately, or sometime in
+      # the future.
+      table.datetime :run_at
+      # Set when a client is working on this object
+      table.datetime :locked_at
+      # Set when all retries have failed (actually, by default, the record is
+      # deleted instead)
+      table.datetime :failed_at
+      # Who is working on this object (if locked)
+      table.string   :locked_by
+      table.timestamps
+    end
+
+    add_index :delayed_jobs, [:priority, :run_at], :name => 'delayed_jobs_priority'
+  end
+
+  def self.down
+    drop_table :delayed_jobs
+  end
+end

script/delayed_job

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
+require 'delayed/command'
+Delayed::Command.new(ARGV).daemonize