resqutils - useful stuff when you have Resque in your app

This is a small library of useful modules and functions that can help dealing with Resque.

Currently:

• Job that kills stale workers
• Means to identify stale workers
• Spec helper :some_queue.should have_job_queued(class: FooJob)
• Methods to introspect queues, including the delayed queue, in your specs
• Simple resque:work task wrapper to better handle exceptions in the worker
• Marker interface to document jobs which should not be retried

Maybe will have more stuff someday.

To use

Add to your Gemfile:

gem 'resqutils'

Stale Workers

It's possible (and, on Heroku, highly likely) that your jobs will appear to be running for "too long". Usually, this happens when a worker exits without cleaning up after itself. Since Resque stores all state in Redis, and is process-based, it's actually fairly easy to create this situation.

The good news is that, if your jobs are idempotent, you can just unregister the "stale" workers, which will kick-off the failed handling (which is hopefully to restart your jobs).

You need a means of identifying these workers, and then killing them.

Resqutils::StaleWorkersKiller.kill_stale_workers

This will queue a WorkerKillerJob for each stale worker. This uses Resqutils::StaleWorkers under the covers to identify which are stale. You can pass it in to StaleWorkersKiller's constructor, or configure it using the environment. By default, a worker running for more than an hour is considered stale.

Setting the RESQUTILS_SECONDS_TO_BE_CONSIDERED_STALE environment variable, you can override that.

The queue that WorkerKillerJob will queue to is worker_killer_job by default, but can be changed by setting the RESQUTILS_WORKER_KILLER_JOB_QUEUE environment variable.

Resqutils::StaleWorkersKiller is also itself a resque job, so you can use this class directly in your resque-scheduler implementation to kill stale jobs on a schedule.

You can, of course, use these building blocks on your own for other purposes.

Spec Helpers

# in spec_helper.rb
require 'resqutils/spec'

# In one of your spec files
describe SomeProcess do
  include Resqutils::Spec::ResqueHelpers

  # ...
end

requireing the resqutils/spec will also set up the have_job_queued matcher, which is likely what you'll want to use.

Clearing Jobs

The most important part of using Resque in tests as making sure the queue has what you think it has in it. To that end, you'll likely need clear_queue in a setup or before block.

before do
  clear_queue(MyImportantJob) # clears whatever queue this job is configured to use
  clear_queue(:foobar)        # clear the "foobar" queue
end

Checking that Jobs Were Queued

# foo_service.rb
class FooService
  def doit(foo)
    Resque.enqueue(:foo,FooJob,foo)
    "bar"
  end
end

# foo_service_spec.rb
describe FooService do
  include Resqutils::Spec::ResqueHelpers
  
  before do
    clear_queue(FooJob) # Looks at what queue FooJob uses and clears before each test
  end
  
  it "queues a job" do
    result = FooService.new.doit("blah")

    expect(result).to eq("bar")
    expect(:foo).to have_job_queued(class: FooJob, args: [ "blah" ])
  end
end

This also works with the delayed queue as provided by resque-scheduler:

# foo_service.rb
class FooService
  def doit(foo)
    Resque.enqueue_in(5.minutes,:foo,FooJob,foo)
    "bar"
  end
end

# foo_service_spec.rb

describe FooService do
  include Resqutils::Spec::ResqueHelpers
  
  before do
    clear_queue(:delayed) # Clears all delayed/scheduled queues
  end
  it "queues a job" do
    result = FooService.new.doit("blah")

    expect(result).to eq("bar")
    # :delayed is special and triggers logic to look into the various scheduled queues
    expect(:delayed).to have_job_queued(class: FooJob, args: [ "blah" ])
  end
end

Executing Jobs

In an integration test, you may wish to execute a job that's on the queue, which will both assert that it's there and perform whatever function it performs.

# foo_service.rb
class FooService
  def doit(foo)
    Resque.enqueue(:foo,FooJob,foo)
    "bar"
  end
end

class FooJob
  def perform(some_value)
    Foo.create!(value: some_value)
  end
end

# the_foo_service_spec.rb
describe "the foo service" do
  include Resqutils::Spec::ResqueHelpers
  it "writes a Foo with the value" do
    result = FooService.new.doit("blah")

    process_resque_job(FooJob)

    expect(Foo.last.value).to eq("blah")
  end
end

The ResqueHelpers module has many more methods, if you need finer control over your tests with respect to resque.

Exception Handling in your Worker

The built-in worker lets exceptions bubble up. In a PaaS setup, or where your Redis is "over the internet", you'll get periodic connection issues from your worker. These self-heal when your worker management system (e.g. monit) restarts the worker after it crashes. Thus, these unhandled exceptions should just be ignored.

Since the built-in resque worker is a rake task, we provide a wrapper rake task to call it and log the exception:

require 'resqutils/worker_task'

To run:

env TERM_CHILD=1 bundle exec rake environment resqutils:work QUEUE=file_uploads --trace

Being clear about not retrying

Although you should design your jobs to automatically retry, some jobs simply should not be retried. Instead of omitting the retry logic or dropping in a comment, you should use a marker interface to communicate intent via code:

class DangerousJob
  include Resqutils::DoNotAutoRetry

  def perform
    # ...
  end
end

This is a more powerful statement that a comment, and communicates intent clearly.