Job uniqueness for ActiveJob

TLDR

Install activejob-uniqueness

bundle add activejob-uniqueness

Get job uniqueness for ActiveJob

class MyJob < ActiveJob::Base
  unique :until_executed, lock_ttl: 3.hours, on_conflict: :log

  def perform(args)
    # regular ActiveJob workload
  end
end

Read documentation.

History

ActiveJob provides abstraction over background processing backend jobs and implements asynchronous jobs in a Rails way with generators, callbacks, action mailer integration, distribution by engines, and other helpful features. It was introduced in Rails 4.2, but it was not actively adopted by the Ruby On Rails community because of popularity of Sidekiq. Usually, there is not much sense in using ActiveJob abstraction over native Sidekiq workers declaration because of reasons such as extra overhead on jobs serialization/deserialisation, losing direct control over full set of Sidekiq options, and incompatibility of some Sidekiq plugins with ActiveJob.

Sidekiq is a brilliant piece of software that had brought threaded processing to the Ruby world and became a default solution for background tasks. As any popular framework, Sidekiq also has plenty of plugins to extend its functionality. It is an ecosystem you don't want to leave. Veeqo actively uses Sidekiq, Sidekiq Pro and Sidekiq plugins to help online retailers to deliver excellent customer experience. But being a fast-growing and scaling business, Veeqo has some specific demands that require us to switch to other solutions.

After a review of possible options, one important requirement was obvious: There should be an abstraction over background processing to make such transitions seamless. ActiveJob came to the rescue! It's common interface gives flexibility to use alternative solutions without dramatic changes in the codebase. In our case it also means that some functionality of Sidekiq plugins that we love should also be present on ActiveJob level.

Particularly, we want something like SidekiqUniqueJobs, but the About ActiveJob section of the Wiki-pages of the project states that ActiveJob is not supported officially. So it is not an option. Unfortunately, we didn't find anything suitable to protect uniqueness of ActiveJob jobs.

Here was an oportunity to implement this functionality and share it with the community!

Implementation

OK, we want high-level strategies of SidekiqUniqueJobs:

ActiveJob also provides callbacks - the perfect match for embedding:

• before_enqueue
• around_enqueue
• after_enqueue
• before_perform
• around_perform
• after_perform

The only piece of puzzle left was the locking mechanism. Luckily, there is a redlock-rb - an implementation of algorythm for distributed locks with Redis, called Redlock.

There were a series of articles which may lead to some finding this choice controversial:

1. "How to do distributed locking" by Martin Kleppmann (distributed system researcher at Cambridge)
2. "Is Redlock safe?" by Antirez (author of Redis)

Nevertheless, we took the decision to stick with Redlock as it was the simplest solution not only in terms of the implementation, but also in terms of requirements for infrastructure.

Having the building bricks (ActiveJob callbacks + Redlock), we put some concrete to join them. With the help of Appraisal and Travis build matrix we tested different versions of ActiveJob to support ActiveJob/Rails v4.2-v6.1

Meet ActiveJob::Uniqueness!

Add the activejob-uniqueness gem to your Gemfile and run bundle install command.

gem 'activejob-uniqueness'

ActiveJob::Uniqueness is ready to work without any configuration. It will use REDIS_URL to connect to Redis instance. To override the defaults, create an initializer config/initializers/active_job_uniqueness.rb using the following command:

rails generate active_job:uniqueness:install

Start making jobs to be unique

class MyJob < ActiveJob::Base
  unique :until_executed

  def perform(args)
    DoSomeImportantThings.new(args).call
  end
end

By default, if you try to enqueue a non-unique job, an ActiveJob::Uniqueness::JobNotUnique error is raised. This behavior can be changed globally by editing config/initializers/active_job_uniqueness.rb or per class:

class MyJob < ActiveJob::Base
  unique :until_executed, on_conflict: :log
  # Or by using a proc for custom behavior
  # unique :until_executed, on_conflict: ->(job) { job.logger.info 'Oops' }

  def perform(args)
    DoSomeImportantThings.new(args).call
  end
end

The same logic is applied for locks expiration (TTL)

class MyJob < ActiveJob::Base
  unique :until_executed, lock_ttl: 3.hours

  def perform(args)
    DoSomeImportantThings.new(args).call
  end
end

The until_and_while_executing strategy also adds extra settings: runtime_lock_ttl and on_runtime_conflict to control "while executing" part.

Sometimes it can be helpful to be able to explicitly unlock some job or group of jobs:

# Remove the lock for particular arguments:
MyJob.unlock!(foo: 'bar')
# or
ActiveJob::Uniqueness.unlock!(job_class_name: 'MyJob', arguments: [{foo: 'bar'}])

# Remove all locks of MyJob
MyJob.unlock!
# or
ActiveJob::Uniqueness.unlock!(job_class_name: 'MyJob')

# Remove all locks
ActiveJob::Uniqueness.unlock!

Most probably you won't want jobs to be locked in tests. Therefore you can add this line to your test suite (rails_helper.rb for Rspec):

ActiveJob::Uniqueness.test_mode!