This post contains some notes I took while reading through the Sidekiq Github wiki. These notes assume that you already have a basic working experience with Sidekiq, including creating and managing jobs. As such, it skips over those basics.

Basics

• Workers get 25 seconds to shutdown in the case of Heroku restart, etc., and any remaining jobs are pushed back to the queue.
  • This entails jobs should not be very long running.
• You can either enqueue jobs by calling perform_now, perform_in, or perform_async on the Job Class, or you can use the generic Sidekiq::Client.push('class' => JobClass, 'args' => [...]).
• Arguments must be simple JSON datatypes since they are serialized into a string in Redis.
• Jobs should be idempotent, meaning that they should be able to be safely executed multiple times.
  • Due to Sidekiq’s retry functionality as well as having the Redis queue live separately from the main app and database, there is no guarantee that jobs are only executed once. Even first-time successful jobs.
• By default, each Sidekiq process has 10 threads, but you can adjust it with parameter or environment variable.
  • Should not go above 10, and 5 is recommended for Heroku 1x or 2x dynos.
• Sidekiq bundles connection_pool to limit database connections if necessary.

Redis

• Cloud-based Redis deployments can face errors from network timeouts. If this is the case, experiment with increasing the timeout which is defaulted to 1 second.
• Set maxmemory-policy noeviction in redis.conf so Redis doesn’t drop Sidekiq’s data silently.

Error handling and debugging

• Connect an error service to Sidekiq and then let Sidekiq catch job exceptions to take advantage of error reporting and automatic retry.
  • Don’t need to manually connect the error service in most cases. Each Sidekiq job spawns our entire Rails env for access to the Rails API, but it has the side benefit that normal error gems are already loaded in. Like how we get Rollbar errors for Sidekiq jobs already.
• Automatic retry follows exponential backoff according to (retry_count ** 4) + 15 + (rand(10) * (retry_count + 1)) which equates to 25 retries over 21 days. After that point, a job is moved to the Dead Job queue where it can be manually retried before being dropped after 6 months.
  • Configure max retries in sidekiq.yml or per job using the retry sidekiq_options in the Job Class
• To send TTIN to a worker process: Sidekiq::ProcessSet.new.each {|ps| ps.dump_threads }
• View rest of the Queue and Process API here
• View signals documentation here

Multiple queues

• Why?
  • Ability to stop one job type but keep others running.
  • Different priorities and time urgencies.
• Each queue can be configured with a weight for processing. Setting each queue with a weight of 1 will give random queue priorities.
• Define which queue in the Job Class by using the queue sidekiq_options
• Do not have many queues per Sidekiq process to avoid complexity (including exponential polling the Redis process for each additional queue). Instead, use multiple Sidekiq processes that handle different queues.

Logging

• Each job can use logger.info or logger.debug to access the Sidekiq logger

Sharding

• Not really necessary since Sidekiq is incredibly efficient (~25,000 jobs/second)
• After that point, view this doc

Extensions

• Some things like job retry, batching, and rate limiting are behind the Sidekiq paywall, but most things have an open source alternative.

Serial execution

For my own use case, serial execution is something unavoidable. I'll detail my approach for this in a different blog post. So long!