Turing 087 Background Workers (a.k.a DAEMONS)
Daemons can be very useful (or scary apparently). For a fun read, both Katrina Owen and myself recommend
In the real world, it is often a good idea to run background workers to perform tasks that would otherwise bog down a server or a user’s experience. A common example is sending off an email when an event happens. Right after the event is triggered, without a background worker, the user would have to wait for the process to complete before continuing on with browsing/usage.
Sending the task off to a background worker to complete allows the user to continue their activity uninterrupted.
While it is certainly possible to write your own background workers and it may help with managing dependancies. As Daemon newbies, lets start off with a gem to help for that.
Enter sidekiq. They tout easy to use and very fast workers using client APIs, server APIs, and Redis. Plugging it in is simple enough.
gem ‘sidekiq’
create your task you want the worker to perform. Note: create one class for each different worker task you are generating. Include Sidekiq::Worker. Then create the perform method. For sidekiq it is always called perform, so do not alter this.
class NameOfTask include Sidekiq::Worker def perform(args) # stuff to do end end
in your controller (or where you are calling the worker task if you are not necessarily using Rails) assign the task. .perform is an instance method, and any of the perform_later options are class methods (NameOfTask.perform_later).
Sidekiq recommends if your tasks are using lots of database calls, to turn up your pool in your database configuration to accommodate asynchronous activity.
make sure you have bundled your gem, and then run bundle exec sidekiq.
To start running the workers, it is helpful to have multiple tabs open on your terminal.
In the first tab, start your Redis server: redis-server
then the client side in tab 2: redis-cli
and finally in a third tab, start sidekiq: sidekiq
The sidekiq server will pull the jobs from the queue in Redis. It will run the tasks as defined in your perform method.
Keep in mind that in a real world operation, there might be multiple different tasks from multiple different job classes running concurrently. Keep the tasks simple and make sure they point to a persistent source of information and not something that could be in flux. Limit the responsibility and power of each worker. Keep yourself protected!
Once you get the hang of what a worker does, it is possible to build some rake tasks that listen for activity or operate on a given schedule if you don’t want to use sidekiq.