Feb 6
i have a tumblr queue problem which is that i keep raising the rate at which my queue drains but someone keeps queuing even more posts
seen from Russia
seen from South Korea
seen from Türkiye
seen from Poland
seen from United States
seen from United States
seen from Switzerland
seen from China
seen from Japan
seen from United States
seen from South Korea
seen from Türkiye
seen from South Korea
seen from South Korea
seen from Indonesia
seen from United States
seen from Japan
seen from South Korea
seen from Türkiye
seen from South Korea
i have a tumblr queue problem which is that i keep raising the rate at which my queue drains but someone keeps queuing even more posts
Unlock power of background jobs in Ruby on Rails with our simple guide. Explore key concepts, implementation and practices for optimal perfo
Sidekiq is a background job processor that helps Rails developers to increase the efficiency and responsiveness of an application. The brilliant Sidekiq gem provides built-in tools for testing the various aspects of your worker's lifecycle. The most important setting that you may choose as a develo
rails conf india 2017!
so i spoke at the indian edition of ruby annual conference here at kochi. more info on the conference here. wonderfully managed by emerging technology trust.
anyone wanting to ask or know more, please feel free leave your comments below. as promised you can find my talk here.
later aligator!
a balancer to manage your background jobs in rails!
well well, hello again! today seems to be a good day to post something - its been exactly 2 years since i started this blog & i think I have just the right hack that should help quite a bunch of you guys.
I know most of you rails developers must've heard (and used as well) about Sidekiq. Its the best option for people who want reliable background processing ability in their applications. Sidekiq is extremely simple to setup and get started (requiring just redis as its datastore). Now lets assume a hypothetical situation - you have two type of tasks that you want to put in background mode - email notifications & daily report computations which are really long running reports. It's usually normal if you run both of them in the "default" queue when you are just getting started and it works as well without any issues.
Fast forward a few months - your app has grown and so has the time taken for the report computation job to finish (with more data comes more computation cycles!). What this in-turn leads to is - your workers are occupied in generating these long running reports, whereas the jobs for email notifications are getting queued. Its okay if the emails aren't time sensitive; but if they are (like password reset email), then its an issue. You can obviously keep on spawning more workers, but it isn't scalable. Sidekiq to the rescue (not resque:P) here again! - you can just go ahead and create a different queue for your long running reporting jobs. What you are essentially doing is - creating 2 sets of workers; one who just keep on sending emails and the other who just work on generating reports. This way you can keep on scaling each set of workers individually depending on their individual load. you are sorted and happy!
Fast forward a few more months - your app is showing huge spikes during certain hours during which you are seeing lot of user activity leading to huge outflow of emails. your current set of workers are quite sufficient to handle the usual email flow, but during these peak hours, they just aren't enough. This leads to 2 things - users experiencing delayed responses from you and your redis might overflow due to large number of jobs being queued & stored if its size is relatively small( worst case scenario; but trust me it happens! personal experience:P) you can obviously spawn more workers for the time being, but you even have to kill them after the spike because well - they are idle doing nothing but eating your resources and increasing your infra bills.
We faced this very issue in our product (Sell.Do) wherein we saw huge temporary spikes for couple of minutes for which we could clearly see the need for more workers to handle the flow. During the initial days it was okay to manually write a script which would login to the server and spawn more workers. But it isn't really viable on a recurring basis for couple of reasons - what about weekends? and also major being we hackers hate doing stuff manually when it can be automated. and guess what - i wrote a sidekiq queue balancer which does this automatically for me!
It essentially uses a set of sidekiq apis & checks for size of a particular queue and its latency; compares it with healthy set of values expected for that queue (you might be okay if latency for welcome emails is upto 10mins, but password reset emails need to be sent within 60seconds). If it finds any mismatch - it spawns new workers which start draining your queue. Once the queue is within healthy limits, the manager again removes those extra workers and maintains a minimum set of workers which you have configured. cool right?
You can find the gist for it here. The library is still very young- there are many changes, optimisations (moving the config of queue meta data to an yml file etc) to be done. Feel free to write back if you have any doubts or need more help understanding it!
catch up soon! kcore out
ice_cube - Ruby Date Recurrence Library - Allows easy creation of recurrence rules and fast querying
recurrence { daily.hour_of_day(2).in_time_zone('Sydney') } So it could run a job for a client in Sydney at a specific time in utc, being aware of the effect of daylight savings as the date changes.