#simpleworker

[honryou-simpleworker-blog]

Pushed 0.3.0 Gem: Much More User Friendly

After using SimpleWorkr for a long time now, I always think to myself, "there's gotta be some way to make this more intuitive". I want it to feel more like it's just like running your normal code without even really realizing that the work is actually offloaded to the cloud. So with that in mind, the gem now has a much more intuitive usage pattern, more like ActiveRecord. The old way:

# Create a Service object @service = SimpleWorker::Service.new(@access_key, @secret_key) # Upload code @service.upload(worker_class_name, path_to_file) # Queue up jobs @service.queue(worker_class_name, data_for_job) # Or schedule jobs @service.schedule(worker_class_name, data_for_job, schedule) # Get status of jobs @service.status(task_id)

Now, here's the shiny new way of doing it: Some one time initialization:

SimpleWorker.configure do |config| config.access_key = @access_key config.secret_key = @secret_key end

Now throughout your app, you can do the following:

# Instantiate a new worker worker = MyWorker.new # Queue directly from the object worker.queue # Schedule directly from the object worker.schedule(my_schedule) # Check status directly from it as well worker.status

All data defined in instance variables in your worker will be sent along and usable in your run method. Also, your code will *automatically* be uploaded if it has been changed!

[honryou-simpleworker-blog]

Including other files in your Worker with "merge"

Previously, your worker file/class had to be self contained (ie: single file including all code for the worker). Now you can use the new "merge" class method for instance:

class TestWorker2 < SimpleWorker::Base merge "models/model_1.rb"

This will automatically upload the merged files to the SimpleWorker service and 'require' them in your Worker class. If you want to use the special methods like "log" in your merged classes, use the SimpleWorker::UsedInWorker mix-in, for instance:

class Model1 include SimpleWorker::UsedInWorker

That's about it, the rest will be taken care of for you. If you want to kick off other workers from within the first worker, use merge_worker instead:

class TestWorker2 < SimpleWorker::Base merge "models/model_1.rb" merge_worker "my_other_worker.rb", "MyOtherWorker" # second parameter is the class name

The merge_worker does things a bit differently so it can be handled properly after uploading to the worker cloud.

[honryou-simpleworker-blog]

Parallelizing Ruby on the Cloud

So I'm sure we've all had the need to want to run multiple threads at once to optimize a part of our applications. There are several options like spinning up new Ruby threads (example from):

pages = %w( www.rubycentral.com www.awl.com www.pragmaticprogrammer.com ) threads = [] pages.each do |page| threads << Thread.new(page) { |myPage| h = Net::HTTP.new(myPage, 80) log "Fetching: #{myPage}" resp, data = h.get('/', nil ) # DO SOME PAGE PROCESSING AND STORE STUFF IN DATABASE } end threads.each { |aThread| aThread.join }

Or you could use a library like delayed_job that works great after a bit of database configuration and starting up the job process from the command line.

But now there is another option that is actually easier and much more scalable and that is sending jobs off into the cloud using SimpleWorkr. Here is an example of the above using SimpleWorkr:

# First we create our Worker class WebPageWorker < SimpleWorker::Base attr_accessor :url def run h = Net::HTTP.new(myPage, 80) log "Fetching: #{myPage}" resp, data = h.get('/', nil ) # DO SOME PAGE PROCESSING AND STORE STUFF IN DATABASE end end # Now lets fire some of those workers up! pages = %w( www.rubycentral.com www.awl.com www.pragmaticprogrammer.com ) pages.each do |page| worker = WebPageWorker.new worker.page = page worker.queue end

That's it. Easy like Sunday morning.

[honryou-simpleworker-blog]

SimpleWorker Usage Pattern Graph

SimpleWorker is a job queuing and scheduling system so while a lot of work comes in at random times usually based on some event in their system (ie: a user clicks a button), there are a lot of scheduled jobs too and those scheduled jobs have a huge affect on the capacity requirements for SimpleWorker. Check out this graph of CPU usage for the past 24 hours across the entire SimpleWorker server farm.

The big spikes start on the hour every hour and this is because most developers schedule their recurring jobs on the hour, not somewhere in the middle of the hour. It would be much better for us if they did so it could spread the load evenly and we could make better use of the unused time and perhaps we will provide incentive to do that at some point, but as it stands, we always have to have enough capacity for the big spikes.

Another thing to note about this information, is that it clearly shows one of the value propositions SimpleWorker provides to our customers. If their jobs only run for a few minutes each hour, they will only pay for those few minutes, not the full hour where the majority of that is unused time.

[honryou-simpleworker-blog]

Autoload Rails Models for Super Easy Rails Integration

SimpleWorker will now include all the models in your Rails app automatically so you don't have to merge them manually. Just use them almost as if they were to be run locally and you're good to go. eg:

class MyWorker < SimpleWorker::Base attr_accessor :user_id def run # User model will automatically be available, so just use it as you normally would. user = User.find(user_id) ...... end end

[honryou-blog]

Autoload Rails Models for Super Easy Rails Integration

SimpleWorker will now include all the models in your Rails app automatically so you don't have to merge them manually. Just use them almost as if they were to be run locally and you're good to go. eg: class MyWorker < SimpleWorker::Base attr_accessor :user_id def run # User model will automatically be available, so just use it as you normally would. user = User.find(user_id) ...... end end