Kirill Zonov (aka graffzon) blog

var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-23872959-2']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })();

[graffzon-blog]

Resque manipulations

Yesterday I came to one production project and saw several Resque Failures. And it seems one of them blocked others, so the queue hungs for more than 20 jobs.

When trying to investigate the bug, we added to the queue a lot of unnecessary and repetitive tasks. In that case, just restart the queue - a bad option, because it'll execute all tasks. Also it's impossible to clean, and because there are important tasks that can't be lost and hard for manually adding.

It was decided to remove unnecessary tasks by hands.

Here I come upon the first disappointment - resque web cannot remove just one job, only everything. In documentation and in stack overflow I can't find any answers. Nowhere is information about how to do this from the console manually (maybe I bad googler, but anyway). By digging into the source code I discovered needed commands. So decided to share it, so that if you have to perform similar actions, it'll be easier to you.

First, make sure, that Redis works:

irb(main):> Resque => Resque Client connected to redis://127.0.0.1:6379/0

Nice. Can go forth. After we should take a look on our queues:

irb(main):> Resque.queues => ["orders_processing", "mailers"]

It's just simple Array of strings, so we can't work with it.

Next command was unnecessary for me, but can be helpful for you, if you want to list your workers:

irb(main):> Resque::Worker.all => [#<Worker our_project:9421:*>, #<Worker our_project:22184:*>]

Next command helps you when you want get list of active workers:

irb(main):> Resque::Worker.working => []

In my case all wokers dead =(

If you have queues separating by workers, you probably want to know, which ones belongs to whom:

irb(main):> Resque::Worker.all.first.queues => ["mailers", "orders_processing"]

If you want, you can get list of failures not in Resque-web, but from console, do that:

irb(main):> Resque::Failure.all => ..there is big array of errors..

It's clear about different Resque queues, workers and etc. So how we can get list of jobs for specified queue? As simple as chunky backon!

irb(main):> Resque.peek("mailers") => {"class"=>"Mailer", "args"=>["registered", {"approved"=>false, "created_at"=>"time", "email"=>"email", "id"=>"}]}

By default that command returns just one element. Don't worry, it has two more arguments for list length:

irb(main):> Resque.peek("mailers", 0, 20) => [{"class"=>"Mailer", "args"=>["registered", {"approved"=>false, "created_at"=>"time", "email"=>"email", "id"=>"}]}, {"class"=>"Notifier", "args"=>["thank_you", "second_param"]}]

Now we can see information for separate jobs and it's simple to remove what you want:

irb(main):> Resque::Job.destroy("mailer", "Notifier", ["thank_you"]) => 0

Ooops =( Nothing removed. Reason is not-fully specified params. It should be absolutely total.

irb(main):> Resque::Job.destroy("mailer", "Notifier", ["thank_you", "second_param"]) => 1

Now you can delete bad jobs from Resque. I hope this information will help you if you have to manually fiddle with Resque!

[graffzon-blog]

fun bug with citydom provider & ubuntu

Пост не технического содержания, посему пишу на русском.

С товарищем произошла забавная история. У него дома интернет дом.ру, подключенный напрямую к ПК (кто не знает, дом.ру привязывается к мак адресу). Вчера он взял себе ноут, попытался подключиться через кабель - у него ничего не вышло.

Заходим в него вимом и что же видим? 

Здесь необходимо немного отойти от повествования и рассказать, что же делает домру с компами с неверным мак адресом. Они их то ли редиректят, то ли просто вместо нужной страницы отдают свою со статусом 200.

Так вот, в его apt листах сплошь и поголовно лежали эти домрушные страницы. Проблема в итоге решилась просто, когда отсмеялись, удалили старые файлы листов и запустили апдейт заново, всё прошло на ура.

Мораль сей басни такова: ошибка порой бывает там, где ты её совсем не ждёшь =)

[graffzon-blog]

Lightweight handmade solution for jquery autocomplete and rails

Hi guys!

Again it was long lull in my blog and now I ready to write small post.

In my current project I had a need to use autocomplete tool but with very specific query to db. Certanly, all you know about excellent gem https://github.com/crowdint/rails3-jquery-autocomplete But there is two ways in which you won't be comfortable using it.

When you (like me) want to use heavy and complicated query for search suggestions

When you want simple solution and don't want unnecessary features from gem 

When I tried to write it with my hands, I found that documentation of jQuery UI for autocomplete plugin is incomplete and it took a while to sort out and find "how it should work". Therefore, I decided to write small manual to you, so now you can do it quick and easy.

So, lets start.

First thing you should do is of course to add jquery-ui to application.js. 

//= require jquery //= require jquery_ujs //= require jquery-ui

Next create controller for your autocomplete resource

class TagsController < ApplicationController def autocomplete_tag_name end end

After that add route for your search action

resources :tags, only: [] do get :autocomplete_tag_name, :on => :collection end

Now it's time to go into frontend. In your view you should have text field, to which you apply autocomplete:

= f.input :tag_list, input_html: { value: '', class: 'tags_with_autocomplete', 'data-autocompleteUrl' => autocomplete_tag_name_tags_path }

In that example I'm using https://github.com/plataformatec/simple_form to build form, so if you using another builder, just follow it's syntax. Here we added input field with specified class and html5 attribute - data, where we placed our autocomplete action url.

What about js? Few words about it. I guess, all of you know that placing js code into application.js is a way to hell. Don't do that. I prefer to create separate file (in my case it common.js.coffee) and connect it in application.js. So next code I'll place into common.js.coffee:

$ -> $('.tags_with_autocomplete').autocomplete minLength: 2 source: (request, response) -> $.ajax url: $('.tags_with_autocomplete').data('autocompleteurl') dataType: "json" data: name: request.term success: (data) -> response(data)

What do we do here? Lets see step by step. We have jquery function autocomplete, in which we pass necessary params (full list of params you can see on http://jqueryui.com/demos/autocomplete/). First we set minLength params. It specifies the minimum number of characters a user has to type before the autocomplete activates. I think, that 2 is good choise, but you can set it to 3. Next param is essential to us. It defines from where script should take suggestions for autocomplete. Certanly best choise is using remote url with JSON response. Source param takes two arguments - request and response. First - is object which you take from user's input. Second - what you should return. So we make ajax request to our server on url specified in url data attribute of our input. In data you can use another param name, but I prefer to use name. Request object has attribute term, in which stores user's string from input. We will give it into params to our server. And really important thing here is success function. It's standart js function which activates when response from server returns with status 200 (OK). And here we should call special autocomplete function response with json data returned by server. Don't forget that, it was little thing, that I didn't write, so nothing worked. 

And last you should make is controller specific code. In my simplified case it looks like:

class TagsController < ApplicationController def autocomplete_tag_name tags = Tag.select([:name]).where("name LIKE ?", "%#{params[:name]}%") result = tags.collect do |t| { value: t.name } end render json: result end end

Few words about that. First we select only name column, because I need search only on it, and it will give us good speed up. Next where clause with like statement. After that we need to form our result to needed format. Jquery autocomplete takes such hash {id: 1, value: 'value', title: 'title'}. Field id is unnecessary, field title too (in that case it will use value instead of title), so I form it to {value: 'value'} hash. That's all.

Now you can use your own autocomplete system and use any complex queries and sorting in your action. Or you can get rid of the excess gem, by adding just one controller method and few js lines. 

Enjoy it!

[graffzon-blog]

Fast async hit counter for your rails app

Hi all!

I hav not written anything for a long time. And now I want to show you solution to implement hit counter in your rails app. Certanly you can just increase field of your model inside your controller, but if you have high attendance of your site, it will cause perfomance problem. 

So my decision is using redis with cron. 

Let's take a look on what we have.

Model Event with field visits; controller EventsController with show action.

First you should install redis on your machine. It's great describet here http://redis.io/topics/quickstart Next you should check that redis installed.

Try to call redis-server inside your console. With default params it will start redis server insite that window. In another console window you can run redis-cli and play a little. Redis is really greate DB and it has really many commands which you can learn on redis website.

So, redis installed, what next. Go into your gemfile and add redis gem there. Now last is 3.0.1

gem 'redis', '~> 3.0.1'

Make bundle install. Next if you want (and you want) to connect to redis in start of your rails server you should create

config/initializers/redis.rb

file with:

$redis = Redis.new(:host => 'localhost', :port => 6379)

This will connect to redis db, started at your machine on default port. Now you can start rails console and play a little. Redis-rb commands fully identical with manual redis commands. So you can just try (don't forget to start redis-server):

1.9.2p290 :001 > $redis.set('foo', 'bar') 

=> "OK" 

1.9.2p290 :002 > $redis.get('foo') 

=> "bar" 

1.9.2p290 :003 > $redis.hset('chunky', 'backon', 1) 

=> true 

1.9.2p290 :004 > $redis.hget('chunky', 'backon') 

=> "1" 

1.9.2p290 :005 > $redis.hincrby('chunky', 'backon', 2) 

=> 3 

1.9.2p290 :006 > $redis.hget('chunky', 'backon') 

=> "3"

Last four commands works with redis Hashes. I said that Redis is fucking cool?

So it's all nice, but let's return to our business task. What we want to do:

1) User visit our event page

2) We check his session (or cookie if you want)

3) If he has appropriate key we do nothing (his visit already counted)

4) If not - set that key for him and add his visit to redis

5) We should have cron task which in some interval (f.e. 1 minute) check if redis have some event visits

6) If it is - cron should call Event's increase method. Let's continue. Here is what we have in EventsController:

def show

  increase_visits_if_needed

end

private

def increase_visits_if_needed

  unless session[visit_event_key]

    session[visit_event_key] = true

    $redis.hincrby("event_visits", resource.id, 1)

  end

end

def visit_event_key

  (resource.url + '-visited').to_sym

end

I think, there is all clear. Redis method hincrby can increase unpresent key too.

So visit added and next we should have cron task. For that I recommend to use Whenever gem. It gives nice interface to add your ruby tasks in cron.

So add it into your gemfile and make bundle install:

gem 'whenever'

Next make

wheneverize .

inside your console. It will create

config/shedule.rb

file for you. There we can add our ruby code.

every 1.minute do

  runner "Event.async_increase_visits"

end

Also you can specify some options, which you can find on https://github.com/javan/whenever/

Next check that everything works all right (in terminal):

master ~/projects/present_work/my_project $ whenever

* * * * * /bin/bash -l -c 'cd /Users/kirillzonov/projects/present_work/my_project && script/rails runner -e production '\''Event.async_increase_visits'\'''

## [message] Above is your schedule file converted to cron syntax; your crontab file was not updated.

## [message] Run `whenever --help' for more options.

This shows you that your task interpreted successfully.

Next add that method in your Event model:

def self.async_increase_visits

  $redis.hkeys('event_visits').each do |id|

    self.find(id).increase_visit($redis.hget('event_visits', id))

    $redis.hdel('event_visits', id)

  end

end

def increase_visit(visits)

  visits.to_i.times { self.increment!(:visits) }

end

Here we go through each redis key for event visits and increase event's visits by given number of visits. After that we clear that keys. So that's all. We have method for async increasing your visits. You can check how it works in your rails console:

1.9.2p290 :018 > Event.last

=> #<Event id: 29, created_at: "2012-06-07 09:42:15", updated_at: "2012-06-18 23:32:02", visits: 0> 

1.9.2p290 :019 > $redis.hincrby("event_visits", 29, 1)

=> 1

1.9.2p290 :020 > Event.async_increase_visits

=> ["29"] 

1.9.2p290 :021 > Event.last

=> #<Event id: 29, created_at: "2012-06-07 09:42:15", updated_at: "2012-06-21 10:49:04", visits: 1> 

1.9.2p290 :022 > Event.async_increase_visits

=> []

Yeah, that's cool. Now you need just compile and run your cron task:

whenever --update-crontab my_project

That's all! You have working cron task for async visits increasing. If you change some of your shedule.rb code, you should just run the same command.

If you want to specify evnironment (f.e. development or staging) you should run:

whenever --set environment=development --update-crontab my_project

Try and check. All should works perfect.

Last what you need - is correctly deploy. First you should add

/etc/redis.conf

file with daemonized option (in development env I use non daemonized redis server, but you can do what you want) in your production server:

daemonize yes

pidfile your_project_folder/shared/pids/redis.pid

port 6379

timeout 300

databases 1

dbfilename dump.rdb

dir your_project_folders/shared/

appendfsync everysec

vm-enabled no

You also can make your custom conf file, just look at http://redis.io Last, what I made for my deploy.rb (capistrano):

before :deploy do

  run "/usr/bin/redis-server /etc/redis.conf"

end

namespace :deploy do

  task :restart_redis do

    run "/usr/bin/redis-server /etc/redis.conf"

    run "cd #{deploy_to}/current; script/rails runner -e production 'Event.async_increase_visits'"

    run "/usr/bin/redis-cli shutdown"

    run "/usr/bin/redis-server /etc/redis.conf"

  end

end

after :deploy do

  run "cd #{deploy_to}/current; bundle exec whenever --update-crontab tixis"

end

Anyway, whenever has own cap task, but it doesn't work for me.

Here I start redis-server before deploy (if it's already started - nothing will happen). And after deploy I update cap tasks. For redis restart I made separate cap task because we don't needed to restart it each time.

That's all, now You have working async hit counting!

[graffzon-blog]

How to calculate sha1 for part of file

In one task I needed to calculate sha1 for first 1mb of my file.

Here is simple solution:

opened = File.open(path) Digest::SHA1.hexdigest opened.read(1048576)     # 1mb.

That's all =)

[graffzon-blog]

Segfault when install 1.9.3 on OSx

When I tried to install ruby 1.9.3 on my mac lion with openssl 1.0.0e I take segfault error.

And here you can take a look, how to avoid this.

rvm remove ruby-1.9.3 export PATH=/usr/bin:/bin:/usr/sbin:/sbin:$HOME/.rvm/bin rvm pkg install iconv rvm pkg install openssl rvm install ruby-1.9.3 --with-openssl-dir=$rvm_path/usr --with-iconv-dir=$rvm_path/usr

Based on Chris Irish post http://www.christopherirish.com/2011/09/02/ruby-1-9-2-segmentation-fault-and-openssl/

[graffzon-blog]

[graffzon-blog]

Nice dynamic scope by status

STATUSES = %w(pending available sold) STATUSES.each do |status| class_eval "scope :#{status}, where(status: :#{status})" end

[graffzon-blog]

[graffzon-blog]

Little notes about multistaging nginx + unicorn

I made this task for quite some time, so I can not reproduce the process in order, so I will describe just how it should looks in the end.

Why Unicorn? It was a rails app, which didn't have mobile interface so all clients are fast. In plans it shouldn't be really highloaded (as I can see now less than 1000 requests per day) so fast and light unicorn is best choise, I think. 

Also we should have 2 rails app running on one physical server (production and staging). So, how I made this:

/etc/nginx/nginx.conf

user www-data; worker_processes 4; pid /var/run/nginx.pid; events { worker_connections 512; } http { ## # Basic Settings ## sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048;

include /etc/nginx/mime.types; default_type application/octet-stream; ## # Logging Settings ## access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; ## # Gzip Settings ## gzip on; gzip_disable "msie6"; ## # Virtual Host Configs ## include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; }

Two files in sites-enabled directory:

/etc/nginx/sites-enabled/staging

server { # We will take access to staging by that port listen 4321; server_name plsnr.ru plsnr.com *.plsnr.ru; access_log /home/user/var/www/staging/shared/logs/access.log; error_log /home/user/var/www/staging/shared/logs/error.log; # Максимальный размер тела запроса клиента client_max_body_size 101M; location ~ /.svn/ { deny all; } location ~ /.git/ { deny all; } location ~ /\.ht { deny all; } location / { root /home/user/var/www/staging/current/public; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect off; if (!-f $request_filename) { proxy_pass http://127.0.0.1:3001; break; } } error_page 500 502 503 504 /500.html; location = /500.html { root /home/user/var/www/staging/current/public; } }

/etc/nginx/sites-enabled/plsnr.ru

server { # listen default http port listen 80; server_name plsnr.ru plsnr.com *.plsnr.ru; access_log /home/user/var/www/production/shared/logs/access.log; error_log /home/user/var/www/production/shared/logs/error.log; # Максимальный размер тела запроса клиента client_max_body_size 101M; location ~ /.svn/ { deny all; } location ~ /.git/ { deny all; } location ~ /\.ht { deny all; } location / { root /home/user/var/www/production/current/public; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect off; if (!-f $request_filename) { proxy_pass http://127.0.0.1:3000; break; } } error_page 500 502 503 504 /500.html; location = /500.html { root /home/user/var/www/production/current/public; } }

We have no any specific unicorn config file and identical RAILS_APP/config/environments/production.rb and RAILS_APP/config/environments/staging.rb

Plsnr::Application.configure do config.cache_classes = true config.consider_all_requests_local = false config.action_controller.perform_caching = true config.action_dispatch.x_sendfile_header = "X-Sendfile" config.serve_static_assets = false config.action_mailer.raise_delivery_errors = false config.action_mailer.default_url_options = { :host => 'plsnr.ru' } config.action_mailer.delivery_method = :ses config.i18n.fallbacks = true config.active_support.deprecation = :notify Paperclip.options[:command_path] = "/usr/bin" end

It's all about nginx+unicorn configuring. Next step is deployment. Certanly, capistrano.

RAILS_APP/config/deploy.rb

require 'bundler/capistrano' require 'capistrano/ext/multistage' set :stages, %w(production staging) set :default_stage, "staging" set :domain, "[email protected]" set :backup_dir, "/home/ouruser/var/www/production/backup" set :use_sudo, false set :application, "plsnr" set :scm, :git set :repository, "[email protected]/plsnr.git" set :deploy_via, :remote_cache role :web, domain role :app, domain role :db, domain, :primary => true after 'deploy:update_code', :roles => :app do run "ln -s #{deploy_to}/shared/public/uploads #{current_release}/public/uploads" backup end # backup db task :backup, :roles => :db, :only => { :primary => true } do filename = "#{backup_dir}/#{application}.dump.#{Time.now.to_f}.sql.bz2" text = capture "cat #{deploy_to}/current/config/database.yml" yaml = YAML::load(text) on_rollback { run "rm #{filename}" } run "mysqldump -u #{yaml['production']['username']} -p #{yaml['production']['database']} | bzip2 -c > #{filename}" do |ch, stream, out| ch.send_data "#{yaml['production']['password']}\n" if out =~ /^Enter password:/ end end namespace :deploy do task :start do run "cd #{deploy_to}/current && bundle exec unicorn_rails -E #{rails_env} -p #{unicorn_port} -D" end task :stop do run "kill `cat #{unicorn_pid}`" end task :graceful_stop do run "kill -s QUIT `cat #{unicorn_pid}`" end task :reload do run "kill -s USR2 `cat #{unicorn_pid}`" end task :restart do stop start end end

RAILS_APP/config/deploy/production.rb

set :branch, "master" set :rails_env, "production" set :unicorn_port, "3000" set :deploy_to, "/home/user/var/www/production" set :unicorn_pid, "#{deploy_to}/shared/tmp/pids/unicorn.pid"

RAILS_APP/config/deploy/staging.rb

set :branch, "staging" set :rails_env, "staging" set :unicorn_port, "3001" set :deploy_to, "/home/user/var/www/staging" set :unicorn_pid, "#{deploy_to}/shared/tmp/pids/unicorn.pid"

Thats all! If I haven't forgotten anythink %)

So production we have on

http://plsnr.ru/

and staging on

http://plsnr.ru:4321

P.S. Yes we needed one domain name %)

P.P.S. Site name isn't real

Good post about Unicorn (if you want to customize it) https://github.com/blog/517-unicorn

[graffzon-blog]

[graffzon-blog]

Презентация с семинара по функциональному языку программирования Haskell. Казань, 2012.

[graffzon-blog]

In one project I worked up, was the man who wrote such migrations. *facepalm*

[graffzon-blog]

Golden hill example (задача о Золотой горе)

Script for programming task "golden hill (exercise of recursion and dynamic programming)" written on Ruby

- At the input we have txt file with integer numbers. In file N rows. In row i: i elements.

- If in row i we have number a[i,k], so in next row adjacent will be numbers: a[i+1,k] and a[i+1,k+1].

- Path starts at single number at first row and ends at last and runs through adjacent numbers (maximum paths is 2^(N-1)).

- Calculate max sum path and show it.

At the input we have file input.txt. At the output we should see max sum and sequence of indexes of elements, which made that path.

This task has been at the international competition for programming in 1994. Here is Ruby realization.

I used that formula: 

So here you can take a look to my realization:

https://github.com/graffzon/golden_hill_exercise

[graffzon-blog]

How to work faster

http://www.targetprocess.com/blog/2012/01/faster-faster-faster.html

[graffzon-blog]

Comparsion of Haml, Slim, Erb.

So, it's my first post in english, will try. Sorry english grammar nazi =)

I think, everybody, who working with Rails, know that words. Here I will try to compare speed of that templating engines. First I will try simple template, after more difficult.

For testing I will use Tilt (1.3.3).

Also:

- Ruby 3.0.3

- Haml 3.1.4

- Slim 1.0.4

1. ERB

If you have just-installed rails app you will have that engine. It's simple and fully html-like. Here template which we will test:

<h1> Just simple erb template</h1>

<p>there is no ruby code or smth else, just text</p>

2. Haml

For me it was first alternative of erb. And as I see many newbie developers also using that engine. So what we will test:

%h1 Just simple erb template %p there is no ruby code or smth else, just text

3. Slim

I think I can call it light-weight haml. Same idea, but more effective. 

h1 Just simple erb template

p there is no ruby code or smth else, just text

Lets see (making 100 requests and take average value):

Haml - 2.7080000000000015e-05

Slim - 0.9199999999999997e-05

ERB - 1.126999999999999e-05

As we see in that test Slim is fastest (for static pages). And as we know Haml is sloooowest. Remember, I used tilt, it's not development or production mode.

So let's try to test templates with part of ruby code:

1. ERB 

<h1> Just simple erb template</h1> <p>there is no ruby code or smth else, just text</p> <% 100.times do %> <%= rand(1000) %> <% end %>

2. Haml

%h1 Just simple erb template

%p there is no ruby code or smth else, just text

- 100.times do

= rand(1000)

3. Slim

h1 Just simple erb template

p there is no ruby code or smth else, just text

- 100.times do

= rand(1000)

Again 100 requests and average value:

Haml - 0.00026751000000000005

Slim - 0.00014951999999999994

ERB - 0.00012785999999999993

Here ERB gave better result than Slim. But Haml again somewhere in ass ;)

I think, that benchmark will give answer for question "what is faster???"

More information:

1. Presentation about haml Haml vs erb

2. You can take a look for full benchmark on https://github.com/stonean/slim

[graffzon-blog]

Обратите внимание на последнюю строчку 0_о