Adam Gotterer - Ramblings of a hacker and founder @agotterer - Tumblr Blog

PG::InFailedSqlTransaction: ERROR: current transaction is aborted, commands ignored until end of transaction block

Was an error I encountered from postgres while trying to resque Sequel::UniqueConstraintViolation. In theory it should have caught the exception and moved on. For whatever reason the transaction seemed to remain opened which caused subsequent queries to fail. Which is all the more odd because there was no transaction explicitly declared. My solution was to wrap the request in a transaction and roll back when that error was encountered.

self.db.transaction(savepoint: true) do begin # Query! rescue Sequel::UniqueConstraintViolation raise(Sequel::Rollback) end end

#ruby #sequel

Omniauth Google force approval prompt

at some oYour config settings may have approval_prompt set to auto which will get you a refresh_token the first time a user authorizes. But you may have a need to get a new refresh_token at some other point in time. With approval_prompt set to auto you don't get new refresh token every time. Unfortunately the obvious argument of approval_prompt was deprecated in favor of prompt, which appears to not be documented anywhere. Here's how you can force the approval prompt from the url without modifying the config settings:

user_omniauth_authorize_path(:google_oauth2, { prompt: 'consent' })

Countries Database

I was really shocked to discover that I couldn't find a comprehensive database with all the countries in the world with full names, iso alpha1, iso alpha2, country code, latitude, and longitude in a reasonable amount of time. I found a ton of lists with various pieces, but non that were complete. Here's a combined version of that list.

https://gist.github.com/adamgotterer/6257755

rake spec is slow

I spent some time debugging rake spec today to try and figure out why it seemed really slow compared to running other rake tasks (and its not because of my tests, those are actually fast). My observation was that the application took a long time to load. So I set out to do some debugging. Durring my benchmarking I realized that the application was getting loaded twice. The first time to load rake and the second time when an external call is made to ruby to run spec. The time through rake is twice as slow.

rake spec: ~27 seconds ruby spec: ~12 seconds

I posted this as a bug on the rspec-rails github. For now the work around is to call rspec directly and skip rake.

/usr/local/rvm/rubies/ruby-1.9.3-p194/bin/ruby -S rspec spec/

Update: This is not an rspec issue. It's a problem with rails and the way environments work. This problem also affects unit/test and minitest. The default rails environment is dev and since tests need to run in a test environment it has no choice but to run as a new process, thus loading it a second time. Thanks @dchelimsky.

#rails #rake #rspec #spec #benchmark #testing

Rails postgres db encoding issues

This error has given me quite the headache today and I could barely find any worthwhile information when searching around. Hopefully this post saves someone the same frustration.

PG::Error: ERROR: encoding "UTF8" does not match locale "en_US" DETAIL: The chosen LC_CTYPE setting requires encoding "LATIN1".

I was running into this when running rake db:create or rake db:setup. I'm not positive, but I believe the issue is that the template and init settings I used to create postgres are different then the rails default. I started messing around with my database.yml file and came up with these additional options:

encoding: utf8 template: template0 collate: en_US.UTF8 ctype: en_US.UTF8

The rails create_database method took the encoding and template settings but was ignoring the lc_ctype and lc_collate setting which was needed to get past that error. After looking at the rails create_database code I realized that as of rails 3.2.8 there is no support for those options. Fortunately someone has added a pull request which adds support and will hopefully be part of the next rails release. As a temporary work-around I've updated the create_database method with the new code.

#rails #ruby on rails #ror #postgres #rake

Twitter bootstrap: Lock the nav bar to top of screen when the page scrolls (CoffeeScript)

Apparently the Twitter bootstap navbar doesn't natively support locking to the top of the screen when the page scrolls. A little bit of googling and I found a lot of solution. For some reason everything I found was dirty and long. Most of them didn't work at all. Then I realized I was spending more time searching then it would take to just write it myself. This was the shortest solution I could come up with. It's in CoffeeScript because thats what I'm learning this week :)

$(document).scroll -> nav = $('.navbar') return if nav.hasClass('navbar-fixed-top') if (nav.offset().top - nav.outerHeight() <= $(this).scrollTop()) nav.addClass('navbar-fixed-top') else nav.removeClass('navbar-fixed-top')

Hacking the .SVN directory (Archive)

This comes from a blog post I wrote on 01/26/2009. I see people are still searching for it and landing at my old site. Unfortunately I can't redirect the URL. But here's the original content in all its glory. Kind of funny to look back at this almost 4 years later. Especially since I haven't used svn in 3 :)

The other day on College Humor and Bustedtees we discovered a fairly serious security vulnerability. Fortunately because of the layout of our code nothing malicious could be exploited (more in another post). We thought our “push” script was skipping .svn folders, it turned out to not be operating correctly.

The hack is simple, documented and easily overlooked. Once the vulnerability was found, I did my best to exploit the shit out of it. I did so very successfully. I even tried it on some other popular websites and was able to access files I should have never been able to. In one instance I gained limited access to a sites admin. I emailed all of these sites to notify them of the security vulnerability. They were most gracious, once company even sent me a gift card!

The hack obviously starts in .svn directory, specifically at the entries file. You can access this file by browsing to:

somedomain.com/.svn/entires?

This document contains all of the files and folders svn manages in that directory. In some instances you can locate admin directories and the same thing applies…

somedomain.com/admin/.svn/entries?

So at this point all you have are a bunch of file names. Sometimes you can get some fun information and access to files that were meant to be hidden. Security by obscurity is not a solution, protect files you don’t want the public to access!

Now this is where things get interesting… Any file that has been checked in I can now execute. Either directly or through an svn folder that holds file revisions. Pick any file in the list and browse to:

somedomain.com/.svn/text-base/filename.php.svn-base

In this example the PHP file will be put through the PHP parser and executed. The results really depend on the layout of the code. Depending on the way the coder uses includes/requires decides on how much access and what kind of output you get. If a file is included using a relative path, the includes won’t be included since your working directory is the text-base dir. If they are using absolute paths, includes will continue through the execution. In one of the sites I poked around in, I found their admin wrapped through some kind of lite template/framework. I was able to bypass the system and go directly to the file without a using password. From there I had limited additional actions, but I still gained access to where I wasn’t welcome.

To do some additional testing I setup a test site to play with other file types. I found that files without a PHP extension, for example .inc files were NOT parsed and instead the contents were spit out to the page. In this test case the .inc file contained passwords and locations to databases. The possible additional damage I could cause from here is endless…

I’m not the first one to discover this hack, although a quick search only revealed obvious prevention methods. Protecting your site is really simple. Add this to your htacces file:

RewriteRule (\.svn)/(.*?) - [F,L]

Another option is blocking .svn folders through your web server config file for all sites.

Update A number of people have mentioned a better prevention technique… They recommend doing an SVN export instead of a checkout or rsync. This was something I thought about after discovering the exploit. But I am by no means a system admin or the person who deals with that stuff at work. I’m glad these people were able to confirm that idea. Thanks!

#hacking #svn #.svn #security

Mountain Lion python command line issues

Looks like Mountain Lion wipes out your site-packages folder. You'll need to sudo easy_install pip and then use pip to reinstall any packages that were wiped out.

How to be a successful tech recruiter: read the resume and proofread

In a given day I receive at least one recruitment email. Sometimes more, lots more. I'm generally very cordial to recruiters. I know they are just doing there job. Usually I'll be pleasant and write back a short note telling them that I'm not considering new opportunities. I honestly believe that you never know when your paths might cross. Also this sometimes limits the number of follow up messages.

But every once in a while I'll receive a message that does't respect me or my time. If your job as a recruiter is to find talent and get them excited about jumping ship to a "better" opportunity, you can't be sending out messages like this:

Adam, Your background looks great and I Have an Awesome Linux Infrastructure Engineer role Mid level 90-140k plus great bonus up to 60% wanted to see if you or any of your friends would be intrested, This is a Great laid back cutting edge environment. Hours 8:30-5:30pm. Great Benefits and perks.. Top Buy & Sell side Financial Services Firm on Park Ave, Your search is confidential. If interested or you know a friend you can send updated word resume and exact comp asap to: [email protected] Interviews are immediate,

I have quite a number of problems with this message. Let's review:

"I Have an Awesome Linux Infrastructure Engineer role Mid level" - To start, besides a footnote or two mentioning Linux usage. I've never worked as a systems engineer. Second, i'd say my most recent experience has been at a fairly senior level. Not entirely sure why you'd think I would want a mid level position after last being a manager? These things would be obvious if you read my resume. I'm just going to assume that a few keywords matched and he sent me this canned message. I guess spray-and-pray is a legit business model.

I don't claim to be an English major or have the greatest grammar and spelling. But my god the whole message has horrible grammar, spelling errors, weird capitalization and comma usage throughout. This is a horrible first impression. A horrible first impression for you personally, for the company you're representing as well as the recruiting firm you work for. This is also a sign of what it would be like to have to interact with you.

"Top Buy & Sell side Financial Services Firm" - Not a single thing on my resume says I would be a good fit at a Financial services firm. I've never worked in finance. Actually most of my resume is consumer web, media and commerce.

"90-140k plus great bonus up to 60%" - Seems like a nice salary for mid level. But that range is all over the place. $140K seems pretty senior to me. But it's nice that you show a range at all.

Last but not least. And this is an offense for almost all recruiting attempts. There is no mention of the company name! No good software engineer (or systems engineer) is going to be so excited by a generic / poorly written / butchered description that they MUST email you back to find out more. No one cares that its a top mobile social geo enabled consumer facing app that just closed another round. We want to know what company it's for. Plain and simple.

What's the takeaway here? If you want to be a successful tech recruiter, start with reading the resume and proofreading your email.

#recruiter #recruiting #tech recruiter #technology recruiter

Exposing Grape entities follow up

After spending some more time thinking about exposing Grape entities. I decided to write a mixin to make it easier. Below is the class and under that the usage. The mixing adds #expose_all and #hide methods. #expose_all can take an optional arguments option like #expose takes. #hide can be used to explicitly prevent an entity from being shown. Order matter when using these methods. The usage section should make it clear.

module EntityMixin def self.included(base) base.extend ClassMethods end module ClassMethods def expose_all? @expose_all ||= false end def expose_all(*args) @expose_all = true @expose_all_options = args.last.is_a?(Hash) ? args.pop : {} end def expose_all_options @expose_all_options ||= {} end def hide(*args) @hide ||= [] args.each { |attr| @hide.push(attr.to_sym) } @hide end end def initialize(object, options={}) # If expose_all? is set to true # run through all the objects attributes and # add send them to #expose. Options will be passed, blocks are not allowed. # If an attribute has already been exposed directly, expose_all # will ignore the attribute and not overwrite it. if self.class.expose_all? args = object.attributes.keys.map { |attr| attr.to_sym } args.each do |attr| if !self.class.exposures.include?(attr) self.class.expose(*[attr].push(self.class.expose_all_options)) end end end super(object, options) # Hide elements by removing them from the exposures list self.class.hide.each { |attr| self.class.exposures.delete(attr) } end end

Usage:

class User < Grape::Entity include EntityMixin expose_all # Must go first, calls to expose overwrite default behavior expose :email, :if { :type => 'admin' } hide :password # hides must go last to remove exposures end

Exposing entities in Grape by default

Entities in Grape are "exposed" instead of hidden by default (white labeling instead of black labeling). For the particular model I'm working with I has dynamic field names in a MongoDB that I don't reliably know the names of. But I do know which fields I want to hide or have additional conditions for. My goal is to white label all fields by default and hide/condition the ones I don't want exposed. Pretty much the opposite of how entities currently work. This is a working example:

module API module Entities class User < Grape::Entity expose :password, :if => {:type => :admin} def initialize(object, options = {}) exposures = self.class.exposures object.attributes.keys.each do |key| key = key.to_sym if !exposures.include?(key) self.class.expose(key) end end super(object, options) end end end end

I'll probably write an entity extension to expose some additional functionality to make it easier to work with these types of scenarios. But in the mean time, this is what I cam up with.

#grape #ruby #programming

A rake task to create Mongoid indexes

If you're not using Mongoid within Rails you don't get access to a rake task that you can run to create indexes. I whipped up this little rake task that automatically runs through a model directory and checks for Mongoid documents and then adds indexes.

# Based on: http://vikinghammer.com/2012/05/16/rake-task-mongoid-index/ # Usage: rake mongoid:create_indexes[ENV_TYPE_HERE] require 'mongoid' require 'logger' logger = Logger.new($stdout) ROOT = File.expand_path('.', File.dirname(__FILE__)) namespace :mongoid do task :create_indexes, :environment do |t, args| def determine_model(path) path =~ /(.*)\/(.*).rb/ $2.camelize.constantize end unless args[:environment] logger.fatal "Mongoid: Must provide an environment" exit end yaml = YAML.load_file("config/mongoid.yml") env_info = yaml[args[:environment]] unless env_info logger.fatal "Mongoid: Unknown environment" exit end Mongoid.configure do |config| config.from_hash(env_info) end Dir.glob(File.join(ROOT, 'models', '*.rb')).each do |model| require model klass = determine_model(model) if klass.ancestors.include?(Mongoid::Document) klass.create_indexes logger.info "Mongoid: Indexes processed for \"#{klass.class.name}\"" end end end end

#rake #mongoid #ruby #programming

Testing Grape without Rails

I was having some trouble getting rspec to work with Grape outside of Rails. While the solution was simple and was mainly about getting the requires correct, I wanted to share an example.

require 'rspec' require 'rack/test' require 'myapi' describe MyAPI do include Rack::Test::Methods def app MyAPI end describe MyAPI do describe 'GET /somepath' do it 'returns a test string' do get '/test', {}, { 'SOME_HEADER' => 'VALUE' } last_response.status.should == 200 JSON.parse(last_response.body).should == [] end end end end

#testing #grape #rails #ruby #programming #rspec

Adding class methods through a module

I've been messing around with some Ruby meta programming the last few days. One of the things I wanted to do was to add functionality through a module include and add methods that would be class methods instead of instance. This is what I came up with:

module Addons module ClassMethods def new_cls_method puts "I'm a class method!" end end def self.included(base) base.extend(ClassMethods) end end class MainClass include Addons end MainClass.new_cls_method # I'm a class method! mc = MainClass.new mc.new_cls_method # undefined method `new_cls_method'

Edit: Worth mentioning that if you are mixing in a module that is only class methods you can just use extend instead of include.

#ruby #programming

Back in action

Hello friends and followers. It's been a while. If you used to read my blog, you will notice I took quite a hiatus. Today I return and have decided to start with a fresh slate.

A lot has happened since the last time I posted. I left Lot18 about a month ago, after spending 2 years running the development team and helping to grow the company (I'll definintely write a bit about Lot18 here). My next adventure is a startup of my own. It's in the e-commerce infastructure space. I'm just getting started, so I'm not going to say too much just yet. But lots of details coming soon...

I'm looking forward to sharing this next adventure with you.

Trending Blogs

Recently Viewed Blogs

Adam Gotterer - Ramblings of a hacker and founder