Martin Emde

This is my subtitle

[martinemde]

Revert part of a large commit based on a regex grep pattern in the code

Let’s say you have a big commit and you have some changes merged into that commit that you would like to revert. Unfortunately, you didn’t commit those changes as separate commits, so reverting is non-trivial.

Here is a concrete example: at The RealReal we are in the process of upgrading from Rails 3.2 to Rails 4.2. A lot of the changes were made and then squashed over time to make rebasing easier.

In the middle of this large squashed commit, a change was made to remove attr_accessible from all the models. We later decided to use the protected_attributes gem, based in part on How to Upgrade to Strong Parameters in Rails and also because of the huge amount of work involved. Now we need all those attr_accessible lines added back in.

Instead of manually finding each line in the diff and copying it back in, I discovered and used the following process. Get ready to resolve some merge conflicts.

The basic idea is to split the 1 big commit into 2 commits:

the commit that removed attr_accessible (which we want to revert)

the rest of the changes on your branch

It’s important here that you turn off rerere in your .gitconfig. The first time you run into the merge conflicts they will be when you are making a commit that remove attr_accessor. The second time through, we will resolve the commits in the opposite way (keeping attr_accessor). rerere confuses this by always picking our first version and throwing out the changes we want.

$ git config --global rerere.enabled false

Now we need patchutils, which gives us tools for grepping within a diff. I’m assuming here that you have homebrew installed.

$ brew install patchutils

Now view the diff, output it as a patch, and look at the output to see if the match string is getting the right changes.

$ git diff master…rails4 -U0 | grepdiff 'attr_accessible' --output-matching=hunk

Make a patch if it looks mostly good. grepdiff is unable to perfectly exclude unmatching changes that are part of the same hunk.

$ git diff master…rails4 -U0 | grepdiff 'attr_accessible' --output-matching=hunk > remove_attr_accessor.patch

Unfortunately, this patch isn't quite right because it includes chunks we don't want that grepdiff is unable to separate.

Next, checkout a fresh branch at the base of the rails4 branch. This keeps us from dealing with a rebase onto the current master.

$ git checkout master...rails4

Create a branch to work on.

$ git checkout -b rails4-attr-accessor

Apply the patch to the new branch.

$ git apply --unidiff-zero < remove_attr_accessor.patch

The next step is more complicated. We need to manually pick only the attr_accessible lines. The only way I know how to do this is by manually adding only the parts of the patch that we truly want. You could also look at the diff and change it so that only the lines we want are actually changed.

$ git add -p

Remember that you’re making a patch that removes attr_accessor. This is the opposite of what we eventually want to do and it can be confusing when you’re working on a file.

Eventually you will have a staged commit that only removed attr_accessor. Check in the commit.

$ git commit -m “Remove attr_accessor"

Now we can just make a revert commit right on top of this branch to make an “inverse” commit that adds attr_accessor back.

$ git revert HEAD

Now checkout the main branch, and cherry-pick the revert commit on top.

$ git checkout rails4

$ git cherry-pick rails4-attr-accessor

This will create a revert commit on the very top of the rails4 branch that adds attr_accessor back in to your branch. I suggest leaving it like this, rather than trying to squash or re-order the commits with a git rebase -i. It can lead to tons of merge conflicts and is just a waste of time.

At this point, you should have the changes restored as commit on the top of your main branch. I hope this helps save you a bunch of manual work and mistakes.

[martinemde]

You will need to divide your current possessions into four major categories. Beautiful things. Emotionally important things. Tools, devices, and appliances that efficiently perform a useful function. Everything else. “Everything else” will be by far the largest category. Anything you have not touched, or seen, or thought about in a year – this very likely belongs in “everything else.” You should document these things. Take their pictures, their identifying makers’ marks, barcodes, whatever, so that you can get them off eBay or Amazon if, for some weird reason, you ever need them again. Store those digital pictures somewhere safe – along with all your other increasingly valuable, life-central digital data. Back them up both onsite and offsite. Then remove them from your time and space. “Everything else” should not be in your immediate environment, sucking up your energy and reducing your opportunities. It should become a fond memory, or become reduced to data.

Worldchanging: Bright Green: The Last Viridian Note

[martinemde]

…remember that the accused must always be assumed innocent, and allowed his day in court. The curious citizen must always have open access to information about anything he wants, and should be able to learn whatever interests him, without having some other person’s ideology superimposed on him during the course of his learning. The maverick must be allowed to retreat to his private domain and live in any manner he finds rewarding, whether his neighbors would find it so or not. He should be free to sit and watch television all day long, if that’s what he chooses to do. Or carry on interminable conversations with his cats. Or use a drug, if he chooses to do that. As long as he does not interfere with the freedom or well-being of any other person, he should be allowed to live as he wishes, and be left alone. I believe that the phasing out of laws regarding drug use by adults, and an increase in the dissemination of truth about the nature and effects—positive and negative—of different drugs, the doing away with random urine testing and the perversion of justice that is its consequence, will certainly lead to smaller prison populations, and to the opportunity to use the “drug-war” funds for desperately needed social improvements and public health matters, such as homelessness, drug dependency and mental illness. And the energies of law-enforcement professionals can once again be directed towards crimes that deserve their skill and attention. Our country might possibly become a more insecure place in some ways, but it will also be a healthier place, in body and spirit, with no further profit to be made on drugs by young men with guns on the streets of our cities. Those who abuse drugs will be able to find immediate help, instead of waiting for six months or more, in confusion and helplessness. And research in the area of drug effects and possible therapeutic use will come alive again in our centers of learning. And we will once again be the free citizens of a free country, a model for the rest of the world.

Lecture at the University by Alexander Shulgin - Very Powerful thoughts on drug law, and our freedoms.

[martinemde]

Is it any wonder that laws prohibiting the use of psychoactive drugs have been traditionally ignored? The monstrous ego (or stupidity!) of a person or group of persons, to believe that they or anyone else have the right, or the jurisdiction, to police the inside of my body, or my mind! It is, in fact, so monstrous a wrong that, were it not so sad indeed, tragic!—it might be humorous. All societies must, it seems, have a structure of laws, of orderly rules and regulations. Only the most hard-core, fanatical anarchist would argue that point. But I, as a responsible, adult human being, will never concede the power, to anyone, to regulate my choice of what I put into my body, or where I go with my mind. From the skin inward is my jurisdiction, is it not? I choose what may or may not cross that border. Here I am the Customs Agent. I am the Coast Guard. I am the sole legal and spiritual Government of this territory, and only the laws I choose to enact within myself are applicable!!! Now, were I to be guilty of invading or sabotaging that same territory in others, then the external law of the Nation has every right—indeed, the responsibility—to prosecute me in the agreed-upon manner. But what I think? Where I focus my awareness? What biochemical reactions I choose to cause within the territorial boundaries of my own skin are not subject to the beliefs, morals, laws or preferences of any other person! I am a sovereign state, and I feel that my borders are far more sacred than the politically drawn boundaries of any country.

Lecture at the University by Alexander Shulgin - Very Powerful thoughts on drug law, and our freedoms.

[martinemde]

The Domain Model of Human Workers: Ticket Tracking Software Must Model Your Engineering Domain

The way humans produce work follows a pattern. This pattern is a company's culture. It's a combination of the way engineers have worked over their own history and the company's history. When using a ticket tracker to organize work, it must accurately match the mental model used by the engineering team in order for it to be effective. When you use the Github issue tracker, Pivotal Tracker, YouTrack, Jira, or your homegrown ticket tracker, you're modeling the work of engineers. It's critical that the real world and your engineering models stay in sync. The only way that the project manager or team lead can really understand the state of the world is through these models. To achieve consistency in the human domain model, you need state, ownership, documentation, and a commitment from engineers to maintain the domain. #### Ticket State Tickets need states that indicate: * In Discussion -- Not ready to start. Work is not scoped. * In Queue -- This ticket is ready to start but work has not begun. * Started -- In progress. * On hold -- Waiting for someone or something. Should be documented in the ticket. * Ready to deploy -- Waiting in a merged state for the next deploy. Work is done. * Deployed -- Code is live in production. * Won't fix -- Not a priority. Won't be worked on. Ticket states allow everyone a quick way to look at and group tickets. #### Ticket Assignment Assign tickets to yourself when they are actively in progress. This indicates that you are responsible for the ticket. When a ticket is assigned, it is a sign to other engineers and to project managers that someone else should not just pick up the ticket and start working on it. Your ticket tracker should also allow you to follow or subscribe to a ticket. If you aren't working on the ticket, but you care about it, subscribe to it. #### Document all of the things! The known information about a piece of work should be documented in a ticket once work has been started. This is the "hit by a bus" safety net. For example, my first child may be born any time now. I might just disappear from the face of the planet with new baby jelly brain for 6 weeks. If the tickets I'm working on are not up to date and suddenly I disappear, then it's absolutely critical that the tickets I'm working on are accurate. My manager can then help the team absorb my work by looking at things that are active and assigned to me. #### Commitment It's critically important that engineers keep their domain model, their ticket system, up to date for the same reason it's important to keep accuracy in any domain model. The engineering team *must* feel well "modeled" by the ticket system they use. The company absolutely needs accurate information to base decisions on and judge the importance and progress of work, and this all hinges on the team's support of the ticket tracker. #### What I use At Engine Yard we use YouTrack. We run it on our own servers and it's far and away the best ticket tracker I've ever used. Our deploys auto-update tickets and build release notes from the tickets. We have chat room bot integration with YouTrack that displays a ticket name and link any time a ticket is mentioned. It allows custom states and focuses on discussion which leads to better solutions and more knowledge sharing. If you don't like your ticket tracker, I highly recommend [YouTrack](http://www.jetbrains.com/youtrack/).

[martinemde]

Dear Management; Trust Us

_"Begin at the beginning and go on till you come to the end: then stop." - Lewis Carroll, Alice in Wonderland_ Dear Management; We've worked together for a while now. There's been ups and downs, but we've always tried to serve our common goal. However, we think lately you may have lost your way. We’re worried and we want you to know: You need to trust us. We invent the things that make the company money. You may think we work for you, but that is not true. We work for the company and we're the rubber that meets the road for our company's product. We're people, with our own ideas, dreams, experiences and goals. If we were computers instead of people, it would be your job to plan out the exact product you want, and we would execute it exactly how you imagined. If we were computers you would write high level management code and out would come "The Product". Even if we were were just typists at those computers, you might expect an extension of your hands on the keyboard, multiplied out as needed. That's not how we work. We’re not computers. We're creative workers. We're novelists and poets. We're the long form journalist to your copy typist. We're painters and sculptors of the expressions of logic, process, flow, and interface. We dream about how to make systems work. It is our passion to create from nothing, and improve what we find. We learn and grow and we incorporated what we know into our work.

White Rabbit Original Artwork on Deviant Art

We know you have important goals that need to be met. We can admit that sometimes we go down rabbit holes and encounter trickster cats that waste our time. Sometimes it takes much longer to find the exact right size than we thought. We're guilty of spawning mad hatters and evil queens but we also know how to deal with them. The world we create is not only folly, but exploration and wonder.

Passion is delicate.

To the extent that you can encourage our passion in the right direction, you can influence what we care about and where we go. You need to know that our passion needs careful handling. When we work with you, we share the delicate crystal of our passionate creativity with you. We trust you with our learning and practice that sustains our lives. It is our driving force, a desire and power of will that we've trained and sharpened. We depend on our passion to feed our families and to live our lives. The reason we're here with you is that we think that we can do something great.

By foot we walk the path over which you fly.

It looks different down here. We're happy to journey towards your destination if it's a goal that we share. You can help us share your passion but we need to find our own way there and we might not arrive exactly where we aimed. A path laid out for us results in a freeway, not a scenic trail. Slowly, by foot, we labor every step of the path over which you fly from high above. It's easy to lose the journey in the quest for the destination. It is paradoxically short sighted to focus only on the distant destination, forgoing the experience of each step in the journey. We cannot plan for terrain which we have not seen, and we cannot see terrain at which we have not arrived. From high above you cannot hope to predict the path, but the changes are gradual when we take our steps one at a time.

Without those of us who walk the path, none of us will ever arrive. Maps of an unexplored frontier are nothing but a fools guide to certain death. With each step we can choose a direction. The soles of our feet will guide us on the right path, and we will tackle hard terrain one step at a time. We may encounter worlds which none have ever seen, but only if we walk, and learn, the path.

We need you to believe we will find our way through the forests and thickets. We've calibrated our compasses and trained our minds for this journey. We're happy to have you point over the mountains and say "the promised land is just over the top!" We've done these journeys before. We know how to ford the rivers and streams, and though we may lose a few brave souls to dysentery along the way, we'll arrive somewhere wonderful if you allow us to blaze our trail.

Listen to us, and we will listen to you. We are happy to work with you and we will take you to wonderful places, but you must walk hand in hand with us.

[martinemde]

BitTorrent — The Distributed Social Network

This is an article I wrote May 12, 2011 on my other blog. I'm removing it and moving all to tumblr for now.

BitTorrent is on to something huge. A way to build a truly distributed social network. From the article on TechDirt:

It appears that BitTorrent is effectively trying to move into the distributed social networking space, while also disrupting the business models of various cloud storage offerings at the same time (a twofer!). On the social networking side of the game, the company recognizes that a big part of the success of Facebook is in sharing personal content such as photos and movies, so it's building off of its underlying technology to let groups and individuals create "channels" in which such content can be shared (not in a weakly compressed format either, but fully) and commented on: Could BitTorrent Be The Distributed Social Network People Have Been Clamoring For? — Tech Dirt

Imagine it working like this: You run your own "profile" through BitTorrent. You serve the images, text, movies, updates; whatever it is that you want to share. Through BitTorrent you share the same thing that you put up on any other profile anywhere else. Standards around torrent sharing will emerge and existing ones will evolve for the best serving of social content. Your friends, people you've chosen to include in the distribution of your profile, will share in the seeding of your profile items. When you're online, you seed content. Friends are there to distribute the load of downloading your content. You can update your profile content offline (ha, like we're ever offline anymore), and it distributes out to your friends whenever you are able to seed. When you are offline, your friends serve whatever content has been shared to the network. If you want your profile to increase in availability, you use a service that seeds your profile to peers. This is the same as serving a website. Companies will emerge that offer service specific data storage as connection speeds and local cache catches up. We've already arrived at this point for smaller amounts of data. If I want to share a few photos it won't be a burden to anyone seeding my profile. However, sharing all the photos and movies I have would exceed 30 GB. No one wants to seed 25, 100, or 300 people's 30 GB of media each, at the cost of their own hard drive space. In the meantime, people will be discouraged from dumping their entire media directories on their torrent friends. Your torrent systems will get smart about what data to cache and what to grab from the network when requested. Don't download and seed everything, just select specific friends that you always seed, and some that are only downloaded on demand. The only people who seed your account are the same people who might want to read it. The necessity to store everyone's data on a huge central server disappears. Profiles are viewable by the people who take part in seeding it. This is more efficient for everyone involved, especially the Facebooks of the new torrented world. Since you have a cache of the most up-to-date information on the profiles you care about, you can read it anywhere and in any format you want. View one person's or everyone's blog posts, images, and status updates in a single view. If Facebook wall style is your thing, view it that way. Perhaps you prefer a tumblog or Twitter format for scanning friends' shared content. New ways of organizing and viewing content will be available. If you want to add someone as a friend, find them through a mutual friend and request access their profile torrent. If you need to add a non mutual friend, send them the link. Services will spring up that seed sets of profiles of your class mates, work mates, or arbitrary groups. Directories will exists to find people, and those directories won't need to serve all the data as well. Imagine Facebook as the index rather than the source. It may even be in their best interest. Serving content requires a hefty infrastructure. This revolution in personal content sharing reduces the need for the old architecture. The central systems that are abused by lawmakers and lobbyists to keep dying business models alive through force. It's harder to sieze the domain of a person that serves their information in a distributed manner to their connected friends. The monolithic service that has everyone's information fades away. The BitTorrent based social network could be the future for personal sharing. It will be as private, verbose, fault tolerant, or available as you want it to be. Now we just need to make it easy and appealing enough that average people will use it.

[martinemde]

Rails database.yml adapter: a failed design

Until the introduction of the mysql2 gem as an adapter in Rails 3, database.yml could be entirely generated by the deploy system — Engine Yard Cloud for example. The addition of mysql2 has thrown a wrench into the way we generate this file, and for no good reason. The original usage of adapter meant essentially "database protocol". It is now more akin to a gem require statement. We need to fix rails to relieve this problem.

Here's how it used to work (rails 2.x and earlier)

When you use capistrano, or any other deploy mechanism, the instructions are to include the generation of database.yml in your deploy script. For security reasons you don't want your production database password stored in your repository. Most of what you'll find on the internet about database.yml suggests not checking it in, and generating it at deploy time.

This capistrano recipe describes how to store the database.yml on your server symlink it on deploy.

This one encourages you to put your database.yml in your home dir on your server and cp it over on deploy.

Since the beginning, Engine Yard has written the database.yml file to a shared directory for your app and symlinked it to your config directory on deploy.

With the generate-on-deploy approach, it is recommended that you don't commit your database.yml file. This is standard practice with rails anyway. In order for different developers to work on the project locally, they usually each need a different database.yml with their own user/password combos. People go to great lengths to solve the problem of shared database.yml files because it's what we're stuck with in Rails.

Database.yml contains connection specific information for the current environment. When you deploy on Engine Yard Cloud, we provide you with a MySQL database by default. We write and symlink a database.yml with the database name, user, password, and everything else needed for the specific environment. All the info you need to connect to our database is given for you.

mysql2 gem destroys this system

When rails decided to add support for mysql2 as an adapter, they built a new mysql2_adapter in to rails. The adapter requires "mysql2" at the top, just like other adapters. What this means is a line in a yaml file directly controls what gem is required. This is not good for generated database.yml files.

Rails 3 was introduced with bundler. With bundler we have a way to declare what gems we want, including our database adapter. When you specify the mysql2 gem, you also need to change your adapter. Just to reiterate, this yaml file controls what gems rails loads for you, and you can't change it without monkey patching or hacks that break gem load paths.

On Engine Yard Cloud, we will be forced to put a hack into the database.yml to determine what adapter you want to use.

$ cat config/database.yml <% def determine_adapter if Gem.loaded_specs.key?("mysql2") 'mysql2' else 'mysql' end end %> production: adapter: <%= determine_adapter %> database: mysql2_app

This is just insane. All we need is for rails to let you specify in your app what adapter you want to use, but you can't because rails explodes if you don't set an adapter in this file. It doesn't matter if the adapter is already loaded or not.

The adapter is a separate piece of information, used to load a particular ruby file. The rest of this file is filled with settings that allow the adapter, once loaded, to connect to your database. They are fundamentally differently used and differently sourced pieces of data.

Rails didn't break postgres for pg

When the pg gem became the new standard for rails instead of the unmaintained postgresql gem, the adapter didn't change. They just decided that you must now use pg instead of postgresql. But postgresql didn't get off too easy.

Let's say you want to use jdbc on your rails project because you're using JRuby. Lucky for you the activerecord-jdbc adapter contains a hack, discussed here by Nick Sieger, to get around Rails' ignorant loading of adapters. There is a pg.rb file in activerecord-jdbc to allow rails to happily require "pg" and continue working properly. Just make sure you never accidentally install the jdbc adapter along side the pg adapter, because you'll get the wrong one when rails requires pg. If you look at the documentation for jruby's jdbc adapter gems, you can see how crazy using the adapter key in database.yml is for choosing which adapter is loaded.

As Nick says in the article, the only solution is to patch rails or add a big warning. Sounds like we need a patch.

Solution: Patch Rails

Stop using adapter in database.yml.

Adapter does not belong in database.yml. Before the fracturing of database adapters, what adapter meant was protocol. The database at HOST with USER@PASSWORD speaks "postgresql" protocol. The internals of rails didn't reflect this, but as far as deploy was concerned, we just write "mysql" when that's the type of database we're providing. Now in Rails 3+ the database protocol and which gem to load are conflated. Everything about this file describes connection setting except the adapter key.

How to fix it:

Allow setting adapter in config/environments/(production|test|development).rb to whatever actual rails adapter you want to use, and require whatever gem you want that implements the adapter's interface.

Rails will load the adapter you specify.

Engine Yard, or any deploy system, will write the database protocol, "mysql" for example, into the database.yml file without having anything to do with what gems you've chosen.

The Rails adapter can assert their supported protocol. For mysql and mysql2, the protocol is "mysql". For pg and jdbc, it's "postgresql". If you set the wrong adapter for the database, you get a self explanatory error (unlike the mysql2 error which will drive you crazy when you already did include the mysql2 gem and the error is just confusing.)

[martinemde]

Pure irony. A blog post draft. Doesn't look like it's going to happen.

[martinemde]

Moving to San Francisco

In July of 2009 the job that brought me out to San Diego ended. I was fortunate enough to be adequately positioned to take some time off and work on personal projects. The result of that is Go Beering which has been and will hopefully continue to be a lot of fun to work on. Eventually it might even be useful for someone besides my wife and me.

Unfortunately, Go Beering isn't in a position to pay the bills, and so I began casually looking for the right job. At that same time a friend of mine mentioned that they just began looking for people to join his team at Engine Yard. I never actually imagined I would have the opportunity to work at Engine Yard so I applied.

After a few weeks and even a lost email (it's 2009 and that still happens), I was on a flight to San Francisco for an all day interviewathon. One of our good friends from San Diego, Taylor (@TheArtofBeer), was on the same flight and made for an excellent pre-interview company. The interview went really well. Within a week I had a job offer and I was faced with the real possibility of moving to San Francisco.

This all happened about 10 days before RubyConf in Burlingame (or San Francisco as they called it). Kewe and I went to San Francisco for a couple extra days an absolutely loved it. It seemed like the right decision to move to San Francisco and I signed the offer letter at the Engine Yard office on the Monday after RubyConf.

In leaving San Diego, I'm leaving behind a lot of dear friends that we've met in the short time we've been here. I look forward to seeing you all before we leave and I hope to visit San Diego regularly.