Fred Wu's Blog

Freelance Ruby on Rails, JavaScript, Front-end, UI Web Developer in Melbourne, Australia

[ifredwu]

Coding and Learning Should Never Stop, Open Sourcing is Caring

I’ve had a productive coding weekend, and so I decided to share my experience. Now, many developers choose to treat their career as a series of 9-5 jobs, but if you’re reading this, I assume you’re like the rest of us who love continuous learning and self improvement.

Preface

About a year ago I started learning Elixir. So as part of the learning experience, I wrote two matching learning related libraries: Stemmer and Simple Bayes. It was a great, really enjoyable experience and I learnt a lot about the concepts of word stemming, naive Bayes classification and of course, functional programming.

These topics have been interesting to learn about, but until one gets to use them on a daily basis for a while, key concepts are less likely to convert from short term memory to long term memory. Given my day job is not about writing Elixir (yet), I needed to find other ways to keep my skill-level up and to continue exploring new things.

So about a month ago, I picked up a project I started a year ago but gave up shortly after: Crawler. At the time GenStage was just announced and I was interested in incorporating it into my project as I thought it'd be a great fit. But due to varies reasons - mostly not having a firm grasp of the GenStage concept and implementation, as well as taking on a CTO role at a startup, I couldn't find enough time and patience to make it work so I had to let it go.

Until now.

Crawler, on Steroids

I'd realised that the way I was trying to incorporate GenStage into my project was never going to work. Not because of GenStage itself, but because of the way I approached learning it. At the time I was so eager to make use of GenStage, and coming off the back of my good streak of releasing the aforementioned machine learning libraries, I thought I could take shortcuts and things would all work out perfectly.

No.

So I licked the wounds, learnt my mistakes and changed tactics. This time, I scoped out and encapsulated my learnings (just as one would in designing a software system), and eventually I came up with another library - OPQ: One Pooled Queue.

OPQ: One Pooled Queue

I knew Crawler could technically work without any queueing system, or GenStage. So I kept on building Crawler, until I felt productive in writing Elixir again, and touched enough areas and concepts that I knew exactly what I needed from GenStage.

And luckily for me too, GenStage over the past year has matured and more importantly, has better documentation with more code examples. Upon closer investigation of the code examples, I found that their GenEvent and RateLimiter examples were almost exactly what I needed. It was an epiphany moment for me after reading and understanding these examples, all of a sudden I "get it".

If you take a look at the source code of OPQ you'll notice that the heavy lifting logic was mostly inspired (or even copy-pasted) from those examples.

Open Sourcing is Contributing and Caring

Up until this point, as far as writing open source Elixir code goes, it had mostly been me writing my own code for my own projects. But open sourcing is much more than just writing one's own code and publishing them on Github.

If you've followed my work you'll know that I'm a big fan of contributing to other projects, some of which are well-known ones like Rails and Slim.

It just so happens, that over the weekend I ran into situations where I needed to contribute to other Elixir projects - and spoiler alert: one of which is Elixir itself.

ElixirRetry

One of the features I set out to add to Crawler, was to allow failed crawls to retry before giving up. Naturally, the first thing I did was to try find an existing package to support the retry functionality.

After some googling and digging into some source code, I've found and settled on ElixirRetry - a neat library that was cleverly built.

Soon I found out that since its last release, it offered both retry/2 and retry/3, the latter of which was to support an extra argument that specifies which exceptions to allow as part of the retry flow. It was a great addition, but it doesn't effect me as Crawler doesn't need it.

As a developer who cares deeply about code quality and clarity, I immediately thought about how the interfaces for retry/2 and retry/3 could be improved, by simply combining them and making the extra argument in retry/3 an option.

So I did, and issued a pull request here: https://github.com/safwank/ElixirRetry/pull/12 You'll notice that I've also made some improvements to the test suite while I was at it. ;)

Elixir Typespec

One issue I ran into while building Crawler, was the static analysis via Dialyzer - some code that actually ran and passed tests functionally, failed the type checking.

As someone who isn't as experienced in Elixir, I first opened an issue and phrased it more as a question seeking validity of the issue. I then jumped on Elixir's Slack group and asked people in the group about this issue.

Fortunately, Ben Wilson immediately came to aid by verifying my issue and validating my suspicion of it being an issue in Elixir's typespec documentation.

And so, a pull request was created and approved shortly after.

Sharing is Caring

The bulk of Crawler as well as the entirety of OPQ were built in the past month or so. I hope some people will benefit from having these libraries around. And, I hope people also enjoyed me sharing my experience, and perhaps be inspired to start sharing more too.

I will leave you all with a Chinese saying: 滴水石穿. The literal translation is "dripping water penetrates the stone", and what it means is "constant perseverance yields success".

[ifredwu]

Elixir and Doctest - Help Writing Better Programs, One Function At A Time

Preface

If memory serves right, it's been several years since I first dabbled in Elixir, but it was about a year ago I really started putting some serious effort into learning Elixir, and as a result I made two libraries in the machine learning space: Simple Bayes, a Naive Bayes text classifier implementation, and Stemmer, an English (Porter2) stemming implementation.

Unfortunately after I've released those two libraries, I hadn't had much opportunities to work with Elixir. My day jobs have been mostly Ruby, JavaScript, PHP and a dash of Golang. And so, after being silent for a year, I've decided to pick up something I had started a year ago - a web Crawler. If you are new to Elixir, feel free to follow this project as I am actively developing it.

Learnings

The preface is to give a bit of background of when and how I started learning Elixir, now, let me talk about one of my favourite features of Elixir, and how it helps me write better code not just in Elixir, but in virtually any other language.

Introducing the topic of today, a really simple feature, and in fact it has been part of Python for years - the doctest.

Doctest

In short, a doctest is pieces of code examples that run as part of the test suite, and show up as part of the documentation. For example:

defmodule Greeting do @doc """ ## Examples iex> hello("world") "hello world" iex> hello("dear") "hello dear" """ def hello(input) do "hello #{input}" end end

And then all you need is in your corresponding test file, to enable doctest:

defmodule GreetingTest do use ExUnit.Case doctest Greeting end

Even though I knew about Python's doctest for years, I'd never realised how impactful it can be given its simplistic nature, probably due to the fact that I've never used Python for anything serious.

Until Elixir.

There are three things I find the most impactful as I write more doctests: clarity, scope, and design.

Clarity

As a ruby programmer, I appreciate greatly the beauty of not just the main code base, but also its test suite. Hence I've always preferred to use RSpec in larger code bases. However, as the application gets more complex, and as the number of test files grows, the cognitive overhead of reading and processing all the files and lines becomes higher and higher.

Doctest solves this perfectly - no longer do we have to crawl through the right file and the right line for a particular test case, all the test cases are neatly presented right in front of you as you read the function itself.

You might think this as trivial, but just like many organisations spend time and effort to optimise for effective communication by studying proxemics, proxemics between different components of a software code base also plays a role in improving the code clarity, and ultimately the code quality.

Scope

Doctest is purposely simple, and is designed for unit tests. There have been many times when I found myself realising my function was too dependant on external states, or are doing too many things because it was hard to write simple doctests. In a way, the constraints of doctests have forced me to rethink the scope of my function, and that would often lead to an overall better designed system.

Design

As much as I'd like to think about the SOLID principles all the time, it is often too easy to dig deep wholes in the midst of building things.

Every now and then I find myself extracting a piece of logic to a private function and calling it a day. In Elixir, only public functions can have doctests - again, this constraint pushes you to think about the importance and the role of a particular function, perhaps it is better to be moved to another module as a public function therefore can have its own doctests. Here is an example when I did some refactorings on Crawler.

Closing

I wanted to write this article for a while now - as I truely love and appreciate Elixir's asthetics and features. Many developers might find functional programming as a barrier, but I can assure you that with Elixir's tooling and ecosystem, and of course doctest (wink), building software feels like a breeze.

Last time when I was this happy building software was when I first discovered ruby. If you haven't given Elixir a try yet, I encourage you to do so sooner rather than later, it will not only give you a functional programming perspective, but will also help you write better code in other languages.

[ifredwu]

I Accidentally Some Machine Learning - My Story of A Month of Learning Elixir

About a month ago I was in-between jobs - I had two weeks to rest up, recharge and get ready for my new job. So I thought, I should use those two weeks to learn something new.

Years ago I briefly looked into Elixir when it was first released to the wild, at the time I wasn’t interested in picking it up due to its syntax similarity to Ruby, despite their vastly different underlying semantics. I love Ruby, and it’s been my weapon of choice for the past 6-7 years, so when it came time for me to learn something new, I naturally wanted to learn something a bit more different than Ruby, syntax-wise.

Fast-forward a few years, I am more mature and open-minded, and are now in a position to welcome Elixir and to embrace the Ruby-like syntax as well as the functional programming mindset with open arms.

Learning Elixir

Given the strong influence of Ruby in Elixir, I can’t help but get nostalgic about learning it by reading another great book by Dave Thomas: Programming Elixir.

Dave Thomas’ original Programming Ruby was instrumental in the success of the Ruby programming language and its communities in the west, and it certainly has helped me to greatly widen my exposure to not only the wonderful world of Ruby but object-oriented programming in general as a PHP developer.

Learning Elixir has been really fun, not only does it share the same developer-friendliness championed by Ruby, it also does so with remarkable strength in concurrency thanks to BEAM (Erlang’s VM).

Many people are drawn to Elixir due to its Ruby influence, its functional and immutability nature, and its Actor-based concurrency model. I would however like to call out one of my favourite features of Elixir that sometimes gets overlooked, and that is how you could write unit tests using ExUnit.DocTest.

Elixir Makes Writing Unit Tests Effortless

Python programmers have been writing doctests for years, but Ruby due to not having an equivalent standard library, has never had writing doctests taken off in its community. I’m glad Elixir has.

Take a look at the code snippet below:

defmodule Stemmer.Step0 do @doc """ ## Examples iex> Stemmer.Step0.trim_apostrophes("'ok") "ok" iex> Stemmer.Step0.trim_apostrophes("o'k") "o'k" iex> Stemmer.Step0.trim_apostrophes("'o'k'") "o'k" """ def trim_apostrophes(word) do word |> String.replace_prefix("'", "") |> String.replace_suffix("'", "") end end

The three test cases given will actually be tested by ExUnit, provided you ask it to do so as below:

defmodule Stemmer.Step0Test do use ExUnit.Case, async: true doctest Stemmer.Step0 end

You may think having doctests is not a big deal, what I can say is that I am really enjoying the fact that unit tests can be written with minimal friction and high visibility (as they sit with the implementation, rather than within a big test suite with a sea of files). Besides, not all of us practice TDD religiously or 100% of the time, and when I don’t or can’t do TDD, having doctests ensures I don’t forget adding test cases. :)

Toy Robot in Elixir

The best way to learn something new is to practice it as you learn it. Instead of diving straight into a complicated system, or to write a hello world program, I thought I would dig out a code test and re-implement it in Elixir.

The code test is the rather infamous Toy Robot test. As an interviewer, I must have reviewed a few hundreds of these tests, most of which in Ruby. I know what a good OO solution looks like, so naturally I was looking forward to “rethink” the problem and have a crack at it using Elixir.

Here is the result: https://github.com/fredwu/toy-robot-elixir

As a learner, I wanted to practice as many language features as possible, so included in the Toy Robot test code I tried:

Different data structures such as Struct and List

Pattern matching

Data piping (|>)

Agent for managing states

Macros for validation rules

I kept the test code small and nimble on purpose to illustrate the readability of a highly expressive language. I may not have exercised all the language features (such as supervisors and protocols), but at that point I felt I could start the Elixir journey with a big smile on my face already.

Learning Phoenix

I have to admit, a big part of the reason that got me started on learning Elixir is to build a side project. Initially I wanted to simply use Ruby and Rails because I can be very productive using them, but in the end I decided on learning Elixir and Phoenix, because most side projects fail, and when they do, how much learning can you extract out of those experiences? My answer to this question, is to learn a new language, a new framework and most importantly a new programming paradigm to drastically increase the amount of learning I could gain.

But please, before you jump on Phoenix, learn Elixir properly first! Over the years I’ve seen far too many cases of people jumping on Rails without understanding the language features of Ruby…

To learn Phoenix, the Programming Phoenix book is pretty much the bible on this subject aside from the official guide, and it is written by Chris McCord, the author of Phoenix, Bruce Tate, and Jose Valim, the author of Elixir.

As an experienced Rails developer, it only took me a day or two to go through the book (I largely skimmed the chapters on Channels as I don’t intend to build a real-time app).

Now, as a Ruby developer, the biggest pain point when working on Rails or other sizeable Ruby MVC projects for me, is ActiveRecord. I got so frustrated to the status quo I even attempted to fix it.

Fortunately, in Elixir and Phoenix we have Ecto. One of my favourite features of Ecto is the concept of Changeset.

Take a look at the code snippet below:

defmodule MySecretApp.User do use MySecretApp.Web, :model schema "users" do field :username, :string field :email, :string field :password, :string, virtual: true field :encrypted_password, :string has_one :profile, MySecretApp.Profile timestamps() end def changeset(struct, params \\ %{}) do struct |> cast(params, [:username, :email]) |> validate_required([:username, :email]) |> validate_length(:username, min: 3, max: 20) |> validate_format(:username, ~r/\A[a-zA-Z0-9_]+\z/, message: "alphanumeric and underscores only") |> validate_format(:email, ~r/@/) |> unique_constraint(:username) |> unique_constraint(:email) end def creation_changeset(struct, params \\ %{}) do struct |> changeset(params) |> password_changeset(params) end defp password_changeset(struct, params) do struct |> cast(params, [:password]) |> validate_required([:password]) |> validate_length(:password, min: 8, max: 200) |> encrypt_password end defp encrypt_password(changeset) do case changeset do %Ecto.Changeset{valid?: true, changes: %{password: password}} -> put_change(changeset, :encrypted_password, Comeonin.Bcrypt.hashpwsalt(password)) _ -> changeset end end end

The changset/2 function can be used when a user needs to be updated, whereas the creation_changeset/2 is to be used only when a user is first created. Sure, you can achieve similar result in Rails by using custom validators, but the fact that this practice is enforced by the library and the framework, is encouraging.

One other thing that I’ve seen in larger Rails apps, is the leaky abstraction of view-level logic, they typically sit in controllers, helpers (which are globally available) or worse, models. Phoenix in this case follows what Hanami, Trailblazer and many other frameworks do: it introduces a “view model” layer.

Something along the lines of:

defmodule MySecretApp.UserView do use MySecretApp.Web, :view def full_name do "#{title} #{first_name} #{last_name}" end end

In a nutshell, Phoenix is just like Rails, but less magical (in a good way) and faster. :) Like the Elixir eco-system, most libraries and concepts feel like their ruby/rails counterparts, but more refined.

There are a few popular Hex packages if you’re familiar with their ruby counterparts:

Elixir Ruby Code analysis credo rubocop Testing espec rspec Browser testing wallaby capybara Test coverage excoveralls simplecov Test factory ex_machina factory_girl Authentication guardian devise

And the list goes on and on… Be sure to check out Awesome Elixir for more community curated Elixir libraries.

Learning Machine Learning

As I started building the foundation of my side project, you know, the usual user management and session management, etc, etc, by chance I came across mentions of Bayesian inference.

One thing led to another, very soon I started looking into varies different machine learning algorithms such as Naive Bayes and Random Forest. These are all subjects I had never come across or even heard of before, as I have no strong mathematics, statistics or computer science background. I was however, intrigued and inspired nonetheless, and wanted to employ some machine learning in my side project.

And the best way to understand and learn about machine learning algorithms? You guessed, is to write one! After some research, I came to the conclusion that Naive Bayes is one of the simplest to implement, is great for text classification which is useful for my side project, and has good accuracy given its simplicity and fast speed.

Introducing Simple Bayes

So, after a few train rides to and from work, on my newly purchased Macbook (yes, the one with only a single USB-C port), I built a Naive Bayes library: Simple Bayes.

As of writing, this is the only Naive Bayes library in Elixir that supports the following features:

Naive Bayes algorithm with different models

Multinomial

Binarized (boolean) multinomial

Bernoulli

No external dependencies

Ignores stop words

Additive smoothing

TF-IDF

Optional keywords weighting

Optional word stemming via Stemmer

Let’s see it in action shall we? :)

SimpleBayes.init |> SimpleBayes.train(:ruby, "I enjoyed using Rails and ActiveRecord for the most part.") |> SimpleBayes.train(:ruby, "The Ruby community is awesome.") |> SimpleBayes.train(:ruby, "There is a new framework called Hanami that's promising.") |> SimpleBayes.train(:ruby, "Please learn Ruby before you learn Rails.") |> SimpleBayes.train(:ruby, "We use Rails at work.") |> SimpleBayes.train(:elixir, "It has Phoenix which is a Rails-like framework.") |> SimpleBayes.train(:elixir, "Its author is a Rails core member, Jose Valim.") |> SimpleBayes.train(:elixir, "Phoenix and Rails are on many levels, comparable.") |> SimpleBayes.train(:elixir, "Phoenix has great performance.") |> SimpleBayes.train(:elixir, "I love Elixir.") |> SimpleBayes.train(:php, "I haven't written any PHP in years.") |> SimpleBayes.train(:php, "The PHP framework Laravel is inspired by Rails.") |> SimpleBayes.classify("I wrote some Rails code at work today.") # => [ # ruby: 0.20761437345986136, # elixir: 0.08101868169313056, # php: 0.019047884912605735 # ]

As you can see, provided with reasonable training data, Naive Bayes can work extremely well.

Introducing Stemmer

Something I discovered as I was building Simple Bayes, is something called stemming. Let’s see another example:

SimpleBayes.init(stem: false) |> SimpleBayes.train(:apple, "buying apple") |> SimpleBayes.train(:banana, "buy banana") |> SimpleBayes.classify("buy apple") # => [ # banana: 0.057143654737817205, # apple: 0.057143654737817205 # ]

Oops, the probabilities of the sentence “buy apple” of being apple and banana are the same, that’s not good, as we know “buying” and “buy” should mean the same thing. This is where stemming comes in handy. Let’s enable stemming and run the example again:

SimpleBayes.init(stem: true) |> SimpleBayes.train(:apple, "buying apple") |> SimpleBayes.train(:banana, "buy banana") |> SimpleBayes.classify("buy apple") # => [ # apple: 0.18096114003107086, # banana: 0.1505149978319906 # ]

Stemming in this case correctly identified “buy” and “buying” as the same thing (they both have the stemmed root of “buy”).

How did I include stemming in Simple Bayes you wonder? Let me introduce you to Stemmer - as of writing this is the only Porter2 stemmer available in Elixir. :)

Elixir and BEAM may not be known for their raw performance, but compared to a similar Porter2 stemmer implemented in pure ruby, my Elixir version runs about five times faster (under 5s to stem 29000+ words, compared to 25s with the ruby version).

The Learning Continues…

I’ve only been writing Elixir for a month, and there are still heaps to learn and to practice. So far though, I have to say my experience with both Elixir and Phoenix has been fantastic - I get the same satisfying and pumped feeling I got when I was learning Ruby and Rails years ago.

If you haven’t checked out Elixir, I strongly encourage you to do so, after all, the world seems to be moving in the concurrency direction at an increasingly rapid speed, including Ruby 3x3. :)

[ifredwu]

Developers, Being Treated Poorly? You Are Not Alone!

UPDATE: Reading Hacker News comments it is apparent to me that I have not explained my position and intention very well. Please allow me to clarify - I do not resent the people involved in these moments, if anything, I thought those encounters were good learning experience for me, as I know there are things I would do and say differently so others around me won't feel I've been too harsh on them. I did not share these moments to complain, but to raise the awareness that kinder communication styles might be better received.

As a software professional who’s been in the industry long enough to have accumulated quite a few battle scars, I tend not to think back about the unpleasant, unfair and sometimes bizarre moments that make me feel depressed or angry. If there is one thing I learnt over the years, it’s to let go and move on.

Despite the effort in staying positive, I still believe it is valuable to share these moments. If you work in software and feel like you are being treated poorly, you are definitely not alone. By openly sharing these moments, I am hoping more people will start taking notice, and perhaps will lend a hand or a shoulder when a colleague is in need of support.

So, I’d like to share a few moments that have happened in my career, in no particular order. There are a few of these moments that I still find hard to swallow, but most of them I simply chuckle whenever I think about them. ;)

One morning I came to work, noticed a few server alerts that required attention. This was pre AWS and the whole Devops automation movement days so I needed to log onto the server. For some reason my access was revoked so I had to log into the hosting dashboard to reset the root password. The senior dev on the team turned up, and when asked about my access, casually told me: "oh you did something on the servers the other day, so I removed your access."

CTO asked me to help building a new mission critical PHP and MySQL service, upon discovering an existing service already built on Ruby and MongoDB by three very senior developers and consultants, CTO said the reason for the rebuild was because “ActiveRecord is too slow.” Let me remind you that the initial Ruby solution was built using MongoDB. At the time the company had about 30-40 experienced Ruby developers, and only 2 junior to mid level PHP developers who were specifically hired to build the PHP/MySQL service which turned out to be a disaster.

As the team lead who was responsible for evaluating performance and setting salary, I was told by one of my team members that they had gotten a pay rise. Surprised, I asked one of the founders of the company who is non-technical what was going on, during the conversation I discovered another pay rise was given to one other member of my team - all without consulting or even informing me.

Excited to have had my overseas conference talk proposal accepted, I asked the new head of our department who had joined our company for only a month or two for travel approval. Fully expected a pad on the back, I was surprised to have been told that I cannot go, with the reason being “it is just not a good time”.

One of our servers needed to be rebuilt, so trying to be a team player and help, I started installing some basic packages. The senior dev on the team turned to me, straight faced and enunciated in a deep and cold voice: “don’t touch anything on it, this is my server!"

The new development manager walked into our meeting, pulled me out, guided me to another meeting room where the new CTO was waiting. The new CTO opened with “so it’s time for us to go separate ways”, and implied the reason being something that I could definitely have pushed for unfair dismissal. Trying to be professional, I walked back into my original meeting, but the new development manager quickly pulled me out and said “you need to go right now”. He then stood behind me, saw me formatted my work laptop, then literally escorted me out of the office. The new development manager and CTO soon drove the company to the ground and left the country.

After submitting the salary increases for the all my direct reports, I was delighted that the CEO was happy with them all. Given I have had no pay increases in over one and half years I asked for a modest pay increase for myself. I was told “no, we have to discuss this after your project delivery.” My direct reports and I were in the same delivery team.

It was near the end of the working day, around 5pm, as the person coordinating the developer recruitment in our area I ping’ed our Slack channel with a lighthearted message to encourage our developers to start reviewing some code tests from job candidates if they're free. One person replied: “My end of day activity is doing the stuff I should have been doing all day instead of the other things that came up."

One day I got really really sick - difficult to walk or even stand still, dizzy and having fever. Upon informing one of the founders, I was told to come into the office, even though there was no urgent tasks need to be done. Reluctantly obliged, I came into the office, went out for lunch with colleagues but I was so sick I could not eat anything and can barely sit straight. I then went home, and was sick for the rest of the week...

My manager, who was the general manager, wanted to fire two of my developers. His tactic for firing the senior developer was to make the senior developer role redundant and offering the dev a junior developer role instead.

I was tasked to investigate setting up a Cassandra database cluster. Upon discovering certain network limitations, I approached the then head of network operations, when asked about the inability to access the Cassandra cluster, he replied confidently that “it works”. Later on I discovered that he successfully telnet’ed to the ports and declared it “working".

The head of HR who is female and a big advocate for workplace diversity walked into my exit interview. The opening line was “Ordinarily I only do exit interviews with female employees..."

New to the big corporation environment, after asking around a few people and could not figure out how to provision an iPad for my team (but knowing some other teams do have company iPads), in an email reply by the head of enterprise IT: “Hey Gents, Fred has escalated this through multiple channels today. Awesome effort on his behalf, to seems like he didn’t like our response.” To which a colleague had to gently remind him that it wasn’t my fault there was no documented process for provisioning hardware.

After having some discussions on our MySQL database, the dev who had recently joined my team all of a sudden raised his voice in an extremely arrogant and dismissive tone: “you clearly don’t understand this feature, do you?"

A senior software architect who I used to respect walked close by in a meetup. We worked for the same company a while back so I smiled, said hi and was about to start a conversation, he quickly cut me off with “I need to get a drink” without looking at me and wondered off.

The general manager who is non-technical, asked me to investigate options to uplift our ageing bespoke ecommerce solution. Upon delivering my findings, I was told that “your findings are biased.” Later it was clear to me that he wanted confirmation of his agenda from the technical perspective and his mind had already been made on the approach we should take.

A new big shot executive joined the company as our new CTO. He had no agile background and our company was transforming and pushing for a lot of agile principles at the time. Two weeks into his new appointment, the new CTO published an internal document titled “Controlled Chaos”. After reading the document everyone immediately realised that he was describing waterfall. The document was shared as a Google Doc and was open for comments, so people started asking hard questions. Weeks later, many of us who were vocal about his document were let go.

There you are, these were my moments. What were yours? Care to share? :)

[ifredwu]

History Text Analysis Over Spreadsheets - A Poker Player and Developer's Road to Agile Project Management

Ever since I started transitioning into a team leadership role over three years ago, I had been trying to find ways to eliminate waste caused by repetitive work and to keep myself on the fringe of pushing the technical boundaries.

Four months ago I started my current role where my official job title is Delivery Lead. People don't often know what a delivery lead is, but in my mind it is a role to ensure the success of the project delivery by identifying and closing the gaps in the team and in the organisation. And in order to do that, one of our responsibilities is to measure, understand and improve our team's agile process.

It is very tempting to rely on the wonderful and powerful Excel formulas to help record and analyse data points and generate metrics such as cycle time. However, punching things into a spreadsheet is tedious, error-prone, time consuming and violates the DRY principle.

The spreadsheet I used to use for tracking cards.

Introducing Amaze Hands

As someone who strives to keep writing code even in a non-technical role, I started building a tool called Amaze Hands to help reduce the amount of waste I accumulate as a delivery lead.

Amaze Hands' simple Web UI.

Analyse Cards Like Poker Hands

I used to play a bit of online poker and one thing you do in online poker is to look at your hand history to understand the game and your opponents.

If you think about it, agile boards are just like poker games - there is history to what has happened in the past, and in order to optimise for future gains, we need to understand what went wrong and what to improve on a case-by-case basis.

One of the teams I am apart of uses LeanKit, whilst it is a good tool, its reporting functionalities are very limited and its XML export function is completely broken. As a result I started building Amaze Hands to parse the copy-pasted card history from LeanKit, and to eventually generate the metrics I care about.

The LeanKit strategy which consists of a parser and a transformer is able to parse the copy-pasted text from a card as shown below:

And below is the high level architecture of Amaze Hands:

+---------------------+ | Text | <- Raw text input. +----------+----------+ | +--------------v--------------+ | Strategies | +-----------------------------+ | +---------------------+ | | | Parser | | <- Parses text into an AST. | +----------+----------+ | | | | | +----------v----------+ | | | Transformer | | <- Transforms the AST into a common AST. | +---------------------+ | +--------------+--------------+ | +----------v----------+ | Builder | <- Builds the dataset from the common AST. +----------+----------+ | +----------v----------+ | Reducer | <- Filters the dataset. +----------+----------+ | +----------v----------+ | Analyser | <- Analyses the dataset for metrics. +----------+----------+ | +----------v----------+ | Producer | <- Produces metrics. +----------+----------+ | +----------v----------+ | Presenter | <- Presents metrics. +---------------------+

Incremental Analysis - Zero In On Metrics That Matter

No projects are equal, and no project teams are equal - the goal of Amaze Hands is to incrementally add intelligence to our agile process that matters to a particular project and its delivery team.

By incrementally adding and/or filtering data points for analysis, we will be able to zero in on the problematic areas of our agile process. The following is a list of potential areas we could perform analysis on:

cycle time

wait time

blocked time

knocked-back time

context switch (between different streams of work)

other factors such as meetings, attrition, etc

As of the time of writing, Amaze Hands supports the following common metrics:

cycle time (mean and median)

cycle time rolling average (mean and median)

wait time (mean and median)

wait time rolling average (mean and median)

standard deviation rolling average

cycle time scatter

It's Just the Beginning

Amaze Hands started off as an REA Hackday project - on the technical level (hey I still see myself as a developer!), the tool was built in a way that it isn't over-engineered (a.k.a. slow to get it out the door and validate its usefulness) but at the same time has multiple layers as shown in the architecture diagram above so I could refactor and optimise each layer independently when necessary.

It is still early stage, but I thought I'd share what I have right now to gather some feedback and perhaps inspire fellow project leaders to look into optimising your own workflow.

When I started Amaze Hands I was only leading one project team that uses LeanKit, but since last week I started leading another team that uses a physical wall - I can't wait to adapt Amaze Hands to support the new input stream.

So, do you have any interesting tools or techniques to help you lead projects? If so, I would love to hear about them!

[ifredwu]

Leading Snowflakes by Oren Ellenbogen - A Pragmatic Book on Software Team Leadership I Wish I Read Sooner

Two weeks ago as I was reading Software Lead Weekly which I had subscribed to for a while, I discovered its curator, Oren Ellenbogen's book - Leading Snowflakes.

It was a moment of discovery that lead to a stream of delightfulness.

It Started with a Long Day at Work...

After a long day at work, I was so beat I couldn't even listen to audio books like I always do on my way to and from work. So I drove home that night in total silence. One thing that was on my mind at the time was - who should I reach out to for some advice and mentorship?

"Probably no one." I shrugged, and decided to simply suck it up and sleep it off, hoping solutions to the problems I had would come to the light of the day.

I got home, opened up my laptop and started reading unread emails. "Software Lead Weekly, ah I haven't read the latest issue yet," I thought to myself, "I wonder who the curator is."

Discovered "Leading Snowflakes"

And that had lead me to discover Oren Ellenbogen's book Leading Snowflakes.

On the book's landing page it says:

This isn't another 350-page book you'll agonize yourself over not reading. It's short, concise and pragmatic. You're busy, I get that.

"This is great!" I thought, with a sense of uncontrollable excitement. After all, these days I found myself hard to justify spending hours and hours on books that talk about subjects in a hand-wavy manner. A short, concise and pragmatic book on leadership was exactly what I needed.

There was a free sample chapter for download, so I clicked on it without hesitation. To my surprise, after downloading the sample chapter, Oren reached out to me with a personal email, and that had quickly lead to a chain of emails of me writing about the things at work that I found challenging, and Oren providing sound advice and suggestions.

I felt not only grateful, but lucky. Over the years I had worked at varies places where I was fortunate enough to have met quite a few people who I consider my mentor. But to run into one who I had never met before, on the same night my spirit was crushed, and who were willing to not only listen to my rant but also to walk me through my problems and offer advice - it was like finding a needle in a haystack.

My decision to purchase the book was made half way through our email exchanges, as I was browsing through the links Oren provided on his book landing page (under the section of "Sharing my lessons learned"), I found them to be very insightful.

Convinced that Oren's book would offer even more value, I quickly bought the complete package with the book and all 10 interviews.

Leading Snowflakes - A Book Every Software Team Manager Should Read

Fast forward to today, I have finally finished reading the book, and I could confidently say that this is a book every software team manager should read.

There is no hand-wavy theories or lectures, what the book offers is exactly what is said on the landing page - short, concise and pragmatic.

The book contains a lot of sound advice, practical tools and techniques as well as a ton of useful links. Reading through it, I can recall many moments where I thought to myself, "hey, that indeed is a very cool technique, I can totally see how I could use it and be better at leading."

I am now ready to implement those techniques and I am hopeful that I would become a better team leader and my team would become a better team as a result. :)

If you are a software team manager, or aspire to be one, I highly recommend this book!

To purchase only the ebook itself, Oren has kindly offered a 20% off discount, click to buy now.

[ifredwu]

Photos from JSConf AU 2014

JSConf AU has been fun - great talks and great venue! It was also a wonderful opportunity for me to test out my new camera kit: SONY A7r + Sony Sonnar T* FE 35mm f/2.8 ZA + Voigtlander Color Skopar 20mm f/3.5 SL-II with Novoflex SONY E-Mount to Nikon adapter.

Here's the kit (taken by Nikon D800 + Tamron SP 24-70mm F/2.8 Di VC USD):

Photos from JSConf AU 2014:

Breakfast in North Melbourne:

Conference:

Check out my Google+ Photo Album for larger versions.

[ifredwu]

On Hiring: Trial Week - Yay or Nay?

Today a blog post titled "Trial Week: Our Hiring Secret" has made to the Hacker News homepage. I naively tweeted my dislike and now I feel obligated to share my thoughts in a more meaningful and constructive way.

First of all, congratulations to the Weebly team, as this trial week strategy is clearly working very well for them.

I, on the other hand, am against using a trial week for vetting candidates, and I am going to share my thoughts.

Let this serve as a reminder to the rest of us: every organisation and team is different, so think carefully before committing to a given strategy.

One Week is a Major Commitment for the Candidate

In Australia, a full time employee typically gets four weeks of annual leave, with one or two weeks of which used up for the Christmas / New year down time. We are looking at asking candidates to spend 33-50% of their vacation time to commit to a trial week for one company - a terrible ROI (Return On Investment) from the candidate's perspective if you ask me.

Candidates who are currently employed, with multiple offers from other organisations are more likely to skip the trial week - from experience, this is often the higher quality candidate pool.

Side Effects

Increases the likelihood of burnout due to the reduced vacation time

Shrinks the candidate pool

Misses top talents who are unable to make the one-week commitment

As a result, the overall quality of the candidate pool drops

Paints an image of "not-caring (enough) about the employee's well being"

Of course, since the trial week is paid for, the employee could always take unpaid leave from their current employer.

Side Effect

Raises alarm bells at current workplace since one week of unpaid leave is significant

One Week is a Major Commitment for the Team

Given the trial only lasts a week - we better make it count! That means one or more current developers need to be assigned to take care of the trial developer - pairing and walking through existing systems, etc. This is assuming we are going to act responsibly, and not simply just direct the trial developers to their desks and ask them to "go for it".

Side Effects

Higher pressure for the team

More difficult to act on other priority tasks

Developer Productivity Curve (One Week is Not Enough)

From my experience of on-boarding new developers, it typically takes 4-8 weeks for a developer to become productive and effective in a new work environment.

According to Weebly, candidates are assigned with a project that is small enough to do in a week, but still resembles what the candidate would be doing if hired. It sounds great if it works, but for many organisations this is unfeasible, for instance:

There is no small projects to assign, unless invented

Navigating documentation and source code would take days, if not weeks

Either way, with one week of trial, the candidate is unlikely to have enough time to contribute as well as to be integrated into the team culture.

Side Effects

Higher chance of misjudging the candidate's ability and productivity

Significantly higher chance of creating solutions misaligned with the team and/or the organisation

Higher maintenance cost should the team decides to keep the solutions created

66% Hire Rate Suggests Deeper Hiring Issue

Weebly at the end of their blog post writes:

Our hire rate out of trial week is around 66%, which feels like the right level.

I respectfully disagree. A 66% hire rate from the trial week is a 34% failure rate on the pre-trial week recruitment process, and this is significant.

Which brings us to...

More Effective Ways to Vet a Candidate

Where I work, we have a simple, three-step recruitment process:

Complete a small and fun code challenge, in your own time and with your own pace. The code challenge usually takes 2-4 hours to complete.

Invited to our office to chat with our developers and founders, optionally done via video chat. This usually takes an hour or so.

Pairing session, usually takes 30-60min.

Step 2 and 3 are sometimes swapped. And we also check out the candidate's Github account if available, and their past projects if public.

In the code challenge we vet the candidate's problem-solving ability, software design sense, code quality, code style and ethics (it's easy to tell whether they cheated).

During the chat we vet the candidate's project experience, depth of knowledge, breadth of knowledge, communication skill and culture fit.

In the pairing session we vet the candidate's development practice, thought process and the ability to articulate.

By the end of the three steps we are usually pretty confident on +1 or -1 to hire the candidate. If we aren't, it's a -1.

But hold on, didn't I mention one week is not enough for a candidate to be productive and effective? Yes! And that's why most places have a three-month probation.

The difference between the long probation period and the short trial period, is not only in duration, but more importantly in commitment. In my opinion, only when both parties are committed can you achieve great result.

So, let's hear your say, what do you think? :)

Poll: Trial Week, Yay or Nay?

[ifredwu]

Protip: Unsync the "Index" Folder of Sublime Text 3 from Dropbox

If you're like me who uses both Sublime Text 3 and Dropbox, chances are you have your Sublime Text 3 folder synced in Dropbox.

I use my laptop as my primary workstation so most of the time it's docked and charged. Occasionally when I do use it on battery power however I notice the extremely poor battery life - typically only 2-3 hours.

Eventually I realised the power consumption was caused by Sublime Text 3 generating a temp file in its "Index" folder every second or so, and that triggers Dropbox to do the syncing - causing it to take 40-50% of CPU constantly.

So, here's a simple fix. Click on the Dropbox icon, then click on the gear icon and select "Preferences...", select the "Advanced" tab, and select "Change Settings..." for Selective Sync to bring up the column-based file dialog, now simply find and untick the "Index" folder and you're all set. :)

[ifredwu]

Protip: Faster Ruby Tests with DatabaseCleaner and DatabaseRewinder

Please also see this blog post on tweaking your ruby GC settings.

I use and love DatabaseCleaner, although historically I had never paid too much attention on the performance of its varies cleaning strategies - I'd always used truncation.

We use Postgres, and after digging around and finding out the difference between DELETE and TRUNCATE, I ended up improving our test suite speed by about 30-40% simply by tweaking the cleaning strategies.

RSpec.configure do |config| config.before :suite do DatabaseCleaner.clean_with :truncation DatabaseCleaner.strategy = :transaction end config.before do if example.metadata[:js] || example.metadata[:type] == :feature DatabaseCleaner.strategy = :deletion else DatabaseCleaner.strategy = :transaction DatabaseCleaner.start end end config.after do DatabaseCleaner.clean end end

Essentially, we want to truncate the DB only once before the whole suite runs to ensure a clean slate DB, then we only want to use deletion on Capybara tests, everything else should just use transaction which is the fastest strategy.

Now, as a bonus, I have just discovered @amatsuda's DatabaseRewinder which is a lightweight alternative that supports only ActiveRecord. It offers comparable performance with a much similar API.

RSpec.configure do |config| config.before :suite do DatabaseRewinder.clean_all end config.after do DatabaseRewinder.clean end end

By the way, we also use parallel_tests to scale our test suite to multiple processes, even on Travis CI and Wercker.

Hooray to faster tests! :)

[ifredwu]

Protip: Ruby Devs, Please Tweak Your GC Settings for Tests!

It was made apparent to me that many ruby devs either aren't aware or couldn't be bothered to tweak their ruby garbage collector settings.

Well, if you are using MRI, please start tweaking your GC settings. Here's what I use (on my 15" Macbook Pro Retina):

export RUBY_GC_MALLOC_LIMIT=90000000 export RUBY_FREE_MIN=200000

Not only can you tweak it for your local dev machine, you can also tweak Jenkins, Travis CI, Wercker and other CI solutions, making instant speed gain for your test suite!

Here's what we get:

Before After Local: 8m22s 5m52s Travis CI: 10m10s 6m0s Wercker: 8m45s 6m24s

YMMV depending on your system and available RAM.

[ifredwu]

Writing Sensible Tests for Happiness

Writing good, sensible tests is hard. As a Rubyist, I feel lucky to be part of a community that embraces tests. Though at the same time, I have come across too many projects that suffered from not having sensible tests.

What are Sensible Tests?

There often isn't a silver bullet when it comes to software development. Technical stuff aside, many things contribute to the solution to a given problem - the team, the project and the business to name a few. This article does not attempt to present any insights into the best practices for testing, rather it collects a few tips I believe would benefit those who are not yet comfortable with writing tests.

To me, sensible tests often have the following characteristics:

it does not replicate implementation details;

it does not provide false sense of security;

it runs reasonably quickly;

it does not slow down the development significantly;

it guides the programmer towards a better architecture;

and, it does not make you sigh every time you want to modify your tests.

Art and Science, TDD or Not TDD

Just like writing production code, writing tests is also a combined form of art and science. It takes not only experience, but also intuition to write sensible tests. You have to remember that not all projects and programmers are equal - take what you get, practise, and reflect on your findings.

Many times I had come across seasoned programmers practising TDD, only to find themselves cornered into a bad design that ultimately had to be thrown away. TDD does not save you from writing bad code, this article is not about TDD, it's about testing in general.

I am most comfortable with using RSpec, FactoryGirl, Capybara and Turnip, so I'm going to use these tools in the code. The principles however apply to any testing framework.

Test as Little as Possible to Reach a Given Level of Confidence

Kent Beck, the inventor (or more correctly, 'rediscoverer') of TDD once said:

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence.

I used to prefer testing almost everything, but over the recent years I find myself increasingly look for key areas of the system that need the test coverage the most. Typically, our systems would have:

unit and functional tests for model behaviours

unit and functional tests for services

integration tests for controller actions

request tests for API endpoints

isolated JavaScript tests

high level integration/acceptance tests in Gherkin

Model and service level tests are arguably the most important ones so we make sure we have really good test coverage for those. For controller tests we rely heavily on reusable production and test code for maintainability and sanity. For API endpoints we mostly test presented data structure - as business logic and data integrity should have been covered in model, service and controller layers. Isolated JavaScript tests take care of both presentational business logic and tricky UI tasks. And finally, acceptance tests handle happy-path user interactions.

Do Not Test Framework and Library Code

Writing application-specific business logic is difficult enough, you really should not test functionalities provided by the framework or libraries. Below is an example of such bad tests:

describe ApprovalStakeholder do it { should belong_to(:approval) } it { should_not validate_presence_of(:approval) } end

Similarly to how you would add useful comments, i.e. describe why instead of what, these tests should be replaced by tests that cover actual functionalities, for instance the reason why an ApprovalStakeholder doesn't need an Approval to be presence should be demonstrated in the tests:

shared_examples_for "non-approval specific stakeholder" do its(:action_that_does_not_care_about_approval) { should be_true } end describe ApprovalStakeholder do let(:approval) { create(:approval) } let(:user) { create(:user) } let(:role) { create(:role) } subject do build(:approval_stakeholder, :user_id => user.id, :role_id => role.id ) end context "with an approval" do before { subject.approval = approval } it_behaves_like "non-approval specific stakeholder" its(:action_that_does_care_about_approval) { should be_true } end context "without an approval" do it_behaves_like "non-approval specific stakeholder" its(:action_that_does_care_about_approval) { should be_false } end end

Ensure What You are Testing Makes Sense

The test case below showcases the original developer's lack of attention and awareness on designing a functional and secure system. It actually tests the reference keys for the ApprovalStakeholder object are allowed to be mass assignable, which is a recipe for disaster.

describe ApprovalStakeholder do it { should allow_mass_assignment_of(:user_id) } it { should allow_mass_assignment_of(:role_id) } end

De-Duplicate Test Cases

Looking at the example below, the first thing you'd notice is the amount of duplication.

describe ApprovalStakeholder do it "#traveller" do stakeholder = create(:approval_stakeholder, :approval => approval, :user_id => traveller.id ) stakeholder.stub(:user).and_return(traveller) approval.stub(:stakeholders_as).and_return([stakeholder]) approval.traveller.should == traveller end it "#authoriser" do stakeholder = create(:approval_stakeholder, :approval => approval, :user_id => authoriser.id ) stakeholder.stub(:user).and_return(authoriser) approval.stub(:stakeholders_as).and_return([stakeholder]) approval.authoriser.should == authoriser end end

It's true that tests act as a form of specification therefore should be optimised for clarity, in this case however, we could still maintain the clarity with significantly reduced duplication:

describe ApprovalStakeholder do let(:stakeholder) do create(:approval_stakeholder, :approval => approval, :user_id => user.id ) end subject { approval } before do stakeholder.stub(:user).and_return(user) approval.stub(:stakeholders_as).and_return([stakeholder]) end describe "#traveller" do let(:user) { traveller } its(:traveller) { should == traveller } end describe "#authoriser" do let(:user) { authoriser } its(:authoriser) { should == authoriser } end end

Do Not Replicate Implementation Details

I am often surprised to see many seasoned developers "enjoy" writing tests that essentially replicate the production code logic without much benefit. See below:

describe ApprovalStakeholder do it "references a user" do approval_stakeholder = build :approval_stakeholder, :user_id => 1 User.should_receive(:find).with(1) approval_stakeholder.user end it "references a role" do approval_stakeholder = build :approval_stakeholder, :role_id => 1 Role.should_receive(:find).with(1) approval_stakeholder.role end end

Rather than creating noisy tests, tests with actual assertions seem much more meaningful and readable:

describe ApprovalStakeholder do subject do build(:approval_stakeholder, :approval => approval, :user_id => user.id, :role_id => role.id, ) end its(:name) { should == "#{user.first_name} #{user.last_name}" } its(:role_name) { should == role.name } end

Reduce the Reliance on Mocks and Stubs

This is a difficult and often-debated subject. In my experience, having too many mocks and stubs even though speeds up the test suite, usually leaves too many holes in your tests and makes the test suite less accurate and effective. Fortunately, by using more service objects (described below), mocking and stubbing become more manageable as you use them mostly on external objects and interfaces.

Take Apart the System, One Service at a Time

If you're a Rails developer, you are already familiar with MVC. But just relying on MVC to hold your application architecture is probably not going to be sufficient for an average modern day web application. Many people like Service-oriented architecture, so do I.

Services are unassociated, loosely coupled units of functionality that are self-contained.

In my experience, as long as you are disciplined in having services do one and only one thing really well, testing becomes much easier.

For instance, we have a Bouncer service that is responsible for safeguarding resources - ensuring read-only attributes don't get overridden.

module Services class Bouncer def self.guard(resource, options = {}) if options[:existing_resource] resource.readonly_attributes.each do |attr_name| resource.send("#{attr_name}=", options[:existing_resource].send(attr_name)) end end resource end end end

The corresponding tests for this service are both fast and self-contained:

describe Services::Bouncer do class BouncerDude include Mos::Entity set_readonly_attributes :age, :gender attribute :name attribute :age attribute :gender end let(:resource) { BouncerDude.new(name: 'Penny', age: 28, gender: 'female') } let(:existing_resource) { BouncerDude.new(name: 'Sheldon Cooper', age: 34, gender: 'male') } subject { Services::Bouncer.guard(resource, existing_resource: existing_resource) } describe "#guard" do its(:name) { should == 'Penny' } its(:age) { should == 34 } its(:gender) { should == 'male' } end end

Recognise Common Patterns and Refactor Them into Services

One of the reasons why service-oriented architecture is so popular is because things are broken down into smaller, more manageable and more testable pieces. It is especially helpful for TDD practitioners as it significantly reduces the amount of coupling between your production code and your tests due to having simpler internals per test subject.

Take a look at the below example, which is hard to read, hard to test and error-prone:

module ApplicationHelper def branch_logo_options(branch) BranchLogo.where(branch_id: branch.id).map { |logo| [logo.file, logo.id] } end def branch_options(agency) BranchRepository.find(agency_id: agency.id, archived: false).map do |b| [b.name, b.id] end end def agency_user_options(agency, filtered_users) filtered_user_ids = filtered_users.compact.map(&:id) || [] AgencyUserRepository.find(agency_id: agency.id, archived: false).select do |u| !filtered_user_ids.include?(u.id) end.map { |u| [u.full_name, u.id] } end def current_agency_user_options(filtered_users = []) agency_user_options(current_agency, filtered_users) end def current_agency_trust_bank_account_options BankAccountRepository.find( agency_id: current_agency.id, archived: false, account_type: BankAccount::TRUST_ACCOUNT).map do |b| [b.account_name, b.id] end end def code_options_for(klass) klass.all.map { |cc| ["#{cc.code} - #{cc.name}", cc.id] }.sort end end

Let's refactor it into something more manageable, by introducing a service ShowGirl for fetching and presenting data collections:

module CollectionOptionsHelper def branch_logo_options(branch) Services::ShowGirl.present(branch, from: BranchLogo, show: :file) end def branch_options Services::ShowGirl.present(current_agency, from: BranchRepository) end def consultant_options(excluded_users = []) Services::ShowGirl.present( current_agency, from: AgencyUserRepository, show: :full_name ) do |collection| collection.reject { |user| user.id.in?(Array.wrap(excluded_users).map(&:id)) } end end def trust_bank_account_options Services::ShowGirl.present( current_agency, from: BankAccountRepository, show: :account_name, filters: { account_type: BankAccount::TRUST_ACCOUNT }, ) end def code_options_for(name) Services::ShowGirl.present( current_agency, from: Admin::Configurations::Essential.descendants.find { |d| d.name =~ /::#{name.to_s.classify}/ }, show: -> (item) { "#{item.code} - #{item.name}" } ) end end

Better yet, we can clean it up even further by introducing another service, BusBoy for just serving the data, and leaving ShowGirl for only presenting the data:

module CollectionOptionsHelper def branch_logo_options(branch) Services::ShowGirl.present( Services::BusBoy.serve(:branch_logos, branch: branch) ) end def branch_options Services::ShowGirl.present( Services::BusBoy.serve(:branches, agency: current_agency) ) end def consultant_options(excluded_users = []) Services::ShowGirl.present( Services::BusBoy.serve(:consultants, agency: current_agency), show: :full_name ) do |collection| collection.reject { |user| user.id.in?(Array.wrap(excluded_users).map(&:id)) } end end def trust_bank_account_options(account_type) Services::ShowGirl.present( Services::BusBoy.serve(:bank_accounts, { agency: current_agency, BankAccount::TRUST_ACCOUNT } ), show: :account_name ) end def code_options_for(name, options = {}) Services::ShowGirl.present( Services::BusBoy.serve(name, agency: current_agency), options ) end end

Basic Controller CRUD Actions

In one of our projects we have lots and lots of forms. Consequently we have lots and lots of CRUD actions. In order to keep our sanity as well as to make basic CRUD controllers maintainable, we have a custom DSL to make CRUD actions portable and testable:

module Profiles class TravellersController < BaseController authorize_resource class: Traveller datamappify_resources entity: Traveller, repository: TravellerRepository, filter_by: :agency_id, filter_value: -> { current_user.agency_id } end end

Most of our controller tests look like this:

require 'spec_helper' describe Profiles::AccountsController do let(:existing_resources) { [] } let(:create_resource) { Mos::Data.create_account } let(:create_resources) { Mos::Data.create_accounts(2) } let(:a_resource) { assigns(:resource) } let(:invalid_param) { { name: '' } } let(:params_key) { :account } let(:redirect_path) { profiles_accounts_path } it_behaves_like 'datamappify resources controller' it_behaves_like 'searchable resources controller', :name, :profile_id, :branch_id, :activated describe "permission" do context 'as a manager' do before do sign_in_as :manager end it_behaves_like 'with write access' it_behaves_like 'with read access' it_behaves_like 'with index access' end context 'as a consultant' do before do sign_in_as :consultant end it_behaves_like 'without write access' it_behaves_like 'with read access' it_behaves_like 'with index access' end end end

API Endpoint Tests

One of our projects at work is an API service that is essential to our platform. Naturally, we not only need to test the models, services and controllers, we also need to ensure the API endpoints do what they are supposed to do - mostly exposing the correct data structure.

During the early stage of the development, I had come up with ApiTaster - a super useful gem for visually testing our Rails application's APIs. Later on, as we continued to grow our API endpoints, we started utilising ApiTaster for our automated test suite too.

In essence, we have one API spec file responsible for describing which endpoints are tested and missed according to the information given by ApiTaster:

describe "API" do load 'db/seeds.rb' load 'spec/api_endpoints.rb' ApiTaster::Route.map_routes ApiTaster::Route.defined_definitions.each do |route| it "api endpoint #{route[:verb]} #{route[:path]}" do params = ApiTaster::Route.params_for(route).first expectation = ApiTaster::Route.metadata_for(route)[:expectation] setup = ApiTaster::Route.metadata_for(route)[:setup] verb = route[:verb].downcase path = parse_path_with_url_params(route[:path], params[:url_params]) setup.call if setup send verb, path, params[:post_params] response.body.should match_json_expression(expectation) end end # warn about undefined definitions ApiTaster::Route.missing_definitions.each do |route| pending "api endpoint #{route[:verb]} #{route[:path]}" end end

Then, we have a bunch of endpoint test files to do the actual testing, like this:

resource_response = ResponseHash[ :response => { :id => Integer, :name => String, :token => String } ] get '/:version/company', {}, { :expectation => resource_response } post '/:version/companies', { :model => FactoryGirl.attributes_for(:company) }, { :expectation => resource_response } put '/:version/companies/:id', { :id => 1, :model => { :name => 'New Company' } }, { :expectation => resource_response.with(:name => 'New Company') } delete '/:version/companies/:id', { :id => 1 }, { :expectation => resource_response }

Notice that for API endpoint tests we don't test the business logic or data integrity - these should be tested in models, services and controllers. What we do test are correct endpoints are exposed, correct parameters are accepted and correct data structures are returned.

Isolated JavaScript Tests

Many developers prefer to rely on their integration test suite to do JavaScript / UI testing. This approach is fine until you start making lots of front-end changes and constantly need to pinpoint the relevant feature spec.

Having an isolated JavaScript test suite (which should be run as part of your continuous integration process) is extremely beneficial and often saves debugging time.

I like Mocha so we use Konacha in our Rails app. Though Mocha with Chai is really not that different to Jasmine.

Custom JavaScript behaviour is obviously a good candidate for isolated testing:

#= require spec_helper describe "form toggle", -> beforeEach -> $("body").append(JST["templates/form/toggle"]) it "hides the collapsible field by default", -> $(".control-group.branch_deactivation_date").hasClass('in').should.be.false it "does not override if there is already a value", -> value = $("input#agency_deactivation_date").val() $("input#agency_activated").click() $("input#agency_deactivation_date").val().should.equal(value)

Sometimes it's also useful to ensure library code is initiated and triggered correctly, if you have other custom JS interact with it:

#= require spec_helper #= require bootstrap-datepicker describe "form dates", -> beforeEach -> @dateFormat = 'DD/MM/YYYY' $("body").append(JST["templates/form/dates"](dateFormat: @dateFormat)) it "has a placeholder", -> $("input").attr("placeholder").should.equal(@dateFormat) it "defaults to today's date", -> $("input#empty").focus() $("input#empty").focus() $("input#empty").val().should.equal(moment().format(@dateFormat)) it "does not override if there is already a value", -> value = $("input#filled").val() $("input#filled").focus() $("input#filled").val().should.equal(value)

"Real" UI Tests

Isolated JavaScript tests are super fast and useful. However, there are times when having pure JavaScript tests simply isn't enough, due to the complicated nature of DOM interaction and template rendering.

A while ago our calendar widget was broken due to a production and UAT environment issue that was not picked up by our JavaScript test suite. Since then we started adding dedicated UI tests in our acceptance test suite (we use Turnip):

@ui Feature: UI Background: Given I am signed in And I go to agency consultants page And I click on "Add New Consultant" Scenario: Calendar When I click "#agency_user_start_date" And I click ".day.active" within ".datepicker" Then I should see today as part of the date field

Effective Acceptance Tests

Writing acceptance tests - also known to many Rubyists as "Cucumber tests", is a double-edged sword - it's extremely useful, but very few developers can write good, maintainable Gherkin-style acceptance tests.

Here's an example of a badly written feature spec with too much implementation details and noise:

Feature: Session Background: Given I visit "/" And there is a user "admin" "password" Scenario: Sign in with valid credentials When I fill in "Username" with "admin" And I fill in "Password" with "password" And I click "Sign In" Then I should be on "/dashboard" Scenario: Sign in with invalid credentials When I fill in "Username" with "admin" And I fill in "Password" with "invalid_password" And I click "Sign In" Then I should not be on "/dashboard" Scenario: Sign out When I fill in "Username" with "admin" And I fill in "Password" with "password" And I click "Sign In" And I click "Sign Out" Then I should be on "/sign_in"

A much cleaner version with only high level, descriptive steps:

Feature: Session Background: Given I am on the homepage And there is a user "admin" with password "password" Scenario: Sign in with valid credentials When I sign in as "admin" with password "password" Then I should be signed in Scenario: Sign in with invalid credentials When I sign in as "admin" with password "invalid_password" Then I should not be signed in Scenario: Sign out Given I am signed in as "admin" with password "password" When I sign out Then I should be signed out

Final Thoughts

Writing good, sensible tests is hard. These examples and tips are by no means the silver bullet, and you might actually find some of them counter-intuitive in your particular situation. So again, take what you get, practise, and reflect on your findings. For Happiness! :)

Do you have any tips to share? If so please feel free to add a few comments!

[ifredwu]

Gotchas in the Ruby Sequel Gem

I haven't really used Sequel much therefore I am definitely a newbie. However, after days and nights of frustration, endless debugging and some search-fu during the development of Datamappify, I have finally arrived at the conclusion that Sequel is a capable library, as long as you are aware of the gotchas.

Gotcha 1: Always use "select"/"select_all", or your data records will mysteriously have wrong IDs!

In ActiveRecord, joining an associated model couldn't be simpler:

Post.joins(:author)

In Sequel, despite having a similar API for models to declare associations and their corresponding primary and foreign keys, you cannot do a join without specifying the keys:

Not good:

Post.join(:authors) # or Post.join(Author)

Better:

Post.join(:authors, :id => :author_id)

You would think the version above works - it doesn't. Even worse, the above example will give you incorrect data - the IDs of the Post records will now contain the IDs from their corresponding Author records! This is because upon a join, Sequel merges attributes from both models into a single hash.

The correct version:

Post.join(:authors, :id => :author_id).select(:posts__id, :posts__title, :posts__body) # or Post.join(:authors, :id => :author_id).select_all(:posts)

Gotcha 2: Always call "all" at the end of the chain, or the chain will present data in a different format.

In ActiveRecord, all of the below examples return an ActiveRecord::Relation collection:

Post.where(:title => 'Hello world') Post.joins(:author) Post.includes(:author)

And indeed, calling first on any of them returns an object of class Post (assuming the result collection is not empty).

Post.where(:title => 'Hello world').first.class #=> Post Post.joins(:author).first.class #=> Post Post.includes(:author).first.class #=> Post

In Sequel, the below examples all return a Sequel::DataSet collection:

Post.where(:title => 'Hello world') Post.eager(:author) Post.eager_graph(:author)

But let's see what we get from calling first.class on them:

Post.where(:title => 'Hello world').first.class #=> Post Post.eager(:author).first.class #=> Post Post.eager_graph(:author).first.class #=> Hash

Huh? Last one is a Hash? It turns out, if you call all at the end of chains to convert them to Arrays, then the returned collections are consistent:

Post.where(:title => 'Hello world').all.first.class #=> Post Post.eager(:author).all.first.class #=> Post Post.eager_graph(:author).all.first.class #=> Post

[ifredwu]

The Future of Computing, The Future of Computer Programmers - An Interview with Yukihiro "Matz" Matsumoto

A while ago I translated an interview with Matz done by a Chinese book publisher. The interview and the translation were well received, so this time I am translating another interview with Matz, done by Ito, the editor-in-chief from Japanese website Engineer Type. Since I don't read Japanese, the translation is based on Turing Book's Chinese translation.

The Chinese translator has done a great job translating the interview, but there are still many words and sentences lack sufficient context and therefore are difficult to grasp. I have put in many hours translating the text as well as doing researches to ensure the final article is readable. I hope you will enjoy it! :)

Ito: Thank you for doing an interview with us, Matz. I have just finished reading your latest book The Future of Computing, could you perhaps talk about the future of programming and software programmers in general?

Matz: Hmmm, this is difficult to answer… but thanks for reading my book!

Ito: In the book you've shared your thoughts on the past, present and future of different programming languages and software design patterns. Would you like to talk about the current state of the software industry? And is there going to be another paradigm shift in software development?

Matz: As discussed in my book, to predict the future of a high tech industry such as computing is not particularly difficult. I believe in the foreseeable future the computing industry is still going to advance based on Moore's law. Although, it is possible that in the next year or two quantum computers become a practical reality, in that case it will change everything! *chuckles* On a serious note, according to Moore's law, the cost of computing will decrease and the performance and capacity of computing will increase - this basic principle is unlikely to change. One thing I did notice in recent years is that due to the advancement in computer hardware, the software industry is subtly changing too.

Software Development in the Era of Multi-core and Cloud Computing

Matz: It was about twenty years ago (in 1993) I invented the Ruby programming language, yet it still runs surprisingly well on modern computers.

What this means is that in the past twenty years the computing environment which the software runs on did not see any fundamental changes. In recent years, we started seeing computing power being shifted from having higher CPU frequencies to being distributed over more CPU cores. And that means software needs to move in that direction too.

Matz: Software has not seen major changes for years.

Ito: And this is covered in the last chapter from the book, right?

Matz: Yes. Similarly to multi-core, cloud computing is advancing in the same direction. The future of computing is all about utilise multiple CPUs or computers effectively.

Ito: So, how does that change software development?

Matz: In the past ten years or so we have been seeing more and more things happen on the Internet, and the Internet is an amazing application platform for extension and distribution. Compared to software engineers working on mainframe computers, web developers are naturally more familiar with the concepts of multi-core and cloud computing.

Ito: After interviewing many web and mobile startups, we realised that the number of software engineers working in PaaS and cloud computing have been increasing rapidly.

Matz: Absolutely. And I do believe that "not needing to purchase and own dedicated hardware" is going to be the mainstream. The idea and thought process of "not owning" is not only important for software development, but also important for business development.

"Owning" Becoming a Liability, Not Asset

Matz: In the past, "owning" was seen as the source of vitality of a corporation - those who own high performance mainframe computers were able to do business transactions in high volume whereas those who do not were not able to compete.

These days the landscape is changing - those corporations who do not "own" expensive hardware have more competitive edge. Let's say it takes five years to break even from the expensive investment of servers, during that time those machines are being put in use to realise their full potential and to justify their cost. It may appear to have saved the business cost but it is not, simply because the value of the hardware decreases as each day passes by.

To put simply, we are now entering the era of "owning" being a liability rather than asset. If you had the most advanced hardware, software engineers were able to develop efficiently. On the contrary, if you didn't, then you might want to get used to the hours-long waiting for the code to compile. *chuckles* The rise of cloud computing platforms like Heroku is making "owning" a thing of the past.

Also, "not owning" has several advantages on the development as well as the commercial front. For instance, it allows many startups to rise. In the past, in order to start a new business you would need capital for purchasing servers and/or renting servers in a data centre. These days, to get started on a platform like Heroku couldn't be easier, for example on Heroku you could start with just one dyno for free. This new way of developing software significantly reduces costs and risks.

Years ago I read an essay called Ramen Profitable by Y Combinator's founder Paul Graham. "Not owning"'s flexibility and agility contribute a great deal to it. And this trend has now grown beyond just relevant to startups, in fact in the recent years many large corporations have begun adapting this approach too.

In the United States, corporations like Disney and Best Buy are indeed utilising Ruby, Rails and Heroku to rapidly grow their internal infrastructure in a cost-effective fashion. What was once considered competitive edges to venture capitalists, like "rapid development" and "development flexibility" are now also possible for these giant corporations.

Ito: What about the giant corporations in Japan?

Matz: I have never worked in a big corporation so I can't tell where they are heading. People have been optimistic, though as an observer I am concerned.

Real Benefits of Innovation in Cloud Computing Not to Be Overlooked

Ito: What makes you concerned about software development?

Matz: The traditional approach of developing software is still the norm. For example, some corporations, even though use Amazon Web Services, still rely on system administrators to handle their infrastructure. It is too common to see a software development team consists of over a dozen people.

This in my opinion defeats the purpose and forfeits the benefits of "not owning" servers. There are simply too many of these case studies whereby only on the surface of cloud computing is explored and understood.

I have to say I am disappointed by some of the so-called "private clouds" owned by large corporations. The advantage of cloud computing is to utilise multiple computers in the cloud, but those private clouds are essentially their internal data centres. Isn't that the same as owning a bunch of servers?

Matz: Many companies barely scratch the surface of emerging technologies.

Ito: Indeed, it is too often to see the real benefits of emerging technologies overlooked or misunderstood. Anything else that makes you concerned about the future?

Matz: Nowadays the speed of development has always been a priority from big development projects in B2B to small development projects in many startups. Yahoo! Japan even coined a term "爆速化" (explosively high speed) to indicate the importance of development speed in the ever more competitive and engaging markets.

Looking at things this way, those so-called "system integrators" are becoming obsolete. Should they just give up what they do or continue? I don't know, but I do know that the gap between them and engineers who have the capability and skills to create real value is increasing.

Career Longevity of Software Engineering

Ito: Who are those engineers who have the capability and skills to create real value?

Matz: The ones who would put in effort to create software or systems from a prototype to a final product. And this has nothing to do with whether they work in web or system integration, or whether it's consumer oriented or corporate oriented.

Matz: Unbalanced skill combination leads to a gloomy future.

Ito: Do you mean the engineers who are capable from design to implementation?

Matz: Yes. Speaking of which, software developers have to know more than just system design - they cannot survive without knowing how to code. Just like in life, you cannot survive without being down to earth. *chuckles*

Despite the fact that it is pointless to have someone doing only the system design, and not the development as well, the System Integration industry is still going strong in Japan - and it is in fact an industry with high profit margin.

Even if the system designers came up with questionable specifications, or if the programmers were sloppy so the software was terrible to use, users would still use it despite whining. Flaws are easily glossed over under high profit margins.

But just as discussed before, as development speed increases, profit margins would undoubtedly become smaller. Flaws are therefore harder to gloss over.

In my opinion, if things don't change, those run-of-the-mill software engineers might not survive in five years. Worse, the junior to mid-level to senior programmer corporate ladder is going to collapse.

Say, you wouldn't want to start a VHS rental shop when DVDs were on the rise, would you?

Difference Between Those Who Control Their Destiny to Those Who Don't

Ito: Do you have any advice for those who do not wish to be in a "gloomy future"? What can they do?

Matz: To innovate and to create new things, I suppose.

It's not all doom and gloom. Even though many ageing technologies have been or are being replaced by the web, jobs will not disappear overnight. I think many software developers will still be employed in those jobs.

Having said that, it is always good to create new things or even invent new programming languages.

Ito: What are these "new things"?

Matz: I see three types of new things.

First of all, new services. If you can create a new service, or a service that offers superior user experience - it would be an innovation.

Secondly, new technologies. To come up with technologies better than the existing ones - and this is what I have been doing.

mruby was released earlier this year on Github.

Or, another way is to invent new algorithms.

The three ways I mentioned have different difficulties, but they share the same goal - to create something that hasn't existed yet. Those who keep working on these kinds of challenges are the true outstanding software engineers.

The ones who do not challenge themselves to create new things are often falling behind - they learn a hip new language today and try a new web framework tomorrow, but still lack the foresight to invent and to improve.

Of course, it is important to learn and try new things, but if you see them as your ultimate goal then you will lose control of your destiny. I believe that the ones who do not get boggled down in every new trendy thing will ultimately be happier.

Software Development is a Punch to Deficiency

Ito: Here is a sharp question: be a follower rather than an inventor is always easier and perhaps makes more money too. What makes you keep inventing?

Matz: My standard answer would be "because writing and running new programs make me happy". But the real reason is because I don't like deficiency.

There are people who have different opinions and thought processes, I would often come up with questions like "why was it done this way" or "this will be too hard to use".

Matz dislikes deficiency, so he invented the ruby programming language.

Ito: True, but all products more or less reflect their producers' preferences, right?

Matz: Absolutely, and I am not saying that this is bad or anything. I just hate to point fingers at other people's preferences - if you don't like something, make your own! This is a basic trait of a good software engineer, and is what makes open source sustainable.

In open source projects, all the source code is publicly available therefore it is very easy to see how a program is designed. As long as you have ideas on how to improve and optimise the design, you are welcome to do so.

Now it is an entirely different story for certain things in the society. *chuckles* At least in software development, we can rely on our skills and knowledge to improve and to change. If it's your own creation, it can be adjusted and adapted to suit the ever changing needs.

This is same for Ruby - I like programming languages and more importantly I like improving programming languages myself, and that's why I still work on Ruby till this day.

Software Development, One of the Rare Careers that Could Make a Change on Your Own

Matz talking about developer happiness, wearing his "[Ruby City MATSUE](https://www.facebook.com/rubycitymatsue)" polo shirt.

Matz: I think I have the right personality for developing software. Only the software industry can tolerate my carefreeness - am I too arrogant for saying that? *chuckles*

In all honesty, software development is one of the rare careers that could bring positive changes to the society on your own. It's a wonderful occupation that brings happiness and fulfillment!

Ito: Many people would predict their software future based on theories, but Matz you always use "happiness".

Matz: That's right. Because only you can control your own destiny. It doesn't matter if you were told to do things in a way just becasue "Matz said so" - ultimately, I cannot be responsible for your destiny. You should make your own decisions.

I would still say things like "the future might look like this", but these are just my personal opinions.

And this is the same even for today's discussion - if someone thinks he does not agree with what Matz has said, he should follow his own decision and the path he chooses.

Exploring the Future: "You" Are the Only Constant

Ito: Having read The Future of Computing, I remember you talked about the inception and development of varies programming languages. But we all know that the IT industry is moving in a rapid pace, it is difficult to rely on history to guide us through to the future. If multi-core and cloud computing are only just the beginning of a paradigm shift, why did you write about the things happened in the past?

Matz: Technologies progress just like a pendulum clock.

Matz: People see things differently - and I believe the IT industry is progressing in a manner similar to the swing motion of a pendulum clock.

As more and more new programming languages, techniques and frameworks pop up, software development related technologies are progressing whilst seeking for balance.

So, how does "the most balanced from the past" become "the most balanced right now"? Think about how pendulum clock swings and in the past how technologies have emerged - you could then predict roughly what would constitute "the most balanced in the future".

Use "centralised computing vs distributed computing" as an example, in the past there was usually only one centralised mainframe computer, later on to increase the processing capability commodity server farms were utilised, and now we are moving towards cloud computing.

There is no point to look at a particular past event. If you wanted to predict a technology in the future, knowing what has contributed to the balance of a past technology's rise and fall is going to help.

Human's ability is one of the factors too, because we have limited capability as a language designer it is useful to look at what others have done to cater for our ability, and therefore improve and evolve the technology.

In the book I briefly talked about Dart and Go. As a programming language inventor I find it really fascinating to explore the thought processes behind those language designers. And it has helped me to gain a deeper understanding of human behaviour.

Ito: I was going to ask why it is so important to study the past, now I know.

Matz: I mentioned this in the beginning - computing has not seen major changes for years.

Programming languages invented over fifty years ago are still in use today, and Ruby has been around for twenty years now. This proves that computing is progressing slower than what a lot of people believe.

On that note, there are many past cases whereby focuses were put on what was cool and new without understanding why. Compared to those "follower" software developers, the ones who command and understand the principles and theories behind changes and progresses have a much longer career longevity.

If you are a software developer who wants a longer career longevity, please read The Future of Computing! *chuckles*

Ito: Thank you Matz for talking to us today!

[ifredwu]

Datamappify - A New Take on Decoupling Domain, Form and Persistence in Rails

This post is about the ruby library we are building - Datamappify, please go check it out.

At Locomote we are building a relatively large web application using Rails. Before we began to lay the foundation, we knew very well that if we wanted the project to be maintainable we had to architect the system with extra care and attention. More specifically, we can't rely on simply using ActiveRecord which combines behaviour and persistence as our domain models.

We began our search for something that would help us decouple our application from the domain layer down to the form handling. We've found a couple of gems that are close to what we were after - Curator, Minimapper, Edr and later on Reform. They are all wonderful gems but unfortunately none of them has everything we need.

Here are the things we need:

Decouple domain logic and data persistence.

Decouple models and forms.

Support ActiveRecord (or at the very least, ActiveModel) so we can still use many of the awesome gems like Devise, SimpleForm and CarrierWave.

Support attributes mapped from different data sources (e.g. remote web services from third-party vendors).

Support lazy loading so that attributes stored on remote data sources will not get triggered upon loading the entities.

All things considered, we bit the bullet and started working on Datamappify.

Before I go into details and examples, here is an extremely simplified overview of Datamappify's architecture:

As you can see, Datamappify is loosely based on Repository pattern and Entity Aggregation pattern.

Datamappify has three main design goals:

To utilise the powerfulness of existing ORMs so that using Datamappify doesn't interrupt too much of your current workflow. For example, Devise would still work if you use it with a UserAccount ActiveRecord model that is attached to a User entity managed by Datamappify.

To have a flexible entity model that works great with dealing with form data. For example, SimpleForm would still work with nested attributes from different ORM models if you map entity attributes smartly in your repositories managed by Datamappify.

To have a set of data providers to encapsulate the handling of how the data is persisted. This is especially useful for dealing with external data sources such as a web service. For example, by calling UserRepository.save(user), certain attributes of the user entity are now persisted on a remote web service. Better yet, dirty tracking and lazy loading are supported out of the box!

You can read more about Datamappify in the project's README. For this blog post I will focus on how we use Datamappify in our Rails application. We are still early days in our application development and Datamappify still has quirks and issues, but I am hoping this post will illustrate some of the key benefits of Datamappify.

Getting Started

Getting started with using Datamappify is really easy. We simply include it in the Gemfile:

gem 'datamappify'

To keep things organised, we put entities and repositories in their respective directories under app:

Entities

The reason we wanted Datamappify to utilise existing ORMs like ActiveRecord, is so that we could still use gems like Devise.

So, we have a UserAccount model that handles authentication:

class UserAccount < ActiveRecord::Base devise :database_authenticatable, :recoverable, :rememberable, :trackable, :validatable end

UserAccount has one and only one purpose - user account authentication. Other user behaviours would be contained in either the User entity itself or service objects. Speaking of the User entity, it looks like this:

class User include Datamappify::Entity attr_accessor :user_account attribute :account_username, String attribute :account_email, String attribute :first_name, String attribute :last_name, String attribute :activated, Boolean, :default => true attributes_from Contact, prefix_with: :work validates :first_name, presence: true validates :last_name, presence: true references :agency def full_name "#{first_name} #{last_name}" end end

We include Datamappify::Entity in the User class to make it an entity. We set the :user_account accessor is so that we could attach the UserAccount object onto the entity.

The attribute DSL is from Virtus - we get attribute type coercion for free, awesome!

attributes_from is a DSL provided by Datamappify - it essentially "mounts" all the attributes from another entity, in this case the Contact entity, which looks like this:

class Contact include Datamappify::Entity attribute :email, String attribute :phone_number, String attribute :fax_number, String validates :phone_number, presence: true validates :fax_number, presence: true end

All the attributes and validation rules from Contact are now "mounted" on User. attributes_from Contact, prefix_with: :work is equivalent to:

attribute :work_email, String attribute :work_phone_number, String attribute :work_fax_number, String validates :work_phone_number, presence: true validates :work_fax_number, presence: true

Validation DSL is provided by ActiveModel::Validations.

references :agency is a convenient DSL provided by Datamappify. It is equivalent to:

attr_accessor :agency attribute :agency_id, Integer

Now that we have entities, we need a way to retrieve and store them. For that we need repositories.

Repositories

Here is the user repository:

class UserRepository include Datamappify::Repository for_entity User default_provider :ActiveRecord map_attribute :account_username, 'ActiveRecord::UserAccount#username' map_attribute :account_email, 'ActiveRecord::UserAccount#email' map_attribute :work_email, 'ActiveRecord::Contact#email' map_attribute :work_phone_number, 'ActiveRecord::Contact#phone_number' map_attribute :work_fax_number, 'ActiveRecord::Contact#fax_number' end

Similarly to an entity, we include Datamappify::Repository to make the class a repository. We specify for_entity to link the repository to an entity, and default_provider to use a specific data provider for unmapped attributes.

Unmapped attributes are the ones not specified in map_attribute, in this case they are first_name, last_name and activated. Unmapped attributes are actually automatically mapped by Datamappify, so the user repository essentially does this:

map_attribute :first_name, 'ActiveRecord::User#first_name' map_attribute :last_name, 'ActiveRecord::User#last_name' map_attribute :activated, 'ActiveRecord::User#activated'

The first argument of map_attribute is the name of the attribute from the User entity (e.g. :first_name).

The second argument is a string containing three things:

ActiveRecord is the data provider.

::User is the ActiveRecord model class.

#first_name is the ActiveRecord attribute from the model class.

Even though the User entity is a representation of a user on the domain level, the underlying data structure does not necessary have to be. As you can see from the user repository example, we are mapping :account_username and :account_email to the UserAccount ActiveRecord model we've seen before. And we have a bunch of contact details attributes mapped to the Contact ActiveRecord model.

The database schema therefore looks like this:

create_table "contacts", force: true do |t| t.string "email" t.string "phone_number" t.string "fax_number" t.integer "user_id" end create_table "user_accounts", force: true do |t| t.string "username", default: "", null: false t.string "email", default: "", null: false t.string "encrypted_password", default: "", null: false t.string "reset_password_token" t.datetime "reset_password_sent_at" t.datetime "remember_created_at" t.integer "sign_in_count", default: 0 t.datetime "current_sign_in_at" t.datetime "last_sign_in_at" t.string "current_sign_in_ip" t.string "last_sign_in_ip" t.integer "user_id" t.datetime "created_at" t.datetime "updated_at" end create_table "users", force: true do |t| t.string "first_name" t.string "last_name" t.boolean "activated" t.integer "agency_id" end

Note that because the Contact entity is mounted on the User entity, we need a foreign key user_id in the contacts table to link them.

Data providers

Because we are allowed to specify a data provider (i.e. ActiveRecord) for each attribute, we can map attributes to entirely different data providers! For instance, we could have:

map_attribute :first_name, 'ActiveRecord::User#first_name' map_attribute :last_name, 'ActiveRecord::User#last_name' map_attribute :activated, 'Sequel::UserActivation#activated'

Now the activated attribute is mapped to the UserActivation Sequel model. This powerful feature would allow us to develop data providers that communicate with remote web services. :)

Repository APIs

Datamappify provides a bunch of APIs for retrieving and storing data. These APIs are being developed as needed during our application development.

For instance, to find a particular user entity by ID:

user = UserRepository.find(1)

To get all the users:

users = UserRepository.all

To search for users:

users = UserRepository.find(:first_name => 'Fred', :activated => true)

To save a user:

UserRepository.save(user)

To delete a user:

UserRepository.destroy(user)

There are some limitations for certain APIs, you can read more about them here.

When saving an entity, because Datamappify has an internal pool for tracking dirty attributes, only those dirty attributes will get saved.

Also, Datamappify supports attribute lazy loading, all we have to do is to tell our entity to become lazy aware:

class User include Datamappify::Entity include Datamappify::Lazy end

Once an entity is lazy aware, a repository will inject the lazy loading mechanism onto the entity when it retrieves such entity (via find).

Both repositories and entities support inheritance, again, you may read more about them in the project's README.

Putting Things Together With A Controller

Remember we have the UserAccount for authentication? So how does that work? Well, here's the piece of code in our application controller:

class ApplicationController < ActionController::Base before_filter :authenticate_user_account! private def current_user unless current_user_account.blank? @current_user ||= UserRepository.find(current_user_account.user_id) end end helper_method :current_user end

Not too different from the normal Devise workflow hey? ;)

Forms

Web applications typically have lots of forms - ours is no different. It turns out, Datamappify can help build form objects too!

Here's the view portion of one of our forms (we use Slim templates):

# our standard form layout = simple_form_for submission_target, url: submission_path, signed: true do |f| = yield(f) .form-actions .pull-left - if archivable? && resource.persisted? = link_to 'Archive', '#', method: :delete, class: 'btn-archive' = f.submit 'Save' = link_to 'Cancel', cancel_path

# actual form content = render layout: 'forms/resource' do |f| h3 Account Details .row-fluid .span4 = f.input :account_type, collection: BankAccount::ACCOUNT_TYPES.invert .span4 = f.input :account_code .span4 = f.input :account_name .row-fluid .span4 = f.input :account_number .span4 = f.input :bank_name .span4 = f.input :bank_branch .row-fluid .span4 = f.input :bank_bsb, label: "Bank BSB" .span4 = f.input :activated, as: :boolean h3 Branch Details .row-fluid .span4 = f.input :branch_address_line_1, label: "Address 1" .span4 = f.input :branch_address_line_2, label: "Address 2" .span4 = f.input :branch_address_line_3, label: "Address 3" .row-fluid .span4 = f.input :branch_state, label: "State" .span4 = f.input :branch_postcode, label: "Postcode" .span4 = f.input :branch_country_code, label: "Country" do = f.country_select :branch_country_code, ['au', 'us', 'gb'], value: 'au' - f.add_signed_fields :branch_country_code .row-fluid .span4 = f.input :branch_phone_number, label: "Phone number" .span4 = f.input :branch_fax_number, label: "Fax number" h3 Merchant Details .row-fluid .span4 = f.input :merchant_number .span4 = f.input :merchant_name .span4 = f.input :merchant_address .row-fluid .span4 = f.input :merchant_biller_code, label: "Biller code" .span4 = f.input :merchant_bpay_method, collection: Merchant::BPAY_METHODS.invert, label: "BPAY method" h3 Payment Details .row-fluid .span4 = f.input :payment_number .span4 = f.input :payment_merchant, label: "Merchant description" .span4 = f.input :payment_reference_type, collection: PaymentAccount::REFERENCE_TYPES.invert, label: 'Reference type'

As you can see, it's a form containing details from bank account, branch address, branch contact information, merchant and payment. And indeed we do have bank_accounts, addresses, contacts, merchants and payment_accounts tables in our database. Yet, the form still remains flat-structured and submits via simple_form_for.

This is thanks to our BankAccount entity and BankAccountRepository repository:

class BankAccount include Datamappify::Entity include Concerns::Archivable ACCOUNT_TYPES = { 'general' => 'General', 'trust' => 'Trust' } attribute :account_type, String attribute :account_code, String attribute :account_name, String attribute :account_number, String attribute :bank_name, String attribute :bank_branch, String attribute :bank_bsb, String attribute :activated, Boolean, default: true attributes_from Address, prefix_with: :branch attributes_from Contact, prefix_with: :branch attributes_from Merchant, prefix_with: :merchant attributes_from PaymentAccount, prefix_with: :payment references :agency validates :account_type, Mos::Validation.in(ACCOUNT_TYPES) validates :account_code, Mos::Validation::STANDARD_TEXT validates :account_name, Mos::Validation::STANDARD_TEXT validates :account_number, Mos::Validation::STANDARD_TEXT validates :bank_name, Mos::Validation::STANDARD_TEXT validates :bank_branch, Mos::Validation::STANDARD_TEXT_OPTIONAL validates :bank_bsb, Mos::Validation::BSB validates :merchant_bpay_method, Mos::Validation.presence_depend_on(:merchant_biller_code) validates :payment_reference_type, Mos::Validation.presence_depend_on(:payment_number) end

class BankAccountRepository include Datamappify::Repository for_entity BankAccount default_provider :ActiveRecord map_attribute :branch_address_line_1, 'ActiveRecord::Address#address_line_1' map_attribute :branch_address_line_2, 'ActiveRecord::Address#address_line_2' map_attribute :branch_address_line_3, 'ActiveRecord::Address#address_line_3' map_attribute :branch_state, 'ActiveRecord::Address#state' map_attribute :branch_postcode, 'ActiveRecord::Address#postcode' map_attribute :branch_country_code, 'ActiveRecord::Address#country_code' map_attribute :branch_phone_number, 'ActiveRecord::Contact#phone_number' map_attribute :branch_fax_number, 'ActiveRecord::Contact#fax_number' map_attribute :branch_email, 'ActiveRecord::Contact#email' map_attribute :branch_website, 'ActiveRecord::Contact#website' map_attribute :merchant_number, 'ActiveRecord::Merchant#number' map_attribute :merchant_name, 'ActiveRecord::Merchant#name' map_attribute :merchant_address, 'ActiveRecord::Merchant#address' map_attribute :merchant_biller_code, 'ActiveRecord::Merchant#biller_code' map_attribute :merchant_bpay_method, 'ActiveRecord::Merchant#bpay_method' map_attribute :payment_number, 'ActiveRecord::PaymentAccount#number' map_attribute :payment_merchant, 'ActiveRecord::PaymentAccount#merchant' map_attribute :payment_reference_type, 'ActiveRecord::PaymentAccount#reference_type' end

Pretty clean and straightforward. What do you think?

Datamappify - Keeping Your Domain Layer Sane

Datamappify's concept isn't new, but there simply isn't anything in the ruby community that solves everything Datamappify tries to solve.

I hope that this post has not only introduced you to Datamappify, but has also made you think about how to make your application's domain layer more decoupled.

Thanks for reading, and if you have any feedback for Datamappify please get in touch with me!

[ifredwu]

Would you be interested in reading such a book?

[ifredwu]

ActiveRecord and DB Migration Ate My Model Attributes!

Update: You might also want to check out reset_column_information.

So a few days ago we started seeing the following errors on our Jenkins builds (swapped with fictional model and attribute names):

NoMethodError: undefined method `attack_power=' for #<Ironman:0x00000008525d20>

attack_power is a new attribute we recently added to the Ironman ActiveRecord model.

I was baffled, as the table column is clearly there but ActiveRecord couldn't see it.

This weird behaviour is confirmed by debugging the model:

Ironman # => Ironman(id: integer, created_at: datetime, updated_at: datetime) Ironman.column_names # => ["id", "created_at", "updated_at"]

And by debugging the schema:

ActiveRecord::Base.connection.structure_dump # => "CREATE TABLE `ironmans` (\n `id` int(11) NOT NULL AUTO_INCREMENT,\n `created_at` datetime NOT NULL,\n `updated_at` datetime NOT NULL,\n `attack_power` int(11) NOT NULL,\n PRIMARY KEY (`id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;\n\n"

So apparently somehow ActiveRecord ate our attack_power attribute!

After some struggling and head-scratching, finally I've discovered that during the migration if we call the Ironman class, then any subsequent attribute changes to the class will not be recognised by ActiveRecord.

It turns out, because we run our test suites by invoking rake tasks:

Rake::Task['spec:something'].invoke Rake::Task['spec:something_else'].invoke

And we manually reset our database beforehand too within the same Rakefile:

Rake::Task['db:reset'].invoke

ActiveRecord will cache its attributes as soon as a model class (Ironman) is called during the migration.

The fix? Simple! Simply use another process to run the database reset:

system 'rake db:reset'