Tom Moor

[tommoorblog]

Ask, Listen, Observe, Distill

This weeks Startup Edition prompt is "How do you discover what people really want?" - at first glance the answer to this question might seem obvious, you just ask, right?! But the way you ask and how you process the feedback can make the difference between a terrible product and a runaway success.

Here i'm thinking from the perspective of developing and improving an existing early-stage product after you have validated the idea. For me, it seems natural to split the process of soliciting feedback and deciding upon production direction into several steps.

Ask

First, is your company really open to receiving feedback? Including a contact link in the footer of your website is no longer enough - you should allow and encourage users to get in touch using whichever channel they prefer. For example at Sqwiggle we include a contact/feedback button in the main menu of all our apps. You can also simply reply to any email sent from the service to reach support (here's a great post on this by Joss, Founder of OpenExchangeRates). We also keep an active presence on Twitter and Facebook, making sure to reply to anyone that reaches out here too.

You'll get the best results by combining and including a variety of channels. As well as email and social - how about blog comments, quora answers, discussion boards, phone calls, surveys and even face-to-face visits (a personal favorite of the Sqwiggle team!).

Listen

In the early days your users will usually be very forthcoming with their thoughts and ideas, even their hopes and dreams for your product. Managing and keeping track of this can be hard, there are a few tricks that I use including:

For smaller items and bug reports we find it easiest to simply add these directly into GitHub issues (or your issue tracker of choice!)

Tagging of emails so that we can find customers later and let them know when their idea has been addressed.

We use our internal discussion tool, to talk about ideas and their potential implementation and priority.

One of the key ideas that I've embraced since my time at Buffer is from 37 Signals - forget feature requests, If a feature is really wanted by your users you won't need to keep track as it it will be mentioned repeatedly! For example in our case, features such as screen sharing and file upload both fall under this category.

Observe

Watching customers use your product can be one of the most painful, frustrating, eyeopening and ultimately useful experiences you will ever have as a software developer.

And finding people to observe is easier now than ever before, services such as User Testing let you perform tests at low cost and watch the resulting video back online. Alternatively you can place adverts on craigslist for testers and bring them into your office to record for a more directed/hands-on test.

Distill

I believe this is the most important yet also the most often missed step in the process. After all, nothing creates a poor product quicker than simply adding everything your users ask for! The great entrepreneurs of our time excel at the craftsmanship of distilling and filtering ideas down to their fundamental principles.

For example, Elon Musk regularly talks about his philosophy of reasoning from first principles rather than by analogy (check out this great interview with Kevin Rose). The same principle applies with product feedback. Users will often make many assumptions with their suggested solutions, the key is to figure out the core problem and think from the ground up how to tackle it within the bounds of your product vision.

For example, with Sqwiggle we rethought video communication in this way - and realised that for our use-case the traditional 'call and answer' model that other software uses just didn't make sense.

In my mind these steps are a great start. It would be great to know your processes for finding out what your users really want - i'd love to know in the comments. You can also read more posts on this topic on Startup Edition!

Photo by Giampaolo Macorig

[tommoorblog]

Pricing That Scales

We recently started charging companies to use Sqwiggle after only a month in private beta. We definitely don't have all of the functionality that customers asked for, there are still bugs and the price is high enough to put-off some companies from using the service.

At first glance this might seem like a terrible idea, but being clear about our pricing and business model from day one has helped to both differentiate Sqwiggle from other perceived competitors and proved that companies are willing to pay to solve the problem we've identified - a crucial part of validating the business idea.

SaaS Kicks Ass

The best pricing strategy balances the needs of your startup and customer perfectly. I believe this is the reason that SaaS has become an increasingly popular pricing choice. Our current model is to charge monthly, per user with a two week free trial. Every factor that makes this a great choice for Sqwiggle also works as an advantage for the customer.

For example, we plan on running servers for hosting video calls with many participants through Sqwiggle. For an app that becomes part of a teams workflow being used all day, every day, the cost of this soon becomes significant. By charging per-user, as the amount of video bandwidth increases, so does our income.

On the flipside, charging per-user allows the cost of using Sqwiggle to grow with the size of your business. If you're paying an employee thousands of dollars in wages per month then a couple of cents per day to make them more efficient and happy is a no-brainer.

There are also a ton of other advantages that have been covered many times; predictable recurring revenue is the lifeblood of a SaaS business allowing us to plan expansion, costs and hiring. A trial with zero upfront payment means that employees can champion Sqwiggle internally and see the benefits without needing to ask higher management for approval.

The Price Point

It's very early days and we're still experimenting with the optimum price to charge per user. At the moment we have a single price of $9 per user, per month. There were a couple of factors that came into our decision to choose this amount:

Value

For a distributed team Sqwiggle offers a lot more value than the competition with persistent presence and instant video chat - we wanted to reflect this in the price, charging more than the typical $2-3 per user.

Psychology

Three studies by MIT and the University of Chicago in 2003 showed "conclusive evidence that $9 price endings can increase demand", even amongst cheaper options. Whilst still charging a premium price and taking into account this rule we wanted to keep the figure as a single digit so that psychologically it still feels like it is relatively much lower than $10 or $9.99 than it is.

Simplicity

One of the key values we have at Sqwiggle is simplicity. We put great pain into ensuring that every part of the product is easy to use and understand. Having a single price keeps things inline with our values and avoids any decision paralysis. "A decision can be treated as over-complicated, with too many detailed options, so that a choice is never made".

What do you think, is this strategy wise? Is there anything you would you do differently? I'd love to hear your comments.

This post was written as part of StartupEdition, a weekly collection of curated posts by startup-minded people. Read more thoughts on pricing strategies here.

[tommoorblog]

How to Build Company Transparency and Reduce Email Using WordPress P2

At Sqwiggle we've been using the P2 WordPress theme for over a month now for a large chunk of our asynchronous internal discussions and I thought it would be interesting to pull apart our time with it a little.

What is P2

For those that might not have heard of P2 before - It is a theme based on Prologue which changes the flow and interface of WordPress to move composition inline. This makes creating posts as easy as Tumblr or Twitter for example, changing the dynamic to that of an internal social network.

In the last month three of us have posted 170 times with 50 different tags, and commented on those posts over 260 times, presuming we would have had these conversations over email we might have saved our individual inboxes of around ~430 emails each and that's not even counting other interactions such as 'liking' a post!

What is it good for

Asynchronous discussions

P2 works like an internal tumblr, you can quickly create different types of posts such as status, link, quote or a more formal blog-style post. Others can comment and comments can be nested leading to good discussions.

Encouraging transparency

With email it is easy to accidentally forget to CC team-mates or not realise that others might have great input on a seemingly unrelated issue. Stripe fights this by having all internal email open, P2 can help you to reduce or cutout email altogether.

Extensibility

Of course WordPress is open-source so in theory it would be possible to make any customisations you like to any part of the theme. There are already some ’must have’ plugins for P2 such as likes, @mentions and more that need to be installed to get the full benefits.

Search and archiving

Of course WordPress comes with tagging, categories and a pretty decent search. Finding that discussion you had on a particular topic is easy with a combination of these (Of course we only have a months worth of content - so only time will tell here!)

Cost

As WordPress is free software the only cost in hosting, we have our P2 installation hosted on a cheap VPS and the cost is negligable. You could use a hosting service like WPEngine if you preferred not to take care of this yourself.

What is it bad for

Realtime discussions

Although not 'bad' per-say, we prefer to keep real time discussions to our Sqwiggle workroom - If everyone is online, then it's just much quicker and easier to make decisions this way.

Items posted in P2 don't necessarily need a reply straight away, or ever in a lot of cases.

Aesthetics

The out-of-the-box experience with P2 isn't incredible, the styling is rough around the edges, as are many of the plugins. We spent a couple of hours customising the theme to match our internal admin tools a little better, but nothing too fanciful.

If you want a finely polished experience then this probably isn't the tool for you unless you like customising your software!

Where next?

I'm really excited to see the development of Inc from the Kippt team as it seems they are tackling this space in a great-looking and easy to use package.

I don't think we could go back to not having a tool like this in our arsenal here at Sqwiggle. An internal social network is a must-have for any distributed team, whether it be a tumblr, twitter or a reddit clone!

If you don't have an avenue for casual rich, asynchronous communication in you company then you might just be missing out.

[tommoorblog]

Why Internal Dogfooding Matters So Much

Dogfooding: A slang term used to reference a scenario in which a company uses its own product to demonstrate the quality and capabilities of the product.

I guess I've always been a little envious of companies like GitHub and 37Signals, not for the many obvious or usual reasons but because they get the joy everyday of the software they create and the software they use to create it being one and the same thing.

For certain products I feel like this actually goes a lot further than just dogfooding, every company knows they should use their own product. But in this case It's recursive positive feedback, an upward spiral of greatness where you improve your product and this in turn makes the team better/happier/more efficient at improving your product. I can't think of a situation more perfect for a newly born startup - where really understanding your customer and market is the most important factor.

At Sqwiggle we're lucky enough to have this possibility, we've been distributed from the day we started working together and have been using the app as our primary form of communication from the moment it became viable. We have morning banter, long discussions, quick questions, rolling laughter and even more on Sqwiggle. Of course, we consistently come across small and big bugs, performance issues, user experience problems and interface improvements. Being the ones discovering the majority of these things also lets us more easily prioritise the pain that they cause.

We recently got to the bottom of a memory leak in Sqwiggle that happened only in the rarest of circumstances, but when it did occur the browser crashed spectacularly leaving no debugging info behind! Had we not been using the app as much this issue may have gone unnoticed or deprioritised for a long time (in fact it had come up in support as a random 'white screen').

Great beta testers and users will let you know of major issues and when they can't understand how to use or find a feature - but there are a myriad of small user interface interactions, improvements and ideas can only be found by exhaustively using your own product all day every day.

Photo by Sh4rp_i

[tommoorblog]

The Future

In February I left Buffer after working on every aspect of the app for over a year and a half (and documenting a lot of the interesting bits here along the way!). I'm immensely proud of everything that we achieved in that time, growing the service to half a million users and the company to 10 awesome team members whilst travelling! Buffer isn't going away any time soon and I'm sure the companies future is very bright with Joel and Leo steering the ship - I can't wait to see what they have in store.

So what now? I'm excited to say that I'm now building a new company to tackle the problems with remote working and distributed teams with a couple of awesome guys, Matt and Eric.

It's a problem I've been thinking about ever since I stopped working in an office environment - how can you bring back the fun, spontaneity, and speed of communication of an office into the online world? It is a huge challenge, both technically and socially - we’re tackling the problem in a unique way and think we have a pretty great solution brewing...

If you have any thoughts on distributed teams then please go ahead and leave them in the comments, I'd love to hear any experiences or problems that you might be facing.

Photo by Sam Howzit

[tommoorblog]

Hacker New

Around a year ago during my time in Hong Kong I started working on a Chrome extension for Hacker News. The aim of the extension was to make the site easier to use and add some (in my mind) much needed functionality.

Today i've released the latest version of the extension which addresses many of the concerns from the previous versions both on HN and Github, here's a little tour of all it has to offer:

User Following

Hover over any username provides the option to "Follow" or "Unfollow" a user, followed users are highlighted wherever the appear, allowing you to pick out articles and comments from friends and thought leaders.

Instant Inline Replies

Previously replying to a comment meant opening a new page, inline replies mean you can type within the page and then submit when you're finished. It also means that long replies won't cause a timeout!

Retina Support

The extension replaces the existing image assets such as vote arrows and icons with higher resolution versions suitable for Retina displays. Thanks to Christian Genco for the idea.

Improved Readability

Hacker New increases the size and colour contrast between links and surrounding elements. It also makes it clearer which articles you have already clicked on.

Quick Profiles

Hover over a username anywhere on the site to instantly see a users Karma and affiliated websites / social profiles. This is a useful indicator of how much respect they have gathered the HN community.

Filters

Filtering allows you to hide articles, users and domains from HN that you would rather not see. I have my filters setup to filter a variety of negative keywords and trashy domains!

Endless Scrolling

The frontpage just got longer, infinitely longer in fact. Endless scrolling kicks in before you reach the bottom so the news keeps flowing. The footer and search bar are always available by hovering near the bottom of the screen.

Collapsible Threads

If you just want to immediately skip past a thread, you can now collapse the entire thing on the right hand side and get directly to the next comment.

Social sharing

Social sharing buttons are built in for Twitter, Facebook, Google+, Buffer. Of course they don't load until you request it so there is no extra bloat on page load times.

I hope the extension improves your HN experience as much as it does mine. If you think something can be done better or would you prefer to tweak the styling go ahead and fork the project over on Github!

[tommoorblog]

There Are Always More Users

 Something I hear often when talking to people and also find myself thinking occasionally is that there are only so many users that your startup can acquire… isn't there a market limit? won't they eventually stop signing up? If I release a feature too soon and some users drop off how will I ever win them back? But the truth is, unless you have the most incredibly niche product (think time-tracking software for ostrich farmers with a penchant for gymnastics) there really are always so many potential users out there that by the time you get anywhere near saturating the market your product will likely be completely different! Think of the biggest sites and apps you know like Facebook, Evernote and Twitter, they have millions (perhaps billions!) of users and yet still every single minute someone, somewhere that has never used the service before signs up for the first time. Don't be afraid to ship early, if the first impression isn't perfect and users don't sign up or quit your app then you can treat it as valuable learning, iterate and try again - **there are always more users**!

Photo by James Cridland

[tommoorblog]

Decisions, Decisions

Recently I was talking to a friend about decision making, she was stuck in a paralysing situation with the knowledge that whatever option she chose it would likely become a major turning point in her life. Mulling over the possibilities for months, weighing the pros and the cons and trying out both options had only left her wracked with the thought that whichever decision she made would ultimately leave her regretting the lost opportunities of the other. The thing is, seemingly innocuous and unrelated actions like where you sit on a bus or who you talk to at a party also have the same potential to completely change your life or the path of your business. Of course these dots can only ever be [connected in hindsight](http://37signals.com/svn/posts/3234-connecting-the-dots-how-my-opinion-made-it-into-the-new-york-times-today) and we don't worry ourselves over the hundreds of decisions we make every day. It’s a problem that entrepreneurs and business owners face constantly, in fact you have to make so many different decisions that the real defining ones often get lost in the noise… Could that feature choice be costing you thousands of dollars a day or your last hire invent the killer feature that rockets your company to success?! All we can do is make the best decision possible with the information we have available at that moment. So how do you make the best decision where everything else is even? Perhaps the answer is to minimize the [possibility of regret](http://en.wikipedia.org/wiki/Regret_(decision_theory)). Not by focusing on the negatives but by choosing the option with the greatest potential payoff and bet on your own ability to achieve it. And the friend? She’s still deciding…

[tommoorblog]

An Amazing Way to Get Customer Feedback and Build a Community

 So you're building a new product or feature and want to gather feedback from your users... If you are building a startup this is probably a situation you have been in many times before and there are tons of ways you could go about doing this. An email list, forums or customer support software like [Get Satisfaction](https://getsatisfaction.com/) / [UserVoice](http://uservoice.com) are popular ways to gather feedback At [Buffer](http://bufferapp.com) we found ourselves wondering if there was a better way, where the whole team could have a proper conversation with users and gather feedback incredibly fast as we rolled out new features and changes every few days. We decided to do something a little different and use [Facebook groups](https://www.facebook.com/about/groups/). We created a group for each of the three parts of our product that we are most actively working on ([Web](http://www.facebook.com/groups/buffer.web), [iOS](http://www.facebook.com/groups/buffer.iphone), [Android](http://www.facebook.com/groups/buffer.android)) and invited anyone that wanted to join and help test the next versions of the apps in return for early access weeks or months before anyone else. Hundreds of people joined after just a few tweets on the company account - It's hard to resist the lure of a beta test! ## The Benefits ### Easy and personal At this point most people that are likely to want to test your product have a Facebook account and feel very comfortable with the interface and idea of groups. This means to get involved they only have to click one button with no extra sign in or learning curve. ### Notifications When you post an update to a group this usually shows up in members notifications and stream, it doesn't clutter up their inbox but makes sure that members are always up to date with what's going on. If the group isn't secret (I would recommend a public group) then when people join their friends may also see this in the stream which spreads the word even further, maybe even introducing new people to your product.

### Speed Facebook is quick, very quick - we've seen comments come in just minutes after a new release has been pushed to testers. Members often see the notification on the web or their mobile phone and check out changes immediately!

### Community But the best aspect has to be the sense of _community_ that is created, group members chat, help each other out and like posts that they agree with. We've run polls, asked questions and given advice - the whole team jumps in to help people out and comment on bug reports and feature requests. It's great having everyone's face and real name, there is an amazing feeling that we are getting to know our users better and i'm sure they feel the same way about talking directly to the team and seeing that they are directly influencing the product. I'm confident that those in the groups will become some of our most loyal and outspoken users.

## The Drawbacks Of course, after using groups for several months we have found some drawbacks. There will always be a couple of people that don't have or want a Facebook account, even for a social startup like ours! ### Deploying features There is also some work in making sure that everyone in the group gets the new features, whether this is enabling them in our admin area, sending out builds or manually adding people to TestFlight. Perhaps in the future this is something we could automate using the Facebook API but at the moment when bug reports come in it can be difficult to know which build someone is using or what version of the website they are seeing. ### Over communicate Ordering of posts in the group is automated, meaning that old posts often get pulled up above newer posts. Posts can be pinned so they stay at the top of the group but you definitely have to over-communicate and assume that previous posts may not have been read. ## Conclusion Overall in the last few months we have found Facebook groups to be an amazing way to connect with our most passionate users no matter where they are located. We've also collected some amazing feedback which has helped to shape the next version of Buffer with is launching very soon. **If you don't currently do anything like this at your startup - give it a try! Start a group, invite your most enthusiastic users to join in and leave a link in the comments :-)**

[tommoorblog]

Sign Up or Sign In, Does it Even Matter?


The hurdle. It's a metaphor!

## The Idea Like any startup, at [Buffer](http://bufferapp.com) we are always looking for ways to reduce friction at every stage of using the product, from the moment you land on the homepage to the daily use of our browser extensions and web app. One of the key points in the user lifecycle is getting people over the hurdle of signing up for your app, of course this should be as seamless as possible, particularly in the consumer internet space. In the last few years this has become easier than ever - with the advent of OAuth solutions such as Facebook Connect and Twitter Sign In users often no longer need to enter any details at all to sign up for your service. This got us thinking could we completely remove the barrier of signing up all together? **Could we make the account creation experience so easy that you might not even notice it had happened?** ## The Approach While there are some intricacies to handling a combined signup / signin on the server the first step is authorising the user with their social network of choice - and of course the UI is simpler than ever. We decided on a familiar 'stack' of social networking buttons that can be consistently reproduced in all of the different sign in locations we have (web, browser extension, mobile). 

Our sign in screen, as seen on mobile.

On the web app the stack sits on the right hand side of our homepage and acts as a bright, strong call to action to. As more networks are added in the future we will likely add a "more options" button combined with a memory of the users last sign in choice which will always be displayed at the top of the stack. Choosing the wording 'Sign In' on the buttons rather than 'connect' or other alternative wording hopefully gives the subtle impression that even new users already have an account and they just need to sign in to access it. The key is with 'Sign In' you don't expect to have to do any more work, which makes the hurdle smaller. ## Key Lessons ### Users are often looking for the words "sign in", "sign up" or "log in" Even (or perhaps especially) with a combined signup / sign in it is important to make it very clear that the user can do both actions with the same buttons. We included a friendly note on every entry point to get this message across 

Let people know that they can do both actions in the same place.

### Many people forget which social network they used previously This is one of the biggest problems we had after rolling out the changes, many users were forgetting which social network they had used to login and were creating two, three and even more accounts. Obviously this is a terrible user experience and it can also completely skew your metrics. We dealt with this situation in a couple of ways; by storing a separate long-life cookie whenever a user signs in we can check on signup if a user has been to Buffer before - we catch people that try to sign up with the cookie and display a screen like this: 

Detect and give users the option to sign in again.

Inevitably some users still fall through the cracks, perhaps because they clear their cookies or simply click on 'create account' anyway. In these cases we need to let them merge their accounts together. In Buffer we allow users to connect additional social media accounts on the dashboard, when the feature was first launched connecting an account already in the system would throw an error with a message to contact support - this would affect anyone who had accidentally created two accounts! As you can imagine, merging accounts quickly became one of our biggest support requests. Thankfully we have now completely removed this friction point also, as authorising with the social network is proof of ownership connecting an account that is already in the database simply merges the two together, seamlessly in the background. ### What about their email? The disadvantage of users signing up with a social network is that you can't always get their email address and other details that may be important for your business. At Buffer we use the email to send alerts if there are any problems or if your Buffer becomes empty as well as optional newsletters and lifecycle triggered retention emails. 

The passive email banner

Rather than immediately hassle the user after signing in with a social network (that kind of defeats the point, doesn't it?) We opted to display a more passive banner at the top the dashboard which lets the user know **the reason** we would like to have their email address, this works well and only displays after a welcome tour which means the user is much more invested in the product than they would have been if asked on a signup page. ## Conclusion So, does it matter? I would say that with the right measures in place to catch and automatically repair duplicate accounts it doesn't matter at all and in fact reducing complexity of the sign up process is well worth these measures. It is now very rare that we get any of the support issues above and because the nature of Buffer is managing and connecting many social accounts we were probably more inclined to get these types of problems than many apps would be. **Have you implemented social sign in differently or had interesting problems with it in the past? I'd love to hear your solutions and thoughts in the comments**

Photo by nickuzma

[tommoorblog]

How We Increased Our Conversion to Paying Users by 171%

At [Buffer](http://bufferapp.com) we are currently focused on driving growth to a new level, in the process we are trying many different channels to increase the number of visitors and signups including content marketing, web stores, mobile markets and more. In the past week I decided to take a quick look at our homepage and how we might improve our conversion instead of driving more new visitors - after all, if we can improve the conversion whilst still increasing visitors it has a compound affect! Previously we have not had much luck with A/B testing, it's something we often do and have infact built an internal library called 'Einstein' which makes it super easy for anyone on the team to deploy a [one-line split test](http://www.startuplessonslearned.com/2008/09/one-line-split-test-or-how-to-ab-all.html) anywhere in the product. We have continuously been amazed at how seemingly large changes can actually make very little difference to conversion and activation, particularly in the early days of the product (more on this later). ### Examples of Some Failed A/B Tests #### Homepage Tagline Changes This is an area we predicted would have a big effect, the thinking being that the main headline on the landing page would be a driving factor in whether people are interested in the value (and we all know not many users read further than the headline!). In the early days [Joel](http://joel.is) tested between several variations of text including "**Be awesome on Twitter**", "**Get 200% more clicks on Tweets**" and "**A Smarter Way to Share**". Unfortunately none of these variations had a significant impact on conversion. #### Colour of Call to Action After meeting [Tim Ferris](http://twitter.com/tferriss), he suggested that we try a higher contrast, orange call to action. At the time we had a single signup button that was a dark shade of green. The difference was similar to below (although unfortunately we don't still have the exact screenshots). 

The two variations used, with a bright central call to action.

#### Positioning Changes We also tried moving the main call to action on the homepage to see if this would have any impact, flipping the page horizontally as well as trying a stark version with no introductory video and a large central signup button! The affect? Negligible. #### Interstitials vs Dropdowns We have also done split tests on internal product features and design variations, one in particular showed 50% of new users an interstitial to install our browser extension whilst the other 50% saw a drop down bar once they began using the product. The hope here was to increase the activity of new users by ensuring they installed the extension, but after running the test for several weeks the cohort analysis showed no discernable difference. ### The First Successful A/B Test In the past week we ran a new A/B test on the homepage that shows potential users more detail about [Buffer](http://bufferapp.com), in particular some of our most popular features run directly below the signup box (multiple accounts, analytics and team members). The hypothesis was that potential users are now discovering Buffer through more channels than just our content marketing efforts and a lot of people might not understand how much Buffer can offer when they first land on the page. 

The two variations used, original on the left and the variation on the right.

We are now tracking all of the tests we perform with [Kissmetrics](http://www.kissmetrics.com) which enables us to easily produce reports for our entire funnel, which currently looks like:  The results (and the trigger for this post!) seem to be for the first time both very positive and statistically significant although entirely unexpected. Every point in the funnel proved to be outperforming the original by multiple percentage points, but in particular upgrades were up a whopping 171%. This outcome was totally unexpected and also shows the importance of tracking your split tests at a [macro level](http://www.startuplessonslearned.com/2008/09/one-line-split-test-or-how-to-ab-all.html), and not just whether there were 'more clicks on the button'. In this case we would have seen that the increase in signups was statistically insignificant and could have even reverted back to the old layout! ### Conclusions #### It Takes Many Tries A/B Testing is difficult! It could be easy to conclude from the many posts highlighting successful A/B tests that getting clear cut results is simple and you too can double your signups by changing the colour of a button but for us that wasn't the case. So far we've found that getting statistically significant results tends to take many iterations and much bigger changes than you might imagine, I don't doubt we will need to perform many more tests to get another significant result of this type. #### Where Your Visitors Come From Makes a Difference Many of the unsuccessful tests listed above were performed in the first eight months of Buffer's life, at this point in time our inbound traffic was much smaller and the traffic we did have often came from early adopters and detailed blog posts explaining how to use Buffer in great detail. I am sure this will have had an affect on the results as a larger percentage of these users will have hit the homepage knowing what Buffer is and knowing they wanted to try it out. This may go someway to explaining why many of the tests in the early days displayed little variation at all. **Have you had a run away split test success or perhaps you've tried hundreds of variations with no results - I'd love to hear your stories in the comments!** ### Resources * [KISSmetrics](http://www.kissmetrics.com) - Particularly tailored to enable easy tracking of SAAS metrics. * [A/B Significance Test](http://getdatadriven.com/ab-significance-test) - Handy tool to measure if your results are statistically significant. * [How not to run an A/B test](http://www.evanmiller.org/how-not-to-run-an-ab-test.html) - Why you shouldn't peak at your results early. * [A Practical Guide to Controlled Experiments on the Web](http://www.exp-platform.com/Pages/hippo.aspx) - The techniques used for experiments at Amazon, Microsoft and NASA.

[tommoorblog]

How to Use Notifications and Alerts to Give Your Startup an Edge


She's alert, get it?

Recently I've come to realise what a critical role notifications and alerts can play in running an internet startup, enabling a small and scrappy team to stay on top of many things at once. By spending a small amount of time identifying and setting up alerts for key metrics you can give yourself a powerful weapon that can be used to minimise problems and wow customers. At [Buffer](http://bufferapp.com) we started with the very basics, using pingdom to send a text message if the website became unavailable - this works great for an MVP. Over the past year as the service has grown we have gradually built up and expanded the notifications to cover many more areas of the architecture and business to make sure we always know what is going on. In my mind notifications are best split into two categories - active and passive. Active notifications are pushed through to email, sms or mobile apps and are used for emergencies or critical problems with the service. Passive notifications are displayed in our internal admin area, team chat or are sent in daily summary emails, these are things that we would ideally action but aren't immediate emergencies. The key notifications that we are using today include: ### Active Alerts #### Macro business metrics With deploying code many times a day there is alway a chance that bugs will slip through unnoticed. It's important that if this happens they are detected as fast as possible and don't affect the things that really matter. We track the core metrics that are most important to our business every minute to ensure they are within preset acceptable bounds. Obviously these will be different for every business, for Buffer these are: * Updates Sent * API requests * Signups * Upgrades If any one of these metrics steps outside of pre-defined bounds then we are immediately notified via push alerts and email, for example if for two consecutive minutes no updates are sent then there is clearly a problem somewhere in the chain that needs immediate attention. #### Brand sentiment Inspired by [an interview with Chuck Rossi](http://arstechnica.com/business/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering/1/) of Facebook's release engineering team we also track sentiment (currently only on Twitter) for people mentioning @[bufferapp](http://twitter.com/bufferapp). If there is a spike in negative tweets we are immediately notified that something may be wrong. This means we can jump into Twitter and both fix whatever the issue may be and also respond with very fast customer support when it's needed most. #### Server status As an internet based business this is a bit of a no-brainer, but mentioned for the sake of completeness. We use [Server Density](http://www.serverdensity.com) to perform minutely checks on all of our public facing services including the website, embeddable buttons and API. These are done in a round-robin fashion from multiple data centres around the world which means a more accurate average response time reading and ensures we are accessible from different regions of the internet. ### Passive Alerts #### Upgrades and downgrades We display a list of the days upgrades and downgrades in the admin area which is visible to everyone on the team. We often reach out individually to users to thank them or in the case of a downgrade to try and trigger a discussion on why they left us and what we might have done differently to keep them.. #### Influencer signups Obviously we would love to send a personal email to every new signup, but with this not being possible we thought it would still be nice to get in personal contact with influencers that signup for the service. At first this was an active notification, we setup the signup system so that [Joel](http://twitter.com/joelgascoigne) would receive an individual email every time a user with a large number of followers signed up. This quickly became a burden and the emails we're auto archived even after the trigger for an influencer was raised time and again. In hindsight it is now obvious that this should have been a passive alert and in a similar fashion to upgrades we now display a list in the admin area of the days influencers and regularly get in contact with users to welcome them personally - this really helps to create the [customer service wow factor](http://www.desk.com/blog/touchpoint-opportunity-customer-wow/). #### Brand mentions Like many businesses and individuals we use [Google Alerts](http://www.google.com/alerts) to inform us in a daily summary where Buffer is being talked about on the web. This lets us find posts that mention the service both in negative and positive ways. Often [Leo](http://twitter.com/LeoWid) can jump into the comments to directly answer questions and provide support. #### Commits and deployments For the past few months we have used [HipChat](http://www.hipchat.com) as the hub of day to day conversation at Buffer. In general HipChat has worked wonders at keeping the team motivated and upto date on what everyone else is working on. As part of this we push passive deployment and commit notifications into HipChat from Github, usually no action is required by anyone but this helps to keep the sense that other people are working and everything is always moving forward fast! I hope some of these examples inspire you to think about how having more notifications and information at your disposal could give your startup an edge! **What have I missed? Is there something you monitor at your business that has proved incredibly valuable? I'd love to hear how we might improve our approach or any aspects that we may be neglecting** Image thanks to [rsteup](http://www.flickr.com/photos/rsteup/3876636138/)

[tommoorblog]

Scaling a Lean Startup, The Story So Far.

Since I joined [Buffer](http://bufferapp.com) in September last year we have seen awesome growth that has pushed both our hardware and mine and [Joel](http://twitter.com/joelgascoigne)'s sys-admin skills to their limits. In this time we have grown from a single Linode VPS shared with Joel's previous startup to a heady total of 10 VPS's and in the last week we undertook the biggest jump so far, moving our infrastructure from this cluster to [Amazon Web Services](https://aws.amazon.com). Our approach to scaling has always been based on necessity - with the [Lean Startup](http://www.startuplessonslearned.com/2008/09/lean-startup.html) methodology ingrained into the company culture we push code fast and try to avoid prematurely scaling our infrastructure beyond it's needs. This has definitely lead to some sleepless nights and the occasional hour of downtime but all in all we feel this approach is much better than over compensating from the beginning and building architecture that you may never need or use. These are a few of the key occasions when scaling became the only option... ### 1 Server: Launch to 20,000 users The first version of Buffer was ([perhaps famously](http://blog.bufferapp.com/idea-to-paying-customers-in-7-weeks-how-we-did-it)) the embodiment of an MVP, a single pricing page that gauged potential interest in the idea before anything was even built. This was hosted on a tiny Linode VPS, shared with several other projects. Once the idea was validated and for the next year Joel was able to scale Buffer to 20,000 users by increasing the size of the VPS and optimising SQL queries, this meant a few hours of downtime each time the server was resized but allowed costs and time investment to remain low while product market fit was reached, or in Joel's words: > Focus on "scaling" too early and you may well forget to focus on "building something people want". Don't make that mistake. ### 2 Servers: Moving to MongoDB As I joined Buffer the first feature we decided to build was support for posting to Facebook - the main decision was how to change the code and database schema to accommodate more than just tweets. Over the course of several weeks the model layer was rewritten and the data migrated table by table from MySQL to collections in the document-based datastore [MongoDB](http://mongodb.org). After the switchover, the new database was kept on the existing server, taking the place of MySQL. This worked well for around a month until early one Sunday morning Buffer suddenly disappeared from the internet. After tailing the error logs we quickly realised that we had reached an inherent limitation of MongoDB - 32-bit processes are limited to ~2gb of data and we had hit that limit. We searched frantically for a way to get the database back in action on the existing hardware. After half an hour with no luck and [Leo](http://twitter.com/leowid) calming distressed users on twitter we realised we had to get in touch with the best person we knew. We called [Eric Lubow](http://twitter.com/elubow) from [SimpleReach](http://simplereach.com) who put us on the right path in minutes - there was no other option but to accept the downtime and spin up another 64-bit server immediately. Thanks to clear, straight forward advice we got off very lightly with less than two hours of disruption in a quiet period as we setup a server, tarballed the existing data directory and shuttled it across! ### 4 Servers: Background Processes By November 2011, Buffer had continued to grow to around 70,000 users and with 20,000 tweets per day being sent the website and browser extension were becoming noticeably slower at the peak time of the day (for us this is around 17:00 GMT, early evening in the UK and morning on the East Coast). At this point it was time to queue the sending tasks and have them processed by their own dedicated server. In hindsight this was the turning point where a Buffer 'architecture' started to emerge, with modular pieces of code and servers that we're independent of each other. As soon as the core task of sending tweets was moved to queues it became obvious that a lot of other processing could be completed this way such as gathering link info, analytics and image processing. We went ahead and moved these tasks to two new utility servers, I'll be covering the details behind how we now use queues extensively at Buffer in a future post. ### 5 Servers: Upgrading MongoDB In December our database server was humming along nicely but quickly filling up it's available RAM headroom and running on an out of date version of MongoDB. In order to upgrade the software and the server to more RAM (as we would end up doing several times of the next few months) we needed to be in a replica set configuration that would allow us to take the individual database servers out of circulation one by one to upgrade them. This proved to be another rocky period with several hours of downtime, we found this was almost entirely caused by issues in the immature PHP drivers which have since been addressed but also partly down to our own lack of experience managing replica sets, firewalls and all of the intricacies of a multi-server setup. ### 7 Servers: DiggDigg Acquisition In January of this year we hit the 100,000 user milestone and completed our acquisition of the popular WordPress plugin [DiggDigg](http://bufferapp.com/diggdigg) which allows any WordPress user to quickly add a sharing bar to their blog or website. Part of the plan behind this move was to add the [Buffer Button](http://bufferapp.com/extras/button) to DiggDigg and have it enabled by default for new installations, adding massively to the button's reach and Buffer branding around the web. In order to handle the millions of new button impressions and isolate the web interface from load generated by third party websites it was time to move the button to it's own servers and introduce our first load balancer. It was also decided to use Nginx AND just for good measure re-write the entire button to become independent of the Buffer framework. We definitely teetered on the edge of our seats after deploying all of this previously unused architecture at once, waiting for the load from DiggDigg to hit the servers. But everything went smoothly, the load balancer acted as expected and impressions gradually rose to new highs as people upgraded their plugin (it seems there is no sudden rush of traffic!). The button servers now handle around 4 million impressions per day and rising! ### 9 Servers: Sleepless Nights Last month we reached 180,000 registered users and completed the acquisition of ShareFeed from [KISSmetrics](http://www.kissmetrics.com). Almost unbelievably, at this time the main website, API, blog and browser extension were all still running from a single Linode VPS (albeit one that had been upgraded on many occasions!). During peak times things we're beginning to feel slow again and complaints started to appear on Twitter at predictable times. Unfortunately late evening Hong Kong time ([where we are currently based](http://techcrunch.com/2011/12/20/sharing-scheduler-app-buffer-raises-400000-gets-kicked-out-of-us)) is the peak time for traffic at Buffer. For almost a week I had incredibly restless sleep, I became acutely aware of every email received, waking up at the slightest sound convinced that the site would be down. It was definitely time to upgrade. Thanks to all we had learnt creating the new environment for the Buttons several months before, we were able to make this move completely seamlessly. By setting up the load balancer, adding the new servers and testing thoroughly before moving the DNS we were able to ensure that the only thing users noticed was a performance increase. ### Thats Not All Folks It's been an amazing ride so far with both highs and some serious lows, at one point working for nearly 30 hours straight just to keep the site online. We have learnt an incredible amount as a team in the last few months and I can't think of any way we could have learnt more, faster. Last week we migrated the whole of Buffer's architecture to Amazon Web Services with less than two hours of scheduled downtime, making a number of huge changes and improvements in the process. I'll cover some details of this migration in a future post. **I'd love to hear any comments your have or similar stories of battling scaling woes as a small startup!**

[tommoorblog]

Realtime Analytics at Buffer with MongoDB

At [Buffer](http://bufferapp.com) we try our best to be a data-driven startup, it can be hard work measuring, testing, correctly reading the data and then acting on these results - we definitely still have a lot to learn. Currently we use a combination of third party tools such as [Kissmetrics](http://kissmetrics.com) and [Google Analytics](http://www.google.com/analytics) as well as our own hand-rolled metrics for internal data. The area where we are most happy with our approach is in measuring internal statistics, for example how many people used the latest Firefox extension last week, or how many API requests to which endpoints were there in the last 2 hours or which website with the Buffer widget provided the most signups last week? **To make informed product decisions we need to easily access and visualise key business data in realtime.** ### Using MongoDB to measure realtime stats Our current tool of choice is [MongoDB](http://www.mongodb.org), using a combination of [atomic 'inc' methods](http://www.mongodb.org/display/DOCS/Atomic+Operations) and 'unsafe' saves we can update key metrics in the database blindingly fast with a negligible affect on performance of the front end (as the driver doesn't wait for save success before continuing). By storing these statistics in minutely, weekly or hourly batches we can also limit data growth to a predictable rate and increase read speeds in the admin area.

db.clients_by_time.update({ 'hour': $hour, 'client_id': $client_id }, { '$inc' => array( 'requests' => 1, 'statuses.'.$status_code => 1, 'endpoints.'.$endpoint => 1, 'response_time' => round($response_time) ) }, true);

The code snippet above is inspired by [this post](http://blog.mongodb.org/post/171353301/using-mongodb-for-real-time-analytics) on the mongodb blog and is used to record the performance usage of Buffer API clients. It allows us to produce detailed graphs like the following in realtime and with upto the second data. 

A typical API client

Several individual values are incremented, one for the total requests as well as individual status codes and endpoints. Sometimes tricks can be used to allow the use of incrementing values where you might not usually, for example in the above code 'response_time' is incremented with the response time of each API request - the value becomes the sum total of all API requests in the hour. It is then easy to calculate the average when we return the document by dividing by the total requests. To store the average in the document itself would require an extra query for every request and this is well worth avoiding! 

Buffer update sources

### Where to go from here We believe that this approach scales exceptionally well and it has certainly done us proud so far. As the dataset grows it is inevitable that we will want to automate weekly, monthly and eventually yearly rollups and with only 168 documents created per collection, per week for data recorded hourly this is a task that can be easily performed in code rather than requiring map reduce. By far the biggest area for improvement is in our display of all the data we are gathering. At the moment this is largely using simple line graphs with raw values over time (we don't even have a way to go back to previous weeks yet!). Some of the analytics would also be much better expressed as percentages of a total, stacked graphs or one of the many other types of charts. **Have any thoughts on our approach? Have you tackled things differently in your company? I would love to hear them in the comments.** ### Further Reading * [Using MongoDB for Real-time Analytics](http://blog.mongodb.org/post/171353301/using-mongodb-for-real-time-analytics) The original mongodb blog post that inspired this approach. * [MongoDB Real-time Data Collection and Stats Generation](http://www.slideshare.net/dacort/mongodb-realtime-data-collection-and-stats-generation) * [Fast, easy, realtime metrics using Redis bitmaps](http://blog.getspool.com/2011/11/29/fast-easy-realtime-metrics-using-redis-bitmaps/) Not a method we are currently using, but a very interesting approach to the problem solved using redis bitmaps.