Yufei (Benny)

These are my notes on C++, Node.js, MongoDB, and programming in general

[bennyyufeichen]

Tech Talk Summary: Perfectly executing the wrong plan

1. Fall in love with the problem, not the solution

2. Don’t ask your friends and families; they are biased

3. Observe what people do, don’t trust what they say

4. They did not test the riskiest assumption in their product

5. Bob-the-builder functionality. They launched an item without knowing what this is for

6. Perfectly executing the wrong plan. A great app that no one needs

Important questions to ask:

Who are my customers

Do people need my product

Is my product usable

Is my product better than competition

How to solve these problems: Lean User Research

Experience Sampling

Ask the same question over and over again

A great sample asks about repeated behavior, something that happens a lot. Ask about behavior

Identify needs, features, pains and delights

Observation

4 pillars: observing, listening, noticing, gathering

5 important things to look for in observing: routines, interruptions, annoyance, delights, transitions (when it is done or on the way to something else)

Fake Doors

Ask people to pay, with a button to nowhere

The full talk here: https://www.youtube.com/watch?v=9TJTbRw4ri8

[bennyyufeichen]

C++14 walkthrough

C++14 is here! And clang has already finished implementing all the features of c++14. We will follow the cxx status for clang and walk through all the changes in the new standard.

Note: all the articles marked WIP are still being worked on

Tweak to certain C++ contextual conversions (WIP)

Binary literals (WIP)

decltype(auto) (WIP)

Return type deduction for normal functions (WIP)

Initialized lambda captures (WIP)

Generic lambdas (WIP)

Variable templates (WIP)

Relaxing requirements on constexpr functions (WIP)

Member initializers and aggregates (WIP)

Clarifying memory allocation (WIP)

[[deprecated]] attribute (WIP)

Single quotation mark as digit separator (WIP)

C++ Sized Deallocation (WIP)

Have fun!

[bennyyufeichen]

Job queue for Node.js: kue

I ran into a problem a few weeks ago regarding delayed processing of a user's request. When a user post something, I want to be able to run some expensive analysis on the story, but I don't want the user to wait for all that time while I do the processing. This is a typical usage case of job queue. You have many options, including the famous celery, but I needed something that would enable me to reuse all the code written in Node.js, so I opted for Kue.

Kue has a great API and relies on Redis. I will walk you through a really basic example, extracted from the README on github. First, you will need to create a connection with Redis:

var kue = require('kue'), jobs = kue.createQueue(redis: { port: 1234, host: '10.0.50.20', auth: 'password', options: {...) } });

Kue relies on node_redis for handling the connection, so you can check the documentation for node_redis if you need to have complex settings such as atomatic fail-over for Kue.

After you have created the connection, you can use the following syntax to create a job:

jobs.create('email', { title: 'welcome email for tj' , to: '[email protected]' , template: 'welcome-email' }).save();

Here, 'email' is the queue name. The second parameter is the data you pass in for the current job. You can chain different commands after create, such as .priority() for a different priority, or .attempts(n) to assign the number of attempts allowed. No processing on the job has been performed yet; you are simply sending data to the broker, redis, and wait for some other machine to pick up the job and process it.

Picking up a job to process is also really easy on Kue. On a different machine, connected to the same redis instance, you can do the following

jobs.process('email', function(job, done) { email({ address: job.data.to, title: job.data.title, content: job.data.template }, done); });

Notice all the data in the second argument of job.create() now resides in job.data. You will be able to access all the data to do whatever processing you want.

There are many more features provided by kue, such as updating progress, delay and promote. Heck, it has a web UI interface so you can see if your jobs are being processed. For all the advanced usage, please check our their github repo. Happy processing!

[bennyyufeichen]

MongoDB admin UI: Genghis

MongoDB has a really easy to use console, and many admin tasks can be performed by writing a script for the console. However, sometimes an admin UI really helps because you need to view your data in an intuitive way, or you just hate to use console. In that case, I highly recommend Genghis. It is by far the easiest admin UI for MongoDB I have used, and it is also dead easy to setup. Check it out!

[bennyyufeichen]

JavaScript Programing #2: Inheritance and Prototype chain

JavaScript do not have the conventional OOP keywords such as 'class', so the syntax is slightly different from C++ and Java if you want to organize your code in terms of objects. There are lots of great tutorials on Google about how to make public and private variables/member functions, so I am not going to cover these today. I will cover a slightly more confusing topic, which is inheritance in JavaScript, achieved through prototype chaining.

Lets say we have a class called Car. Nice and simple.

'use strict'; var sys = require('sys'); function Car(plate, owner) { // you can assign the property to this this._plate = plate; this._owner = owner; } Car.prototype.details = function() { console.log('plate: ', this._plate, ', Owner:', this._owner); }; function Toyota(model, owner) { Car.call(this, 'no plate!', owner); this._model = model; } // we are basically assigning what methods Toyota should have based on // what methods Car has Toyota.prototype = Object.create(Car.prototype); // we can assign extra methods to the prototype of Toyota Toyota.prototype.recall = function() { console.log(this._plate, 'is getting recalled'); }; var myToyota = new Toyota('yaris', 'Alice'); myToyota.details(); myToyota.recall();

If you run this code, you will see 

plate: Rand , Owner: BennyChen

Now if I want to inherit from this definiion of car, and make a Toyota out of it?

Think of prototype in JavaScript as assignments to the prototype property. 

If we do some system inspecting, as the following

console.log("Toyota's prototype", sys.inspect(Toyota.prototype, true), "\n"); console.log("Toyota's properties", sys.inspect(Object.getPrototypeOf(Toyota), true), "\n"); console.log("Toyota's prototype's prototype", sys.inspect(Object.getPrototypeOf(Toyota.prototype), true), "\n");

You will see the following output

Toyota's prototype { recall: { [Function] [length]: 0, [name]: '', [arguments]: [Getter/Setter], [caller]: [Getter/Setter], [prototype]: { [constructor]: [Circular] } } } Toyota's properties { [Function: Empty] [length]: 0, [name]: 'Empty', [arguments]: null, [caller]: null, [constructor]: { [Function: Function] [length]: 1, [name]: 'Function', [arguments]: null, [caller]: null, [prototype]: [Circular] }, [bind]: { [Function: bind] [length]: 1, [name]: 'bind', [arguments]: null, [caller]: null }, [toString]: { [Function: toString] [length]: 0, [name]: 'toString', [arguments]: null, [caller]: null }, [call]: { [Function: call] [length]: 1, [name]: 'call', [arguments]: null, [caller]: null }, [apply]: { [Function: apply] [length]: 2, [name]: 'apply', [arguments]: null, [caller]: null } } Toyota's prototype's prototype { [constructor]: { [Function: Car] [length]: 2, [name]: 'Car', [arguments]: [Getter/Setter], [caller]: [Getter/Setter], [prototype]: [Circular] }, details: { [Function] [length]: 0, [name]: '', [arguments]: [Getter/Setter], [caller]: [Getter/Setter], [prototype]: { [constructor]: [Circular] } } }

Toyota's prototype, recall, is easy enough to understand, since we declared that. Toyota's propertis are mostly the default properties associated with the JavaScript Function (you can get more details about the built-in Function here.

The important takeaway for the properties are, details function is not present in the properties or the prototype of Toyota.

So how does JavaScript know which details to call? As it turns out, it actually inspect the prototype of the prototype of the Toyota object to find a details function to call. When we do

Toyota.prototype = Object.create(Car.prototype);

the statement essentially copy Car.prototype to the properties of Toyota.prototype. By doing this, you would be able to find a details function to call. Note that Toyota.prototype.prototype is actually null; but as long as Toyota.prototype has a property 'details', it will be able to call the function you expect.

This is what we call prototype chaining. JavaScript will keep on inspecting the properties of prototype to the next level, until there are no more prototype to look for. The following extreme example fully illustrate the idea of prototype chaining.

'use strict'; var sys = require('sys'); function Car(plate, owner) { this._plate = plate; this._owner = owner; } Car.prototype.details = function() { console.log('plate: ', this._plate, ', Owner:', this._owner); }; function Toyota(model, owner) { Car.call(this, 'no plate!', owner); this._model = model; } Toyota.prototype = Object.create(Car.prototype); Toyota.prototype.recall = function() { console.log(this._plate, 'is getting recalled'); }; function Corolla() { Toyota.call(this, 'Corolla', 'Bob'); } Corolla.prototype = Object.create(Toyota.prototype); Corolla.prototype.insurance = function() { console.log('this car is not ensured!'); }; var myCorolla = new Corolla(); console.log("Corolla's prototype", sys.inspect(Corolla.prototype, true), "\n"); console.log("Corolla's prototype's prototype", sys.inspect(Object.getPrototypeOf(Corolla.prototype), true), "\n"); console.log("Corolla's prototype's prototype", sys.inspect(Object.getPrototypeOf(Object.getPrototypeOf(Corolla.prototype)), true), "\n");

The output of those inspection statements are

Corolla's prototype { insurance: { [Function] [length]: 0, [name]: '', [arguments]: [Getter/Setter], [caller]: [Getter/Setter], [prototype]: { [constructor]: [Circular] } } } Corolla's prototype's prototype { recall: { [Function] [length]: 0, [name]: '', [arguments]: [Getter/Setter], [caller]: [Getter/Setter], [prototype]: { [constructor]: [Circular] } } } Corolla's prototype's prototype { [constructor]: { [Function: Car] [length]: 2, [name]: 'Car', [arguments]: [Getter/Setter], [caller]: [Getter/Setter], [prototype]: [Circular] }, details: { [Function] [length]: 0, [name]: '', [arguments]: [Getter/Setter], [caller]: [Getter/Setter], [prototype]: { [constructor]: [Circular] } } }

Note now Car's prototype becomes Corolla's prototype's prototype. JavaScript will basically perform the same set of action as we did here to find a member function to call.

This programming model has significant implications. Since JavaScript has to go up the chain to find the right prototype to call, complicated inheritance pattern would not be so good for JavaScript code that has a strict performance requirement. It also means a programmer would be able to inspect the inheritance hierary of a JavaScript object with ease.

Hopefully I have made prototype chaining very clear in the tutorial. If it is still not clear, please leave a comment and I will try by best to answer. All the code in this blog can be found here Have fun with JavaScript!

[bennyyufeichen]

How to tame your vim for JavaScript

When you log into a system, one thing you will definitely encounter would be a wild vim, with all the default plugins and no smart indent. You must tame it and bend it towards your need. The tutorial will be an in-depth tutorial on all the good plugins for JavaScript development.

bunch of tamed dogs. src: Minecraft Wiki

TL;DR: You can download the script here. The plugins involved will be pathogen, jshint, vim-javascript-syntax, syntastic, YouCompleteMe, vim-distinguished. It WILL override your .vimrc and .jshintrc, so only use it for fresh installation.

To make your life easier, you need a plugin organizer for vim. Pathogen is one of the older and better supported plugin system for vim. Installation is very easy:

mkdir -p ~/.vim/autoload ~/.vim/bundle; \ curl -Sso ~/.vim/autoload/pathogen.vim \ https://raw.github.com/tpope/vim-pathogen/master/autoload/pathogen.vim

and add the following line 

execute pathogen#infect()

From this point onwards, to install a plugin, all you need to do is to clone the repo into ~/.vim/bundle, and it will be loaded into vim. You can check out more about pathogen here.All the following plugins will be installed through pathogen, except for jshint which will be covered later.

Syntastic

Syntastic is a plugin for syntax checking. I will assume you have installed jshint already. If not, it will be covered in a completely separate blog.

I might be repeating myself, but installing a plugin under pathogen is really simple. All you need to do is clone the repo into ~/.vim/bundle like the following

cd ~/.vim/bundle git clone https://github.com/scrooloose/syntastic.git

To enable syntastic to check when a file opens, and use jshint as the syntax checker, you need to add the following to your .vimrc

let g:syntastic_jslint_checkers=['jslint'] let g:syntastic_check_on_open=1 let g:syntastic_enable_signs=1

Done! check out Syntastic's github here for more information

vim-javascript-syntax

vim-javascript-syntax is a syntax plugin for vim. Installation is a no-brainer

git clone https://github.com/jelera/vim-javascript-syntax.git ~/.vim/bundle/vim-javascript-syntax

The main reason I use the plugin is that it can help me fold javascript functions, by putting the following line into .vimrc

au FileType javascript call JavaScriptFold()

All the actions associated with folding and unfolding under vim can be found here.

vim-distinguished (or any other color scheme for vim)

I may be getting a little repetitive here, but installing any plugin, including colorschemes, simply needs you to clone the repo of the plugin into ~/.vim/bundle. The same goes for my faviorite color scheme, vim-distinguished

cd ~/.vim/bundle git clone https://github.com/Lokaltog/vim-distinguished

To enable the colorschem, put the following line into .vimrc

colorscheme distinguished

note that the name of the plugin does not necessarily match the actual name of the colorscheme, so make sure you enabled the right colorscheme by going through their documentation.

YouCompleteMe

YouCompleteMe is a great auto-completer under vim. A lot of other autocompletion plugin requires you to use special keystrokes to make the list of possible matches show up, but YouCompleteMe does everything automatically, which is closer to what you would expect on xcode or Visual Studio. Installing the plugin is a little more work than before

git clone https://github.com/Valloric/YouCompleteMe.git cd YouCompleteMe cd third_party git submodule update --init --recursive cd .. ./install.sh --clang-completer

you need to go down to the /third_party folder to pull the submodules, and compile the whole project.

Result

After you have installed all the plugins, vim should look like this

Notice the folds for all the functions, and the autocompletion coming into play.

If we make a syntactical mistake, jshint and syntastic will catch it and remind us

Pay attention to the smaller error message on the very bottom of the screen, and the bright yellow arrows on the side of the code. You have successfully tamed your first wild vim!

[bennyyufeichen]

Website that scales #3: profiling and optimization

Premature optimization is the root of all evil

-- Donald Knuth

You have read "Website that scales" #1 and #2, and made your first prototype of your application. People tell you EventMachine and Node.js can server tens of thousands of requests on a single instance, and your server only serve a hundred. The key to identify your bottleneck is to setup realistic benchmark and profiling the whole system.

Stressing your server with a realistic benchmark is not a trivial problem. A lot of benchmark blogs talk about apache benchmark and its variants. You bash one url as hard as you can with as many sockets as possible, but apache bench does not realistically emulate the actions of user. Locust.io is a great framework for load testing that solves all the problems above. You will write python scripts for each action, which means basically whatever user action you are expecting can be achieved, and you can assign different load to each action. For example, if you want to tests a social network site with user often getting stories, and sometimes check out other people's profile, you can do the following

from locust import HttpLocust, TaskSet, task class WebsiteTasks(TaskSet): def on_start(self): self.client.post("/login", { "username": "Alice", "password": "Bob" }) @task(10) #weight on /stories is 10 def index(self): self.client.get("/stories") @task(2) #weight on /profile is 2 def about(self): self.client.get("/profile") class WebsiteUser(HttpLocust): task_set = WebsiteTasks

The load to put on each end point will be proportional to the weight you put on for each task, but every time the requests are makde, the are some element of randomness to make sure it is not exacly 10:2 all the time. After writing a comprehensive load test, you can choose how many user you want to emulate in the web terminal, and monitor the stats as they come back. Coming up with a comprehensive locust.io stress test can really help you identify the bottleneck in your system.

With a great load testing framework, you are in store to identify all the bottlenecks of your system. The job after this point is relatively easy; profile and identify bottleneck. Every single language have its own way of profiling:

C/C++: gperf,and loads of commercial solutions

Node.js: node --prof, or commercial solution such as Nodetime and strongloop

Python: cProfile, profile, and the list goes on

Most commercial databases also have their own profiling system. Every database systems I know have 'explain' option to show you if you are using an index to query or not. All of the profiling systems are well documented, and you will have to read them.

There are many, many toys you can play around with, but I strongly suggest isolate every server for a single function when profiling, and use a really simple monitoring service such as Munin to find every detail associated with the system when identifying bottleneck. It is very, very easy to get over-excited and write a new framework to solve your bottleneck, but that is almost always not the right option unless you are a huge company. Most of the time, there are many other easier solutions that you can go for. It could be dirty, but maintaining less code means less bug, and lower cost overall. These decisions are strongly situational, and you will have to make the call, stress the system, profile, and repeat.

After identifying all the bottleneck, you will see your modern web application server and database server doing its magic, serving lots of requests at ease. If you have any suggestions about any content I didn't cover enough, please ping me about it. Have fun with your servers!

[bennyyufeichen]

JavaScript Programing #1: async and its programming patterns

"JavaScript Programming" will be a series of blogs about useful JavaScript programming tricks and patterns. Taking about JavaScript programming, what is better than introducing async to kick things off?

Quoting from the module's site

Async is a utility module which provides straight-forward, powerful functions for working with asynchronous JavaScript.

and those functions are really, really powerful. Suppose I want to do something in series, e.g. when a user login, I want to validate his password, and check if it is from the same phone, then pass it to a backend infrastructure to make sure the ip address associated with the login is reasonable. How do I do that?

async.series([ function(callback) { validatePW(loginData, callback); }, function(cb) { checkSamePhone(loginData, callback); }, function(cb) { checkFraud(loginData, callback); }, ], function(e, results) { if(e) { // handle the error } // return the result });

If any callback was returned with first parameter being null, the execution will pause and you will jump to the last callback, where you will be able to check your error. Nice and easy. Did I just save you from callback hell? All of a sudden the code looks utterly familiar with Java and C++, right?

Of course, the example I gave was a bad example. If you want to return the result to the user as soon as possible, and assuming checkSamePhone and checkFraud does not have a very high cost, you would want to execute all the code in parallel and return the result to the user as soon as possible. That is also easy; all you need to do is change async.parallel to async.series, like the following

async.parallel([ function(callback) { validatePW(loginData, callback); }, function(callback) { checkSamePhone(loginData, callback); }, function(callback) { checkFraud(loginData, callback); }, ], function(e, results) { ... do whatever you are suppose to do here });

However, keep in mind that Node.js is single threaded throughout. For a more detailed explanation for how async.parallel works, check out this stack overflow post, but it will do what you expect it to do: pipeline calls to other services.

There are many, many other control flow usages for async, and one of my favorite is async.waterfall. Suppose your instructions not only needs to execute in parallel, but the subsequent calls relies on the data returned from the previous calls. When a user post something, you want to reformat the story, and check the formatted story for spam, and then post it to the the database. To check the story for spam you might need the formatted data, and to post the story you might need to know it is not a spam. THe call would look like

async.waterfall([ function(callback) { formatPostData(loginData, callback); }, function(formattedPost, callback) { checkForSpam(formattedPost, callback); }, ], function(error, formattedPost, isSpam) { if (error) { ... return the error } else if(isSpam) { postStory(formattedPost, callback); } else { warnTheUser(userData, formattdPost, callback); } });

And those 3 are calls I use most often. There are also many other more advanced async calls for control flow that I strongly suggest you to check out. We will now move onto the collections operation for async.

JavaScript comes with great native collection support for Array. Array.map, Array.filter and many other functions are built in to the language (you can check it out here). To some degree, most of basic calls for collection offered by async are already offered by the language itself. However, what JavaScript can't do are

limit the number of functions being called at the same time

call those functions in series

((which is often needed on server side programming. For every collection call such as async.function, there are async.functionSeries, and async.functionLimit. Lets say you want to check every user's friend and see how many friends they have, you would do

async.eachLimit(allUsers, limit, apiCall, function(e, results) { ... carry on counting friends of everyone. Why do I always make up awful examples.. });

and you are ready to roll! Make sure to checkout async here. It is one of the most popular package on npm for a reason :P

[bennyyufeichen]

Website that scales #2: pick a database system that fits your need

You have picked an web application server language, now its time to pick a database. There aren't that many tricks to picking a database systems, so I will keep it short and simple

Check out http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis. It lists out most of the current popular NoSQL databases. After reading it, you should be able to come up with a decision.

DO NOT RESORT TO SHARDING/PARTITIONING BEFORE YOU HAVE TRIED ABSOLUTELY EVERY OTHER OPTIMIZATION TRICK! This blog explain this point in detail. Although the blog is about MySQL, the idea applies to almost all database systems I know. Just don't do it unless you really have to.

If not, these are the following rule of thumb

If you are in a startup and you need to get your work done, go for something that has everything packaged up for you, such as MongoDB and CouchDB

Don't use Redis as cache. Use memcached. You don't need the persistence offered by Redis, and memcached is designed to be a distributed caching system

If you really have no idea what to pick, go for MySQL. It is still the most widely used and the most well known database, and you will definitely be able to hire a MySQL DB admin with 10+ year experienced.

If you hate MySQL and want to blindly follow the trend like me, then go for MongoDB. If it is popular, it must work well for a lot of people.

If latency/performance is key, go for Kyoto Cabinat/Kyoto Tycoon. Crazy speed, but not that much love from the community, so not something you can easily plug into Python/Node.js/Ruby. If you want some database that is fast but not as much hassle as Kyoto Cabinat, use Redis.

If you want a database that can handle heavy, heavy write volume and stays up then go for Cassandra. If will do what you expect it to do: handle huge volume.

That's it, nice and short. Have fun with databases!

[bennyyufeichen]

Website that scales #1: pick your web application server language

I am starting a new series covering my knowledge about how to build a scalable website. Where does it start? The very first step is to pick a language for your application server. You have to have an application before you can scale, right?

First thing you must keep in mind is that you really have to be careful of your choice of your application server language. I have heard of the statement "I will rewrite this code later" countless times, but the truth is, unless the code breaks, no one wants to touch it. Most of the time, when the code is written in a startup, it is basically carved onto a stone tablet. When you set off using Node.js for a computationally intensive service, or when you set off using PHP as a social network site, the mistake has been made and you are going to carry mistake for a very long time. There are always tricks you can do to boost the performance of the scripting language you like. Heck, Facebook wrote HHVM so their code can run faster, but I am assuming most of the reader do not have the resources that Facebook has. So, pick carefully.

(logo for HHMV, just so you know what I am talking about :P)

Secondly, pick a language you are comfortable with. If most of the websites uses Python to power their website (Dropbox, Pinterest, Google, Yahoo), it is not because Python is crazily efficient, and the panacea for all. Python was their choice because it is good enough for their usage, and they are comfortable with it. If you pick a language that your whole teams are not familiar with, you are going to run into a lot of pain. If your team are all front-end engineers and want to build an SNS website, pick Node.js. If you are all big rails fan and someone told you Node is more efficient, you should still go for rails. Building a website will not be easy, so make sure you know what you are doing by using a language you are comfortable with.

(Wow Python and Java is big. Which circle do you belong to? source: codeEval)

Picking a language your team is comfortable with could be hard if you all know many languages. If you cannot come to a consensus for which one to pick, pick the one that has the most support. If you all know Erlang and JavaScript, I would advise you to go for JavaScript, not because JavaScript is more efficient, but it is efficient enough, and there are many, many people on Stack Overflow who knows JavaScript well. I know a specific example where a team deployed Erlang into production, experienced problem with Erlang, and had to seek help from a programmer in Poland to fix their problem. The problem turned out to be relatively small, and the Polish programmer fixed it in a fortnight, yet the team was stuck on this problem for almost a year. Support also means you can easily get many open source packages so that you don’t have to reinvent the wheels. Many reason why scripting languages are popular because there are tools like pip, npm and gem. If you all know Ruby and Python well, and you are developing a web application, then flip a coin.

Hopefully you weren’t one of those programmer who knows both python and ruby, and read all the way till the end to find out that I told you to flip a coin. However, for most of the readers, you need to make sure you are comfortable with your choice, and pick a language other people know well. Bear in mind that once you make the choice, you will be stuck with the choice, so pick carefully.

[bennyyufeichen]

Better use of Munin

Munin is one of the most popular open source solution for server monitoring, and there are several tricks to make Munin easier to use. In this tutorial, we are going to use a different template for Munin, install some plugin for Munin, and setup email warnings for our server.

If you are not sure about how to setup munin, there are many great tutorials you can follow such as here and here

Getting a better template

The first time I tuned into my munin master, I felt like I was in the 80s. The interfaces are hard to navigate, and there is no such thing as dashboard. To improve that, we are going to install a jQuery template for munin.

clone from https://github.com/mallocator/Munin-JQuery-Template

move the cloned folder under a folder munin can read from, suc as /etc/munin

change the parameter tmpldir to point to the new template

load all necessary assets to apache

wait for 5 munites....

and viola! Here is a screenshot:

(look! its a dashboard!)

Install a plugin to monitor process memory for Munin

Munin shows a lot of graph for you my default, but there is no way to monitor the memory usage of a program by default. There is a plugin for munin called multips_memory, 

multips_memory - Munin plugin to monitor memory usage of processes. Which processes are configured in a file in plugin-conf.d.

An example for that would be: I have a MMS monitor running on oen of my server. How much memory does it take?

The source code of the plugin actually gives great example, so I will not spend more time on this :P (http://munin-monitoring.org/browser/munin/plugins/node.d/multips_memory.in)

A step by step for munin-nginx will be coming up soon

Setup warning emails

One of another feature that monitoring tools should provide is alerts. If the CPU or memory usage on a web server exceeds 75%, you probably need to do something about it. Reading the default munin.conf, you can see that there are directives where you can specify multiple emails:

# Drop [email protected] and [email protected] an email everytime # something changes (OK -> WARNING, CRITICAL -> OK, etc) #contact.someuser.command mail -s "Munin notification" [email protected] #contact.anotheruser.command mail -s "Munin notification" [email protected]

Uncomment contact.* and setup your list of emails. A working example:

contacts benny contact.benny.command mail -s "Munin notification for ${var:group} :: ${var:host}" [email protected] contact.benny.text "Check myserver" contact.benny.always_send warning critical

To setup the alert, there are often default values for warning and critical for the plugins that came with munin, but we can also specify them ourselves. An example to define a warning level for each host would be:

[Webnode001] address *.*.*.* use_node_name yes cpu.user.warning 20

[bennyyufeichen]

Index and shardkey choice for social network type query using MongoDB

People often use MongoDB for social networking application, and one of the most important pattern for social networking application would be getting your friend's latest status. Assuming you do not have a complex internal infrastructure for a pre-built activity queue for each user, you will need to query MongoDB in the following way:

db.posts.find({ userID: { $in: […] } }).sort({ datetime: -1 }).limit(10)

Nice an easy isn't it. But how do you index them? If you are using the old MongoDB, then follow this post (http://java.dzone.com/articles/mongodb-indexing-tip-1-find?page=0,1). If you are using MongoDB 2.4 and beyond, its simple:

db.posts.ensureIndex({userID: 1, datetime: 1})

I didn't put down {userID: 1, datetime: -1} because it doesn't matter which field you choose. MongoDB has to find all your friend's post, and sort them in a chronological order anyways. Keeping datetime in the index helps because now MongoDB doesn't have to scan all the documents to do the sorting; it will sort using this index.

Nice and easy isn't it. There is a catch. If you use userID based on MongoDB's default objectID, and use the above multikey index as the shard key, then your shard will be completely out of balance, because the objectID is a monotonically increasing number. (Don't believe me? checkout http://www.slideshare.net/mongodb/hash-based-sharding-21289231, start from slide 41). The solution MongoDB proposed is hash based sharding (http://www.mongodb.com/presentations/hash-based-sharding-mongodb-24), but that doesn't work on the key we just created.

You may not create compound indexes that have hashed index fields. (http://docs.mongodb.org/manual/tutorial/create-a-hashed-index/)

So the bottom line: you cannot use the objectID as the userID if you want to shard it later. Picking a right shard key is always hard because there are a lot of things you need to consider. As you would imagine, random key would result in users all over the place, and you will be queries all the database to get the feed for one person. A solution for that are describe in this paper(http://5harad.com/papers/netshard.pdf), but there is no panacea. It is not easy solution to host a site with millions of users :P

[bennyyufeichen]

Nice and easy initial vimrc setup

Sometimes I go into a completely new vim environment, and vanilla vim doesn't indent properly. Here is my nice and easy vimrc setting file.

set smartindent

set tabstop=2

set shiftwidth=2

set expandtab

set hlsearch

[bennyyufeichen]

Profiling Node.js/Express

Today I was trying to resolve the CPU bottleneck on my web servers. The usage for the CPU was really high, so I wanted to profile my node.js setup. It is very easy.

Start your node.js script with the command

node --prof app.js

install node-tick-processor (npm -g install node-tick-processor)

call node-tick-processor, and it will show you the profiled result!

Note the profiled result can be a little weird as on ubuntu, a lot of ticks are on libc because of epoll_wait (https://groups.google.com/forum/#!topic/nodejs/_U0MmS6rUl4)

After profiling Express, and found out I waste a lot of cycles on crc32..

 [JavaScript]:    ticks  total  nonlib   name    1416    0.2%    1.8%  KeyedLoadIC: A keyed load IC from the snapshot    1146    0.1%    1.5%  LazyCompile: EventEmitter.emit events.js:53    1121    0.1%    1.5%  Stub: CEntryStub    1089    0.1%    1.4%  LazyCompile: APPLY_PREPARE native runtime.js:438     940    0.1%    1.2%  Stub: StringAddStub     929    0.1%    1.2%  LazyCompile: next /root/photo-assault-db-code/node_modules/express/node_modules/connect/lib/proto.js:111     889    0.1%    1.2%  Stub: InstanceofStub     873    0.1%    1.1%  LazyCompile: BSON.deserialize /root/photo-assault-db-code/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:1154     848    0.1%    1.1%  LazyCompile: *exports._unrefActive timers.js:425     811    0.1%    1.1%  LazyCompile: *_crc32 /root/photo-assault-db-code/node_modules/express/node_modules/buffer-crc32/index.js:64     782    0.1%    1.0%  Stub: FastNewClosureStub

Look at the second line from bottom: _crc32?! I did a search and found the source of crc32:

   1088    0.1%      2    0.0%                                      LazyCompile: *exports.etag /root/photo-assault-db-code/node_modules/express/lib/utils.js:23    1074    0.1%      1    0.0%                                        LazyCompile: *crc32.signed /root/photo-assault-db-code/node_modules/express/node_modules/buffer-crc32/index.js:81    1072    0.1%    442    0.0%                                          LazyCompile: *_crc32 /root/photo-assault-db-code/node_modules/express/node_modules/buffer-crc32/index.js:64

turns out I don't need HTTP caching, so I can disable etags. the decrease in CPU was dramatic.

[bennyyufeichen]

Query cache for MongoDB

MongoDB is amazingly fast because of all the corners it cuts. However, one of the thing it does not have is query cache, probably because it wants to you all available memory for data storage.

MongoDB does not implement a query cache: MongoDB serves all queries directly from the indexes and/or data files. (http://docs.mongodb.org/manual/faq/fundamentals/)

Since the backend I am trying to optimize is for a social networking site, and some of the queries are going to be repeated, and some of the queries are going to be repeated. Since MongoDB does not provide a query cache, I tried to look for some query cache already implemented, and there are a few solutions:

Mongoose Cachebox: https://github.com/deremer/cachebox

Mongoose Redis Cache: https://github.com/conancat/mongoose-redis-cache

but I simply cannot find anything that uses Memcached as the engine. Why redis? I don't need persistence. I only wanted cache. In the end, I wrote a simple implementation for MongoDB that is fairly significant. These are the graphs for the database before I implemented query cache

And here is the result after I implemented query cache

The graph that went down most significantly is opcounter. The number of  queries dropped to about a third, which is what we are expecting. I will try to clean up and release my implementation to npm. Comment on the post if you have any better suggestions.

[bennyyufeichen]

Perform stress tests in a realistic manner

Whenever you want to test the upper limit of your system, ideally you want to stress test. The old way is to use apache benchmark, or the command ab. However, it is difficult to provide realistic stress test result using ab, since a stress test should

hammer server on multiple endpoints

simulate user behavior, sometimes perform complex operation

use lots, and lots, and lots of connections

All hail Locust.io! It is a stress test suite written in python. You will be able to write tasks for the user to perform, with different weight on each task. Locust also uses gevent to simulate threads, so it is able to open 10s of thousands of connections on a single machine easily. You also get a pretty web UI to monitor things. It is a pleasure to work with.

You can download locust.io using pip. Check out more information and API from www.locust.io

[bennyyufeichen]

Making a secondary hidden

Yesterday I was trying to increase the read throughput of my system, and for some reason the throughput flattens out. I checked the graphs and could not find the bottleneck. Here are the graphs for my secondary that I was reading from

when the CPU on secondary reaches around 80%, the read capability of the server flattens out. The secondary have 4 cores, and the ssd disks should be able to take around 30k writes. I was puzzled by these numbers.

Today I decided to compare the primary and the secondary side by side, and the bottleneck jumped at my face.

The reason why the read was bottled because I was reading from all the secondaries of the replica set, even though I only specified the secondary on the right in my MongoDB connections. MongoDB was nice that it automatically distributed all the reads to all the nodes, except the node in the middle is suppose to be a cold replica, with a very low system resource and only one core. As you can see from teh opcounters graph, the node in the middle received as much traffic as the node on the left. When we look at the CPU graph for all the nodes in the replica set, the story becomes very clear

The CPU for the cold replica approached 100%, which bottled the read capacity of the system.

In order to resolve the problem, I had to set the cold replica to be 'hidden'. The detailed tutorial is covered in the official MongoDB doc: http://docs.mongodb.org/manual/tutorial/configure-a-hidden-replica-set-member/. I will come back and post the result of the