Random forests for predictive analytics
By Aravind Natarajan, SunFunder’s Data Scientist
A major difficulty that solar companies face in developing countries is the absence of a verifiable credit score for their customers. Many solar companies in Africa operate on the “pay-as-you-go” system -- this means the customer obtains a solar powered home system largely financed by the solar company, with monthly installments made by the customer. Although a sound concept in principle, the pay-as-you-go approach is fraught with difficulty. The large fraction of customers who default on their payments causes solar companies to incur losses and potentially drives them out of business. As a result, many solar companies require that customers pay the entire cost of the system at the time of purchase, limiting its utility to a small percentage of relatively wealthy customers.
Solar companies are increasingly turning to data science to help them figure out which customers are financially stable, and which are too risky to support. This is done by measuring a large number of socio-economic factors: age, gender, marital status, number of dependents, income, number of sources of income, etc. A machine learning algorithm then utilizes this information to distinguish between good customers and risky customers.
The technique of random forests has rapidly become one of most powerful techniques for working with supervised datasets (i.e. customers labeled as being ‘good’ or ‘bad’ in our case). The building block of this method is the decision tree or flowchart. We construct a tree with a number of nodes and at each node, ask questions designed to filter the data into ‘good’ and ‘bad’ customers.
Decision trees work well as classifiers but they have several problems. For example, a single decision tree is prone to overfitting, which occurs when the classifier fits for noise as well as data. The solution is to build several decision trees--a forest of them--and to consider an aggregate of the results of all trees. While each tree (also called a learner) may be weak, the ensemble of many learners will be far more robust.
This is the principle used in the random forest method. The term `random’ comes about due to two reasons: Each decision tree in the forest is provided with a data sample that is randomly chosen with replacement (this is called a bootstrapped sample). At every decision node, the condition to be tested is randomly chosen from a subsample of features. This ensures that not every decision tree considers the same data and not every decision tree utilises the same features for classification. This introduced randomness along with the ensemble approach ensure that the collective wisdom of the forest is superior to the output of any individual tree, and results in a classifier robust to the overfitting problem.
Another highly prized quality of the random forest technique is its ability to choose features intelligently. Let us say we are provided with 1,000 customers, each having 5 features (which could possibly be gender, income, marital status, rent, and size of plot). With such a small data set, it is straightforward to plot each feature against every other feature and search for relevant features visually (the number of plots to be examined is 10). The visual technique becomes increasingly difficult as the number of features grows. (With n features, the number of plots is n(n-1)/2 which grows quadratically with n. With 25 features, 300 plots would need to be examined). Needless to say, a more systematic and mathematically sound technique is desirable.
Consider the following toy example data set with 3 features (called ‘Rent’, ‘Income’, and ‘Marital’) along with 2 possible outcomes (labeled ‘Paid’ and ‘Delinquent’).
Since our aim is to best separate ‘Paid’ and ‘Delinquent’ customers, which features should we choose to perform the split? We could choose to split the dataset on the node ‘Rent = High?’. For those customers whose rent is high, 2 out of 3 are delinquent, while 1 out of 3 has paid. We therefore have a probability = 66.7% of finding delinquent customers by splitting on the ‘rent’ node.
How about the ‘Income = High?’ node? Choosing customers whose Income = High results in all ‘Paid’ customers, i.e. a 100% purity in the split! On the other hand, the node ‘Marital’ results in as many ‘Paid’ customers as ‘Delinquent’ customers and is therefore useless. The gini impurity is a quantitative measure of the deviation from purity defined as:
where k is the number of output classes (2 in our case), and pi is the probability of obtaining class i as a result of the split. The node ‘Rent’ has a gini impurity of 1- [(2/3)^2 + (1/3)^2] = 4/9. The node ‘Income’ has a gini impurity of 0, while the node ‘Marital’ has a gini impurity of 0.5. It is clear that the gini impurity lies between 0 (all samples belong to the the same class, and hence a perfectly pure split) and 1.0 (a large number of classes, each with equal probability). The decrease in the gini impurity is then calculated as:
where N refers to the node in question, N-L, the node to the left of N, and N-R, the node to the right of N. n is the number of samples to the left of node N. The decrease in gini impurity is computed for each feature, and averaged over the forest. A large decrease in gini impurity implies that the feature being considered is useful in distinguishing between classes. The random forest algorithm thus tells us which features are best suited to distinguish between good and bad customers. The figure below shows useful features ranked by their gini importance:
