Data Analytics @genaraprescendo-blog - Tumblr Blog

Defining my new Research Question

Title:

Predicting blood donations based on past donation patterns

Research Question:

The purpose of this study is to analyse each donor’s information, such as month since last donation, number of donations, total volume donated, and months since first donation; to predict whether a blood donor will donate again in March 2007.

Motivation:

Help improve the health-related processes and system has always been my area of interest given the importance and impact to the society. Blood donation is a fairly simple process but it is crucial for a variety of situations in hospitals. By predicting blood donations, we will be able to focus on donation strategies in a timely manner.

Implications:

Predicting blood donations and supply needs can improve the entire supply chain, making sure that more patients get the blood transfusions they need. According to the American Red Cross, our society is currently being faced by a scarcity of blood. For this reason, a better understanding of future patterns can ensure blood availability and save lives.

Exploratory Data Analysis

The assignment of the week is about running a basic frequency distribution in Python using the Gapminder data set.

Here is my code:

import pandas import numpy

data = pandas.read_csv("gapminder.csv") print("\n" + "The total number of observations is " + str(len(data))) print("The total number of variables is " + str(len(data.columns)) + "\n")

def conv_num(x): #convert to numeric to have the distribution in order data[x] = pandas.to_numeric(data[x], errors = 'coerce')

conv_num("breastcancerper100th") conv_num("alcconsumption") conv_num("femaleemployrate")

#continuous values can be discretized using the cut function sub0 = data.copy()

def print_quartil(x): sub0["categories"] = pandas.cut(sub0[x], 4) print("\n" + x + " frequency table") print(sub0["categories"].value_counts())

print_quartil("breastcancerper100th") print_quartil("alcconsumption") print_quartil("femaleemployrate")

#creating a subsest data sub1 = data[(data["femaleemployrate"]>= 50) & (data["breastcancerper100th"]>=20)]

del sub1["incomeperperson"] del sub1["armedforcesrate"] del sub1["breastcancerper100th"] del sub1["co2emissions"] del sub1["femaleemployrate"] del sub1["hivrate"] del sub1["internetuserate"] del sub1["lifeexpectancy"] del sub1["oilperperson"] del sub1["polityscore"] del sub1["relectricperperson"] del sub1["suicideper100th"] del sub1["employrate"] del sub1["urbanrate"] del sub1["country"]

new_data = sub1.copy()

new_data["categories"] = pandas.cut(new_data["alcconsumption"], bins=[-1, 2, 5, 7, 10, 12, 25], labels=False) labels = numpy.array("0_Rarely 1_Sometimes 2_Often 3_Very_Often 4_Almost_Daily 5_Daily".split()) new_data["categories"] = labels[new_data["categories"]]

group_data = new_data.groupby(["categories"], as_index= True).count() group_data["percent"] = 100*group_data/group_data.sum()

print("\n" + "Frequency Table from Subset Data:") print(group_data.sort_index())

The output:

As you can see from python output, I chose to display the frequency table of following three variables: breastcancerper100th, alcconsumption and femaleemployrate.

I decided for redefining my research question by narrowing down the subset of data. The new subset data was defined by those countries that have high female employment rate (>=50) and high incidence of breast cancer (>=20). This is represented in the last frequency table.

Running a K-Means Cluster Analysis

A k-means cluster analysis was conducted to identify subgroups of countries based on their similarity of responses on 5 variables that represent characteristics that could have an impact on breast cancer incidence. Clustering variables in this case are all quantitative, measuring the consumption of alcohol, CO2 emissions, female employment rate, income per person and life expectancy.

All clustering variables were standardized to have a mean of 0 and a standard deviation of 1.

Data were not split into training and test set since there is a relatively small number of observations (214 countries).

A series of k-means cluster analyses were conducted on the data specifying k=1-9 clusters, using Euclidean distance. The variance in the clustering variables that was accounted for by the clusters (r-square) was plotted for each of the nine cluster solutions in an elbow curve to provide guidance for choosing the number of clusters to interpret.

Here is my Python code:

from pandas import Series, DataFrame import pandas as pd import numpy as np import matplotlib.pylab as plt from sklearn.cross_validation import train_test_split from sklearn import preprocessing from sklearn.cluster import KMeans from scipy.spatial.distance import cdist

#loading dataset data = pd.read_csv("gapminder.csv")

def analyse(ax): global data #setting variables you will be working with to numeric because python read from the csv file as strings(objects) data[ax] = pd.to_numeric(data[ax], errors='coerce')

analyse("breastcancerper100th") analyse("alcconsumption") analyse("co2emissions") analyse("femaleemployrate") analyse("incomeperperson") analyse("lifeexpectancy")

#data management data_clean = data.dropna()

#Subset clustering variables cluster = data_clean[[ "alcconsumption", "co2emissions", "femaleemployrate","lifeexpectancy", "employrate", "incomeperperson"]] cluster.describe()

#standardize clustering variables to have mean=0 and sd=1 clustervar = cluster.copy()

def standardizing(x): clustervar[x] = preprocessing.scale(clustervar[x].astype("float64")) standardizing("alcconsumption") standardizing("co2emissions") standardizing("femaleemployrate") standardizing("incomeperperson") standardizing("lifeexpectancy")

# k-menas cluster analysis for 1-9 clusters clusters = range(1,10) meandist= []

for k in clusters: model = KMeans(n_clusters=k) model.fit(clustervar) clausassign = model.predict(clustervar) meandist.append(sum(np.min(cdist(clustervar, model.cluster_centers_, "euclidean"), axis=1))/clustervar.shape[0])

#Plot average distance from observations from the cluster centroid to sue the Elbow Method to identify number of clusters to choose

plt.plot(clusters, meandist) plt.xlabel("Number of clusters") plt.ylabel("Average distance") plt.title("Selecting k with the Elbow Method") plt.show()

The figure below shows the elbow curve of r-square values for the nine cluster solution.

So what this plot shows is the decrease in the average minimum distance of the observations from the cluster centroids for each of the cluster solutions. We can see that the average distance decreases as the number of clusters increases. Since the goal of cluster analysis is to minimize the distance between observations and their assigned clusters we want to chose the fewest numbers of clusters that provides a low average distance. What we're looking for in this plot is a bend in the elbow that kind of shows where the average distance value might be leveling off such that adding more clusters doesn't decrease the average distance as much.

We can see there appears to be a couple of bends at the line at two clusters and at three clusters, but it's not very clear. To help us figure out which of the solutions is best we should further examine the cluster solutions for at least the two and three cluster solutions to see whether they do not overlap, whether the patterns of means on the clustering variables are unique and meaningful, and whether there are significant differences between the clusters on our external validation variable “breastcancerper100th”.

Here is my additional code to the tree cluster solution:

#interpret 3 cluster solution model3 = KMeans(n_clusters=3) model3.fit(clustervar) clusassign = model3.predict(clustervar)

#plot clusters pca_2 = PCA(2) plot_columns = pca_2.fit_transform(clustervar) plt.scatter(x=plot_columns[:,0], y=plot_columns[:,1], c=model3.labels_,) plt.xlabel('Canonical variable 1') plt.ylabel('Canonical variable 2') plt.title('Scatterplot of Canonical Variables for 3 Clusters') plt.show()

#multiple steps to merge cluster assignment with clustering variables to examine cluster variable means by cluster # create a unique identifier variable from the index for the # cluster training data to merge with the cluster assignment variable clustervar.reset_index(level=0, inplace=True) # create a list that has the new index variable cluslist=list(clustervar['index']) # create a list of cluster assignments labels=list(model3.labels_) # combine index variable list with cluster assignment list into a dictionary newlist=dict(zip(cluslist, labels)) newlist # convert newlist dictionary to a dataframe newclus=DataFrame.from_dict(newlist, orient='index') newclus # rename the cluster assignment column newclus.columns = ['cluster']

# now do the same for the cluster assignment variable # create a unique identifier variable from the index for the # cluster assignment dataframe # to merge with cluster training data newclus.reset_index(level=0, inplace=True) # merge the cluster assignment dataframe with the cluster training variable dataframe # by the index variable merged_train=pd.merge(clustervar, newclus, on='index') merged_train.head(n=100) # cluster frequencies merged_train.cluster.value_counts()

# calculate clustering variable means by cluster clustergrp = merged_train.groupby('cluster').mean() print ("Clustering variable means by cluster") print(clustergrp)

# validate clusters by examining cluster differences in breastcanerper100th using ANOVA # first have to merge breast cancer with clustering variables and cluster assignment data bc_data=data_clean["breastcancerper100th"] # split breastcancer data into train and test sets bc_train, bc_test = train_test_split(bc_data, test_size=.3, random_state=123) bc_train1=pd.DataFrame(bc_train) bc_train1.reset_index(level=0, inplace=True) merged_train_all=pd.merge(bc_train1, merged_train, on='index') sub1 = merged_train_all[['breastcancerper100th', 'cluster']].dropna()

import statsmodels.formula.api as smf import statsmodels.stats.multicomp as multi

bcmod = smf.ols(formula='breastcancerper100th ~ C(cluster)', data=sub1).fit() print (bcmod.summary())

print ('means for breast cancer by cluster') m1= sub1.groupby('cluster').mean() print (m1)

print ('standard deviations for breast cancer by cluster') m2= sub1.groupby('cluster').std() print (m2)

mc1 = multi.MultiComparison(sub1["breastcancerper100th"], sub1['cluster']) res1 = mc1.tukeyhsd() print(res1.summary())

Results:

Canonical discriminant analyses was used to reduce the 5 clustering variable down a few variables that accounted for most of the variance in the clustering variables. Plot of the first two canonical variables for the clustering variables by cluster is showed below:

This scatter plot indicates that the clusters are not densely packed and they did not overlap very much each other, which means the observations within the clusters are not highly correlated, and there is a high variance between clusters.

Analysis:

The means on the clustering variables showed that compared to the other clusters, countries in the second cluster, cluster 1, clearly includes countries more likely to have high incidence of breast cancer due to high levels of alcohol consumption and high rates of life expectancy.

In order to externally validate the clusters, an Analysis of Variance (ANOVA) was conducting to test for significant differences between the clusters on breast cancer incidence. The results showed they do differ significantly (p-value= 0.00254).

The tukey post hoc comparisons showed significant differences between clusters on breast cancer incidence, with the exception that clusters 0 and 1 were not significantly different from each other. Countries in cluster 2 had the highest breast cancer incidence (mean=43.94, sd=22.64), and cluster 3 had the lowest (mean=23.06, sd=16.84).

K-Means Cluster Analysis

The goal of cluster analysis is to group or cluster observations into subsets based on the similarity of responses on multiple variables. Observations that have similar response patterns are grouped together to form clusters.

Cluster analysis is an unsupervised machine learning method, which means there is no specific response variable included in the analysis. Cluster analysis is often used in marketing, to develop targeted advertising campaigns.This is often referred to as market segmentation. Likewise, health researchers might use cluster analysis to identify individuals at greatest risk for health problems, and to develop targeted health messages based on patterns of health behavior.

With cluster analysis, what we want is to obtain clusters that have less variance within clusters and more variance between clusters.

Less variance within clusters means that the observations within a particular cluster are similar to each other in their pattern of response on the clustering variables. More variance between clusters means the clusters are distinct.

Cluster analysis can also be used as a data reduction technique that allows us to take many variables and reduce them down to a single categorical variable that has as many categories as the number of clusters identified in the dataset. This categorical variable can then be used in other analysis to predict some response variable of interest.

There are many types of clustering algorithms. K-means cluster analysis, is one of the most commonly uses clustering algorithms. Cluster analysis measures the distance between points in the p-dimensional space, and groups together those observations that are close to each other.

There are multiple ways to calculate the distance between observations. The most commonly used distance measuring, K-means cluster analysis, is call Euclidean distance. The Euclidian distance measure determines how close observations are to each other by drawing a straight line between pairs of observations, and calculating the distance between them based on the length of this line.

To demonstrate how cluster analysis works, let's look at a simple example. Say we have p=2 variables and we want to create K=2 clusters. The first step is to randomly choose two points in the two dimensional space. These points will start as the center or centroid for each of the two clusters. Then the distance between each point and cluster centroids is calculated and the point is assigned to the cluster that has the smallest distance.

After the initial formation of the clusters, the centroid for each cluster is recalculated based on the location of the points that were assigned to it. Then the process starts all over again by calculating the distances between the points and the new locations of both centroids.

This process is repeated using multiple iterations until the location of the centroids doesn't change very much. During the process, observations that were originally assigned to one cluster may end up in a different cluster.

#machine learning #k-means clustering #market segmentation #k-means cluster analysis #wesleyan university

Confusion Matrix for Binary Classification

A confusion matrix is a table that is often used to describe the performance of a classification model on a set of test data for which the true values are known.

Let's start with an example confusion matrix for a binary classifier (though it can easily be extended to the case of more than two classes).

When we have binary outcomes and a binary classification, we can be right in two different ways and we can be wrong in two different ways.

Example:

Consider that the matrix below is predicting the presence of a disease. There are two possible predicted classes: "yes" and "no". “Yes” means they have the disease, and "no" means they don't have the disease. The classifier made a total of 165 predictions (e.g., 165 patients were being tested for the presence of that disease). Out of those 165 cases, the classifier predicted "yes" 110 times, and "no" 55 times. In reality, 105 patients in the sample have the disease, and 60 patients do not.

Let's now define the most basic terms, which are whole numbers (not rates):

true positives (TP): These are cases in which we predicted yes (they have the disease), and they do have the disease.

true negatives (TN): We predicted no, and they don't have the disease.

false positives (FP): We predicted yes, but they don't actually have the disease. (Also known as a "Type I error.")

false negatives (FN): We predicted no, but they actually do have the disease. (Also known as a "Type II error.")

This is a list of rates that are often computed from a confusion matrix for a binary classifier:

Accuracy: Overall, how often is the classifier correct?

(TP+TN)/total = (100+50)/165 = 0.91

Misclassification Rate: Overall, how often is it wrong?

(FP+FN)/total = (10+5)/165 = 0.09 (1-Accuracy) also known as "Error Rate"

True Positive Rate: When it's actually yes, how often does it predict yes?

TP/Condition “+” = 100/105 = 0.95 also known as "Sensitivity" or "Recall"

False Positive Rate: When it's actually no, how often does it predict yes?

FP/Condition “-” = 10/60 = 0.17

Specificity: When it's actually no, how often does it predict no?

TN/Condition “-” = 50/60 = 0.83 equivalent to 1 minus False Positive Rate

Precision: When it predicts yes, how often is it correct?

TP/Classification “+” = 100/110 = 0.91

Prevalence: How often does the yes condition actually occur in our sample? condition “+”/total = 105/165 = 0.64

Source: Duke University through Coursera, http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/

Case Study Analysis

Retaking the case study from the previous post, it is time to evaluate the most relevant business metric that could help Happy Life company increase revenues and restore profitability.

I strongly suggest a business metric related to inventory.

Considering that the number of visitor has held constant and the revenues are down, it means the ticket average size must have decreased due to unavailability of customers' favorite flavors. On the other hand, the fact that many stores are no longer profitable makes me believe (believe because I do not have access to the database) some other flavors must have been thrown away, otherwise they should be able to keep the stores profitable as they use to be in the past with the same number of visitors (and of course considering no costs change).

Taking into account the current tracking of data, this business metric would be measured weekly for each store and for each flavor: the quantity sold divided by the quantity delivered times 100.

The result would tell us the percentage of each flavor is thrown away per week (when <100%), or indicate that the store may have been losing sales due to insufficient stock (when =100%).

Logical conclusion: This business metric would allow us to adjust the quantity of each ingredient that should be delivered to each store in order to neither cause wastage of products nor sales lost. In another words, this metric would help us eliminate wastes, increase revenues and operation efficiency.

Note that we are talking about a dynamic metric, a metric that can be significantly affected by specific actions. I another words, the kind of metric a Business Analyst should work with (I think it is worth an exclusive post!).

A relevant business process change would be to start tracking the quantity of product made per day. Currently, they just have the information about the quantity delivered per week. However, as far as I am concerned, the frozen yogurt is made daily and if not sold by the end of the day, it must be thrown away. Tracking the quantity effectively produced by day would be useful to account the daily wastage of each ingredient and then, to generate a more accurate production planning.

*Of course we could do some other improvements and analysis even with the same available data. However, the challenge here is to choose just one business metric.

A Business Metric Case Study

Happy Life, a U.S. national chain of frozen yogurt stores with about 500 stores in 40 states is asking for assistance with its business processes. The average number of visitors per store has held constant over the past several years, but revenues per store are down by an average of 10%, and many stores are no longer profitable. The client suspects that a large amount of inventory is being thrown away unused at the end of each day. At the same time, customer polling suggests that the yogurt flavor customers want is often not available, even when the flavor is posted on the menu. People also complain about stores being closed when they visit. Now, the chain is facing increased competition from frozen yogurt sold in 24-hour grocery stores. Happy Life has asked your team to recommend business process changes that can help them increase revenues and restore profitability.

Happy Life currently has the following data available:

4 years of cash register data for every store that has, for each transaction, the date and time of the purchase, the specific items purchased, and the sales price of each item purchased.

4 years of delivery data that lists how much of each kind of yogurt mix, flavoring, and topping was delivered to each store each week.

Typical public company accounting data, including annual revenues, annual cost of goods sold, in-store inventory on hand at the end of the year.

No other company data is available.

You are the Business Consultant and you were asked to identify one business metric that could be extracted from the available data in order to suggest a business process change that could be related to improving Happy Life’s revenue or store profitability.

What would you choose and why?

In the next post I’ll be sharing my point of view... See you there!

#business metrics #data analysis #business case study

Running a Lasso Regression Analysis

Let’s see how Lasso Regression performed with the Gapminder dataset.

Lasso regression analysis is a shrinkage and variable selection method for linear regression models. The goal of lasso regression is to obtain the subset of predictors that minimizes prediction error for a quantitative response variable. The lasso does this by imposing a constraint on the model parameters that causes regression coefficients for some variables to shrink toward zero. Variables with a regression coefficient equal to zero after the shrinkage process are excluded from the model. Variables with non-zero regression coefficients variables are most strongly associated with the response variable. Explanatory variables can be either quantitative, categorical or both.

Here is my Python code:

import pandas as pd import numpy as np import matplotlib.pylab as plt from sklearn.cross_validation import train_test_split from sklearn.linear_model import LassoLarsCV

from sklearn import preprocessing

#loading the dataset data = pd.read_csv("gapminder.csv")

def analyse(ax): global data #setting variables you will be working with to numeric because python read from the csv file as strings(objects) data[ax] = pd.to_numeric(data[ax], errors='coerce')

analyse("breastcancerper100th") analyse("alcconsumption") analyse("armedforcesrate") analyse("co2emissions") analyse("femaleemployrate") analyse("hivrate") analyse("lifeexpectancy") analyse("polityscore") analyse("suicideper100th") analyse("employrate") analyse("urbanrate")

#data management data_clean = data.dropna()

#I set my explanatory (predictors) and response variable (target). predvar = data_clean[[ "alcconsumption", "armedforcesrate", "co2emissions", "femaleemployrate", "hivrate", "lifeexpectancy", "polityscore", "suicideper100th", "employrate", "urbanrate" ]]

#standardize predictors to have mean=0 and sd=1 predictors = predvar.copy()

def standardizing(x): global predictors predictors[x] = preprocessing.scale(predictors[x].astype("float64")) standardizing("alcconsumption") standardizing("armedforcesrate") standardizing("co2emissions") standardizing("femaleemployrate") standardizing("hivrate") standardizing("lifeexpectancy") standardizing("polityscore") standardizing("suicideper100th") standardizing("employrate") standardizing("urbanrate")

target = data_clean["breastcancerper100th"] pred_train, pred_test, tar_train, tar_test = train_test_split(predictors, target, test_size=.3, random_state=123)

#specify the Lasso Regression Model model = LassoLarsCV(cv=10, precompute= False).fit(pred_train, tar_train)

#The dict object creates a dictionary, and the zip object creates lists. my_dict = dict(zip(predictors.columns, model.coef_)) for k, v in my_dict.items(): print(k, v)

#plot coefficient progression m_log_alphas = -np.log10(model.alphas_) ax = plt.gca() plt.plot(m_log_alphas, model.coef_path_.T) plt.axvline(-np.log10(model.alpha_), linestyle="--", color = "k", label = "alpha CV") plt.ylabel("Regression Coefficients") plt.xlabel("-log(alpha)") plt.title("Regression Coefficients Progression for Lasso Paths") plt.show()

# plot mean square error for each fold m_log_alphascv = -np.log10(model.cv_alphas_) plt.figure() plt.plot(m_log_alphascv, model.cv_mse_path_, ':') plt.plot(m_log_alphascv, model.cv_mse_path_.mean(axis=-1), 'k', label='Average across the folds', linewidth=2) plt.axvline(-np.log10(model.alpha_), linestyle='--', color='k', label='alpha CV') plt.legend() plt.xlabel('-log(alpha)') plt.ylabel('Mean squared error') plt.title('Mean squared error on each fold') plt.show()

# MSE from training and test data from sklearn.metrics import mean_squared_error train_error = mean_squared_error(tar_train, model.predict(pred_train)) test_error = mean_squared_error(tar_test, model.predict(pred_test)) print ('\ntraining data MSE') print(train_error) print ('\ntest data MSE') print(test_error)

# R-square from training and test data rsquared_train=model.score(pred_train,tar_train) rsquared_test=model.score(pred_test,tar_test) print ('\ntraining data R-square') print(rsquared_train) print ('\ntest data R-square\n') print(rsquared_test)

The output:

Analysis:

Predictors with regression coefficients that do not have a value of zero are included in the selected model. Predictors with regression coefficients equal to zero means that the coefficients for those variables had shrunk to zero after applying the LASSO regression penalty, and were subsequently removed from the model.

Python console results shows that of the 10 variables, 5 were selected in the final model.

So we can also use the size of the regression coefficients to tell us which predictors are the strongest predictors of breast cancer incidence. For example, urban rate and alcohol consumption had the largest regression coefficients, and were most strongly associated with breast cancer incidence, followed by life expectancy and CO2 emissions.

From the first plot we can see the relative importance of the predictor selected at any step of the selection process, how the regression coefficients changed with the addition of a new predictor at each step, as well as the steps at which each variable entered the model.

The second plot shows the change in the mean square error for the change in the penalty parameter alpha at each step in the selection process.

The R-square values were 0.65 and 0.51, indicating that the selected model explained 65% and 51% of the variance in breast cancer incidence for the training and test sets, respectively.

#lasso regression #machine learning #linear regression

What is Lasso Regression?

Lasso regression is a supervised machine learning method often used in to select the subset of variables. Specifically, LASSO is a shrinkage and variable selection method for linear regression models. LASSO stands for “Least Absolute Selection and Shrinkage Operator”.

The LASSO imposes a constraint on the sum of the absolute values of the model parameters, where the sum has a specified constant as an upper bound. This constraint causes regression coefficients for some variables to shrink towards zero. This is the shrinkage process. The shrinkage process allows for better interpretation of the model and identifies the variables most strongly associated with the target corresponds variable. That is the variable selection process. It goes to obtain the subset of predictors that minimizes prediction error. So why use Lasso instead of just using Ordinary Least Squares (OLS) multiple regression? Well, first, it can provide greater prediction accuracy. If the true relationship between the response variable and the predictors is approximately linear and you have a large number of observations, then OLS regression parameter estimates will have low bias and low variance. However, if you have a relatively small number of observations and a large number of predictors, then the variance of the OLS perimeter estimates will be higher. In this case, Lasso Regression is useful because:

1) Shrinking the regression coefficient can reduce variance without a substantial increase in bias.

2) It can increase model interpretability. Often times, at least some of the explanatory variables in an OLS multiple regression analysis are not really associated with the response variable. As a result, we often end up with a model that's over fitted and more difficult to interpret. With Lasso Regression, the regression coefficients for unimportant variables are reduced to zero which effectively removes them from the model and produces a simpler model that selects only the most important predictors.

In Lasso Regression, a tuning parameter called lambda is applied to the regression model to control the strength of the penalty. As lambda increases, more coefficients are reduced to zero that is fewer predictors are selected and there is more shrinkage of the non-zero coefficient.

With Lasso Regression, when lambda is equal to zero we have an OLS regression analysis. Bias increases and variance decreases as lambda increases.

#lasso regression #machine learning #linear regression model #statistics #regression anlysis #cousera

Business Metrics

I have no doubt about the importance of analyzing data to find out patterns, trends and unexpected variances that could show new business directions.

By the way, it is exactly the kind of thing I am keen to do. However, I do know that all these efforts may fall apart if we build our analysis on the wrong questions.

That’s why today I chose to write about business metrics.

Business metrics are important to help us to ask and answer the right questions. Metrics are numbers that can be impacted when business processes are changed.

All business metrics can be classified into three broad categories.

So the question is:

What changes in our business processes can we make right now to increase revenues, maximize profitability and reduce risk?

In order to help you answer this question about your business, I’ll be sharing in the next post a hypothetical situation and the rationale to choose the most relevant business metric for this case study.

#business metrics #data analys

Running a Random Forest

Random forest analysis was performed in Python3 to evaluate the importance of a series of explanatory variables in predicting a binary, categorical response variable. Random forests provide importance scores for each explanatory variable and also allow us to evaluate any increases in correct classification with the growing of smaller and larger number of trees.

The following explanatory variables were included as possible contributors to a random forest to evaluate my response variable breast cancer incidence (moderate/strong): income per person, alcohol consumption, CO2 emissions, female employ rate, life expectancy and urban rate.

Here is my Python code:

from pandas import Series, DataFrame import pandas as pd import numpy as np import os import matplotlib.pylab as plt import matplotlib.pylab as plt from sklearn.cross_validation import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import classification_report import sklearn.metrics # Feature Importance from sklearn import datasets from sklearn.ensemble import ExtraTreesClassifier from sklearn.ensemble import RandomForestClassifier

#to indicate where my data set is located os.chdir("/home/genara/Documents/Coursera/4_Machine_Learning/Random Forests")

#loading the dataset AH_data = pd.read_csv("gapminder.csv")

def analyse(ax): #setting variables you will be working with to numeric because python read from the csv file as strings(objects) AH_data[ax] = pd.to_numeric(AH_data[ax], errors='coerce')

analyse("breastcancerper100th") analyse("incomeperperson") analyse("alcconsumption") analyse("co2emissions") analyse("femaleemployrate") analyse("lifeexpectancy") analyse("urbanrate")

#grouping breast cancer(response variable) into 2 categories AH_data["breastcategories"] = pd.cut(AH_data["breastcancerper100th"], bins=[1, 40, 102], labels=False) labels = np.array("0_Moderate 1_Strong".split()) AH_data["breastcategories"] = pd.to_numeric(AH_data["breastcategories"], errors='coerce')

#Because decision tree analyses cannot handle any NA's in our data set, my next step is to create a clean data frame that drops all NA's. data_clean = AH_data.dropna()

#I set my explanatory (predictors) and response variable (target). predictors = data_clean[["incomeperperson", "alcconsumption", "co2emissions", "femaleemployrate", "lifeexpectancy", "urbanrate" ]]

targets = data_clean.breastcategories

#And then include the train test split function for predictors and target. And set the size ratio to 60% for the training sample and 40% for the test sample by indicating test_size=.4. pred_train, pred_test, tar_train, tar_test = train_test_split(predictors, targets, test_size=.4)

print("The shape of the predictor training sample:") print(pred_train.shape) print("The shape of the predictor test sample:") print(pred_test.shape) print("The shape of the target training sample:") print(tar_train.shape) print("The shape of the target and test sample:") print(tar_test.shape)

#Build model on training data #n_estimators are the number of trees you would build with the random forest algorithm. classifier = RandomForestClassifier(n_estimators = 25)

#Then use this classifier.fit function which you pass the training predictors and training targets to. It's this code that will fit our model. classifier=classifier.fit(pred_train,tar_train)

#Next we include the predict function where we predict for the test values predictions=classifier.predict(pred_test)

#and then call in the confusion matrix function which we passed the target test sample to. print("Confusion Matrix: ") print(sklearn.metrics.confusion_matrix(tar_test,predictions))

#We can also look at the accuracy score print("Accuracy Score: ") print(sklearn.metrics.accuracy_score(tar_test, predictions))

#fit an Extra Trees model to the data model = ExtraTreesClassifier() model.fit(pred_train, tar_train) # display the relative importance of each attribute print(model.feature_importances_)

#Impact of tree size on prediction accuracy #To determine what growing larger number of trees has brought us in terms of correct classification. We're going to use code that builds for us different numbers of trees, from one to 25, and provides the correct classification rate for each.

trees = range(25) accuracy = np.zeros(25)

for idx in range(len(trees)): classifier = RandomForestClassifier(n_estimators = idx + 1) classifier = classifier.fit(pred_train, tar_train) predictions = classifier.predict(pred_test) accuracy[idx] = sklearn.metrics.accuracy_score(tar_test, predictions)

plt.cla() plt.plot(trees,accuracy) plt.show()

The Output:

Data Analysis:

From the confusion matrix, we see the true negatives and true positives on the diagonal. The 3 and the 7 represent the false negatives and false positives, respectively. The accuracy of the random forest was 84.37%, which means that roughly 84% of the countries were classified correctly, as having a moderate or a strong breast cancer incidence.

Given that we don't interpret individual trees in a random forest, the most helpful information to be gotten from a forest is arguably the measured importance for each explanatory variable. The explanatory variables with the highest relative importance scores were, as expected, life expectancy (with 0.3487...) and alcohol consumption (with 0.1982...).

From the graph, it can be seen that the subsequent growing of multiple trees rather than a single tree, added considerably (approximately 5 p.p.) to the overall accuracy of the model, and suggesting that interpretation of a single decision tree may be inappropriate.

#random forest #data mining #python3

What Is A Random Forest?

Random Forest is a machine learning method. This data mining algorithm is based on decision trees, but proceeds by growing many trees. While decision trees are not very reproducible on future data and are proceed by searching for a split on every variable in every node, Random Forests searches for a split on only one variable in a node. The variable that has the largest association with the target among candidate explanatory variables but only among those explanatory variables that have been randomly selected to be tested for that node.

How does it work?

First, a small subset of explanatory variables is selected at random.

Next, the node is split with the best variable among the small number of randomly selected variables.

Then, a new list of eligible explanatory variables is selected on random to split on the next node. This continues until the tree is fully grown, and ideally there is one observation in each terminal mode.

The eligible variables set will be quite different from node to node.

However, important variables will eventually make it into the tree. Their relative success in predicting the target variable will begin to get them larger and larger numbers of "votes" in their favor.

The growing of each tree in a random forest is not only based on subsets of explanatory variables at each node, but also based on a random subset of the sample for each tree in the forest.

This process of selecting a random sample of observations is known as Bagging. Importantly, each tree is growing on a different randomly selected sample of Bagged data with the remaining Out of Bag data available to test the accuracy of each tree. For each tree, the Bagging Process selects about 60% of the original sample, while the resulting tree is tested against the remaining 40% of the sample. Thus, the randomly selected bag data and out of bag data, will be a different 60% and 40% of observations for each tree. The most important thing to know when interpreting results of random forests is that the trees generated are not themselves interpreted. Instead, they are used to collectively rank the importance of variables in predicting our target of interest.

#machine learning #random forests #data mining

Running a Classification Tree

Decision trees are predictive models that allow for a data driven exploration of nonlinear relationships and interactions among many explanatory variables in predicting a response or target variable. When the response variable is categorical (two levels), the model is a called a classification tree. Explanatory variables can be either quantitative, categorical or both. Decision trees create segmentations or subgroups in the data, by applying a series of simple rules or criteria over and over again which choose variable constellations that best predict the response (i.e. target) variable.

In order to perform a classification tree with the Gapminder data set, I had to first bin my response variable (breastcancerper100th) into 2 categories since this data is quantitative.

Here is the complete Python code:

from pandas import Series, DataFrame import pandas as pd import numpy as np import os import matplotlib.pylab as plt from sklearn.cross_validation import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import classification_report import sklearn.metrics from sklearn import tree from io import StringIO from IPython.display import Image import pydot

#to indicate where my data set is located os.chdir("/home/genara/Documents/Coursera/4_Machine_Learning")

#loading the dataset AH_data = pd.read_csv("gapminder.csv")

#setting variables you will be working with to numeric because python read from the csv file as strings(objects)

def analyse(ax): AH_data[ax] = pd.to_numeric(AH_data[ax], errors='coerce')

analyse("breastcancerper100th") analyse("incomeperperson") analyse("alcconsumption") analyse("co2emissions") analyse("femaleemployrate") analyse("lifeexpectancy") analyse("urbanrate")

#Because decision tree analyses cannot handle any NA's in our data set, my next step is to create a clean data frame that drops all NA's. data_clean = AH_data.dropna()

#Setting the new data frame called data_clean, I can now take a look at various characteristics of my data by using the d types and describe functions to examine data types and summary statistics print(data_clean.dtypes) print(data_clean.describe())

#I set my explanatory (predictors) and response variable (target). predictors = data_clean[["incomeperperson", "alcconsumption", "co2emissions", "femaleemployrate", "lifeexpectancy", "urbanrate" ]]

targets = data_clean.breastcategories

#Build model on training data #Once training and testing data sets have been created, we will initialize the DecisionTreeClassifier from SKLearn. classifier=DecisionTreeClassifier()

#Then use this classifier.fit function which you pass the training predictors and training targets to. It's this code that will fit our model. classifier=classifier.fit(pred_train,tar_train)

#Next we include the predict function where we predict for the test values predictions=classifier.predict(pred_test)

#and then call in the confusion matrix function which we passed the target test sample to. sklearn.metrics.confusion_matrix(tar_test,predictions)

#We can also look at the accuracy score sklearn.metrics.accuracy_score(tar_test, predictions)

#request the picture of our decision tree. out = StringIO() tree.export_graphviz(classifier, out_file=out) graph=pydot.graph_from_dot_data(out.getvalue()) graph.write_pdf("breastcancer.pdf")

Here is the output:

Here is the decision tree:

Analysis:

Let’s first look at the accuracy score, which is approximately 0.89. It suggests that the decision tree model has classified 89% of the sample correctly as either moderate cancer incidence or high cancer incidence.

For our predictor variable, the training sample has 95 rows and 6 explanatory variables, while the test sample has 64 rows and 6 explanatory variables.

The resulting tree starts with the split on X[4], our fifth explanatory variable, life expectancy. If the the value is greater than 74.231 years, we move to the right side of the split that include 33 of the 95 observations in the training sample. From this node, another split is made on our second explanatory variable, alcconsumption. In the same way, if the the alcohol consumption is less than 8.3, we move to the left side of the split that include 12 of the 33 observations in the training sample. From this node, another split is made on our third explanatory variable, co2emissions, and so the splits continue for the others explanatory variables.

It can be seen that among those countries that have life expectancy greater than 74.231 and alcohol consumption levels between 8.6 and 17.8, all of them were classified with strong cancer incidence [20, 0].

#decision trees

What is the Process of Growing a Decision Tree?

Based on the variables considered by the model, the decision tree method works by making binary splits in a sample to maximize correct classification of those with and without the target of interest. All possible separations or cut-points are tested and the separation yielding the minimum impurity or error is selected. And subgroups showing similar outcomes, but different explanatory variable constellations, are generated.

Decision tree methods also commonly employ a cross-validation procedure to guard against an overfit model, in which splits are included in the tree that do no actually improve the fit of the model. Following an initial growing of the tree, a random subset of the data is tested, that is, validation sample. And only branches of the tree that improve the correct classification rate are retained. Generally, the data is divided into training and validation or test sub-samples. And the training sample is used to grow an overly large tree, while the validation or test sample is then used to estimate the rate at which the cases are misclassified. The misclassification rate is calculated for every sized tree, and the selected sub-tree represents the lowest probability of misclassification.

#data mining #decision tree

Decision Trees

When our data has large numbers of explanatory variables that may interact in very complicated ways, building a global linear model can be difficult if not foolish. Decision trees, a data mining method that allows us to explore the presence of potentially complicated interactions within our data by creating segmentation or sub groups.

Like liner regression, decision trees are statistical models designed for what are known as supervised prediction problems. In supervised prediction, a set of explanatory variables also known as predictors, inputs or features is used to predict the value of a response variable, also called the outcome or target variable.

Decision trees are read from the top down, starting at the root node. Each internal node in blue represents a split based on the values of one of the explanatory variables. Importantly, an explanatory variable can appear in more than one split throughout the tree, as variable B does here. The green terminal nodes of the tree are called leaves and represent final subgroups based on the combination of previous splits.

#decision tree #data mining #data analytics

Machine Learning and the Bias-Variance Trade off

In a linear regression analysis, the accuracy of the model is assessed by the mean square error, or MSE, which is the difference between the model estimated value of a response variable denoted as Y hat, and the observed value of the Y response variable. This difference is computed for each observation, where each observation is denoted by the i subscript, and then the difference is squared. Finally, the squared errors for all the observations are summed, then divided by n, the number of observations, to get the mean square error.

There are two characteristics that we need to consider when selecting an accurate statistical model, their variance and bias.

Variance refers to the amount by which the model parameter estimates would change if we estimated them using a different training data set. If a method has high variance, then small changes in the training data can result in large changes in the parameter estimates. Ideally, the parameter estimates are stable across data sets meaning that the method has low variance.

Bias refers to the error that is introduced by using a statistical model to approximate a real life phenomenon. It is a measure of how far off the model estimated values are from the true values.

What we would like to do is find a statistical model that has both low variance and low bias. The problem is that these two properties are negatively associated. Increases in one result in decreases in the other, hence the bias, variance trade-off.

Generally as model complexity increases, the variance tends to increase, and bias tends to decrease. Simpler models will be more stable across samples, meaning that they will have low variance, but they are also likely to be more biased.

The figure bellow shows a complex model. The observed values are all very close to the estimated regression line. So the error rate is low, meaning that bias is low. However, more complex models attempt to capture every pattern in the training data set even those that occur by chance. The chance patterns are specific to the sample on which the model is fit and are not likely to exist in the test data set. As a result, the model will not fit as well in the test data set which means that the test error rate will be high in the test data set. When a model has a small training mean square error but a large test mean square error, the model is said to be overfitted. When we overfit the training data, the test mean square error will be very large because the patterns that the method found in the training data set simply won't exist in the test data.

On the other hand, the simpler model bellow has a high training mean square error. This simpler model may be considered underfitted. Basically, the simpler model ignores many of the patterns in the training data set, which lead to increased bias. The high mean square error means that it is not taking into account many patterns that are likely to be real, so is also likely to result in a high test error rate. On the other hand, the simpler model is also likely to overlook random sample specific patterns which means that variance will be low.

The figure bellow shows how model complexity impacts training and test error rate. In a really simple model, there's a lot of predication error. Bias is high, but variance is low. In this case the model is underfitted. As model complexity increases, you can see that the prediction error, or bias, decreases in the training sample. Similar to the training sample, prediction error, or bias, decreases, and variance increases as the model becomes more complex. However you can see that there's a point which an overfitted or increasingly complex model will actually increase the test error rate. In this situation a model that is overfitted on the training sample leads to a low error rate in the training sample at the cost of fitting poorly in the test data set. The ideal model complexity is where the test error rate bottoms out. The model at this point will have low bias and low variance, both of which will provide the lowest possible test error rate.

Machine Learning for Data Analysis

Machine learning encompasses a wide range of statistical methods, including linear and logistic regression. These methods can be used to describe associations, search for patterns in a data set, or make predictions about outcomes, also called target or response variables, from a set of explanatory variables. Pattern detection, or describing associations among variables without a specific outcome variable, is referred to as unsupervised learning. When the goal is prediction of the value of a response variable based on a number of predictors, then we are using a supervised learning approach. Unlike the hypothesis testing approach, we typically do not go into a machine learning application with specific hypotheses in mind. Instead we learn from the data. We use a subset of observations from our data set, which we call the training set, to learn about the data. And then test the statistical model we get from the training data set on a different set of observations which we call the test set. We split the data this way because the model that is fit using a machine learning approach is going to fit best in the data set on which it was developed. But it might not perform as well when we try to test it on a different set of observations. If the model doesn't work for a different data set then it's not of much use even if it fits well in the training data set. When we apply our statistical model from the training data set to the test data set, we're interested primarily in the accuracy of our statistical model. Accuracy can be assessed by what is called the test error rate, which is a measure of the extent to which a model correctly classifies observations into categories; or correctly estimates the value of a different variable of interest for each observation in the test data set. The goal then is to identify a model that minimizes the test error rate. That is, a model that accurately reflects true population associations or patterns.

#machine learning #data analytics #data analysis #statistics

Trending Blogs

Recently Viewed Blogs

Data Analytics