ML Researcher

[altaireon]

K-Means Cluster Analysis

Cluster analysis is an unsupervised machine learning method that partitions the observations in a data set into a smaller set of clusters where each observation belongs to only one cluster. The goal of cluster analysis is to group, or cluster, observations into subsets based on their similarity of responses on multiple variables. Clustering variables should be primarily quantitative variables, but binary variables may also be included.

Data were randomly split into a training set that included 70% of the observations (N=3201) and a test set that included 30% of the observations (N=1701). Categorical Variables are kept as it is and standard scaling performed on quantitative variables. Clustering is performed on a range from 1 to 10 number of clusters and their accuracy measured.

Code for KMeans:-

from google.colab import drive,files from sklearn.decomposition import PCA import pandas as pd import numpy as np import matplotlib.pyplot as plt import os import pydotplus import io from IPython.display import Image from sklearn.cluster import KMeans from sklearn.metrics import confusion_matrix from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split from sklearn.preprocessing import scale drive.mount('/content/gdrive') corpusName ="_d21b2085472fd467f689f21cd421b13b_tree_addhealth.csv" folderPath = "/content/gdrive/My Drive/MachineLearningForDataAnalysis" corpus = os.path.join(folderPath, corpusName) df_train = pd.read_csv(corpus) print("Dataset shape = ") print(df_train.shape)

Output:  Dataset shape = (6504, 25)   

# print(df_train.head(10)) # print(df_train.info()) df_train.dropna(inplace=True) non_categorical_columns = [] for col in df_train.drop("TREG1",axis=1).columns:  if df_train[col].nunique() > 10:    df_train[col]=scale(df_train[col]) x_train,x_test,y_train,y_test = train_test_split(df_train.drop("TREG1",axis=1),df_train.TREG1,train_size=0.8,random_state=42) from scipy.spatial.distance import cdist clusters=range(1,10) accuracy=np.zeros(len(clusters)) meandist=[] for k in clusters:    model=KMeans(n_clusters=k,random_state=42)    model.fit(x_train,y_train)    clusassign=model.predict(x_test)    meandist.append(sum(np.min(cdist(x_train, model.cluster_centers_, 'euclidean'), axis=1))    / x_train.shape[0])    accuracy[k-1]=accuracy_score(y_test, clusassign) plt.xlabel("Number of Estimators") plt.ylabel("Accuracy") plt.plot(clusters, accuracy) plt.savefig(folderPath+'/KMeans.png') plt.show() model2=KMeans(n_clusters=2,random_state=42) model2.fit(x_train,y_train) print("Accuracy = {}".format(accuracy_score(y_test,model2.predict(x_test))))

Output: Accuracy = 0.7289617486338797   

plt.plot(clusters, meandist) plt.xlabel('Number of clusters') plt.ylabel('Average distance') plt.title('Selecting k with the Elbow Method') plt.savefig(folderPath+'/KMeans_Elbow.png') plt.show()

pca_2 = PCA(2) plot_columns = pca_2.fit_transform(x_train) plt.scatter(x=plot_columns[:,0], y=plot_columns[:,1], c=model2.labels_,) plt.xlabel('Canonical variable 1') plt.ylabel('Canonical variable 2') plt.title('Scatterplot of Canonical Variables for 2 Clusters') plt.savefig(folderPath+'/KMeans_groups.png') plt.show()

Elbow method has been used to determine the number of clusters best for the model. As per the first image the elbow can be observed on k=2. Model gives the accuracy of 72% approx. Principal Component Analysis(PCA) was used to scale down 24 dimensional input to 2 dimensional space with two classes 0 and 1 for visualization.

[altaireon]

Lasso Regression

A lasso regression analysis was conducted to identify a subset of variables from a pool of 24 categorical and quantitative predictor variables that best predicted a quantitative response variable measuring school connectedness in adolescents. Categorical predictors included gender and a series of 5 binary categorical variables for race and ethnicity (Hispanic, White, Black, Native American and Asian) to improve interpretability of the selected model with fewer predictors. Binary substance use variables were measured with individual questions about whether the adolescent had ever used alcohol, marijuana, cocaine or inhalants. Additional categorical variables included the availability of cigarettes in the home, whether or not either parent was on public assistance and any experience with being expelled from school. Quantitative predictor variables include age, alcohol problems, and a measure of deviance that included such behaviors as vandalism, other property damage, lying, stealing, running away, driving without permission, selling drugs, and skipping school. Another scale for violence, one for depression, and others measuring self-esteem, parental presence, parental activities, family connectedness and grade point average were also included. All predictor variables were standardized to have a mean of zero and a standard deviation of one.

Data were randomly split into a training set that included 70% of the observations (N=3201) and a test set that included 30% of the observations (N=1701). The least angle regression algorithm with k=10 fold cross validation was used to estimate the lasso regression model in the training set, and the model was validated using the test set. The change in the cross validation average (mean) squared error at each step was used to identify the best subset of predictor variables.

Code for Lasso Regression:

from google.colab import drive,files import pandas as pd import numpy as np import matplotlib.pyplot as plt import os import pydotplus import io from IPython.display import Image from sklearn.metrics import confusion_matrix from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split from sklearn.linear_model import LassoLarsCV from sklearn import preprocessing drive.mount('/content/gdrive') corpusName ="_d21b2085472fd467f689f21cd421b13b_tree_addhealth.csv" folderPath = "/content/gdrive/My Drive/MachineLearningForDataAnalysis" corpus = os.path.join(folderPath, corpusName) df_train = pd.read_csv(corpus) print("Dataset shape = ") print(df_train.shape)

Output:  Dataset shape = (6504, 25)

print(df_train.columns)

Output:  Index(['BIO_SEX', 'HISPANIC', 'WHITE', 'BLACK', 'NAMERICAN', 'ASIAN', 'age',       'TREG1', 'ALCEVR1', 'ALCPROBS1', 'marever1', 'cocever1', 'inhever1',       'cigavail', 'DEP1', 'ESTEEM1', 'VIOL1', 'PASSIST', 'DEVIANT1',       'SCHCONN1', 'GPA1', 'EXPEL1', 'FAMCONCT', 'PARACTV', 'PARPRES'],      dtype='object') 

# print(df_train.head(10)) # print(df_train.info()) df_train.dropna(inplace=True) for col in df_train.columns:  df_train[col]=preprocessing.scale(df_train[col].astype('float64'))

x_train, x_test, y_train, y_test = train_test_split(df_train.drop("TREG1",axis=1), df_train.TREG1,                                                              test_size=.3, random_state=123)

model=LassoLarsCV(cv=10, precompute=False).fit(x_train,y_train)

dict(zip(x_train.columns, model.coef_))

m_log_alphas = -np.log10(model.alphas_) ax = plt.gca() plt.plot(m_log_alphas, model.coef_path_.T) plt.axvline(-np.log10(model.alpha_), linestyle='--', color='k',            label='alpha CV') plt.ylabel('Regression Coefficients') plt.xlabel('-log(alpha)') plt.title('Regression Coefficients Progression for Lasso Paths') plt.savefig(folderPath+"/Reg_Coeff.png")

m_log_alphascv = -np.log10(model.cv_alphas_) plt.figure() plt.plot(m_log_alphascv, model.mse_path_, ':') plt.plot(m_log_alphascv, model.mse_path_.mean(axis=-1), 'k',         label='Average across the folds', linewidth=2) plt.axvline(-np.log10(model.alpha_), linestyle='--', color='k',            label='alpha CV') plt.legend() plt.xlabel('-log(alpha)') plt.ylabel('Mean squared error') plt.title('Mean squared error on each fold') plt.savefig(folderPath+"/Mean_Sq_Err.png")

# MSE from training and test data from sklearn.metrics import mean_squared_error train_error = mean_squared_error(y_train, model.predict(x_train)) test_error = mean_squared_error(y_test, model.predict(x_test)) print ('training data MSE') print(train_error)

Output:  training data MSE 0.7103773580520552 

print ('test data MSE') print(test_error)

Output:  test data MSE 0.721859543809532

# R-square from training and test data rsquared_train=model.score(x_train,y_train) rsquared_test=model.score(x_test,y_test) print ('training data R-square') print(rsquared_train)

Output:  training data R-square 0.2745661967781555

print ('test data R-square') print(rsquared_test)

Output:  test data R-square 0.31071521636196964

print("Number of feature Used = {}".format(np.sum(model.coef_!=0)))

Output:  Number of feature Used = 22

print("Coeff = {}".format(out_dict))

Output: Coeff = {'BIO_SEX': 0.025628290838580477, 'HISPANIC': -0.030002808614276108, 'WHITE': 0.09522057486885759, 'BLACK': -0.051547164889418545, 'NAMERICAN': -0.006659984878718197, 'ASIAN': -0.007331382053598405, 'age': 0.03877631368977216, 'ALCEVR1': 0.07576516459955492, 'ALCPROBS1': 0.05009414437125331, 'marever1': 0.24568705171878635, 'cocever1': 0.03243612799667629, 'inhever1': 0.015269660628021415, 'cigavail': 0.08109571776580017, 'DEP1': 0.045763557239754234, 'ESTEEM1': 0.007312169922740399, 'VIOL1': 0.0, 'PASSIST': -0.012550380318644493, 'DEVIANT1': 0.06974156846904282, 'SCHCONN1': -0.038455747385471306, 'GPA1': -0.1040514070440718, 'EXPEL1': 0.006879433295868557, 'FAMCONCT': 0.0, 'PARACTV': -0.038665425414240076, 'PARPRES': -0.01951092512912656}

As we can see VIOL1 and  FAMCONCT are 0 they can be assumed to be related to some other variable.

Of the 24 predictor variables, 22 were retained in the selected model.

[altaireon]

RandomForest

Random forest analysis was performed to evaluate the importance of a series of explanatory variables in predicting a binary, categorical response variable. The following explanatory variables along with their importance mostly contributed to a classification tree model evaluating smoking experimentation (my response variable):- Bio_sex(.016), age(.083), White(.024), Black(.020), Alcohol use(.036), marijuana use(.1), cocaine use(.016), inhalant use(.012), availability of cigarettes in the home(.023), alcohol problems(.050), deviance(.076), whether or not either parent was on public assistance(.011), violence(0.044), self-esteem(.063), depression(.075), parental presence(.040), parental activities(.060), family connectedness(.069), school connectedness(.078) and grade point average(.078). 

The total model classified 84% of the sample correctly, 87% of experimenters (sensitivity) and 61% of nonsmokers (specificity).

——————————————Code———————————-

Imports & Setup:

from google.colab import drive,files import pandas as pd import numpy as np import matplotlib.pyplot as plt import os import pydotplus import io from IPython.display import Image from sklearn import ensemble from sklearn.metrics import confusion_matrix from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split drive.mount('/content/gdrive') corpusName ="_d21b2085472fd467f689f21cd421b13b_tree_addhealth.csv" folderPath = "/content/gdrive/My Drive/MachineLearningForDataAnalysis" corpus = os.path.join(folderPath, corpusName) df_train = pd.read_csv(corpus) print("Dataset shape = ") print(df_train.shape)

Output:  Dataset shape = (6504, 25)

df_train.dropna(inplace=True) x_train,x_test = train_test_split(df_train,train_size=0.65,random_state=42) y_train,y_test = x_train.TREG1,x_test.TREG1 model=ensemble.RandomForestClassifier(n_estimators=25,max_features='sqrt',random_state=42) model.fit(x_train.drop("TREG1",axis=1),y_train) y_pred=model.predict(x_test.drop("TREG1",axis=1)) print("Confusion matrix = ") print(confusion_matrix(y_test,y_pred))

Output:  Confusion matrix = [[1254   62] [ 189   97]]

print("Accuracy = ") print(accuracy_score(y_test,y_pred))

Output:  Accuracy = 0.8433208489388264

trees=range(30) accuracy=np.zeros(30)

for idx in trees:  classifier=ensemble.RandomForestClassifier(n_estimators=idx + 1)  classifier=classifier.fit(x_train.drop("TREG1",axis=1),y_train)  predictions=classifier.predict(x_test.drop("TREG1",axis=1))  accuracy[idx]=accuracy_score(y_test, predictions)

plt.cla() plt.xlabel("Number of Estimators") plt.ylabel("Accuracy") plt.plot(trees, accuracy) plt.savefig(folderPath+'/RandomForest.png') dict_ = dict() for i,col in enumerate(x_train.drop("TREG1",axis=1).columns):  dict_[col]=model.feature_importances_[i] print("Feature Importance : ") print(dict_)

Output: Feature Importance : {'BIO_SEX': 0.01601269662881169, 'HISPANIC': 0.007692345837090267, 'WHITE': 0.02408407358221776, 'BLACK': 0.019755648335386778, 'NAMERICAN': 0.004265882556590864, 'ASIAN': 0.003965075691786905, 'age': 0.083141953002379, 'ALCEVR1': 0.036341236077591405, 'ALCPROBS1': 0.05053293071513981, 'marever1': 0.0987686322473449, 'cocever1': 0.01576007604679264, 'inhever1': 0.012339850211915809, 'cigavail': 0.023161593771540735, 'DEP1': 0.07536426665259273, 'ESTEEM1': 0.06386308089292135, 'VIOL1': 0.044516749158471175, 'PASSIST': 0.010875447978273207, 'DEVIANT1': 0.07555092297679514, 'SCHCONN1': 0.07854454267326157, 'GPA1': 0.07787836461238645, 'EXPEL1': 0.007537762692643415, 'FAMCONCT': 0.06900183297339084, 'PARACTV': 0.06065052154076747, 'PARPRES': 0.040394513143908066}

[altaireon]

DecisionTree

Decision tree analysis was performed to test nonlinear relationships among a series of explanatory variables and a binary, categorical response variable. All possible separations (categorical) or cut points (quantitative) are tested. For the present analyses, “the entropy- goodness of split” criterion was used to grow the tree and 5% min_samples_split was used to decide whether to split further branch or not.  

The following explanatory variables along with their importance mostly contributed to a classification tree model evaluating smoking experimentation (my response variable):- age(.029), White(.077), Black(.013), Alcohol use(.115), marijuana use(.500), availability of cigarettes in the home(.012), alcohol problems(.010), deviance(.043),  depression(.017), parental presence(.014), parental activities(.031), family connectedness(.022), school connectedness(.047) and grade point average(.052).

The most important feature for marijuana use with 50% contribution. The total model classified 85% of the sample correctly, 90% of experimenters (sensitivity) and 55% of nonsmokers (specificity). 

Note:- The Code given below can be run directly in Google Colab. Test data size is set to 915 and training data 3660.

------------------------------------------Code----------------------------------

Imports:-

from google.colab import drive,files import pandas as pd import matplotlib.pyplot as plt import os import pydotplus import io from IPython.display import Image from sklearn import tree from sklearn.metrics import confusion_matrix from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split drive.mount('/content/gdrive') corpusName ="_d21b2085472fd467f689f21cd421b13b_tree_addhealth.csv" folderPath = "/content/gdrive/My Drive/MachineLearningForDataAnalysis" corpus = os.path.join(folderPath, corpusName) df_train = pd.read_csv(corpus) print("Dataset shape = ") print(df_train.shape) 

Output:  Dataset shape =  (6504, 25)

df_train.dropna(inplace=True) x_train,x_test = train_test_split(df_train,train_size=0.8,random_state=42) y_train,y_test = x_train.TREG1,x_test.TREG1 model=tree.DecisionTreeClassifier(random_state=42,criterion="entropy",min_samples_split=(int)(x_train.shape[0]*0.05),) model.fit(x_train.drop("TREG1",axis=1),y_train) y_pred=model.predict(x_test.drop("TREG1",axis=1)) print("Confusion matrix = ") print(confusion_matrix(y_test,y_pred))

Output:  Confusion matrix = [[714  56] [ 77  68]] 

print("Accuracy = ") print(accuracy_score(y_test,y_pred))

Output:  Accuracy = 0.8546448087431694 

out = io.StringIO() tree.export_graphviz(model, out_file=out) graph=pydotplus.graph_from_dot_data(out.getvalue()) img = Image(graph.create_png()) filePath = os.path.join(folderPath, "DecisionTree.png") with open(filePath, "wb") as png:    png.write(img.data)    png.close()

dict_ = dict() for i,col in enumerate(x_train.drop("TREG1",axis=1).columns):  dict_[col]=model.feature_importances_[i] print(dict_)

Output:  {'BIO_SEX': 0.0, 'HISPANIC': 0.0, 'WHITE': 0.07689800188375509, 'BLACK': 0.013329278083582821, 'NAMERICAN': 0.0, 'ASIAN': 0.0, 'age': 0.028676800427653523, 'ALCEVR1': 0.11531998044836722, 'ALCPROBS1': 0.010359831147465094, 'marever1': 0.5003916650012903, 'cocever1': 0.0, 'inhever1': 0.0, 'cigavail': 0.011564724963814329, 'DEP1': 0.017293445565594052, 'ESTEEM1': 0.003972967897391851, 'VIOL1': 0.008243966063649775, 'PASSIST': 0.005773466454035425, 'DEVIANT1': 0.042695886446359736, 'SCHCONN1': 0.046592301917146584, 'GPA1': 0.05240826234607083, 'EXPEL1': 0.0, 'FAMCONCT': 0.022239864016842078, 'PARACTV': 0.030525880193369718, 'PARPRES': 0.013713677143611642}