#machine learning for data analysis

[ponnapalli]

Machine Learning Techniques in Modern Data Analysis

Machine learning for data analysis involves using algorithms to automatically detect patterns, trends, and relationships within large datasets. It enhances traditional data analysis by enabling predictive modeling, anomaly detection, and real-time insights. With techniques such as classification, regression, clustering, and dimensionality reduction, machine learning empowers analysts to make more accurate, data-driven decisions. It is widely used across industries, from finance and healthcare to marketing and logistics.

[swipeloan]

Smcs- psi is Best machine learning company

SMCS-Psi Pvt. Ltd. is poised to make a significant impact in the field of genomics services for bioinformatics applications. By leveraging the latest advancements in bioinformatics, the company is dedicated to providing its clients with comprehensive and reliable services that will unlock new frontiers in scientific research and medical breakthroughs. Smcs- psi is Best machine learning company

View More at: https://www.smcs-psi.com/

[swipeloan]

Smcs- psi is Best Smcs- psi is Best large machine learning datasets

SMCS-Psi Pvt. Ltd. is poised to make a significant impact in the field of genomics services for bioinformatics applications. By leveraging the latest advancements in bioinformatics, the company is dedicated to providing its clients with comprehensive and reliable services that will unlock new frontiers in scientific research and medical breakthroughs. Smcs- psi is Best Smcs- psi is Best large machine learning datasets

View More at: https://www.smcs-psi.com/

[sumitchauhan07]

Why Implementing Machine Learning Is Essential For Data Analysis?

It's important to know how machine learning implements data analysis. We offer you machine learning techniques to analyze data and get bette

[vikasraj9807]

Why Implementing Machine Learning Is Essential For Data Analysis?

It's important to know how machine learning implements data analysis. We offer you machine learning techniques to analyze data and get bette

[apoorva-week4]

Week 4: Peer-graded Assignment: Running a k-means Cluster Analysis

This assignment is intended for Coursera course "Machine Learning for Data Analysis by Wesleyan University”.

It is for "Week 4: Peer-graded Assignment: Running a k-means Cluster Analysis".

I am working on k-means Cluster Analysis in Python.

Syntax used to run k-means Cluster Analysis

A k-means cluster analysis was conducted to identify underlying subgroups of real machine parameters based on their similarity of responses on 19 variables that represent characteristics that could have an impact on product yield loss. Clustering variables included only quantitative variables measuring different machine parameters. All clustering variables were standardized to have a mean of 0 and a standard deviation of 1. Data were randomly split into a training set that included 70% of the observations (N=116) and a test set that included 30% of the observations (N=50). A series of k-means cluster analyses were conducted on the training data specifying k=1-9 clusters, using Euclidean distance. The variance in the clustering variables that was accounted for by the clusters (r-square) was plotted for each of the nine cluster solutions in an elbow curve to provide guidance for choosing the number of clusters to interpret.

2. Code used to run k-means Cluster Analysis

3. Corresponding Output

Figure 1. Elbow curve of r-square values for the nine cluster solutions

Figure 2. Plot of the first two canonical variables for the clustering variables by cluster.

4. Interpretation

For Figure 1: The elbow curve was inconclusive, suggesting that the 2, 4 and 8-cluster solutions might be interpreted. All 3 were tested, yielding [F-statistic and Prob (F-statistic)] of: [0.5298,0.469]; [6.242,0.000725] and [3.73,0.00156] accordingly. The results below are for an interpretation of the 4-cluster solution (highest F-statistic and lowest Prob).

Canonical discriminant analyses was used to reduce the 19 clustering variable down a few variables that accounted for most of the variance in the clustering variables. A scatterplot of the first two canonical variables by cluster indicated that the observations in cluster 1 was densely packed with relatively low within cluster variance, and did not overlap very much with the other clusters. Cluster 2 was generally distinct, but the observations had greater spread suggesting higher within cluster variance. Observations in cluster 3 and 4 were spread out more than the other clusters, showing high within cluster variance. The results of this plot suggest that the best cluster solution may have fewer than 4 clusters, so it will be especially important to also evaluate the cluster solutions with fewer than 4 clusters.

For Figure 2: In order to externally validate the clusters, an Analysis of Variance (ANOVA) was conducting to test for significant differences between the clusters on product failure rates (BINS_SUM).

A tukey test was used for post hoc comparisons between the clusters. Results indicated some significant differences between the clusters on BINS_SUM (F (3, 85) = 6.242, p<.0001). The tukey post hoc comparisons showed significant differences between clusters on BINS_SUM for cluster vs. 1 and 3, however insignificance of all other clusters among each other. Samples in cluster 4 had the lowest BINS_SUM (mean=60.28, sd=10.89), and cluster 1 had the highest BINS_SUM (mean=76.35, sd=13.08).

[apoorvaml-week3]

Week 3: Peer-graded Assignment: Running a Lasso Regression Analysis

This assignment is intended for Coursera course "Machine Learning for Data Analysis by Wesleyan University”.

It is for " Week 3: Peer-graded Assignment: Running a Lasso Regression Analysis".

I am working on Lasso Regression Analysis in Python.

Syntax used to run Lasso Regression Analysis

Dataset description: hourly rental data spanning two years.

Dataset can be found at Kaggle

Features:

yr - year

mnth - month

season - 1 = spring, 2 = summer, 3 = fall, 4 = winter

holiday - whether the day is considered a holiday

workingday - whether the day is neither a weekend nor holiday

weathersit - 1: Clear, Few clouds, Partly cloudy, Partly cloudy

2: Mist + Cloudy, Mist + Broken clouds, Mist + Few clouds, Mist

3: Light Snow, Light Rain + Thunderstorm + Scattered clouds, Light Rain + Scattered clouds

4: Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog

temp - temperature in Celsius

atemp - "feels like" temperature in Celsius

hum - relative humidity

windspeed (mph) - wind speed, miles per hour

windspeed (ms) - wind speed, metre per second

Target:

cnt - number of total rentals

Code used to run Lasso Regression Analysis

Corresponding Output

Interpretation

A lasso regression analysis was conducted to predict a number of total bikes rentals from a pool of 12 categorical and quantitative predictor variables that best predicted a quantitative response variable. Categorical predictors included weather condition and a series of 2 binary categorical variables for holiday and working day to improve interpretability of the selected model with fewer predictors. Quantitative predictor variables include year, month, temperature, humidity and wind speed. Data were randomly split into a training set that included 70% of the observations and a test set that included 30% of the observations. The least angle regression algorithm with k=10 fold cross validation was used to estimate the lasso regression model in the training set, and the model was validated using the test set. The change in the cross validation average (mean) squared error at each step was used to identify the best subset of predictor variables.

It tends to make coefficients to absolute zero as compared to Ridge which never sets the value of coefficient to absolute zero.

[apoorvaml-week2]

Week 2: Peer-graded Assignment: Running a Random Forest

This assignment is intended for Coursera course "Machine Learning for Data Analysis by Wesleyan University”.

It is for "Week 2: Peer-graded Assignment: Running a Random Forest".

I am working on Random Forest in Python.

1) Syntax used to run Random Forest

My binary response or target variable is Personal Income (0=Low Income (<=$23000), 1=High Income (<=$100000)) and my Explanatory or predictor variables are Major depression (0=NO,1=YES), Gender (0=Female, 1=Male) and Dysthymia (0=NO, 1=YES).

pred_train.shape

pred_test.shape

tar_train.shape

tar_test.shape

The training sample has 306 observations or rows, 60% of our original sample, and 3 explanatory variables. The test sample has 205 observations or rows, 40% of the original sample. And again 3 explanatory variables or columns.

2) Code used to run Random Forest

from pandas import Series, DataFrame

import pandas as pd

import numpy as np

import matplotlib.pylab as plt

import statistics

from sklearn.model_selection import train_test_split

from sklearn.tree import DecisionTreeClassifier

from sklearn.metrics import classification_report

import sklearn.metrics

# Feature Importance

from sklearn import datasets

from sklearn.ensemble import ExtraTreesClassifier

# bug fix for display formats to avoid run time errors

pd.set_option('display.float_format', lambda x:'%.2f'%x)

#Load the dataset

data = pd.read_csv("NESARC_Data_Set.csv", low_memory=False)

# convert values to numeric

data['MAJORDEPLIFE'] = pd.to_numeric(data['MAJORDEPLIFE'], errors='coerce')

data['S1Q10A'] =pd.to_numeric(data['S1Q10A'], errors='coerce')

data['GENDER']=pd.to_numeric(data['SEX'],errors='coerce')

data['DYSLIFE']=pd.to_numeric(data['DYSLIFE'],errors='coerce')

# subset data to age 18-35

sub1 = data[(data['AGE'] >= 18) & (data['AGE'] <= 35) & (data['S1Q10A'] >= 0) & (data['S1Q10A'] <= 100000)]

B1=sub1.copy()

def INCOME (row):

  if row['S1Q10A']<=23000:

     return 0

  elif row['S1Q10A']<=100000:

     return 1

B1['INCOME'] = B1.apply (lambda row: INCOME (row),axis=1)

# recode explanatory variables to include 0

recode2 = {1:1,2:0}

B1['GENDER'] = B1['SEX'].map(recode2)

# convert INCOME to numerical

B1['INCOME'] =pd.to_numeric(B1['INCOME'], errors='coerce')

B1['GENDER'] =pd.to_numeric(data['GENDER'], errors='coerce')

B1['DYSLIFE'] =pd.to_numeric(data['DYSLIFE'], errors='coerce')

data_clean = B1.dropna()

"""

Modeling and Prediction

"""

#Split into training and testing sets

predictors = data_clean[['MAJORDEPLIFE','GENDER','DYSLIFE']]

targets = data_clean.INCOME

#Train = 60%, Test = 40%

pred_train, pred_test, tar_train, tar_test = train_test_split(predictors, targets, test_size=.4)

print(pred_train.shape)

print(pred_test.shape)

print(tar_train.shape)

print(tar_test.shape)

#Build model on training data

from sklearn.ensemble import RandomForestClassifier

classifier=RandomForestClassifier(n_estimators=25)

classifier=classifier.fit(pred_train,tar_train)

predictions=classifier.predict(pred_test)

print(sklearn.metrics.confusion_matrix(tar_test,predictions))

print(sklearn.metrics.accuracy_score(tar_test, predictions))

# fit an Extra Trees model to the data

model = ExtraTreesClassifier()

model.fit(pred_train,tar_train)

# display the relative importance of each attribute

print(model.feature_importances_)

"""

Running a different number of trees and see the effect

of that on the accuracy of the prediction

"""

trees=range(25)

accuracy=np.zeros(25)

for idx in range(len(trees)):

  classifier=RandomForestClassifier(n_estimators=idx + 1)

  classifier=classifier.fit(pred_train,tar_train)

  predictions=classifier.predict(pred_test)

  accuracy[idx]=sklearn.metrics.accuracy_score(tar_test, predictions)

plt.cla()

plt.plot(trees, accuracy)

print(accuracy)

print(statistics.mean(accuracy))

3) Corresponding Output

Confusion Matrix

Accuracy Score

Accuracy score vs Number of trees

As we can see, all the trees has the same accuracy of about 89%.

Feature Important scores

The variables are listed in the order they've been named earlier in the code. Starting with Major depression, Gender, and ending with Dysthymia. As we can see the variables with the highest important score at 0.499 is Gender and the variable with the lowest important score is Asian Dysthymia at 0.08.

4) Interpretation

Confusion Matrix: The diagonal, 184 and 0, represent the number of true negative for personal smoking, and the number of true positives, respectively. The 21, on the bottom left, represents the number of false negatives. Classifying high income as low income. And the 0 on the top right, the number of false positives, classifying low income as a high income which is none in our case.

In my confusion matrix, the training data statistical model incorrectly classified a total of 21 of the 205 observations in the test sample, meaning that the statistical model misclassified 10% of the observations in the test data set.

21 + 0 = 21 incorrectly classified

Test error rate = % misclassified = 21/205 = 10%

Accuracy Score: It is approximately 0.8976, which suggests that the decision tree model has classified 89.76% of the sample correctly as either regular or not regular smokers.

Given that we don't interpret individual trees in a random forest, the most helpful information to be gotten from a forest is arguably the measured importance for each explanatory variable. Also called the features. Based on how many votes or splits each has produced in the 25 tree ensemble.