Sem título

[eduardotleite]

Week 4 - Data Analysis Tools

Dataset Description:

The dataset contains variables about boughts in some regions, to determin if some age group buy more in store or outside

Qualitative variables: In.Store, region

Quantitative variables: Age, Amount, items

Source: https://www.kaggle.com/datasets/thedevastator/demographical-shopping-purchases-data

The Code

import pandas as pd import statsmodels.formula.api as smf import seaborn import matplotlib.pyplot as plt

def ageGroup(row): if row["age"]>60: return "Senior " elif row["age"]>30: return "Adult" elif row["age"]>=18: return "Young" elif row["age"]>9: return "Teenage" else: return "Child"

data = pd.read_csv('Demographic_Data_Orig.csv', low_memory=False)

data['age'] = data['age'].apply(pd.to_numeric, errors='coerce') data['items'] = data['items'].apply(pd.to_numeric, errors='coerce') data['amount'] = data['amount'].apply(pd.to_numeric, errors='coerce')

data["AgeGroup"]=data.apply(ageGroup, axis=1)

data = data[["region","items","in.store","amount", "AgeGroup"]].dropna()

model1 = smf.ols(formula='amount ~ C(region)', data=data).fit() print (model1.summary())

recode1 = {1: 'In', 0: 'Out'} data['in.store']= data['in.store'].map(recode1)

sub1 = data[['amount', 'in.store']].dropna()

print ("means for Amount by Region") m1= sub1.groupby('in.store').mean() print (m1)

print ("standard deviation for mean Amount by In vs Out Store") st1= sub1.groupby('in.store').std() print (st1)

bivariate bar graph

seaborn.barplot(x="in.store", y="amount", data=data, ci=None) plt.xlabel('In Store') plt.ylabel('Amount')

sub2=data[(data['AgeGroup']=='Young')] sub3=data[(data['AgeGroup']=='Adult')]

print ('association between in-store and out-of-store purchases - Young') model2 = smf.ols(formula='amount ~ C(region)', data=sub2).fit() print (model2.summary())

print ('association between in-store and out-of-store purchases - Adult') model3 = smf.ols(formula='amount ~ C(region)', data=sub3).fit() print (model3.summary())

print ("means - Young") m3= sub2.groupby('in.store').mean() print (m3) print ("Means - Adult") m4 = sub3.groupby('in.store').mean() print (m4)

General Result

Young Group

Adult Group

Mean

[eduardotleite]

Week 3 - Data Analysis Tools - Correlation

This is the third task (Week 3) of the course Data Analysis Tools from the Wesleyan University at the Coursera Platform. The challenge is to generate a Correlation Coefficient. This type of coefficient is used when you have two quantitative variables. For this assignment, as asked, I will use the correlation between Life Expectancy, CO2 emissions and Breast cancer. The dataset used is gap Minder (https://www.gapminder.org/.)

The Code

import pandas as pd import numpy import seaborn import scipy import matplotlib.pyplot as plt

df = pd.read_csv('gapminder.csv', low_memory=False)

df['co2emissions'] = df['co2emissions'].apply(pd.to_numeric, errors='coerce') df['breastcancerper100th'] = df['breastcancerper100th'].apply(pd.to_numeric, errors='coerce') df['lifeexpectancy'] = df['lifeexpectancy'].apply(pd.to_numeric, errors='coerce')

clear values Outliers

df=df[(df['lifeexpectancy']>=1) & (df['lifeexpectancy']<=100) & (df['co2emissions']<1000000000) ]

print(max(df['co2emissions'])) print(min(df['co2emissions']))

df['lifeexpectancy']=df['lifeexpectancy'].replace(' ', numpy.nan) df['co2emissions']=df['co2emissions'].replace(' ', numpy.nan) df['breastcancerper100th']=df['breastcancerper100th'].replace(' ', numpy.nan)

Adjust each parameters to compare

df.breastcancerper100th *=10000 df.co2emissions /=1000

df1=df.dropna() df2=df1

graf1= seaborn.regplot(x="breastcancerper100th", y="lifeexpectancy", fit_reg=True, data=df1)

plt.xlabel('Breast cancer') plt.ylabel('Life Expectancy') plt.title('Scatterplot for the Association Between Life Expectancy and Breast cancer')

graf2 = seaborn.regplot(x="co2emissions", y="lifeexpectancy", fit_reg=True, data=df1) plt.xlabel('CO2 emissions') plt.ylabel('Life Expectancy') plt.title('Scatterplot for the Association Between Life Expectancy and CO2 emissions')

data_clean=df.dropna()

print ('association between Life Expectancy and co2emissions') print (scipy.stats.pearsonr(data_clean['co2emissions'], data_clean['lifeexpectancy']))

print ('association between Life Expectancy and Breast cancer') print (scipy.stats.pearsonr(data_clean['breastcancerper100th'], data_clean['lifeexpectancy']))

[eduardotleite]

Data Management and Visualization - Week 2 - Chi

The Dataset contains variables of Watch television and age group. To examine the relationship of watch television with each age group people.

Python Code

Importing required libraries

import pandas as pd import numpy import scipy.stats import matplotlib.pyplot as plt import seaborn as sns

def faixa_etaria(row): if row["PA2"]>=65: return "Idoso" elif row["PA2"]>=41: return "Adulto" elif row["PA2"]>=18: return "Adulto Jovem" elif row["PA2"]>=11: return "Adolescencia" else: return "Infância"

Getting data in

data = pd.read_csv('addhealth_pds.csv', low_memory=False)

data['PA2'] = pd.to_numeric(data['PA2'], errors='coerce') data['H1DA3'] = pd.to_numeric(data['H1DA3'], errors='coerce')

sub1=data[(data['PA2']>=1) & (data['PA2']<=99) & (data['H1DA3']<7)]

sub1["faixa_etaria"]=sub1.apply(faixa_etaria, axis=1)

sub1 = sub1.dropna()

sub2 = sub1.copy()

recode1 = {1: 10, 2: 20, 3: 30, 0: 1} sub2['USFREQMO']= sub1['H1DA3'].map(recode1)

contingency table of observed counts

ct1=pd.crosstab(sub2['USFREQMO'], sub2['faixa_etaria']) print (ct1)

column percentages

colsum=ct1.sum(axis=0) colpct=ct1/colsum print(colpct)

chi-square

print ('chi-square value, p value, expected counts') cs1= scipy.stats.chi2_contingency(ct1) print (cs1)

set variable types

sub2["USFREQMO"] = sub2["USFREQMO"].astype('category')

new code for setting variables to numeric:

sub2['USFREQMO'] = pd.to_numeric(sub2['USFREQMO'], errors='coerce')

sns.barplot(data=sub2, x="USFREQMO", y="faixa_etaria",errorbar=None)

plt.xlabel('Days per month') plt.ylabel('Age Group')

Python Output

faixa_etaria Adulto Adulto Jovem Idoso USFREQMO 1 97 68 3 10 475 387 3 20 653 578 9 30 1735 1574 28 faixa_etaria Adulto Adulto Jovem Idoso USFREQMO 1 0.032770 0.026084 0.069767 10 0.160473 0.148446 0.069767 20 0.220608 0.221711 0.209302 30 0.586149 0.603759 0.651163 chi-square value, p value, expected counts Chi2ContingencyResult(statistic=8.655741129293999, pvalue=0.19388454866325733, dof=6, expected_freq=array([[8.86417112e+01, 7.80705882e+01, 1.28770053e+00], [4.56399287e+02, 4.01970588e+02, 6.63012478e+00], [6.54260250e+02, 5.76235294e+02, 9.50445633e+00], [1.76069875e+03, 1.55072353e+03, 2.55777184e+01]]))

Interpretation: As p-value (0.19) is greater than 0.05, there exists no relationship between the time wacthing television and age group people. The bar diagram also shows that have no signaficative differences in the graph which further suggests no relationship between the two variables. This way, the Post-hoc test is not necessary for examining the relationship as there is no significant relationship.

[eduardotleite]

ANOVA Analysis

This is the task of Week 1 of the course Data Analysis Tools at the Coursera Plataform. The challenge is to execute an Analysis of Variance using the ANOVA Statistical Test. This type of analysis assesses whether the means of two or more groups are statistically different from each other. Is used whenever you want to compare the means (quantitative variables) of groups (categorical variables). The null hypothesis is that there is no difference in the mean of the quantitative variable across groups (categorical variable), while the alternative is that there is a difference.

DataSet Used – Gap Minder Gapminder identifies systematic misconceptions about important global trends and proportions and uses reliable data to develop easy to understand teaching materials to rid people of their misconceptions. Gapminder is an independent Swedish foundation with no political, religious, or economic affiliations. should visit it: https://www.gapminder.org/.

The dataset used has 16 variables and 213 rows. I choosed to analyze income per person (incomeperperson) and life expectancy (lifeexpectancy).

And how is the Question?

Is the life expectancy different among four categories of income per person (A,B,C,D,E)?

Since the income per person is a quantitative variable, I transformed it into a categorical variable, using parameters sugested by IBGE to classify the social class of according of income. For the parameters, I analyzed the boxplot posted below.

the data in image is in portuguese, because the IBGE is an Brazilian institute.

The Code

I used the Anaconda to code in Python for this task. The code is posted below.

import numpy import pandas as pd import statsmodels.api as sm import statsmodels.formula.api as smf import statsmodels.stats.multicomp as multi import matplotlib.pyplot as plt import seaborn as sns import researchpy as rp import pycountry_convert as pc

df = pd.read_csv('gapminder.csv') df = df[['lifeexpectancy', 'incomeperperson']]

df['lifeexpectancy'] = df['lifeexpectancy'].apply(pd.to_numeric, errors='coerce') df['incomeperperson'] = df['incomeperperson'].apply(pd.to_numeric, errors='coerce')

def income_categories(row): if row["incomeperperson"]>15000: return "A" elif row["incomeperperson"]>5000: return "B" elif row["incomeperperson"]>3000: return "C" elif row["incomeperperson"]>1000: return "D" else: return "E"

df=df[(df['lifeexpectancy']>=1) & (df['lifeexpectancy']<=120) & (df['incomeperperson'] > 0) ]

df["Income_category"]=df.apply(income_categories, axis=1)

df = df[["Income_category","incomeperperson","lifeexpectancy"]].dropna()

df["Income_category"]=df.apply(income_categories, axis=1)

print (rp.summary_cont(df['lifeexpectancy']))

fig1, ax1 = plt.subplots() df_new = [df[df['Income_category']=='A']['lifeexpectancy'], df[df['Income_category']=='B']['lifeexpectancy'], df[df['Income_category']=='C']['lifeexpectancy'], df[df['Income_category']=='D']['lifeexpectancy'], df[df['Income_category']=='E']['lifeexpectancy']] ax1.set_title('life expectancy') ax1.boxplot(df_new) plt.show()

results = smf.ols('lifeexpectancy ~ C(Income_category)', data=df).fit() print (results.summary())

print ("Tukey") mc1 = multi.MultiComparison(df['lifeexpectancy'], df['Income_category']) print (mc1) res1 = mc1.tukeyhsd() print (res1.summary())

print ('means for for life expectancy by Income') m1= df.groupby('Income_category').mean() print (m1)

print ('Results') print ('standard deviations for life expectancy by Income') sd1 = df.groupby('Income_category').std() print (sd1)

Results – ANOVA Analysis

Aiming to answer the question of the task, I ran a test ANOVA. As shown below, from the 176 rows, 171 were used for the test, i have used a filter to remove some wrong values, as non numeric, negative, etc, reducing the rows of the original dataset

The ANOVA analysis shows a graph for each category (above) and, as we can see, the life expectancy of A class, have the life expectative of 80.39 years while the E class have the life expectative of 59.15 years.