In the previous assignment (ANOVA), I discovered there was a lack of statistical significance in the association between female employment and urbanisation (Gapminder). This prompted an adjustment to my research question by elevating my third variable ‘internetuserate’ in place of ‘urbanrate’.
The ANOVA suggested statistical significance with a Probability F Statistic of 0.00137. The Tukey Post Hoc test highlighted three paired comparisons where we should accept statistical significance.
For this assignment, I have continued to apply a modified research question: Is there an association between the rate of female employment (response variable) and the rate of internet usage (explanatory variable). My hypothesis is that there is a positive association.
To perform the chi square test for independence, I have had to convert my quantitative research variables to categoric.
The code for this assignment follows below.
2.1 Convert quantitative variables to categoric
#Quantile (5) split using the pandas.qcut function. (Explanatory variable)
sub2['internet5'] = pandas.qcut(sub2.internetuserate_rd, 5, labels = \
["Q1", "Q2", "Q3", "Q4", "Q5"])
'''
Create two level categorical variable for female employment using 42% as the top level of the 25th percentile representing middling to high rates of female employment
'''
def FEMCAT (row):
if row['femaleemployrate_rd'] < 42:
return 0
else :
return 1
sub2['FEMCAT'] = sub2.apply (lambda row: FEMCAT (row), axis = 1)
2.2 New dataframe with categorical variables of interest
# New Dataframe for explanatory categorical variable with 5 levels
sub3 = sub2[['FEMCAT','internet5']].dropna()
2.3 Chi Square Test of Independence
#Contingency table of observed counts
ct1 = pandas.crosstab(sub3['FEMCAT'], sub3['internet5'])
print(ct1)
#Column percentages. Axis = 0 sums values in each column
colsum = ct1.sum(axis = 0)
colpct = ct1/colsum
print(colpct)
#Request chi-square calculations which include chi-square, p value and exp
print('Chi-square valuee, p value and the expected counts')
cs1 = scipy.stats.chi2_contingency(ct1)
print(cs1)
2.4 Post Hoc Comparison | Bonferroni Adjustment
#Bonferroni adjustment. Performed by recode pair (x 10). P value < 0.005
#Recode1
recode1 = {'Q1': 'Q1', 'Q2': 'Q2'}
sub3['COMP1v1'] = sub3['internet5'].map(recode1)
#Contingency table of observed counts
print('Observed counts and column percentage for recode1')
ct2 = pandas.crosstab(sub3['FEMCAT'], sub3['COMP1v1'])
print(ct2)
#column percentage
colsum = ct2.sum(axis = 0)
colpct = ct2 / colsum
print(colpct)
#chi-square
print('Chi-square value, p value, expected counts')
cs2 = scipy.stats.chi2_contingency(ct2)
print(cs2)
#Recode2
recode2 = {'Q1': 'Q1', 'Q3': 'Q3'}
sub3['COMP1v2'] = sub3['internet5'].map(recode2)
#Contingency table of observed counts
print('Observed counts and column percentage for recode2')
ct3 = pandas.crosstab(sub3['FEMCAT'], sub3['COMP1v2'])
print(ct3)
#column percentage
colsum = ct3.sum(axis = 0)
colpct = ct3 / colsum
print(colpct)
#chi-square
print('Chi-square value, p value, expected counts')
cs3 = scipy.stats.chi2_contingency(ct2)
print(cs3)
#Recode3
recode3 = {'Q1': 'Q1', 'Q4': 'Q4'}
sub3['COMP1v3'] = sub3['internet5'].map(recode3)
#Contingency table of observed counts
print('Observed counts and column percentage for recode3')
ct4 = pandas.crosstab(sub3['FEMCAT'], sub3['COMP1v3'])
print(ct4)
#column percentage
colsum = ct4.sum(axis = 0)
colpct = ct4 / colsum
print(colpct)
#chi-square
print('Chi-square value, p value, expected counts')
cs4 = scipy.stats.chi2_contingency(ct4)
print(cs4)
#Recode4
recode4 = {'Q1': 'Q1', 'Q5': 'Q5'}
sub3['COMP1v4'] = sub3['internet5'].map(recode4)
#Contingency table of observed counts
print('Observed counts and column percentage for recode4')
ct5 = pandas.crosstab(sub3['FEMCAT'], sub3['COMP1v4'])
print(ct5)
#column percentage
colsum = ct5.sum(axis = 0)
colpct = ct5 / colsum
print(colpct)
#chi-square
print('Chi-square value, p value, expected counts')
cs5 = scipy.stats.chi2_contingency(ct5)
print(cs5)
#Recode5
recode5 = {'Q2': 'Q2', 'Q3': 'Q3'}
sub3['COMP1v5'] = sub3['internet5'].map(recode5)
#Contingency table of observed counts
print('Observed counts and column percentage for recode5')
ct6 = pandas.crosstab(sub3['FEMCAT'], sub3['COMP1v5'])
print(ct6)
#column percentage
colsum = ct6.sum(axis = 0)
colpct = ct6 / colsum
print(colpct)
#chi-square
print('Chi-square value, p value, expected counts')
cs6 = scipy.stats.chi2_contingency(ct6)
print(cs6)
#Recode6
recode6 = {'Q2': 'Q2', 'Q4': 'Q4'}
sub3['COMP1v6'] = sub3['internet5'].map(recode6)
#Contingency table of observed counts
print('Observed counts and column percentage for recode6')
ct7 = pandas.crosstab(sub3['FEMCAT'], sub3['COMP1v6'])
print(ct7)
#column percentage
colsum = ct7.sum(axis = 0)
colpct = ct7 / colsum
print(colpct)
#chi-square
print('Chi-square value, p value, expected counts')
cs7 = scipy.stats.chi2_contingency(ct7)
print(cs7)
#Recode7
recode7 = {'Q2': 'Q2', 'Q5': 'Q5'}
sub3['COMP1v7'] = sub3['internet5'].map(recode7)
#Contingency table of observed counts
print('Observed counts and column percentage for recode7')
ct8 = pandas.crosstab(sub3['FEMCAT'], sub3['COMP1v7'])
print(ct8)
#column percentage
colsum = ct8.sum(axis = 0)
colpct = ct8 / colsum
print(colpct)
#chi-square
print('Chi-square value, p value, expected counts')
cs8 = scipy.stats.chi2_contingency(ct8)
print(cs8)
#Recode8
recode8 = {'Q3': 'Q3', 'Q4': 'Q4'}
sub3['COMP1v8'] = sub3['internet5'].map(recode8)
#Contingency table of observed counts
print('Observed counts and column percentage for recode8')
ct9 = pandas.crosstab(sub3['FEMCAT'], sub3['COMP1v8'])
print(ct9)
#column percentage
colsum = ct9.sum(axis = 0)
colpct = ct9 / colsum
print(colpct)
#chi-square
print('Chi-square value, p value, expected counts')
cs9 = scipy.stats.chi2_contingency(ct9)
print(cs9)
#Recode9
recode9 = {'Q3': 'Q3', 'Q5': 'Q5'}
sub3['COMP1v9'] = sub3['internet5'].map(recode9)
#Contingency table of observed counts
print('Observed counts and column percentage for recode9')
ct10 = pandas.crosstab(sub3['FEMCAT'], sub3['COMP1v9'])
print(ct10)
#column percentage
colsum = ct10.sum(axis = 0)
colpct = ct10 / colsum
print(colpct)
#chi-square
print('Chi-square value, p value, expected counts')
cs10 = scipy.stats.chi2_contingency(ct10)
print(cs10)
#Recode10
recode10 = {'Q4': 'Q4', 'Q5': 'Q5'}
sub3['COMP1v10'] = sub3['internet5'].map(recode10)
#Contingency table of observed counts
print('Observed counts and column percentage for recode10')
ct11 = pandas.crosstab(sub3['FEMCAT'], sub3['COMP1v10'])
print(ct11)
#column percentage
colsum = ct11.sum(axis = 0)
colpct = ct11 / colsum
print(colpct)
#chi-square
print('Chi-square value, p value, expected counts')
cs11 = scipy.stats.chi2_contingency(ct11)
print(cs11)
2.5 Graph proportions within explanatory category
'''
Graph the percent of middling/high rates of female employment ( > 42%) within each category (quantile) for rates of internet usage
'''
#Set variable types. First set explanatory variable to cat and response to num
sub3['internet5'] = sub3['internet5'].astype('category')
sub3['FEMCAT'] = pandas.to_numeric(sub3['FEMCAT'], errors = 'coerce')
#Bivariate bar chart, internet categories on x axis, mean for female employment
#(proportion of 1's on y axis)
seaborn.factorplot(x = 'internet5', y = 'FEMCAT', data = sub3, kind = 'bar',\
ci = None)
plt.xlabel('Rate of internet usage grouped by quantile')
plt.ylabel('proportion of middling/high rate of female employment')
3.1 Chi Square Test of Independence (prior to post hoc)
3.2 Bonferroni Adjustment (p < 0.005)
3.3 Graph of proportions within categories
Prior to performing a Bonferroni Adjustment, the chi square test for independence revealed a chi square value of 13.2 (rounded), and a p value of 0.0103 with one degrees of freedom.
This suggests we should reject a null hypothesis that the rate of female employment and the rate of internet usage are independent. We should therefore accept the alternate hypothesis that the rate of female employment and the rate of internet usage are dependent (statistically associated).
Since the null hypothesis was rejected, I performed a post hoc test using the Bonferroni Adjustment. The explanatory variable has been grouped into five categories. As a result, the post hoc test was performed across 10 pairwise comparisons. To reduce the risk of type one error, the p value for statistical significance was lowered to 0.005 (0.05/10).
Using the adjusted Bonferroni Adjustment post hoc comparison, none of the pairwise comparisons satisfies the threshold for statistical significance. This suggests the initial chi square test for independence was subject to overall type one error. We should therefore accept the null hypothesis that female employment and internet usage rates are independent.