Untitled @theluckystudentperson - Tumblr Blog

Machine Learning For Data Analysis: Assignment 4

Introduction: The K-means cluster analysis technique is used to analyze the Gapminder database. The per capita income or “incomeperperson” is the response variable, while 10 quantitative and 1 binary explanatory variables are chosen.

-------------------------------------------------------------------

Python Code Below:

----------------------------------------------------------------

Python Output Below:

---------------------------------------------------------------------

Background and Summary:

Background: In this assignment, the k-means cluster analysis technique is used on the gapminder database to analyze the relationship between the quantitative response variable (”incomeperperson” or per capita income) and10 quantitative and one binary explanatory variable (”RICHORPOOR”). Only 56 countries had information for all 10 quantitative variables. The 10 quantitative explanatory variables include all the variables recorded in the Gapminder database except for the following variables: ’oilperperson”, “urbanrate”, “suicideper100th’”, and “relectricperperson”. The binary response variable, RICHORPOOR, is defined to have a value of 0 for countries with per capita income < $2000 and a value of 1 for countries with per capita income > $2000. This cutoff (per capita income=$2000) was chosen so as to have a large enough list of countries for which RICHORPOOR=0 (18 countries) and for which RICHORPOOR=1 (38 countries). The following data management techniques were used in preparation for the K-means cluster analysis: (a) removing missing or non-numeric values, (b) removing values that are recorded as NaN, and (c) centering all explanatory variables. Also, all variables are pre-processed (standardized) to have a mean of zero and an unit value for standard deviation. A test size factor of 0.35 was used to randomly split the total number of 56 countries into training and test data sets. A series of k-means cluster analyses were conducted on the training data specifying different number of clusters (k=1-9), using Euclidean distance to obtain the clustering. The r-squared value was plotted for each of the 9 cluster solutions to obtain an inflection point that provides guidance for choosing the number of clusters to interpret. Four of the variables (”armedforcesrate”, “employmentrate”, “hivrate”, and “internetusedrate”) were inverted. For example, the armed forces rate was substituted by the unarmedforcesrate such that unarmedforcesrate = 100 - armedforcesrate. The full list of variables used in the k-means cluster analysis is: 'unarmedforcesrate’,‘alcconsumption’,'breastcancerper100th’,'co2emissions’,'femaleemployrate’,

'unemployrate’,'nonhivrate’,'internetunusedrate’,'lifeexpectancy’,'RICHORPOOR’,'polityscore’. The stats for all the quantitative explanatory variables (mean, max, min, standard deviation) are computed and printed.

Summary: The elbow curve was not conclusive in that no clear inflection point can be observed in the plot of average distance versus number of clusters. Consequently, the choice of number of clusters was made almost arbitrarily to be n=3. Canonical discriminant analysis was used to reduce the 11 clustering variables (10 quantitative and one binary) down to the most important variables that explain most of the variance. The scatterplot attached here indicates that clusters are not fully distinct from each other. Only one of the 3 clusters is clearly separated from the other two.

Description of the three clusters: Cluster 1 is the poorest cluster. Cluster 1 has the lowest percentage of its population in the armed forces (Note: the “unarmedforcesrate” = 100 – “armedforcesrate”, so a high value of “unarmedforcesrate” translates to a lower percentage of the population in the armed forces). Also, Cluster 1 has moderate levels of alcohol consumption; lowest levels of breast cancer; lowest levels of Co2 emissions; highest levels of female employment; lowest levels of unemployment; highest levels of HIV infection (since “nonHIVrate” = 100 – “HIVrate”; i.e., low levels of “nonHIVrate” translates to higher HIV infection rates); lowest levels of internet use rates (again, “internetunusedrate” = 100 – “internetuserate”, so high values of “internetunusedrate” translates to lower levels of internet usage by the population); lowest life expectancies and moderate levels of democratic governance (since “polityscore” measures democratic governance). In general, Cluster 2 is the exact opposite of Cluster 1 (highest alcohol consumption, highest breast cancer rates< highest CO2 emissions, highest internet use rates, highest life expectancies and highest levels of democratic governance. Cluster 0 is in between Cluster 1 and Cluster 2 in most dimensions. Also, the per capita income also supports the idea that Cluster 1 and Cluster 2 are on opposite ends, since Cluster 1 has the lowest per capita income (mean per capita income = $1823; standard deviation = $1609.75), while Cluster 2 has the highest per capita income (mean per capita income = $11419; standard deviation = $8938.76). Cluster 0 is in between (mean per capita income= $4266; standard deviation=$7614) .

An analysis of Variance (ANOVA) test indicates significant differences between the clusters (F=4.337, p=0.0213), which indicates that the differences are significant since the p-value is less than 0.05). The Tukey post-hoc test indicates that only the differences between clusters 0 and 2 are significant and the null-hypothesis can be rejected. For the other two pairs of clusters (Clusters 0 and 1, and Clusters 1 and 2), the null-hypothesis can NOT be rejected.

Machine Learning for Data Analysis: Assignment 3

Background: In this assignment, the Lasso Regression analysis technique is used on the gapminder database to analyze the relationship between the quantitative response variable (”incomeperperson” or per capita income) and13 quantitative and one binary explanatory variable (”RICHORPOOR”). Only 51 countries had information for all 13 quantitative variables. The 13 quantitative explanatory variables include all the variables recorded in the Gapminder database except for one variable, ’oilperperson”, which was not included because of a lack of data points. The newly introduced binary response variable, RICHORPOOR, is defined to have a value of 0 for countries with per capita income < $2000 and a value of 1 for countries with per capita income > $2000. This cutoff (per capita income=$2000) was chosen so as to have a large enough list (15 in number) of countries for which RICHORPOOR=0 and a similarly-sized list (36 in number) of countries for which RICHORPOOR=1. The following data management techniques were used in preparation for the Lasso Regression analysis: (a) removing missing or non-numeric values, (b) removing values that are recorded as NaN, and © centering all explanatory variables. Also, all variables are pre-processed (standardized) to have a zero-mean and an unit value for standard deviation. A test size factor of 0.2 was used to randomly split the total number of 51 countries into training and test data sets.

_______________________________________

Python Code:

_________________________

Python Output Below:

Summary:

The Gapminder database is used to examine the relationship between a quantitative response variabe (”incomeperperson”) and 13 quantitative and one newly-defined categorical variable (”RICHORPOOR”). The data set is randomly spit into a training set that contains 80% of the data samples and a test set that contains 20% of the data samples. The total number of data samples (countries) is 51. Four of the variables (”armedforcesrate”, “employmentrate”, “hivrate”, and “internetusedrate”) were inverted so as to obtain a fair sample of variables for whom the regression coefficients are negative. For example, the HIV rate was substituted by the nonHIVrate (nonHIV rate = 100 - HIVrate). The full list of variables is 'unarmedforcesrate','alcconsumption','breastcancerper100th','co2emissions','femaleemployrate','unemployrate','nonhivrate','internetunusedrate','lifeexpectancy','RICHORPOOR','polityscore','relectricperperson','suicideper100th', 'urbanrate'. The stats (mean, max, min, standard deviation) for all the explanatory variables is included in this report.

The Lasso regression reveals that the regression coefficients for two variables (female employment rate and suicide rate) drop to zero. Thus, these two variables are not retained in the final analysis from the list of 13 quantitative and one binary response variables. The least-angle regression algorithm with k=10 fold cross-validation was used to estimate the lasso regression model in the training set, and the model was validated using the 20 % of countries (roughly 10) in the test set. The change in the cross-validation mean squared error was used to identify the best subset of predictor variables. This resulted in excluding the two variables mentioned above. The 11 quantitative and one categorical explanatory variable together account for 91.4% of the variance in the response variable in the training data set and 46% of the variance in the test data set. The training data mean squared error is 6948282, which is significantly less than the test data mean squared error (36174463). I believe that the significant increase in mean squared error in the lasso regression for the test data set is because of the small test data set size factor (test size factor=0.2). The regression coefficient is largest for the electric consumption per person (coeff=4310), followed by lack of participation in the armed forces (coeff=-2584). Other important variables are internet unused rate (coeff=-2126), alcohol consumption (coeff=-1523), unemployment rate (coeff=-1447), life expectancy (coeff=1401), etc.

theluckystudentperson

Machine Learning for Data Analysis: Assignment 2

Background: Random Forest analysis is used in this assignment. The gapminder data base is used to analyze the relationship between a new binary variable (”RICHORPOOR”) based on per capita income and four explanatory factor variables. The response and four explanatory variables (and their names in the gapminder database) are: (1) per capita income, “incomeperperson”; (2) the female labor force participation rate, “femaleemployrate”; (3) the rate of internet participation, “internetuserate”; (4) the percentage of the population residing in urban areas, “urbanrate”; and (5) the strength of democratic governance, “polityscore”.

The newly introduced binary response variable, RICHORPOOR, is defined to have a value of 0 for countries with per capita income < $1500 and a value of 1 for countries with per capita income > $1500. This cutoff (per capita income=$1500) was chosen so as to have a large enough list of countries for which RICHORPOOR=0 and a similarly sized list of countries for which RICHORPOOR=1. The following data management techniques were used in preparation for the random forest analysis: (a) removing missing or non-numeric values, (b) removing values that are recorded as NaN, and © centering all explanatory variables. Please note that the results shown in this study are quite sensitive and vary significantly in reruns of the python code and in response to any changes in the test size factor.

——————————————————

Python Code Below:

(Note: Please use browser settings and zoom in to read the small font)

——————————————————

Python Output Below:

runfile(‘C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.SEP2017/COURSERA/DataAnalysisInterPret/COURSE4/WEEK2/Assignment1.6.py’, wdir='C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.SEP2017/COURSERA/DataAnalysisInterPret/COURSE4/WEEK2’)

==============RICH OR POOR=================== 0 62 1 54 Name: RICHORPOOR, dtype: int64 ==============RICH OR POOR=================== STATS OF EXPLANATORY VARIABLES BEFORE CENTERING

femaleemployrate urbanrate count 116.000000 116.000000 mean 48.224138 50.298448 std 16.124589 21.831924 min 12.400000 10.400000 25% 38.550000 31.855000 50% 48.200001 50.740000 75% 57.200001 68.015000 max 83.300003 100.000000 internetuserate polityscore count 116.000000 116.000000 mean 22.841045 2.448276 std 21.264054 5.895237 min 0.210066 -9.000000 25% 5.849793 -3.000000 50% 13.799561 5.000000 75% 36.125220 8.000000 max 82.515928 9.000000 ________________________________ STATS OF EXPLANATORY VARIABLES AFTER CENTERING

femaleemployrate_c urbanrate_c count 1.160000e+02 1.160000e+02 mean -1.396584e-14 -9.616828e-15 std 1.612459e+01 2.183192e+01 min -3.582414e+01 -3.989845e+01 25% -9.674138e+00 -1.844345e+01 50% -2.413705e-02 4.415517e-01 75% 8.975863e+00 1.771655e+01 max 3.507587e+01 4.970155e+01 internetuserate_c polityscore_c count 1.160000e+02 1.160000e+02 mean -4.471519e-15 -1.761043e-16 std 2.126405e+01 5.895237e+00 min -2.263098e+01 -1.144828e+01 25% -1.699125e+01 -5.448276e+00 50% -9.041483e+00 2.551724e+00 75% 1.328418e+01 5.551724e+00 max 5.967488e+01 6.551724e+00 ________________________________ RICHORPOOR count 116.000000 mean 0.465517 std 0.500974 min 0.000000 25% 0.000000 50% 0.000000 75% 1.000000 max 1.000000 Printing Image Imgtree [ 0.17760183 0.31359873 0.32013215 0.1886673 ]

Accuracy Diagram: Accuracy versus number of decision Trees

Number of Decision Trees

————————————————————-

A Decision Tree Below:

(Note: Please set browser settings to zoom to read the small font)

——————————————————————————————

Summary: The total number of countries is 116, of which 62 have RICHORPOOR=0 and 54 have RICHORPOOR=1 (i.e, 62 countries have per capita income less than $1500, while 54 countries have per capita income greater than $1500). I use a test size factor of 0.45, so my training sample has 63 countries, since (1-0.45)*116=63; my test sample has 53 countries, since 0.45*116=53. Using the random forest code, 25 different decision trees are generated. However, the accuracy score is 86% with one decision tree and only increases slightly with more decision trees added. One of the 25 decision trees is shown in this assignment. Note that the decision tree is called a classification tree because the response variable, RICHORPOOR, is binary. Four explanatory variables were used. The four explanatory variables are: femaleemployrate (denoted as X[0] in decision tree), internetuserate (denoted as X[1] in decision tree), urbanrate (denoted as X[2] in decision tree), and polityscore (denoted as X[3] in decision tree). All explanatory variables are centered to have a zero mean.

The decision tree appended to this submission is rather complicated, but the explanation offered here is necessarily sketchy since the focus is on random forests.

The biggest terminal nodes (nodes which have gini index=0) are indicated with colored arrows. One terminal node on the left (indicated by red arrow) has 18 samples. This terminal node satisfies the following conditions:

X[2] <= -7.75; X[3] > -6.45 ; X[1] <= -3.85, which translates to

urban rate_c < = -7.75 ; polity score_c > -6.45 and internet use rate_c < = -3.85

Another terminal node on the right (indicated by a blue arrow) has 17 samples

and satisfies the following conditions:

X[2] > -7.75 ; X[1] > -11.13; X[0] < = -0.22, which translates to

urban rate_c > -7.75; internet use rate_c > -11.13 and female labor force participation rate_c < = -0.22.

The importance scores matrix is:

[female labor force participation; internet use rate; urban rate; polity score] = [0.17760183; 0.31359873; 0.32013215; 0.1886673 ]

It is clear that the urban rate has the highest relative importance score, while female labor force participation rate has the lowest relative importance score. Internet use rate is the second most important variable, followed by the polity score. I noticed when re-running this code that the numbers would change, but the urban rate was almost always the most important variable.

Accuracy Scores: The accuracy scores show an accuracy of more than 86% for one decision tree. This number only increases slightly to approximately 90% as more decision trees are added.

Machine Learning for Data Analysis: Assignment 2

The newly introduced binary response variable, RICHORPOOR, is defined to have a value of 0 for countries with per capita income < $1500 and a value of 1 for countries with per capita income > $1500. This cutoff (per capita income=$1500) was chosen so as to have a large enough list of countries for which RICHORPOOR=0 and a similarly sized list of countries for which RICHORPOOR=1. The following data management techniques were used in preparation for the random forest analysis: (a) removing missing or non-numeric values, (b) removing values that are recorded as NaN, and (c) centering all explanatory variables. Please note that the results shown in this study are quite sensitive and vary significantly in reruns of the python code and in response to any changes in the test size factor.

------------------------------------------------------

Python Code Below:

(Note: Please use browser settings and zoom in to read the small font)

------------------------------------------------------

Python Output Below:

runfile('C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.SEP2017/COURSERA/DataAnalysisInterPret/COURSE4/WEEK2/Assignment1.6.py', wdir='C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.SEP2017/COURSERA/DataAnalysisInterPret/COURSE4/WEEK2')

==============RICH OR POOR=================== 0 62 1 54 Name: RICHORPOOR, dtype: int64 ==============RICH OR POOR=================== STATS OF EXPLANATORY VARIABLES BEFORE CENTERING

Accuracy Diagram: Accuracy versus number of decision Trees

Number of Decision Trees

-------------------------------------------------------------

A Decision Tree Below:

(Note: Please set browser settings to zoom to read the small font)

------------------------------------------------------------------------------------------

The decision tree appended to this submission is rather complicated, but the explanation offered here is necessarily sketchy since the focus is on random forests.

X[2] <= -7.75; X[3] > -6.45 ; X[1] <= -3.85, which translates to

urban rate_c < = -7.75 ; polity score_c > -6.45 and internet use rate_c < = -3.85

Another terminal node on the right (indicated by a blue arrow) has 17 samples

and satisfies the following conditions:

X[2] > -7.75 ; X[1] > -11.13; X[0] < = -0.22, which translates to

urban rate_c > -7.75; internet use rate_c > -11.13 and female labor force participation rate_c < = -0.22.

The importance scores matrix is:

[female labor force participation; internet use rate; urban rate; polity score] = [0.17760183; 0.31359873; 0.32013215; 0.1886673 ]

Accuracy Scores: The accuracy scores show an accuracy of more than 86% for one decision tree. This number only increases slightly to approximately 90% as more decision trees are added.

Machine Learning For Data Analysis: Assignment 1

---------------------------------------------------------

Background: In this assignment, A Decision Tree is used to analyze the relationship between a binary variable based on per capita income and four explanatory factor variables. The gapminder database is used in this assignment. The important response and explanatory variables and their names in the gapminder database are: (1) per capita income, “incomeperperson”; (2) the female labor force participation rate, “femaleemployrate”; (3) the rate of internet participation, “internetuserate”; (4) the percentage of the population residing in urban areas, “urbanrate”; and (5) the strength of democratic governance, “polityscore”. A new binary response variable, RICHORPOOR, is defined with a value of 0 for countries with per capita income < $2000 and a value of 1 for countries with per capita income > $2000. I use all the data management techniques used in past assignments (e.g., removing missing values, removing values that are recorded as NaN, etc.). Also, in preparation for the decision tree analyses, all explanatory variables are centered.

An important point needs to be stressed here: The results shown in this study are quite sensitive and vary significantly in reruns of the python code and in response to any changes in the test size factor.

-----------------------------------------------------------------------

Python Code:

-----------------------------------------------------------

Python Output:

-----------------------------------------------------------------------

Decision Tree:

----------------------------------------------------------------------------

Summary: The total number of countries is 116, of which 70 have RICHORPOOR=0 and 46 have RICHORPOOR=1 (i.e, 70 countries have per capita income less than $2000, while 46 countries have per capita income greater than $2000). I use a test size factor of 0.333, so my training sample has 77 countries, since (1-0.333)*116=77; my test sample has 39 countries, since 0.333*116=39. The decision tree shown here is called a classification tree because the response variable, RICHORPOOR, is binary.

Four explanatory variables are used. These are femaleemployrate (denoted as X[0] in decision tree), internetuserate (denoted as X[1] in decision tree), urbanrate (denoted as X[2] in decision tree), and polityscore (denoted as X[3] in decision tree). All explanatory variables are centered to have a zero mean.

This decision tree is quite elaborate, so only a limited explanation is given here. I focus on the biggest terminal nodes (nodes which have gini index=0). One terminal node on the second row has 20 countries that have RICHORPOOR=1, while a large terminal node at the second-to-last row has a terminal row with 31 countries for which RICHORPOOR=0.

At the right on the second row of the decision tree is a split based on urban rate (denoted as X[2] in the decision tree) less than or greater than 17.75. Note that all explanatory variables are centered, so X[2] > 17.75 means an urban rate that is greater by 17.75 or more than the mean urban rate of the full list of countries.

Looking at the terminal node right at the top of the decision tree tells me that there are 20 countries that have their urban rates greater than 17.75 (i.e., 17.75% more than the mean urban rate for all countries) and RICHORPOOR=1 (i.e., have per capita income > $2000). This makes intuitive sense because a country that has high rates of urbanization is most probably not poor, since urban centers are typically areas of great economic activity. Thus, the biggest list of rich countries ( RICHORPOOR=1 or per capita income > $2000 ) has urban rates that are significantly greater than the mean.

At the second to last row, we see a terminal node with 31 countries that have RICHORPOOR=0 (i.e, countries that have per capita income less than $2000). This terminal node of 31 countries for which per capita income is less than $2000 has the following characteristics: X[1] < 22.99; X[2] < 17.75; -10.57 < X[0] ; -5.95 < X[3] < 6.05 (i.e., internet use rate is less than 22.99; urban rate is less than 17.75; female employ rate is greater than -10.57; and polity score is greater than -5.95 and less than 6.05).Note again that these cutoffs are all based on variables that are centered. For example, -5.95 < X[3] < 6.05 reflects a polity score that is greater than -5.95 and less than 6.05 compared to the mean polity score for the full list of countries. Thus, the biggest list of poor countries (RICHORPOOR=0 or per capita income < $2000 ) has the following characteristics: internet use rate not much higher than the mean, urban rates not much higher than the mean, female labor force participation rates that are not much below the mean and polity scores that are not very different from the mean.

Assignment 4

Background: In this assignment, the logistic regression tool is used for the same topic investigated in the previous assignment: i.e., the link between per capita income and female labor force participation rates. As in the previous submission, the gapminder database is used for the assignment. The important variables and their names in the gapminder database are: (1) per capita income, “incomeperperson”; (2) the female labor force participation rate, “femaleemployrate”; (3) the rate of internet participation, “internetuserate”; (4) the percentage of the population residing in urban areas, “urbanrate”; and (5) the strength of democratic governance, “polityscore”. To make the study a little more illuminating, I only look at countries that are neither highly democratic nor highly undemocratic (i.e, countries whose polity scores are greater than -8 and less than + 8). Logistic regression requires a binary response variable, so a new binary variable, RICHORPOOR, is defined with a value of 0 for countries with per capita income < $2000 and a value of 1 for countries with per capita income > $2000. Also, a new categorical variable, DEMSCORE, is defined with values from 0 to 6, with higher values denoting more democratic governance. I use all the data management techniques used in past assignments (e.g., removing missing values, removing values that are recorded as NaN, etc.). Also, in preparation for the multiple regression and logistic regression analyses, all explanatory variables are centered.

An important point needs to be stressed here: The results shown in this study are quite sensitive to my choice of the binary response variable, RICHORPOOR, and to the allowed polity scores ( greater than -8 and less than + 8). My experiments revealed significant variation as a result of any change in these choices.

Summary:

The total number of data points is 82. These countries met the criteria of having -8 < polity score < 8. A total of 59 countries qualified to have binary variable RICHORPOOR=0, 23 countries qualified to have the binary variable RICHORPOOR=1. The binary response variable, RICHORPOOR, was regressed against the explanatory variable, femaleemployrate . I find that the association between RICHORPOOR and femaleemployrate is negative (coefficient= -0.0369) with p-value=0.013 (statistically significant). Also, internetuserate is a confounding variable and with internetuserate included in the logistic regression, the regression between femaleemployrate and RICHORPOOR has a coefficient of -0.0244 and the p-value is now 0.160 (not statistically significant any longer).

Similarly, when the response variable, incomeperperson, is regressed against the explanatory variable, RICHORPOOR, I find that the link is negative (coeff= -0.1174) and statistically significant (p=0.023). Also, urban rate is a clear confounder, and with urbanrate included, the p-value for association between RICHORPOOR and polityscore is p=0.071 (not statistically significant any longer) and the coefficien t= -0.1163. The final summary has:

(1) After adjusting for the confounding variable (internetuserate), the association between femaleemployrate and RICHORPOOR yields the following numbers (OR= 0.98, 95% CI = 0.94 -- 1.01, p=0.160). This denotes a weak negative association between female labor force participation rate and per capita income which is not statistically significant because of the confounding variable (internet use rates).

(2) After adjusting for the confounding variable (urbanrate), the association between polityscore and the binary response variable, RICHORPOOR, yields the following numbers (OR=0.89, 95% CI = 0.78 -- 1.01, p=0.071). This denotes a weak negative association between democracy and per capita income which is not statistically significant because of the confounding variable (urban rate).

A very important point needs to be made here: These results are very sensitive to my choice of the binary response variable, RICHORPOOR. My experiments revealed significant variation as a result of any change in these choices.

-------------------------------------------------------------------------

Python code and output below:

NOTE: YOU MAY NEED TO ZOOM IN TO READ THE SMALL FONT

--------------------------------------------

--------------- END OF ASSIGNMENT 4 -------------

Assignment 3

Background: In assignment 3, I look at the association between “income per person” (per capita income) and “female employ rate” (female labor force participation rate), using data from the Gapminder database.

__________________________________________

Python Code below:

# -*- coding: utf-8 -*-

""" @author: jrose01; modified by Sudhir """

import scipy import numpy import pandas import matplotlib.pyplot as plt import statsmodels.api as sm import statsmodels.formula.api as smf import seaborn import warnings warnings.filterwarnings('ignore')

# bug fix for display formats to avoid run time errors pandas.set_option('display.float_format', lambda x:'%.2f'%x)

#call in data set data = pandas.read_csv('SudhirCodebook.csv')

# convert variables to numeric format using convert_objects function data['incomeperperson'] = pandas.to_numeric(data['incomeperperson'], errors='coerce') data['internetuserate'] = pandas.to_numeric(data['internetuserate'], errors='coerce') data['incomeperperson']=data['incomeperperson'].replace(' ', numpy.nan) data['internetuserate']=data['internetuserate'].replace(' ', numpy.nan)

data['polityscore'] = pandas.to_numeric(data['polityscore'], errors='coerce') data['urbanrate'] = pandas.to_numeric(data['urbanrate'], errors='coerce') data['polityscore']=data['polityscore'].replace(' ', numpy.nan) data['urbanrate']=data['urbanrate'].replace(' ', numpy.nan) data['femaleemployrate'] = pandas.to_numeric(data['femaleemployrate'], errors='coerce') data['femaleemployrate']=data['femaleemployrate'].replace(' ', numpy.nan)

data_clean=data.dropna() print (' ') print ('association between femaleemployrate and incomeperperson for all countries') print (scipy.stats.pearsonr(data_clean['femaleemployrate'], data_clean['incomeperperson'])) print (' ')

#For the current 2017 fiscal year, low-income economies are defined as those with a #GNI per capita, calculated using the World Bank Atlas method, of $1,025 or less in 2015; #lower middle-income economies are those with a GNI per capita between $1,026 and $4,035; #upper middle-income economies are those with a GNI per capita between $4,036 and $12,475; #high-income economies are those with a GNI per capita of $12,476 or more. #https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups

def incomegrp (row): if row['incomeperperson'] < 1025 : return 4 elif row['incomeperperson'] <= 4035 : return 3 elif row['incomeperperson'] <= 12475 : return 2 elif row['incomeperperson'] > 12476 : return 1

data_clean['incomegrp'] = data_clean.apply (lambda row: incomegrp (row), axis=1) chk1 = data_clean['incomegrp'].value_counts(sort=False, dropna=False) print(chk1)

sub1=data_clean[(data_clean['incomegrp']== 1)] sub2=data_clean[(data_clean['incomegrp']== 2)] sub3=data_clean[(data_clean['incomegrp']== 3)] sub4=data_clean[(data_clean['incomegrp']== 4)]

print (' ') print ('association between female employment rate and incomeperperson for low income countries') print (scipy.stats.pearsonr(sub4['femaleemployrate'], sub4['incomeperperson'])) print (' ') print ('association between female employment rate and incomeperperson for lower middle-income countries') print (scipy.stats.pearsonr(sub3['femaleemployrate'], sub3['incomeperperson'])) print (' ') print (' ') print ('association between female employment rate and incomeperperson for upper middle-income countries') print (scipy.stats.pearsonr(sub2['femaleemployrate'], sub2['incomeperperson'])) print (' ') print ('association between female employment rate and incomeperperson for high income countries') print (scipy.stats.pearsonr(sub1['femaleemployrate'], sub1['incomeperperson'])) print (' ') print('________________________________') print(' REST OF ASSIGNMENT IS FOCUSED ON LOW INCOME COUNTRIES ') print(' LOW INCOME COUNTRIES have Income Per Person < $1025') print(' ') print('STATISTICS OF EXPLANATORY AND RESPONSE VARIABLES BEFORE CENTERING EXPLANATORY VARIABLE') print(' ') sub4['internetuserate_c'] = (sub4['internetuserate'] - sub4['internetuserate'].mean()) sub4['incomeperperson_c'] = (sub4['incomeperperson'] - sub4['incomeperperson'].mean()) sub4['urbanrate_c'] = (sub4['urbanrate'] - sub4['urbanrate'].mean()) sub4['polityscore_c'] = (sub4['polityscore'] - sub4['polityscore'].mean()) sub4['femaleemployrate_c'] = (sub4['femaleemployrate'] - sub4['femaleemployrate'].mean())

chk2=sub4[["femaleemployrate", "incomeperperson"]].describe() print(chk2) print('________________________________') print('STATISTICS OF EXPLANATORY AND RESPONSE VARIABLES AFTER CENTERING EXPLANATORY VARIABLE') print('OLD EXPLANATORY VARIABLE=femaleemployrate') print('NEW CENTERED EXPLANATORY VARIABLE=femaleemployrate_c') print(' ') chk3=sub4[["femaleemployrate_c", "incomeperperson"]].describe() print(chk3) print('________________________________') ###################################################### # BASIC LINEAR REGRESSION ######################################################

scat1 = seaborn.regplot(x="femaleemployrate_c", y="incomeperperson", scatter=True, data=sub4) plt.xlabel('Female Employment Rate') plt.ylabel('Income Per Person') plt.title ('Scatterplot for the Association Between Female Employment Rate and Income Per Person') print(scat1)

print ("OLS regression model for the association between Female Employment Rate and Income Per Person") reg9 = smf.ols('incomeperperson ~ femaleemployrate_c', data=sub4).fit() print (reg9.summary()) # linear regression analysis reg1 = smf.ols('incomeperperson ~ femaleemployrate_c', data=sub4).fit() print (reg1.summary())

# quadratic (polynomial) regression analysis

# run following line of code if you get PatsyError 'ImaginaryUnit' object is not callable #del I reg2 = smf.ols('incomeperperson ~ femaleemployrate_c + I(femaleemployrate_c**2)', data=sub4).fit() print (reg2.summary())

####################################################### # EVALUATING MODEL FIT #######################################################

# adding polity score reg3 = smf.ols('incomeperperson ~ femaleemployrate_c + I(femaleemployrate_c**2) + polityscore_c',data=sub4).fit() print (reg3.summary())

#Q-Q plot for normality fig4=sm.qqplot(reg3.resid, line='r')

# simple plot of residuals stdres=pandas.DataFrame(reg3.resid_pearson) plt.plot(stdres, 'o', ls='None') l = plt.axhline(y=0, color='r') plt.ylabel('Standardized Residual') plt.xlabel('Observation Number')

#fig2 = sm.graphics.plot_regress_exog(reg3, "femaleemployrate_c", fig=fig2) fig2 = sm.graphics.plot_regress_exog(reg3, "femaleemployrate_c") # leverage plot fig3=sm.graphics.influence_plot(reg3, size=8) print(' TESTING EACH VARIABLE SEPARATELY FOR CONFOUNDING EFFECT') reg4 = smf.ols('incomeperperson ~ femaleemployrate_c + internetuserate_c', data=sub4).fit() print (reg4.summary()) reg5 = smf.ols('incomeperperson ~ femaleemployrate_c + urbanrate_c', data=sub4).fit() print (reg5.summary()) reg6 = smf.ols('incomeperperson ~ femaleemployrate_c + polityscore_c', data=sub4).fit() print (reg6.summary()) # NOW ADD VARIAABLES, ONE AT A TIME print(' TESTING SUMS OF VARIABLES FOR CONFOUNDING EFFECT') reg7 = smf.ols('incomeperperson ~ femaleemployrate_c + internetuserate_c + urbanrate_c', data=sub4).fit() print (reg7.summary()) reg8 = smf.ols('incomeperperson ~ femaleemployrate_c + internetuserate_c + urbanrate_c + polityscore_c', data=sub4).fit() print (reg8.summary())

___________________________________________________________

Python Output Below:

In [18]: runfile('C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.AUG2017/COURSERA/DataAnalysisInterPret/COURSE3/WEEK3/Assignment3.4.femaleemp.3.py', wdir='C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.AUG2017/COURSERA/DataAnalysisInterPret/COURSE3/WEEK3')

association between femaleemployrate and incomeperperson for all countries (-0.019164036830870472, 0.81594119524247255)

1 28 2 30 3 42 4 50 Name: incomegrp, dtype: int64

association between female employment rate and incomeperperson for low income countries (-0.35384770067156085, 0.011704159674753694)

association between female employment rate and incomeperperson for lower middle-income countries (-0.026846132068958046, 0.8659834103019749)

association between female employment rate and incomeperperson for upper middle-income countries (-0.15392250424505802, 0.41673339118750996)

association between female employment rate and incomeperperson for high income countries (0.48958067333164995, 0.0081866599492884679)

________________________________ REST OF ASSIGNMENT IS FOCUSED ON LOW INCOME COUNTRIES LOW INCOME COUNTRIES have Income Per Person < $1025

STATISTICS OF EXPLANATORY AND RESPONSE VARIABLES BEFORE CENTERING EXPLANATORY VARIABLE

femaleemployrate incomeperperson count 50.00 50.00 mean 55.71 437.01 std 17.19 215.91 min 12.40 103.78 25% 48.50 275.96 50% 54.10 377.23 75% 68.58 583.73 max 83.30 952.83 ________________________________ STATISTICS OF EXPLANATORY AND RESPONSE VARIABLES AFTER CENTERING EXPLANATORY VARIABLE OLD EXPLANATORY VARIABLE=femaleemployrate NEW CENTERED EXPLANATORY VARIABLE=femaleemployrate_c

femaleemployrate_c incomeperperson count 50.00 50.00 mean -0.00 437.01 std 17.19 215.91 min -43.31 103.78 25% -7.21 275.96 50% -1.61 377.23 75% 12.87 583.73 max 27.59 952.83 ________________________________ Axes(0.125,0.125;0.775x0.755) OLS regression model for the association between Female Employment Rate and Income Per Person OLS Regression Results ======================================================== Dep. Variable: incomeperperson R-squared: 0.125 Model: OLS Adj. R-squared: 0.107 Method: Least Squares F-statistic: 6.870 Date: Sun, 06 Aug 2017 Prob (F-statistic): 0.0117 Time: 15:45:34 Log-Likelihood: -335.84 No. Observations: 50 AIC: 675.7 Df Residuals: 48 BIC: 679.5 Df Model: 1 Covariance Type: nonrobust ========================================================== coef std err t P>|t| [0.025 0.975] -------------------------------------------------------------------------------------- Intercept 437.0122 28.855 15.145 0.000 378.995 495.030 femaleemployrate_c -4.4457 1.696 -2.621 0.012 -7.856 -1.035 ========================================================= Omnibus: 1.298 Durbin-Watson: 1.880 Prob(Omnibus): 0.523 Jarque-Bera (JB): 1.240 Skew: 0.360 Prob(JB): 0.538 Kurtosis: 2.725 Cond. No. 17.0 =========================================================

Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. OLS Regression Results ========================================================= Dep. Variable: incomeperperson R-squared: 0.125 Model: OLS Adj. R-squared: 0.107 Method: Least Squares F-statistic: 6.870 Date: Sun, 06 Aug 2017 Prob (F-statistic): 0.0117 Time: 15:45:34 Log-Likelihood: -335.84 No. Observations: 50 AIC: 675.7 Df Residuals: 48 BIC: 679.5 Df Model: 1 Covariance Type: nonrobust ========================================================= coef std err t P>|t| [0.025 0.975] -------------------------------------------------------------------------------------- Intercept 437.0122 28.855 15.145 0.000 378.995 495.030 femaleemployrate_c -4.4457 1.696 -2.621 0.012 -7.856 -1.035 ========================================================= Omnibus: 1.298 Durbin-Watson: 1.880 Prob(Omnibus): 0.523 Jarque-Bera (JB): 1.240 Skew: 0.360 Prob(JB): 0.538 Kurtosis: 2.725 Cond. No. 17.0 =========================================================

Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. OLS Regression Results ========================================================= Dep. Variable: incomeperperson R-squared: 0.141 Model: OLS Adj. R-squared: 0.104 Method: Least Squares F-statistic: 3.852 Date: Sun, 06 Aug 2017 Prob (F-statistic): 0.0282 Time: 15:45:34 Log-Likelihood: -335.39 No. Observations: 50 AIC: 676.8 Df Residuals: 47 BIC: 682.5 Df Model: 2 Covariance Type: nonrobust ========================================================= coef std err t P>|t| [0.025 0.975] ---------------------------------------------------------------------------------------------- Intercept 415.6192 37.019 11.227 0.000 341.147 490.091 femaleemployrate_c -3.9629 1.777 -2.230 0.031 -7.538 -0.388 I(femaleemployrate_c ** 2) 0.0739 0.080 0.925 0.360 -0.087 0.235 ========================================================= Omnibus: 2.572 Durbin-Watson: 1.944 Prob(Omnibus): 0.276 Jarque-Bera (JB): 2.197 Skew: 0.511 Prob(JB): 0.333 Kurtosis: 2.910 Cond. No. 610. =========================================================

Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. OLS Regression Results ========================================================= Dep. Variable: incomeperperson R-squared: 0.143 Model: OLS Adj. R-squared: 0.088 Method: Least Squares F-statistic: 2.567 Date: Sun, 06 Aug 2017 Prob (F-statistic): 0.0659 Time: 15:45:34 Log-Likelihood: -335.32 No. Observations: 50 AIC: 678.6 Df Residuals: 46 BIC: 686.3 Df Model: 3 Covariance Type: nonrobust ========================================================= coef std err t P>|t| [0.025 0.975] ---------------------------------------------------------------------------------------------- Intercept 416.8687 37.514 11.112 0.000 341.357 492.380 femaleemployrate_c -4.0999 1.831 -2.239 0.030 -7.786 -0.414 I(femaleemployrate_c ** 2) 0.0696 0.082 0.854 0.398 -0.094 0.234 polityscore_c -2.2092 5.946 -0.372 0.712 -14.177 9.759 ========================================================== Omnibus: 2.135 Durbin-Watson: 1.977 Prob(Omnibus): 0.344 Jarque-Bera (JB): 1.890 Skew: 0.468 Prob(JB): 0.389 Kurtosis: 2.827 Cond. No. 613. ==========================================================

Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. TESTING EACH VARIABLE SEPARATELY FOR CONFOUNDING EFFECT OLS Regression Results ========================================================= Dep. Variable: incomeperperson R-squared: 0.230 Model: OLS Adj. R-squared: 0.197 Method: Least Squares F-statistic: 7.019 Date: Sun, 06 Aug 2017 Prob (F-statistic): 0.00215 Time: 15:45:35 Log-Likelihood: -332.65 No. Observations: 50 AIC: 671.3 Df Residuals: 47 BIC: 677.0 Df Model: 2 Covariance Type: nonrobust ========================================================= coef std err t P>|t| [0.025 0.975] -------------------------------------------------------------------------------------- Intercept 437.0122 27.359 15.973 0.000 381.974 492.051 femaleemployrate_c -3.7826 1.629 -2.321 0.025 -7.061 -0.505 internetuserate_c 8.3381 3.297 2.529 0.015 1.705 14.971 ========================================================= Omnibus: 2.945 Durbin-Watson: 2.218 Prob(Omnibus): 0.229 Jarque-Bera (JB): 2.812 Skew: 0.535 Prob(JB): 0.245 Kurtosis: 2.548 Cond. No. 17.1 =========================================================

Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. OLS Regression Results ========================================================= Dep. Variable: incomeperperson R-squared: 0.177 Model: OLS Adj. R-squared: 0.142 Method: Least Squares F-statistic: 5.069 Date: Sun, 06 Aug 2017 Prob (F-statistic): 0.0102 Time: 15:45:35 Log-Likelihood: -334.30 No. Observations: 50 AIC: 674.6 Df Residuals: 47 BIC: 680.3 Df Model: 2 Covariance Type: nonrobust ========================================================= coef std err t P>|t| [0.025 0.975] -------------------------------------------------------------------------------------- Intercept 437.0122 28.277 15.455 0.000 380.127 493.898 femaleemployrate_c -2.8619 1.898 -1.508 0.138 -6.681 0.957 urbanrate_c 4.1524 2.404 1.728 0.091 -0.683 8.988 ========================================================= Omnibus: 1.694 Durbin-Watson: 1.705 Prob(Omnibus): 0.429 Jarque-Bera (JB): 1.388 Skew: 0.406 Prob(JB): 0.499 Kurtosis: 2.913 Cond. No. 18.9 =========================================================

Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. OLS Regression Results ========================================================= Dep. Variable: incomeperperson R-squared: 0.130 Model: OLS Adj. R-squared: 0.093 Method: Least Squares F-statistic: 3.506 Date: Sun, 06 Aug 2017 Prob (F-statistic): 0.0381 Time: 15:45:35 Log-Likelihood: -335.71 No. Observations: 50 AIC: 677.4 Df Residuals: 47 BIC: 683.2 Df Model: 2 Covariance Type: nonrobust ========================================================= coef std err t P>|t| [0.025 0.975] -------------------------------------------------------------------------------------- Intercept 437.0122 29.083 15.026 0.000 378.504 495.521 femaleemployrate_c -4.5901 1.734 -2.647 0.011 -8.078 -1.102 polityscore_c -2.9328 5.868 -0.500 0.620 -14.737 8.872 ========================================================= Omnibus: 1.022 Durbin-Watson: 1.922 Prob(Omnibus): 0.600 Jarque-Bera (JB): 1.038 Skew: 0.314 Prob(JB): 0.595 Kurtosis: 2.677 Cond. No. 17.0 =========================================================

Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. TESTING SUMS OF VARIABLES FOR CONFOUNDING EFFECT OLS Regression Results ========================================================= Dep. Variable: incomeperperson R-squared: 0.269 Model: OLS Adj. R-squared: 0.222 Method: Least Squares F-statistic: 5.649 Date: Sun, 06 Aug 2017 Prob (F-statistic): 0.00221 Time: 15:45:35 Log-Likelihood: -331.34 No. Observations: 50 AIC: 670.7 Df Residuals: 46 BIC: 678.3 Df Model: 3 Covariance Type: nonrobust ========================================================= coef std err t P>|t| [0.025 0.975] -------------------------------------------------------------------------------------- Intercept 437.0122 26.941 16.221 0.000 382.784 491.241 femaleemployrate_c -2.4427 1.817 -1.344 0.185 -6.100 1.214 internetuserate_c 7.8413 3.262 2.404 0.020 1.275 14.408 urbanrate_c 3.6165 2.301 1.572 0.123 -1.015 8.248 ========================================================= Omnibus: 2.817 Durbin-Watson: 2.057 Prob(Omnibus): 0.245 Jarque-Bera (JB): 2.466 Skew: 0.541 Prob(JB): 0.291 Kurtosis: 2.880 Cond. No. 19.0 =========================================================

Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. OLS Regression Results ========================================================= Dep. Variable: incomeperperson R-squared: 0.274 Model: OLS Adj. R-squared: 0.209 Method: Least Squares F-statistic: 4.236 Date: Sun, 06 Aug 2017 Prob (F-statistic): 0.00539 Time: 15:45:35 Log-Likelihood: -331.20 No. Observations: 50 AIC: 672.4 Df Residuals: 45 BIC: 682.0 Df Model: 4 Covariance Type: nonrobust ======================================================= coef std err t P>|t| [0.025 0.975] -------------------------------------------------------------------------------------- Intercept 437.0122 27.158 16.092 0.000 382.313 491.711 femaleemployrate_c -2.5782 1.850 -1.393 0.170 -6.305 1.148 internetuserate_c 7.8123 3.289 2.375 0.022 1.188 14.437 urbanrate_c 3.6329 2.320 1.566 0.124 -1.039 8.305 polityscore_c -2.8305 5.481 -0.516 0.608 -13.869 8.208 ======================================================== Omnibus: 2.786 Durbin-Watson: 2.090 Prob(Omnibus): 0.248 Jarque-Bera (JB): 2.396 Skew: 0.534 Prob(JB): 0.302 Kurtosis: 2.912 Cond. No. 19.0 ========================================================

Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

__________________________________________________________

Summary: The association between female labor force participation and per capita income for the full list of countries (no of observations=150) is slightly negative (coeff = -0.019) but not statistically significant (p=0.816)

I then segment the full list of countries into four sub-groups based on income. These are : (1) low-income countries, which have per capita income < $1025; (2) lower middle income countries which have $1025 < per capita income < $4035; (3) upper middle income countries which have $4036 < per capita income < 12475; and (4) rich countries, which have per capita income > $12476. This follows World Bank guidelines (See https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups). The explanatory variable (female labor force participation) is centered and the means are computed before and after centering. I then look at the association between female labor force participation and per capita income for these four groups and find a negative correlation for the first three groups (low income, lower middle income, and upper middle income) and a positive correlation only for rich countries. Note that the relationship between female labor force participation is statistically significant (p < 0.05) only for (1) low income and (4) high income countries. For (2) lower middle income and (3) upper middle income countries, the relationship does not pass the null test (i.e, p > 0.05). The results indicated here show that the relationship between per capita income and female labor force participation still follows the U-shape indicated by Claudia Goldin in 1995 (see https://scholar.harvard.edu/files/goldin/files/the_u-shaped_female_labor_force_function_in_economic_development_and_economic_history.pdf ).

The rest of my analysis focuses on low-income countries to try and understand why the association (pearson correlation) between female labor force participation rate and per capita income is negative (coeff= -0.354) and statistically significant (p=0.012, i.e., p < 0.05). The total number of low-income countries is 50. When I look at the linear regression, I find a parameter value of t = -2.621 with p= 0.012 (statistically significant).

Now, l add more variables to the multiple regression and keep note of the parameter estimate, t, and the p-value for regression of femaleemployrate and incomeperperson. When I add the quadratic term (femaleemployrate_c * femaleemployrate_c), I note that the R-squared value goes up from 0.125 to 0.141. The parameter estimate, t = - 2.23 and p=0.031 for regressing femaleemployrate with incomeperperson when the quadratic term is added in the multiple regression.

Then, three other variables are added to the analysis. These are (a) internet use rate, (b) urban rate, and (c ) polity score. The internet use rate and urban rate measure the percentage of the population that uses the internet and resides in urban areas, respectively, while the polity score measures the strength of democratic governance in the respective countries. I then rerun the multiple regression adding these three variables one at a time. I find that the urban rate is a crucial confounding variable (i.e, the p-value for female labor force participation and income per person is greater than 0.05 when urban rate is added as a variable in the multiple regression). Also, internet use rate and polity scores are not confounding variables, since the p-value for the association between female labor force participation and per capita income remains below 0.05. So, (a) adding internetuserate only (regression between female employrate and incomeperperson is statistically significant with p=0.025, so internetuserate is not a confounding variable), (b) adding urbanrate only (regression between female employrate and incomeperperson is not statistically significant with p=0.138, so urbanrate is a confounding variable) , and (c) adding polityscore only( regression between female employrate and incomeperperson is statistically significant with p=0.011, so polityscore is not a confounding variable).

Finally, I use q-q plots and other plots of residuals to look at the residuals and leverage of outliers. For the regression relation, incomeperperson ~ femaleemployrate_c + I(femaleemployrate_c**2) + polityscore_c, I find that the observations differ significantly from the regression predictions at very low and very high values. Thus, the residuals of this regression relation do not perfectly follow a normal distribution. Also, both the plot of residuals versus female employment rate and the partial regression plot show higher residual values for low values of female employment rate. This indicates that the regression model does not fully explain the variability at low values of female employment. Possibly, the confounding variable (urban rate) needs to be added to the above regression relation. In the final plot of leverage, I notice that there are two outliers, numbers 140 and 205, which have studentized residuals > 2. Note that outliers are defined as having studentized residuals greater than 2 or less than -2. I also note number 89 which has H leverage score of approximately 0.40. However, this observation data point has a studentized residual of zero.

The final summary: In this assignment, the regression analysis for female employment rate and income per person is presented for low income countries. I find that urban rate is a confounding variable. The regression relation shows that female employment rate is negatively associated with income but this relation is not statistically significant (beta = - 2.58, p=0.170) due to the confounding variable, urban rate. The variable, urban rate (beta=3.63, p=0.124) is positively but not significantly associated with income per person, while polity score (beta=-2.83, p=0.608) is negatively but not significantly associated with income per person. The internet use rate is significantly associated with income per person (beta=7.81, p=0.022).

Assignment 2

Background: In this assignment, I look at the association between income per person and internet use rate using a subset of the gapminder data set. The linear regression algorithms in statsmodels are used.

Python Code Below:

___________________________________

# -*- coding: utf-8 -*-

""" Created on Tue Jun 16 15:46:09 2015; modified 07/29/2017 @author: jrose01; modified by Sudhir """

import numpy import pandas import seaborn import statsmodels.formula.api as smf import scipy import matplotlib.pyplot as plt

# bug fix for display formats to avoid run time errors pandas.set_option('display.float_format', lambda x:'%.2f'%x)

#call in data set data = pandas.read_csv('SudhirCodebook.csv')

data_clean=data.dropna() print (' ') print ('association between internetuserate and incomeperperson for all countries') print (scipy.stats.pearsonr(data_clean['internetuserate'], data_clean['incomeperperson'])) print (' ')

def incomegrp (row): if row['incomeperperson'] < 1025 : return 1 elif row['incomeperperson'] <= 4035 : return 2 elif row['incomeperperson'] <= 12475 : return 3 elif row['incomeperperson'] > 12476 : return 4

data_clean['incomegrp'] = data_clean.apply (lambda row: incomegrp (row), axis=1) chk1 = data_clean['incomegrp'].value_counts(sort=False, dropna=False) print(chk1)

sub1=data_clean[(data_clean['incomegrp']== 1)] sub2=data_clean[(data_clean['incomegrp']== 2)] sub3=data_clean[(data_clean['incomegrp']== 3)] sub4=data_clean[(data_clean['incomegrp']== 4)]

print (' ') print ('association between internetuserate and incomeperperson for low income countries') print (scipy.stats.pearsonr(sub1['internetuserate'], sub1['incomeperperson'])) print (' ') print ('association between internetuserate and incomeperperson for lower middle-income countries') print (scipy.stats.pearsonr(sub2['internetuserate'], sub2['incomeperperson'])) print (' ') print (' ') print ('association between internetuserate and incomeperperson for upper middle-income countries') print (scipy.stats.pearsonr(sub3['internetuserate'], sub3['incomeperperson'])) print (' ') print ('association between internetuserate and incomeperperson for high income countries') print (scipy.stats.pearsonr(sub4['internetuserate'], sub4['incomeperperson'])) print (' ') print('________________________________') print(' REST OF ASSIGNMENT IS FOCUSED ON HIGH INCOME COUNTRIES ') print(' HIGH INCOME COUNTRIES have Income Per Person > $12476') print(' ') print('STATISTICS OF EXPLANATORY AND RESPONSE VARIABLES BEFORE CENTERING EXPLANATORY VARIABLE') print(' ') sub4['internetuserate_zeromean'] = (sub4['internetuserate'] - sub4['internetuserate'].mean()) sub4['incomeperperson_zeromean'] = (sub4['incomeperperson'] - sub4['incomeperperson'].mean()) chk8=sub4[["internetuserate", "incomeperperson"]].describe() print(chk8) print('________________________________') print('STATISTICS OF EXPLANATORY AND RESPONSE VARIABLES AFTER CENTERING EXPLANATORY VARIABLE') print('OLD EXPLANATORY VARIABLE=internetuserate') print('NEW CENTERED EXPLANATORY VARIABLE=internetuserate_zeromean') print(' ') chk9=sub4[["internetuserate_zeromean", "incomeperperson"]].describe() print(chk9) print('________________________________') ############################################################################################ # BASIC LINEAR REGRESSION ############################################################################################ scat5 = seaborn.regplot(x="internetuserate_zeromean", y="incomeperperson", scatter=True, data=sub4) plt.xlabel('Internet Use Rate') plt.ylabel('Income Per Person') plt.title ('Scatterplot for the Association Between Internet Use Rate and Income Per Person') print(scat5) print ("OLS regression model for the association between Internet Use Rate and Income Per Person") reg5 = smf.ols('incomeperperson ~ internetuserate_zeromean', data=sub4).fit() print (reg5.summary())

_______________________________________________________

Python Output Below:

runfile('C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.AUG2017/COURSERA/DataAnalysisInterPret/COURSE3/WEEK2/Assignment2.5.py', wdir='C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.AUG2017/COURSERA/DataAnalysisInterPret/COURSE3/WEEK2')

association between internetuserate and incomeperperson for all countries (0.7509393764769613, 1.8911408665067692e-34)

1 52 2 53 3 39 4 39 Name: incomegrp, dtype: int64

association between internetuserate and incomeperperson for low income countries (0.34632262422958809, 0.011902443581392723)

association between internetuserate and incomeperperson for lower middle-income countries (0.18726481632388575, 0.17936489903503308)

association between internetuserate and incomeperperson for upper middle-income countries (0.38644166704358984, 0.015096836416322831)

association between internetuserate and incomeperperson for high income countries (0.46893241554164006, 0.0026019753237851153)

________________________________ REST OF ASSIGNMENT IS FOCUSED ON HIGH INCOME COUNTRIES HIGH INCOME COUNTRIES have Income Per Person > $12476

STATISTICS OF EXPLANATORY AND RESPONSE VARIABLES BEFORE CENTERING EXPLANATORY VARIABLE

internetuserate incomeperperson count 39.00 39.00 mean 73.45 28245.60 std 14.47 13853.95 min 42.69 12505.21 25% 65.60 19306.40 50% 77.50 25575.35 75% 82.76 33927.57 max 95.64 81647.10 ________________________________ STATISTICS OF EXPLANATORY AND RESPONSE VARIABLES AFTER CENTERING EXPLANATORY VARIABLE OLD EXPLANATORY VARIABLE=internetuserate NEW CENTERED EXPLANATORY VARIABLE=internetuserate_zeromean

internetuserate_zeromean incomeperperson count 39.00 39.00 mean -0.00 28245.60 std 14.47 13853.95 min -30.76 12505.21 25% -7.85 19306.40 50% 4.05 25575.35 75% 9.31 33927.57 max 22.19 81647.10 ________________________________ Axes(0.125,0.125;0.775x0.755) OLS regression model for the association between Internet Use Rate and Income Per Person OLS Regression Results ============================================================================== Dep. Variable: incomeperperson R-squared: 0.220 Model: OLS Adj. R-squared: 0.199 Method: Least Squares F-statistic: 10.43 Date: Sun, 30 Jul 2017 Prob (F-statistic): 0.00260 Time: 08:40:41 Log-Likelihood: -421.91 No. Observations: 39 AIC: 847.8 Df Residuals: 37 BIC: 851.1 Df Model: 1 Covariance Type: nonrobust ============================================================================================ coef std err t P>|t| [0.025 0.975] -------------------------------------------------------------------------------------------- Intercept 2.825e+04 1985.674 14.225 0.000 2.42e+04 3.23e+04 internetuserate_zeromean 448.8168 138.974 3.230 0.003 167.229 730.405 ============================================================================== Omnibus: 33.776 Durbin-Watson: 1.914 Prob(Omnibus): 0.000 Jarque-Bera (JB): 90.744 Skew: 2.115 Prob(JB): 1.97e-20 Kurtosis: 9.160 Cond. No. 14.3 ==============================================================================

Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

______________________________________

Note: Python warnings deleted from output

______________________________________

Summary:

In this assignment, I look at the regression relation between income per person (response variable) and internet use rate (explanatory variable) for all countries first. I find that the pearson correlation between income per person and internet use rate is 0.751 and the p-value is 1.89e-34, which is much smaller than 0.05. This means that we can safely reject the null hypothesis. Thus, we find a statistically significant relationship between income per person and internet use rate for the full list of countries.

Then, I segment the list of countries into four groups based on income per person. These are: (1) low-income countries with income per person < $1025; (2) lower middle-income countries with $1025 < income per person < $4035; (3) upper middle-income countries with $4036 < income per person < $12475; and (4) high-income countries with income per person > $12476. I find that the relationship between income per person and internet use rate is strong and statistically significant (i.e., p < 0.05) for low income, upper middle-income, and high income countries. However, for lower middle-income countries, the relationship between income per person and internet use rate is not statistically significant (i.e., the null hypothesis can NOT be rejected, since p=0.179, which is much greater than the cutoff of p=0.05).

Next, I examine the relationship between income per person and internet use rate for high-income countries in more depth. The total number of countries in this group of high-income countries is 39. The mean internet use rate is 73.45, the min is 42.69, and the max is 95.64. The mean income per person is $28250, the min is $12505, and the max is $81647. Both explanatory (internet use rate) and response (income per person) variables are quantitative. As discussed in the course, the explanatory variable (but not the response variable) needs to be centered at zero. A new variable, “internetuserate_zeromean”, centered at zero, is created from the old variable “internetuserate”. This new variable is used in the OLS function to obtain the regression relation. For this centered variable, internetuserate_zeromean, the mean is 0.00, min is -30.76, and max is 22.19. The regression relation I obtain is

Incomeperperson = 28250 + 448.8 * internetuserate_zeromean

Also, the F-statistic is 10.43, and the P-value is 0.00260, which is much smaller than p=0.05 , so we can safely reject the null hypothesis and accept that there is a relationship between income per person and internet use rate. The R-squared value is 0.220, which means that roughly 22% of the variability in income per person can be explained by differences in internet use rate.

FINAL RESULT: The linear regression analysis indicates that income per person is significantly and positively associated with internet use rate (slope or beta_1= 448.8; p=0.003, and the intercept or beta_0 is 28250).

FINAL CHECKLIST:

Instructions for the assignment:

1) If you have a categorical explanatory variable, make sure one of your categories is coded "0" and generate a frequency table for this variable to check your coding. If you have a quantitative explanatory variable, center it so that the mean = 0 (or really close to 0) by subtracting the mean, and then calculate the mean to check your centering.

Answer: I have a quantitative explanatory variable (internetuserate), and I centered it to create a new explanatory variable (internetuserate_zeromean). I also checked that the mean of this centered explanatory variable is zero.

2) Test a linear regression model and summarize the results in a couple of sentences. Make sure to include statistical results (regression coefficients and p-values) in your summary.

Answer: I tested a linear regression model between income per person (response variable) and a centered explanatory variable (internetuserate_zeromean). I included the regression coefficients and p-value in my summary.

WHAT TO SUBMIT: Create a blog entry where you 1) post your program and output, and 2) post a frequency table for your (recoded) categorical explanatory variable or report the mean for your centered explanatory variable. 3) Write a few sentences describing the results of your linear regression analysis.

Answer: This blog entry contains the program, output and mean of the centered explanatory variable. I have also described the results of the linear regression analysis.

theluckystudentperson

Assignment 1

For this assignment, I will follow the template provided by the instructors: ( https://d18ky98rnyall9.cloudfront.net/_a89f644116bccca879c14b59b603165b_C3W1-Assignment-Example—Writing-About-Your-Data.pdf?Expires=1501027200&Signature=DZOlOOHeb9LAQh1G-hn2tzOC6ikZUyHkMmBNiiqk~82219PvoXTXLJxX79fJ9AYzWNflnv0FOJqHNiUeHfaaidFpHTiC9Nvw~w9so58E0g3pw5A7YUkJtl35Ya~X0jHPoMTtIJFj~dWCwZHGVZ6Z2zmrYpC0TexOfEtgYn63i~8_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A)

Sample: The sample is from the Gapminder data set. This data set has important data on all 193 (N=193) UN member countries (http://www.gapminder.org/data/countries-territories-in-gapminder-world/ ) Also, included are 51 other non-UN member countries, listed in Wikipedia in the list of countries. These entities are not fully sovereign countries. This list of entities includes the Vatican, dependent territories, special entities and disputed territories. Two sub-dependencies that were excluded even though they are listed in Wikipedia are: (a) Ascension Island, and (b) Tristan da Cunha. Also included in the Gapminder data set are: (1) 4 French overseas territories – Guadeloupe, Martinique, Reunion and French Guyana, (2) 10 former states, (3) 2 ad-hoc areas – Serbia excluding Kosovo and the Channel Islands (the collective name of Guernsey and Jersey. The total number of countries and territories included in the Gapminder data set is 193+51+4+10+2=260.

Procedure: It is important to note the wide variety of variables (see section on measures below) and income statistics among the different countries in the Gapminder data set. In this assignment, it is not possible to detail the procedures adopted to measure all the different variables in the data set. Instead, I choose to detail the procedures adopted for measuring just one variable: HIV Prevalence rate among ages 15-49. In high-income countries, HIV is mainly found among persons who: (a) use drugs intravenously, and/or (b) sex workers and homosexual men. In these countries, the HIV rate is reported using (1) reporting of new HIV cases by labs/clinicians, and (2) by HIV testing in risk groups. This process of recording new HIV cases does not work well for for low and middle-income countries, presumably because of lack of good governance and available public health resources. In these countries, the HIV rate is calculated by using repeated surveys of different population groups, especially for groups at risk for higher levels of infection. One typical method to estimate HIV rate is to survey and estimate the percentage rate of infection of intravenous drug users. This rate is then multiplied by the estimate of the total percentage of the total population that engages in intravenous drug usage. IT is easy to see that the first part (obtaining the rate of infection among intravenous drup users) is more easily done and accurate than the second part (estimating the fraction of the total population that engages in intravenous drug use). There is a danger that the intravenous drug users who are easily found are typically already infected. Thus, there is a danger of overestimating the total HIV rate in the general population. Another way to estimate HIV rates in such countries is to monitor HIV prevalence rates among pregnant women. These estimates are then extrapolated to the whole population. Another (but more expensive method) is to test a representative sample of men and women of different ages and in different geographical areas, using “two-stage cluster” sampling techniques. These sophisticated and more expensive methods of HIV estimation have led to revising downwards the HIV estimates.

Measures (Variables): The Gapminder data set is an extraordinary collection of data points from across the globe. The (http://www.gapminder.org/data/) current data set lists 519 different variables. The Gapminder data set provided in the codebook made available for this course lists 15 important variables. These variables are : (1) income per person in constant 2000 US dollars for the year 2010; (2) alcohol consumption for ages 15+ for the year 2008; (3) armedforcesrate (i.e., total percentage of total labor force in the armed forces); (4) new breast cancer cases per 100, 000 female residents for the year 2002; (5) total co2 emissions in metric tons from 1751 till 2006; (6) 2007 female labor force participation rate for ages 15+; (7) 2007 total labor force participation for ages 15+ ; (8) HIV prevalence rate for the year 2009 ; (9) 2010 internet users per 100 people; (10) 2011 life expectancy at birth in years; (11) 2010 oil consumption in tonnes per year per person ; (12) 2009 polity score on a scale of -10 [least democratic] to + 10 [ most democratic] ; (13) 2008 residential electricity consumption per person in Kilowatt hour ; (14) 2005 suicide rate per 100, 000 of population ; and (15) 2008 urban rate as a percentage of the total population.

Management of Data:

To manage the data, I use python code, such as

data[‘femaleemployrate’] = pandas.to_numeric(data['femaleemployrate’], errors='coerce’)

data['incomeperperson’] = pandas.to_numeric(data['incomeperperson’], errors='coerce’) data['polityscore’] = pandas.to_numeric(data['polityscore’], errors='coerce’)data['incomeperperson’]=data['incomeperperson’].replace(’ ’, numpy.nan) data['femaleemployrate’]=data['femaleemployrate’].replace(’ ’, numpy.nan) data['polityscore’]=data['polityscore’].replace(’ ’, numpy.nan)data_clean=data.dropna()

In this case, data management consists of: (1) dropping data points with missing data or non-numeric data, and (2) converting data points to numeric.

theluckystudentperson

Specific Questions To Answer:

Step 1:

1) Describe the study population (who or what was studied).

Answer: The population studied are different UN member countries.

2) Report the level of analysis studied (individual, group, or aggregate).

Answer: The dataset comprises an aggregate of countries.

3) Report the number of observations in the data set.

Answer: There are 260 observation points. These include 193 UN member countries and 67 non-countries (e.g., territories, dependencies, etc.)

4) Describe your data analytic sample (the sample you are using for your analyses).

Answer: The data analytic sample used by me are typically middle income and rich countries (per capita income > $4035 ), which are democratic (polity score > 0).

Step 2:

1) Report the study design that generated that data (for example: data reporting, surveys, observation, experiment).

Answer: surveys

2) Describe the original purpose of the data collection.

Answer: Gapminder is focused on promoting a fact-based understanding of the world.

3) Describe how the data were collected.

Answer: By individual countries, typically by surveys.

4) Report when the data were collected.

Answer: The period is indicated in the section under measures (above).

5) Report where the data were collected.

Answer: The data were collected by each country locally.

Step 3:

1) Describe what your explanatory and response variables measured.

Answer: I have focused on understanding how polity scores (explanatory variable) influences per capita income (response variable). In other words, are democratic countries associated with higher per capita incomes. Another question that I have focused on : How female labor force participation (explanatory variable) leads to greater per capita incomes (response variables).

2) Describe the response scales for your explanatory and response variables.

Answer: Both the polity score and per capita income are numeric variables. Thepolity scores range from -10 to +10, while per capita income ranges from approximately $500 to $50,000 or more. We can convert these to categorical variables by separating democratic (polity score > 0) and non-democratic countries (polity score < 0). Similarly, the per capita income can be converted to a categorical variable by separating countries into categories: (a) low-income, (b) middle-income, and (3) rich countries.

3) Describe how you managed your explanatory and response variables.

Answer: I removed data points with missing or non-numeric data. Then, as required for chi-square or Anova or correlation analysis, the variables were left as numeric variables or converted to categorical variables. Also, recoding was done using a mapping if necessary.

Assignment 1

For this assignment, I will follow the template provided by the instructors: ( https://d18ky98rnyall9.cloudfront.net/_a89f644116bccca879c14b59b603165b_C3W1-Assignment-Example---Writing-About-Your-Data.pdf?Expires=1501027200&Signature=DZOlOOHeb9LAQh1G-hn2tzOC6ikZUyHkMmBNiiqk~82219PvoXTXLJxX79fJ9AYzWNflnv0FOJqHNiUeHfaaidFpHTiC9Nvw~w9so58E0g3pw5A7YUkJtl35Ya~X0jHPoMTtIJFj~dWCwZHGVZ6Z2zmrYpC0TexOfEtgYn63i~8_&Key-Pair-Id=APKAJLTNE6QMUY6HBC5A)

Sample: The sample is from the Gapminder data set. This data set has important data on all 193 (N=193) UN member countries (http://www.gapminder.org/data/countries-territories-in-gapminder-world/ ) Also, included are 51 other non-UN member countries, listed in Wikipedia in the list of countries. These entities are not fully sovereign countries. This list of entities includes the Vatican, dependent territories, special entities and disputed territories. Two sub-dependencies that were excluded even though they are listed in Wikipedia are: (a) Ascension Island, and (b) Tristan da Cunha. Also included in the Gapminder data set are: (1) 4 French overseas territories -- Guadeloupe, Martinique, Reunion and French Guyana, (2) 10 former states, (3) 2 ad-hoc areas -- Serbia excluding Kosovo and the Channel Islands (the collective name of Guernsey and Jersey. The total number of countries and territories included in the Gapminder data set is 193+51+4+10+2=260.

Procedure: It is important to note the wide variety of variables (see section on measures below) and income statistics among the different countries in the Gapminder data set. In this assignment, it is not possible to detail the procedures adopted to measure all the different variables in the data set. Instead, I choose to detail the procedures adopted for measuring just one variable: HIV Prevalence rate among ages 15-49. In high-income countries, HIV is mainly found among persons who: (a) use drugs intravenously, and/or (b) sex workers and homosexual men. In these countries, the HIV rate is reported using (1) reporting of new HIV cases by labs/clinicians, and (2) by HIV testing in risk groups. This process of recording new HIV cases does not work well for for low and middle-income countries, presumably because of lack of good governance and available public health resources. In these countries, the HIV rate is calculated by using repeated surveys of different population groups, especially for groups at risk for higher levels of infection. One typical method to estimate HIV rate is to survey and estimate the percentage rate of infection of intravenous drug users. This rate is then multiplied by the estimate of the total percentage of the total population that engages in intravenous drug usage. IT is easy to see that the first part (obtaining the rate of infection among intravenous drup users) is more easily done and accurate than the second part (estimating the fraction of the total population that engages in intravenous drug use). There is a danger that the intravenous drug users who are easily found are typically already infected. Thus, there is a danger of overestimating the total HIV rate in the general population. Another way to estimate HIV rates in such countries is to monitor HIV prevalence rates among pregnant women. These estimates are then extrapolated to the whole population. Another (but more expensive method) is to test a representative sample of men and women of different ages and in different geographical areas, using "two-stage cluster" sampling techniques. These sophisticated and more expensive methods of HIV estimation have led to revising downwards the HIV estimates.

Management of Data:

To manage the data, I use python code, such as

data['femaleemployrate'] = pandas.to_numeric(data['femaleemployrate'], errors='coerce')

data['incomeperperson'] = pandas.to_numeric(data['incomeperperson'], errors='coerce') data['polityscore'] = pandas.to_numeric(data['polityscore'], errors='coerce')data['incomeperperson']=data['incomeperperson'].replace(' ', numpy.nan) data['femaleemployrate']=data['femaleemployrate'].replace(' ', numpy.nan) data['polityscore']=data['polityscore'].replace(' ', numpy.nan)data_clean=data.dropna()

In this case, data management consists of: (1) dropping data points with missing data or non-numeric data, and (2) converting data points to numeric.

Assignment 4

Background: In this assignment, I analyze the relationship between female employment rate and polity score (i.e., whether higher female labor force participation is associated with higher levels of democracy). The question is: does a country become more democratic as the female labor force participation increases? The data analyses show that the answer depends on the income level of the country. For upper middle-income and rich countries, there is an unambiguous (statistically significant) relationship between female labor force participation and democracy -- i.e., higher female labor force participation is associated with more democratic governance. However, for low-income countries, the relationship may be reversed. It is important to stress that the relationship is not statistically significant for low-income countries. I break up the full list of countries into three groups based on per capita incomes: 1) low-income, for countries with per capita income < $1025; 2) lower middle-income countries, for $1026 < per capita incomes < $4035; and 3) high income countries for per capita income > $4035. These cutoffs are based on World Bank statistics ( https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups)

_______________________

Python Code Below:

_______________________

# -*- coding: utf-8 -*-

#Created on Fri Jul 17 10:13:31 2017 #@author: sudhir

import numpy import pandas import seaborn import scipy.stats import matplotlib.pyplot as plt data = pandas.read_csv('SudhirCodebook.csv', low_memory=False)

data['femaleemployrate'] = pandas.to_numeric(data['femaleemployrate'], errors='coerce') data['incomeperperson'] = pandas.to_numeric(data['incomeperperson'], errors='coerce') data['polityscore'] = pandas.to_numeric(data['polityscore'], errors='coerce')

data['incomeperperson']=data['incomeperperson'].replace(' ', numpy.nan) data['femaleemployrate']=data['femaleemployrate'].replace(' ', numpy.nan) data['polityscore']=data['polityscore'].replace(' ', numpy.nan)

data_clean=data.dropna() print (' ') print ('association between femaleemployrate and polityscore for all countries') print (scipy.stats.pearsonr(data_clean['femaleemployrate'], data_clean['polityscore'])) print (' ')

data_clean['incomegrp'] = data_clean.apply (lambda row: incomegrp (row), axis=1) chk1 = data_clean['incomegrp'].value_counts(sort=False, dropna=False) print(chk1)

sub1=data_clean[(data_clean['incomegrp']== 1)] sub2=data_clean[(data_clean['incomegrp']== 2)] sub3=data_clean[(data_clean['incomegrp']== 3)]

print (' ') print ('association between femaleemployrate and polityscore for low income countries') print (scipy.stats.pearsonr(sub1['femaleemployrate'], sub1['polityscore'])) print (' ') print ('association between femaleemployrate and polityscore for lower middle-income countries') print (scipy.stats.pearsonr(sub2['femaleemployrate'], sub2['polityscore'])) print (' ') print ('association between femaleemployrate and polityscore for rich and upper middle-income countries') print (scipy.stats.pearsonr(sub3['femaleemployrate'], sub3['polityscore'])) print (' ')

#scat1 = seaborn.regplot(x="femaleemployrate", y="polityscore", data=sub1) plt.xlabel('Female Employment Rate') plt.ylabel('Polity Score') plt.title('Association Between Female Employment Rate and Polity Score: LOW income countries') #print (scat1) #scat2 = seaborn.regplot(x="femaleemployrate", y="polityscore", fit_reg=False, data=sub2) plt.xlabel('Female Employment Rate') plt.ylabel('Polity Score') plt.title('Association Between Female Employment Rate and Polity Score: LOWER MIDDLE-income countries') #print (scat2) scat3 = seaborn.regplot(x="femaleemployrate", y="polityscore", data=sub3) plt.xlabel('Female Employment Rate') plt.ylabel('Polity Score') plt.title('Association Between Female Employment Rate and Polity Score: RICH and HIGHER MIDDLE-income countries') print (scat3)

__________________________

Python Output Below:

__________________________

WEEK4/Sudhir.Assignment4.FINAL.2.py', wdir='C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/COURSE2/WEEK4')

association between femaleemployrate and polityscore for all countries (0.067377537326284098, 0.40950412939371428)

1 52 2 42 3 58 Name: incomegrp, dtype: int64

association between femaleemployrate and polityscore for low income countries (-0.11201895225879215, 0.42915591325306024)

association between femaleemployrate and polityscore for lower middle-income countries (0.13091959970432215, 0.40856791054395381)

association between femaleemployrate and polityscore for rich and upper middle-income countries (0.52368624515056217, 2.4599057808171647e-05)

C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/COURSE2/WEEK4/Sudhir.Assignment4.FINAL.2.py:41: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy sub3=data_clean[(data_clean['incomegrp']== 3)] Axes(0.125,0.125;0.775x0.755)

________________________________________

Summary:

The relationship between democracy and female labor force participation is clearly moderated/modulated by the income level of countries. For high-income countries, there is a clear statistically significant relationship between female labor force participation and polity score (correlation=0.52, p-value=2.46e-6), but for low-income countries, the relationship is in the opposite direction (i.e., higher female labor force participation is associated with lower democratic governance) and the relationship is not statistically significant (correlation=-0.11, p-value=0.43). For lower middle-income countries, the correlation is positive but not statistically significant (correlation=0.13, p-value=0.41). Also, for the full list of countries, the correlation is very weak and not statistically significant (correlation=0.067, p-value=0.41).

Data Analysis Tools Assignment 3: Correlation

Background: In this assignment, I look at the correlation between female employment rate and per capita income in the gapminder data set. An influential paper by Claudia Goldin ( https://scholar.harvard.edu/files/goldin/files/the_u-shaped_female_labor_force_function_in_economic_development_and_economic_history.pdf) from 1995 found an U-shaped relation between female labor force participation and per capita income. In this assignment, I show this to be true even in 2017. I perform a correlation analysis between female employment rate and per capita income for two groups of countries: (a) Low-Income Countries with per capita income less than $4036, and (b) High-Income countries with per capita income greater than $12476. I find that higher female labor force participation is associated with lower per capita income for low-income countries, but the reverse is true for high-income countries.

_____________________________________________________

Python Code below for High-Income Countries:

______________________________________________________

# -*- coding: utf-8 -*- """ Created on Sun Aug 30 10:50:43 2015; modified on Sat Jun 17, 2017

@author: ldierker; modified by Sudhir """ import pandas import numpy import seaborn import scipy.stats import matplotlib.pyplot as plt

data = pandas.read_csv('SudhirCodeBook.csv', low_memory=False) print ('_______________________________________________________________________________________') print ('OUTPUT PRINTOUT STARTS HERE') print ('Total Number of Countries') print (len(data)) #number of observations (rows)

#setting variables you will be working with to numeric data['incomeperperson'] = pandas.to_numeric(data['incomeperperson'],errors='coerce') data['polityscore'] = pandas.to_numeric(data['polityscore'],errors='coerce') data['femaleemployrate'] = pandas.to_numeric(data['femaleemployrate'],errors='coerce') #For the current 2017 fiscal year, low-income economies are defined as those with a #GNI per capita, calculated using the World Bank Atlas method, of $1,025 or less in 2015; #lower middle-income economies are those with a GNI per capita between $1,026 and $4,035; #upper middle-income economies are those with a GNI per capita between $4,036 and $12,475; #high-income economies are those with a GNI per capita of $12,476 or more. #https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups print ('_______________________________________________________________________________________') print('Rest of Assignment will Focus on High Income Countries') print('High Income Countries have Income Per Person > $12476') print ('_______________________________________________________________________________________') data_clean=data.dropna()

# Sub-sampling entire data set to only pick middle-income countries sub1=data[(data['incomeperperson'] > 12475)] #make a copy of my new subsetted data sub2 = sub1.copy() sub2_clean=sub2.dropna() print ('Total Number of High-Income Countries') print (len(sub2)) #number of observations (rows)

scat1=seaborn.regplot(x="femaleemployrate",y="incomeperperson",fit_reg=True, data=sub2) plt.xlabel('Female Employment Rate') plt.ylabel('Income Per Person') plt.title('Scatterplot for High-Income Countries: Female Employment Rate & Income Per Person') print('Correlation for High-Income Countries: Female Employment Rate & Income Per Person') print(scipy.stats.pearsonr(sub2_clean['femaleemployrate'], sub2_clean['incomeperperson'])) # bug fix for display formats to avoid run time errors - put after code for loading data above pandas.set_option('display.float_format', lambda x:'%f'%x) print ('OUTPUT PRINTOUT ENDS HERE') _________________________________________________________

Python Output for High-Income Countries Below:

_________________________________________________________

runfile('C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/COURSE2/WEEK3/SudhirAssignment3.HIGHINCOME.py', wdir='C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/COURSE2/WEEK3') _______________________________________________________________________________________ OUTPUT PRINTOUT STARTS HERE Total Number of Countries 213 _______________________________________________________________________________________ Rest of Assignment will Focus on High Income Countries High Income Countries have Income Per Person > $12476 _______________________________________________________________________________________ Total Number of High-Income Countries 41 Correlation for High-Income Countries: Female Employment Rate & Income Per Person (0.48958067333164995, 0.0081866599492884679) OUTPUT PRINTOUT ENDS HERE

_____________________________________________________

Python Code for Low-Income Countries Below:

______________________________________________________

# -*- coding: utf-8 -*- """ Created on Sun Aug 30 10:50:43 2015; modified on Sat Jun 17, 2017

@author: ldierker; modified by Sudhir """ import pandas import numpy import seaborn import scipy.stats import matplotlib.pyplot as plt

#setting variables you will be working with to numeric data['incomeperperson'] = pandas.to_numeric(data['incomeperperson'],errors='coerce') data['polityscore'] = pandas.to_numeric(data['polityscore'],errors='coerce') data['femaleemployrate'] = pandas.to_numeric(data['femaleemployrate'],errors='coerce') #For the current 2017 fiscal year, low-income economies are defined as those with a #GNI per capita, calculated using the World Bank Atlas method, of $1,025 or less in 2015; #lower middle-income economies are those with a GNI per capita between $1,026 and $4,035; #upper middle-income economies are those with a GNI per capita between $4,036 and $12,475; #high-income economies are those with a GNI per capita of $12,476 or more. #https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups print ('_______________________________________________________________________________________') print('Rest of Assignment will Focus on Low Income Countries') print('Low Income Countries have Income Per Person < $4036') print ('_______________________________________________________________________________________') data_clean=data.dropna()

# Sub-sampling entire data set to only pick middle-income countries sub1=data[(data['incomeperperson'] < 4036)] #make a copy of my new subsetted data sub2 = sub1.copy() sub2_clean=sub2.dropna() print ('Total Number ofLow Income Countries') print (len(sub2)) #number of observations (rows)

scat1=seaborn.regplot(x="femaleemployrate",y="incomeperperson",fit_reg=True, data=sub2) plt.xlabel('Female Employment Rate') plt.ylabel('Income Per Person') plt.title('Scatterplot for Low-Income Countries: Female Employment Rate & Income Per Person') print('Correlation for Low-Income Countries: Female Employment Rate & Income Per Person') print(scipy.stats.pearsonr(sub2_clean['femaleemployrate'], sub2_clean['incomeperperson'])) # bug fix for display formats to avoid run time errors - put after code for loading data above pandas.set_option('display.float_format', lambda x:'%f'%x) print ('OUTPUT PRINTOUT ENDS HERE')

________________________________________________________

Python Output For Low-Income Countries Below:

________________________________________________________

runfile('C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/COURSE2/WEEK3/SudhirAssignment3.LOWINCOME.py', wdir='C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/COURSE2/WEEK3') _______________________________________________________________________________________ OUTPUT PRINTOUT STARTS HERE Total Number of Countries 213 _______________________________________________________________________________________ Rest of Assignment will Focus on Low Income Countries Low Income Countries have Income Per Person < $4036 _______________________________________________________________________________________ Total Number ofLow Income Countries 108 Correlation for Low-Income Countries: Female Employment Rate & Income Per Person (-0.34162797700474612, 0.00075201236232819309) OUTPUT PRINTOUT ENDS HERE

__________________________________________________________

Summary: The output above shows that: (a) low-income countries have a negative correlation [correlation coefficient = -0.34] between female labor force participation and per capita income [i.e, higher female labor force participation is associated with lower per capita incomes], but (b) high-income countries have a positive correlation [correlation coefficient = 0.49] between female labor force participation and per capita incomes [i.e, higher female labor force participation is associated with higher per capita incomes]. For both groups, the p-value is very small [p-value=0.00075 for low-income countries and p-value=0.0082 for high-income countries], which means that the associations are statistically significant.

Assignment 2

Data Analysis Tools, Assignment 2 The Chi-square test is used when the explanatory and response variables are both categorical. In this assignment, I look at the dependence if any between Per Capita Income and Democratic Governance. I use the GapMinder data set. The countries with Polity Score > 5 are examined, i.e, countries that are highly democratic. I also divide the countries into two categories: (a) upper middle- income and rich countries, and (b) poor and lower middle-income countries. The cutoff for a country to qualify as upper middle- income and rich is per capital income > $4035 [ #https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank- country-and-lending-groups ]. I introduce two new variables: (1) INCOME, which is 1 for upper middle-income and rich countries and 0 for poor and lower middle-income countries, and (2) DEMSCORE, which ranges from 1 to 5 when Polity Score ranges from 6 to ten. The chi-square test is performed on the whole group of countries initially. The original statistical test is significant, so I then perform post-hoc paired comparisons on the ten pairs separately. The Bonferroni adjustment is used to compute a significance level of (p=0.05/10 or 0.005, since n=10 in this case). I find that the null test can be safely rejected for four of the ten pairings (i.e., there is a statistically significant relationship between INCOME and DEMSCORE), but in six of the ten pairings, the null hypothesis can NOT be rejected (i.e., there is not a statistically significant relationship between INCOME and DEMSCORE). ___________________________________________________ _______________ Python Code: # -*- coding: utf-8 -*- """ Created on Sun Aug 30 10:50:43 2015; modified on Sat Jun 17, 2017 @author: ldierker; modified by Sudhir """ import pandas import numpy import seaborn import scipy.stats import matplotlib.pyplot as plt data = pandas.read_csv('SudhirCodeBook.csv', low_memory=False) print ('_______________________________________________________________________________________') print ('OUTPUT PRINTOUT STARTS HERE') print ('Total Number of Countries') print (len(data)) #number of observations (rows) print ('Total Number of Variables') print (len(data.columns)) # number of variables (columns) #setting variables you will be working with to numeric data['incomeperperson'] = pandas.to_numeric(data['incomeperperson'],errors='coerce') data['polityscore'] = pandas.to_numeric(data['polityscore'],errors='coerce') data['femaleemployrate'] = pandas.to_numeric(data['femaleemployrate'],errors='coerce') print ('_______________________ _______________________________________________________________') print('Rest of Assignment will Focus on Highly Democratic Countries') print('Highly Democratic Countries Have PolityScores > 5') print ('_______________________________________________________________________________________') # Sub-sampling entire data set to only pick middle-income countries sub1=data[(data['polityscore'] > 5)] #make a copy of my new subsetted data sub2 = sub1.copy() # SETTING MISSING DATA sub2['femaleemployrate']=sub2['femaleemployrate'].replace(-9.0, numpy.nan) sub2['polityscore']=sub2['polityscore'].replace(-9.0, numpy.nan) sub2['incomeperperson']=sub2['incomeperperson'].replace(-9.0, numpy.nan) # Introduce a New Variable, "DEMSCORE" recode1 = { 6: 1, 7: 2, 8: 3, 9: 4, 10: 5} sub2['DEMSCORE']=sub2['polityscore'].map(recode1) print ('_______________________________________________________________________________________') print('Upper Middle-Income and Rich Countries have Per Capita Income > $4035') print('New Variable "INCOME"= 0 for poor and lower-middle income countries') print('New Variable "INCOME"= 1 for upper-middle income and rich countries' ) print ('_______________________________________________________________________________________') def INCOME (row): if row['incomeperperson'] < 4036.0 : return 0 else : return 1 sub2['INCOME'] = sub2.apply (lambda row: INCOME (row),axis=1) #sub2['DEMSCORE'] = pandas.cut(sub2.polityscore,[-10, -6, -2, 2, 6, 10]) ct1=pandas.crosstab(sub2['INCOME'], sub2['DEMSCORE']) print (ct1) colsum=ct1.sum(axis=0) colpct=ct1/colsum print(colpct) print('chi-square, p value, expected counts') cs1=scipy.stats.chi2_contingency(ct1) print(cs1) sub2['DEMSCORE']=sub2['DEMSCORE'].astype('category') sub2['INCOME'] = pandas.to_numeric(sub2['INCOME'],errors='coerce') seaborn.factorplot(x='DEMSCORE', y='INCOME', data=sub2,kind="bar", ci=None) plt.xlabel('DEMSCORE') plt.ylabel('INCOME') recode12 = {1: 1, 2: 2} sub2['DEM1v2']=sub2['DEMSCORE'].map(recode12) ct12=pandas.crosstab(sub2['INCOME'], sub2['DEM1v2']) print(ct12) colsum=ct12.sum(axis=0) colpct=ct12/colsum print(colpct) print('chi-square value, p value, expected counts') cs12=scipy.stats.chi2_contingency(ct12) print(cs12) recode13 = {1: 1, 3: 3} sub2['DEM1v3']=sub2['DEMSCORE'].map(recode13) ct13=pandas.crosstab(sub2['INCOME'], sub2['DEM1v3']) print(ct13) colsum=ct13.sum(axis=0) colpct=ct13/colsum print(colpct) print('chi-square value, p value, expected counts') cs13=scipy.stats.chi2_contingency(ct13) print(cs13) recode14 = {1: 1, 4: 4} sub2['DEM1v4']=sub2['DEMSCORE'].map(recode14) ct14=pandas.crosstab(sub2['INCOME'], sub2['DEM1v4']) print(ct14) colsum=ct14.sum(axis=0) colpct=ct14/colsum print(colpct) print('chi-square value, p value, expected counts') cs14=scipy.stats.chi2_contingency(ct14) print(cs14) recode15 = {1: 1, 5: 5} sub2['DEM1v5']=sub2['DEMSCORE'].map(recode15) ct15=pandas.crosstab(sub2['INCOME'], sub2['DEM1v5']) print(ct15) colsum=ct15.sum(axis=0) colpct=ct15/colsum print(colpct) print('chi-square value, p value, expected counts') cs15=scipy.stats.chi2_contingency(ct15) print(cs15) recode23 = {2: 2, 3: 3} sub2['DEM2v3']=sub2['DEMSCORE'].map(recode23) ct23=pandas.crosstab(sub2['INCOME'], sub2['DEM2v3']) print(ct23) colsum=ct23.sum(axis=0) colpct=ct23/colsum print(colpct) print('chi-square value, p value, expected counts') cs23=scipy.stats.chi2_contingency(ct23) print(cs23) recode24 = {2: 2, 4: 4} sub2['DEM2v4']=sub2['DEMSCORE'].map(recode24) ct24=pandas.crosstab(sub2['INCOME'], sub2['DEM2v4']) print(ct24) colsum=ct24.sum(axis=0) colpct=ct24/colsum print(colpct) print('chi-square value, p value, expected counts') cs24=scipy.stats.chi2_contingency(ct24) print(cs24) recode25 = {2: 2, 5: 5} sub2['DEM2v5']=sub2['DEMSCORE'].map(recode25) ct25=pandas.crosstab(sub2['INCOME'], sub2['DEM2v5']) print(ct25) colsum=ct25.sum(axis=0) colpct=ct25/colsum print(colpct) print('chi-square value, p value, expected counts') cs25=scipy.stats.chi2_contingency(ct25) print(cs25) recode34 = {3: 3, 4: 4} sub2['DEM3v4']=sub2['DEMSCORE'].map(recode34) ct34=pandas.crosstab(sub2['INCOME'], sub2['DEM3v4']) print(ct34) colsum=ct34.sum(axis=0) colpct=ct34/colsum print(colpct) print('chi-square value, p value, expected counts') cs34=scipy.stats.chi2_contingency(ct34) print(cs34) recode35 = {3: 3, 5: 5} sub2['DEM3v5']=sub2['DEMSCORE'].map(recode35) ct35=pandas.crosstab(sub2['INCOME'], sub2['DEM3v5']) print(ct35) colsum=ct35.sum(axis=0) colpct=ct35/colsum print(colpct) print('chi-square value, p value, expected counts') cs35=scipy.stats.chi2_contingency(ct35) print(cs35) recode45 = {4: 4, 5: 5} sub2['DEM4v5']=sub2['DEMSCORE'].map(recode45) ct45=pandas.crosstab(sub2['INCOME'], sub2['DEM4v5']) print(ct45) colsum=ct45.sum(axis=0) colpct=ct45/colsum print(colpct) print('chi-square value, p value, expected counts') cs45=scipy.stats.chi2_contingency(ct45) print(cs45) #upper-case all DataFrame column names - place afer code for loading data aboave data.columns = map(str.upper, data.columns) # bug fix for display formats to avoid run time errors - put after code for loading data above pandas.set_option('display.float_format', lambda x:'%f'%x) print ('OUTPUT PRINTOUT ENDS HERE') print ('__________________________________________________ ____________________________________') PYTHON CODE OUTPUT: runfile('C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/COURSE2/WEEK2/Sudhir Assignment2.14.py', wdir='C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/COURSE2/WEEK2') _________________________________________________ _____________________________________ OUTPUT PRINTOUT STARTS HERE Total Number of Countries 213 Total Number of Variables 6 _________________________________________________________________________ _____________ Rest of Assignment will Focus on Highly Democratic Countries Highly Democratic Countries Have PolityScores > 5 _______________________________________________________________________________________ _______________________________________________________________________________________ Upper Middle-Income and Rich Countries have Per Capita Income > $4035 New Variable "INCOME"= 0 for poor and lower-middle income countries New Variable "INCOME"= 1 for upper-middle income and rich countries _______________________________________________________________________________________ DEMSCORE 1 2 3 4 5 INCOME 0 9 11 10 11 1 1 1 2 9 4 32 DEMSCORE 1 2 3 4 5 INCOME 0 0.9 0.846154 0.526316 0.733333 0.030303 1 0.1 0.153846 0.473684 0.266667 0.969697 chi-square, p value, expected counts (44.870704952941793, 4.2298182800549206e-09, 4, array([[ 4.66666667, 6.06666667, 8.86666667, 7. , 15.4 ], [ 5.33333333, 6.93333333, 10.13333333, 8. , 17.6 ]])) DEM1v2 1.0 2.0 INCOME 0 9 11 1 1 2 DEM1v2 1.0 2.0 INCOME 0 0.9 0.846154 1 0.1 0.153846 chi-square value, p value, expected counts (0.059711538461538413, 0.80695244786968012, 1, array([[ 8.69565217, 11.30434783], [ 1.30434783, 1.69565217]])) DEM1v3 1.0 3.0 INCOME 0 9 10 1 1 9 DEM1v3 1.0 3.0 INCOME 0 0.9 0.526316 1 0.1 0.473684 chi-square value, p value, expected counts (2.5644113573407199, 0.10929323221158412, 1, array([[ 6.55172414, 12.44827586], [ 3.44827586, 6.55172414]])) DEM1v4 1.0 4.0 INCOME 0 9 11 1 1 4 DEM1v4 1.0 4.0 INCOME 0 0.9 0.733333 1 0.1 0.266667 chi-square value, p value, expected counts (0.26041666666666663, 0.6098340436734595, 1, array([[ 8., 12.], [ 2., 3.]])) DEM1v5 1.0 5.0 INCOME 0 9 1 1 1 32 DEM1v5 1.0 5.0 INCOME 0 0.9 0.030303 1 0.1 0.969697 chi-square value, p value, expected counts (27.833615702479332, 1.3220892501034591e-07, 1, array([[ 2.3255814, 7.6744186], [ 7.6744186, 25.3255814]])) DEM2v3 2.0 3.0 INCOME 0 11 10 1 2 9 DEM2v3 2.0 3.0 INCOME 0 0.846154 0.526316 1 0.153846 0.473684 chi-square value, p value, expected counts (2.2259845417740154, 0.13570612619827002, 1, array([[ 8.53125, 12.46875], [ 4.46875, 6.53125]])) DEM2v4 2.0 4.0 INCOME 0 11 11 1 2 4 DEM2v4 2.0 4.0 INCOME 0 0.846154 0.733333 1 0.153846 0.266667 chi-square value, p value, expected counts (0.069619269619269647, 0.79189193141124448, 1, array([[ 10.21428571, 11.78571429], [ 2.78571429, 3.21428571]])) DEM2v5 2.0 5.0 INCOME 0 11 1 1 2 32 DEM2v5 2.0 5.0 INCOME 0 0.846154 0.030303 1 0.153846 0.969697 chi-square value, p value, expected counts (28.101912793089262, 1.1509205162825137e-07, 1, array([[ 3.39130435, 8.60869565], [ 9.60869565, 24.39130435]])) DEM3v4 3.0 4.0 INCOME 0 10 11 1 9 4 DEM3v4 3.0 4.0 INCOME 0 0.526316 0.733333 1 0.473684 0.266667 chi-square value, p value, expected counts (0.77085020242914948, 0.37995416385495884, 1, ar ay([[ 11.73529412, 9.26470588], [ 7.26470588, 5.73529412]])) DEM3v5 3.0 5.0 INCOME 0 10 1 1 9 32 DEM3v5 3.0 5.0 INCOME 0 0.526316 0.030303 1 0.473684 0.969697 chi-square value, p value, expected counts (14.936504736948196, 0.00011119053484556204, 1, array([[ 4.01923077, 6.98076923], [ 14.98076923, 26.01923077]])) DEM4v5 4.0 5.0 INCOME 0 11 1 1 4 32 DEM4v5 4.0 5.0 INCOME 0 0.733333 0.030303 1 0.266667 0.969697 chi-square value, p value, expected counts (23.563636363636363, 1.2084991155284946e-06, 1, array([[ 3.75, 8.25], [ 11.25, 24.75]])) OUTPUT PRINTOUT ENDS HERE

Summary: The chi-square test was performed on the relationship between per capital income and polity score for countries that have polity score > 5 (highly democratic). The null test was rejected, since the p-value is 4.23E-9, which is less than 0.05. Then, I introduce two new categorical variables, (INCOME which is 1 for rich and upper middle-income countries and 0 for poor countries) and DEMSCORE, which ranges from 1 to 5 when polity scores range from 6 to 10. There are ten pairings in this test, so I run post-hoc paired comparisons. The Bonferroni adjustment requires that the p-value must be less that 0.05/10 (i.e., 0.005) for the null hypothesis to be rejected. I find that the null hypothesis can be rejected for four pairings out of ten. The four pairings for which the null hypothesis can be rejected have these DEMSCORES (1 and 5; 2 and 5; 3 and 5; 4 and 5). Thus, for the category (DEMSCORE=5), we can safely state that there is a relationship between INCOME and DEMSCORE. On the other hand, the null-hypothesis can not be rejected for the six pairings out of ten for which DEMSCORES=1 and 2, 1 and 3, 1 and 4, 2 and 3, 2 and 4, 3 and 4. Thus, for these pairings, there is no relationship between INCOME and DEMSCORE.

Sudhir’s Assignment 4

In this assignment, I continue with the program used in assignment 3. In the first half of the python code, I use the gapminder data set to look at democratic lower middle-income countries. Democratic countries are defined as having positive ( greater than or equal to 1) polity score. Lower middle-income countries have $1026 < income per person < $4036. I use this data set to look at the important statistics pertaining to democratic score, which is a new variable derived from polity score. The democratic score ranges from 10 to 100 for democratic countries, with higher values denoting more democracy. In the second half of the python code, I look at univariate and bivariate plots that map the relationship between (i) income per person and urban rate, (ii) income per person and internet use rate, (iii) income per person and hiv rate. The code and output are appended below. The first part of the Python code is below:

--------------------------------------------------------------------------------

# -*- coding: utf-8 -*- """ Created on Sun Aug 30 10:50:43 2015; modified on Sat Jun 17, 2017

@author: ldierker; modified by Sudhir """ import pandas import numpy import seaborn import matplotlib.pyplot as plt # any additional libraries would be imported here data = pandas.read_csv('SudhirCodeBook.csv', low_memory=False) print ('_____________________________________________________') print ('OUTPUT PRINTOUT STARTS HERE') print ('Total Number of Countries') print (len(data)) #number of observations (rows) print ('Total Number of Variables') print (len(data.columns)) # number of variables (columns) #setting variables you will be working with to numeric print ('The Variables in this Database:') print('(1) country, (2) incomeperperson, (3) polityscore') print('(4) femaleemployeerate, (5) internetuserate, and (6) urbanrate') data['incomeperperson'] = data['incomeperperson'].convert_objects(convert_numeric=True) data['polityscore'] = data['polityscore'].convert_objects(convert_numeric=True) data['femaleemployrate'] = data['femaleemployrate'].convert_objects(convert_numeric=True) data['internetuserate'] = data['internetuserate'].convert_objects(convert_numeric=True) data['urbanrate'] = data['urbanrate'].convert_objects(convert_numeric=True)

#counts and percentages (i.e. frequency distributions) for each variable) print ('Examples of Sorting by Frequency and by Values') print ('List Polity Scores for Entire Data Set, Sorted by Frequency') c2 = data['polityscore'].value_counts(sort=True) print(c2) print ('List Polity Scores for Entire Data Set, sorted by Values') ct2= data.groupby('polityscore').size().sort_index(ascending=True) print (ct2) ######################################################################################## # REST OF ASSIGNMENT FOCUSES ON DEMOCRATIC (Polity Score > 0 ) # LOWER MIDDLE-INCOME COUNTRIES ($1025 < GNI < $4036) # #For the current 2017 fiscal year, low-income economies are defined as those with a #GNI per capita, calculated using the World Bank Atlas method, of $1,025 or less in 2015; #lower middle-income economies are those with a GNI per capita between $1,026 and $4,035; #upper middle-income economies are those with a GNI per capita between $4,036 and $12,475; #high-income economies are those with a GNI per capita of $12,476 or more. #https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups # Polity scores range from -10 for undemocratic to +10 for highly democratic countries. # # REST OF ASSIGNMENT FOCUSES ON DEMOCRATIC (Polity Score > 0 ) # LOWER MIDDLE-INCOME COUNTRIES ($1025 < GNI < $4036) ########################################################################################

print ('_____________________________________________________') print('Rest of Assignment will Focus on Highly Democratic Lower Mid-Income Countries') print('Highly Democratic Countries Have Polity Scores > 0 ') print('Lower Middle-Income Countries have $1025 < Per Capital Income < $4036 ') print ('_____________________________________________________') # Sub-sampling entire data set to only pick up Highly Democratic lower middle-income countries sub1=data[(data['incomeperperson']>1025) & (data['incomeperperson']<4036) & (data['polityscore']> 0)] #make a copy of my new subsetted data sub2 = sub1.copy() # frequency distritions on new sub2 data frame # First Fill in Blanks with NaN. Then print out original counts print ('Original Counts for FemaleEmployRate For Lower Democratic Lower Mid-Income Countries, Sorted by Values') c5 = sub2['femaleemployrate'].value_counts(sort=True,dropna=False).sort_index(ascending=True) print(c5)

sub2['femaleemployrate']=sub2['femaleemployrate'].replace(-9.0, numpy.nan) sub2['femaleemployrate'].fillna(-99.0,inplace=True) print ('New Counts for FemaleEmployRate For Democratic Lower Mid-Income Countries, Sorted by Values') c55 = sub2['femaleemployrate'].value_counts(sort=True,dropna=False).sort_index(ascending=True) print(c55) #sub2['femaleemployeerate']=sub2['femaleemployrate'].replace(-99.0,numpy.nan) #c2=sub2['femaleemployrate'].value_counts(sort=True,dropna=False) #print(c2) #chk2=sub2['femaleemployrate'].value_counts(sort=True,dropna=True) #print(chk2) ds2=sub2['femaleemployrate'].describe() print('Statistics for Female Employment Rate') print(ds2) print('____________________________________________') print('I will now delete data points for which the female Employment Rate is Missing or NaN' ) sub1=data[(data['incomeperperson']>1025) & (data['incomeperperson']<4036) & (data['polityscore']> 0) & (data['femaleemployrate']>0)] sub2 = sub1.copy() recode1={1:10,2:20,3:30,4:40,5:50,6:60,7:70,8:80,9:90,10:100} print('Introducing Democratic Score, which is mapped from Polity Score') print('Polity Score Ranges from 1 to 10, and Democratic Score Ranges from 10 to 100') sub2['democraticscore']=sub2['polityscore'].map(recode1)

print('Introducing New Variable: Governance Score') print('GOVERNANCE SCORE=Democratic Score+Female Employment Rate+Urban Rate') sub2['GOVERNANCESCORE']=sub2['democraticscore'] + sub2['femaleemployrate']+sub2['urbanrate'] c3=sub2['GOVERNANCESCORE'].value_counts().sort_index(ascending=True) print ('GOVERNANCE SCORES') print(c3) sub2['INCOMEGROUP']=pandas.qcut(sub2.incomeperperson, 5, labels=["1=0-20","2=20-40","3=40-60","4=60-80","5=80-100"]) c4 = sub2['INCOMEGROUP'].value_counts(sort=False, dropna=True) print(c4) print('I break up the Middle-Income group of countries into four sub-groups') print('The four sub-groups have per capital income: $1025-$1525,$1525-$2025,$2025-$2525,$2525-$3025,$3025-3525,$3525-$4036') print('INCOME-6 categories: 1025-1525, 1525-2025, 2025-2525, 2525-3025, 3025-3525, 3525-4036') sub2['INCOMEGROUP']=pandas.cut(sub2.incomeperperson,[1025,1525,2025,2525,3025,3525,4036]) c5=sub2['INCOMEGROUP'].value_counts(sort=True,dropna=True) print(c5) print('printing crosstabs') print(pandas.crosstab(sub2['INCOMEGROUP'],sub2['incomeperperson']))

print('____________________________________________')

print('Plotting Income Per Person Univariate Bar Graph') ax = seaborn.countplot(x="INCOMEGROUP", data=sub2) plt.xlabel('income per person') plt.title('INCOME PER PERSON') print('BASIC STATISTICS ABOUT INCOME PER PERSON GROUP') ds10 = sub2['INCOMEGROUP'].describe() print(ds10) c10 = sub2.groupby('INCOMEGROUP').size() print(c10)

print('mode of INCOME GROUP') mode1 = sub2['INCOMEGROUP'].mode() print(mode1)

print('mean of income per person') mean1 = sub2['incomeperperson'].mean() print(mean1)

print('std of income per person') std1 = sub2['incomeperperson'].std() print(std1)

print('min of income per person') min1 = sub2['INCOMEGROUP'].min() print(min1)

print('max of income per person') max1 = sub2['INCOMEGROUP'].max() print(max1)

print('median of income per person') median1 = sub2['incomeperperson'].median() print(median1)

c1 = sub2.groupby('INCOMEGROUP').size() print(c1)

p1 = sub2.groupby('INCOMEGROUP').size() * 100 / len (data) print(p1)

print('__________________________________________________') print('LOOKING AT POLITY SCORE') print('Plotting POLITY SCORE Bar Graph') #ax = seaborn.countplot(y="democraticscore", data=sub2) #plt.xlabel('DEMOCRATIC SCORE') #plt.title('DEMOCRATIC SCORE OF LOWER MIDDLE-INCOME DEMOCRATIC NATIONS') print('BASIC STATISTICS ABOUT INCOME PER PERSON GROUP') ds10 = sub2['democraticscore'].describe() print(ds10) c10 = sub2.groupby('democraticscore').size() print(c10)

print('mode of DEMOCRATIC SCORE') mode1 = sub2['democraticscore'].mode() print(mode1)

print('mean of DEMOCRATIC SCORE') mean1 = sub2['democraticscore'].mean() print(mean1)

print('std of democratic score') std1 = sub2['democraticscore'].std() print(std1)

print('min of democraticscore') min1 = sub2['democraticscore'].min() print(min1)

print('max of democraticscore') max1 = sub2['democraticscore'].max() print(max1)

print('median of democraticscore') median1 = sub2['democraticscore'].median() print(median1)

c1 = sub2.groupby('democraticscore').size() print(c1)

p1 = sub2.groupby('democraticscore').size() * 100 / len (data) print(p1)

print ('OUTPUT PRINTOUT ENDS HERE') print ('_____________________________________________________') #upper-case all DataFrame column names - place afer code for loading data aboave data.columns = map(str.upper, data.columns) # bug fix for display formats to avoid run time errors - put after code for loading data above pandas.set_option('display.float_format', lambda x:'%f'%x) print('END OF OUTPUT FROM FIRST HALF OF PYTHON CODE FOR ASSIGNMENT 3') print('_______________________________________________________________')

Now, the output from the first half of the python code:

-----------------------------------------------------------------------------------

runfile('C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/WEEK4/SudhirAssignment4.Part1.Part1.1.py', wdir='C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/WEEK4') _____________________________________________________ OUTPUT PRINTOUT STARTS HERE Total Number of Countries 213 Total Number of Variables 6 The Variables in this Database: (1) country, (2) incomeperperson, (3) polityscore (4) femaleemployeerate, (5) internetuserate, and (6) urbanrate Examples of Sorting by Frequency and by Values List Polity Scores for Entire Data Set, Sorted by Frequency 10.000000 33 8.000000 19 9.000000 15 7.000000 13 -7.000000 12 6.000000 10 5.000000 7 -4.000000 6 0.000000 6 -3.000000 6 -2.000000 5 -1.000000 4 -9.000000 4 4.000000 4 1.000000 3 -6.000000 3 2.000000 3 -8.000000 2 3.000000 2 -5.000000 2 -10.000000 2 Name: polityscore, dtype: int64 List Polity Scores for Entire Data Set, sorted by Values polityscore -10.000000 2 -9.000000 4 -8.000000 2 -7.000000 12 -6.000000 3 -5.000000 2 -4.000000 6 -3.000000 6 -2.000000 5 -1.000000 4 0.000000 6 1.000000 3 2.000000 3 3.000000 2 4.000000 4 5.000000 7 6.000000 10 7.000000 13 8.000000 19 9.000000 15 10.000000 33 dtype: int64 _____________________________________________________ Rest of Assignment will Focus on Highly Democratic Lower Mid-Income Countries Highly Democratic Countries Have Polity Scores > 0 Lower Middle-Income Countries have $1025 < Per Capital Income < $4036 _____________________________________________________ Original Counts for FemaleEmployRate For Lower Democratic Lower Mid-Income Countries, Sorted by Values 26.799999 1 31.700001 1 34.200001 2 34.299999 1 37.299999 1 39.200001 1 39.900002 1 41.799999 1 42.099998 2 43.799999 1 44.000000 1 44.099998 1 44.799999 1 45.500000 1 46.799999 1 47.500000 1 48.500000 1 49.400002 1 51.299999 1 53.799999 1 54.900002 1 59.799999 1 61.599998 1 65.000000 1 65.300003 1 nan 2 Name: femaleemployrate, dtype: int64 New Counts for FemaleEmployRate For Democratic Lower Mid-Income Countries, Sorted by Values -99.000000 2 26.799999 1 31.700001 1 34.200001 2 34.299999 1 37.299999 1 39.200001 1 39.900002 1 41.799999 1 42.099998 2 43.799999 1 44.000000 1 44.099998 1 44.799999 1 45.500000 1 46.799999 1 47.500000 1 48.500000 1 49.400002 1 51.299999 1 53.799999 1 54.900002 1 59.799999 1 61.599998 1 65.000000 1 65.300003 1 Name: femaleemployrate, dtype: int64 Statistics for Female Employment Rate count 29.000000 mean 35.575862 std 38.482256 min -99.000000 25% 37.299999 50% 44.000000 75% 49.400002 max 65.300003 Name: femaleemployrate, dtype: float64 ____________________________________________ I will now delete data points for which the female Employment Rate is Missing or NaN Introducing Democratic Score, which is mapped from Polity Score Polity Score Ranges from 1 to 10, and Democratic Score Ranges from 10 to 100 Introducing New Variable: Governance Score GOVERNANCE SCORE=Democratic Score+Female Employment Rate+Urban Rate GOVERNANCE SCORES 104.380002 1 114.300001 1 116.920001 1 130.179999 1 134.139999 1 138.320000 1 148.060001 1 149.259999 1 152.080001 1 161.080000 1 161.240000 1 166.639999 1 172.580000 1 175.559998 1 178.819998 1 183.699999 1 185.039999 1 185.499999 1 187.380002 1 188.039999 1 190.800000 1 191.719999 1 197.179998 1 199.400002 1 203.199998 1 205.600003 1 221.199999 1 Name: GOVERNANCESCORE, dtype: int64 1=0-20 6 2=20-40 5 3=40-60 5 4=60-80 5 5=80-100 6 Name: INCOMEGROUP, dtype: int64 I break up the Middle-Income group of countries into four sub-groups The four sub-groups have per capital income: $1025-$1525,$1525-$2025,$2025-$2525,$2525-$3025,$3025-3525,$3525-$4036 INCOME-6 categories: 1025-1525, 1525-2025, 2025-2525, 2525-3025, 3025-3525, 3525-4036 (1025, 1525] 11 (2525, 3025] 6 (1525, 2025] 4 (3525, 4036] 2 (3025, 3525] 2 (2025, 2525] 2 Name: INCOMEGROUP, dtype: int64 printing crosstabs incomeperperson 1036.830725 1143.831514 1144.102193 1200.652075 \ INCOMEGROUP (1025, 1525] 1 1 1 1 (1525, 2025] 0 0 0 0 (2025, 2525] 0 0 0 0 (2525, 3025] 0 0 0 0 (3025, 3525] 0 0 0 0 (3525, 4036] 0 0 0 0

incomeperperson 1232.794137 1258.762596 1295.742686 1324.194906 \ INCOMEGROUP (1025, 1525] 1 1 1 1 (1525, 2025] 0 0 0 0 (2025, 2525] 0 0 0 0 (2525, 3025] 0 0 0 0 (3025, 3525] 0 0 0 0 (3525, 4036] 0 0 0 0

incomeperperson 1326.741757 1383.401869 ... 2549.558474 \ INCOMEGROUP ... (1025, 1525] 1 1 ... 0 (1525, 2025] 0 0 ... 0 (2025, 2525] 0 0 ... 0 (2525, 3025] 0 0 ... 1 (3025, 3525] 0 0 ... 0 (3525, 4036] 0 0 ... 0

incomeperperson 2557.433638 2636.787800 2667.246710 2712.517199 \ INCOMEGROUP (1025, 1525] 0 0 0 0 (1525, 2025] 0 0 0 0 (2025, 2525] 0 0 0 0 (2525, 3025] 1 1 1 1 (3025, 3525] 0 0 0 0 (3525, 4036] 0 0 0 0

incomeperperson 2923.144355 3180.430612 3233.423780 3665.348369 \ INCOMEGROUP (1025, 1525] 0 0 0 0 (1525, 2025] 0 0 0 0 (2025, 2525] 0 0 0 0 (2525, 3025] 1 0 0 0 (3025, 3525] 0 1 1 0 (3525, 4036] 0 0 0 1

incomeperperson 3745.649852 INCOMEGROUP (1025, 1525] 0 (1525, 2025] 0 (2025, 2525] 0 (2525, 3025] 0 (3025, 3525] 0 (3525, 4036] 1

[6 rows x 27

____________________________________________ Counts for Income Per Person for Democratic Lower Middle-Income Countries, Sorted by Values 1036.830725 1 1143.831514 1 1144.102193 1 1200.652075 1 1232.794137 1 1258.762596 1 1295.742686 1 1324.194906 1 1326.741757 1 1383.401869 1 1392.411829 1 1621.177078 1 1728.020976 1 1860.753895 1 1914.996551 1 2221.185664 1 2231.993335 1 2549.558474 1 2557.433638 1 2636.787800 1 2667.246710 1 2712.517199 1 2923.144355 1 3180.430612 1 3233.423780 1 3665.348369 1 3745.649852 1 Name: incomeperperson, dtype: int64 Percentages for Income Per Person for Democratic Lower Middle-Income Countries, Sorted by Values 1036.830725 3.703704 1143.831514 3.703704 1144.102193 3.703704 1200.652075 3.703704 1232.794137 3.703704 1258.762596 3.703704 1295.742686 3.703704 1324.194906 3.703704 1326.741757 3.703704 1383.401869 3.703704 1392.411829 3.703704 1621.177078 3.703704 1728.020976 3.703704 1860.753895 3.703704 1914.996551 3.703704 2221.185664 3.703704 2231.993335 3.703704 2549.558474 3.703704 2557.433638 3.703704 2636.787800 3.703704 2667.246710 3.703704 2712.517199 3.703704 2923.144355 3.703704 3180.430612 3.703704 3233.423780 3.703704 3665.348369 3.703704 3745.649852 3.703704 Name: incomeperperson, dtype: float64 Counts for Female Employ Rate for Democratic Lower Middle-Income Countries, Sorted by Values 26.799999 1 31.700001 1 34.200001 2 34.299999 1 37.299999 1 39.200001 1 39.900002 1 41.799999 1 42.099998 2 43.799999 1 44.000000 1 44.099998 1 44.799999 1 45.500000 1 46.799999 1 47.500000 1 48.500000 1 49.400002 1 51.299999 1 53.799999 1 54.900002 1 59.799999 1 61.599998 1 65.000000 1 65.300003 1 Name: femaleemployrate, dtype: int64 Percentages for Female Employ Rate for Democratic Lower Middle-Income Countries, Sorted by Values 26.799999 3.703704 31.700001 3.703704 34.200001 7.407407 34.299999 3.703704 37.299999 3.703704 39.200001 3.703704 39.900002 3.703704 41.799999 3.703704 42.099998 7.407407 43.799999 3.703704 44.000000 3.703704 44.099998 3.703704 44.799999 3.703704 45.500000 3.703704 46.799999 3.703704 47.500000 3.703704 48.500000 3.703704 49.400002 3.703704 51.299999 3.703704 53.799999 3.703704 54.900002 3.703704 59.799999 3.703704 61.599998 3.703704 65.000000 3.703704 65.300003 3.703704 Name: femaleemployrate, dtype: float64 Counts for Polity Scores for Democratic Lower Middle-Income Countries, Sorted by Values 2.000000 1 3.000000 1 4.000000 2 5.000000 2 6.000000 4 7.000000 4 8.000000 6 9.000000 7 Name: polityscore, dtype: int64 Percentages for Polity Scores for Democratic Lower Middle-Income Countries, Sorted by Values 2.000000 3.703704 3.000000 3.703704 4.000000 7.407407 5.000000 7.407407 6.000000 14.814815 7.000000 14.814815 8.000000 22.222222 9.000000 25.925926 Name: polityscore, dtype: float64 Plotting Income Per Person Univariate Bar Graph BASIC STATISTICS ABOUT INCOME PER PERSON GROUP count 27 unique 6 top (1025, 1525] freq 11 Name: INCOMEGROUP, dtype: object INCOMEGROUP (1025, 1525] 11 (1525, 2025] 4 (2025, 2525] 2 (2525, 3025] 6 (3025, 3525] 2 (3525, 4036] 2 dtype: int64 mode of INCOME GROUP 0 (1025, 1525] Name: INCOMEGROUP, dtype: category Categories (6, interval[int64]): [(1025, 1525] < (1525, 2025] < (2025, 2525] < (2525, 3025] < (3025, 3525] < (3525, 4036]] mean of income per person 2044.0420212962965 std of income per person 836.1734117910694 min of income per person (1025, 1525] max of income per person (3525, 4036] median of income per person 1860.7538949999998 INCOMEGROUP (1025, 1525] 11 (1525, 2025] 4 (2025, 2525] 2 (2525, 3025] 6 (3025, 3525] 2 (3525, 4036] 2 dtype: int64 INCOMEGROUP (1025, 1525] 5.164319 (1525, 2025] 1.877934 (2025, 2525] 0.938967 (2525, 3025] 2.816901 (3025, 3525] 0.938967 (3525, 4036] 0.938967 dtype: float64 __________________________________________________ LOOKING AT POLITY SCORE Plotting POLITY SCORE Bar Graph BASIC STATISTICS ABOUT INCOME PER PERSON GROUP count 27.000000 mean 68.888889 std 20.064000 min 20.000000 25% 60.000000 50% 70.000000 75% 85.000000 max 90.000000 Name: democraticscore, dtype: float64 democraticscore 20 1 30 1 40 2 50 2 60 4 70 4 80 6 90 7 dtype: int64 mode of DEMOCRATIC SCORE 0 90 dtype: int64 mean of DEMOCRATIC SCORE 68.88888888888889 std of democratic score 20.06400016357911 min of democraticscore 20 max of democraticscore 90 median of democraticscore 70.0 democraticscore 20 1 30 1 40 2 50 2 60 4 70 4 80 6 90 7 dtype: int64 democraticscore 20 0.469484 30 0.469484 40 0.938967 50 0.938967 60 1.877934 70 1.877934 80 2.816901 90 3.286385 dtype: float64

The Second Half of the python Code is below:

----------------------------------------------------------------------------------

# -*- coding: utf-8 -*- """ Created on Sat Jun 17 23:50:31 2017

@author: sudhir """

#ADDHEALTH EXAMPLE

import pandas import numpy import seaborn import matplotlib.pyplot as plt

data = pandas.read_csv('gapminder.csv', low_memory=False)

#making individual PROGRESS variables numeric data['incomeperperson'] = data['incomeperperson'].convert_objects(convert_numeric=True) data['employrate'] = data['employrate'].convert_objects(convert_numeric=True) data['femaleemployrate'] = data['femaleemployrate'].convert_objects(convert_numeric=True) data['internetuserate'] = data['internetuserate'].convert_objects(convert_numeric=True) data['urbanrate'] = data['urbanrate'].convert_objects(convert_numeric=True) data['armedforcesrate'] = data['armedforcesrate'].convert_objects(convert_numeric=True) data['hivrate'] = data['hivrate'].convert_objects(convert_numeric=True) #Set missing data to NAN data['employrate']=data['employrate'].replace('', numpy.nan) data['femaleemployrate']=data['femaleemployrate'].replace('', numpy.nan) data['internetuserate']=data['internetuserate'].replace('', numpy.nan) data['urbanrate']=data['urbanrate'].replace('', numpy.nan) data['armedforcesrate']=data['armedforcesrate'].replace('', numpy.nan) data['incomeperperson']=data['incomeperperson'].replace('', numpy.nan) #count of number of PROGRESS categories endorsed, GENDEREQUITY data['GENDEREQUITY']=data['femaleemployrate'] - data['employrate'] print('GENDEREQUITY')

# subset variables in new data frame, sub1 sub1=data[['country','employrate', 'femaleemployrate', 'internetuserate', 'urbanrate', 'armedforcesrate', 'GENDEREQUITY']] sub1=data[(data['femaleemployrate']>=40.0) & (data['internetuserate']>60)] print ('printing a before PROGRESS variable,n=5') a = sub1.head (n=10) print(a)

#new PROGRESS variable, categorical 1 through 6 def PROGRESS (row): if (row['GENDEREQUITY'] > 0) & (row['femaleemployrate'] > 50) : # if GENDEREQUITY > 0, then female employment rate is greater # than male employment rate, indicating great gender equity return 1 # if gender equity < 0, we look at employment rate. If that is # greater than 50%, then the country has high employment if (row['GENDEREQUITY'] > 0) & (row['femaleemployrate'] <= 50) : # if GENDEREQUITY > 0, then female employment rate is greater # than male employment rate, indicating great gender equity return 2 # if gender equity < 0, we look at employment rate. If that is # greater than 50%, then the country has high employment if (row['GENDEREQUITY'] < 0) & (row['GENDEREQUITY'] >= -10) & (row['femaleemployrate'] >= 50) : return 3 if (row['GENDEREQUITY'] < 0) & (row['GENDEREQUITY'] >= -10) & (row['femaleemployrate'] <= 50): return 4 if (row['GENDEREQUITY'] < -10) & (row['femaleemployrate'] >= 50): return 5 if (row['GENDEREQUITY'] < -10) & (row['femaleemployrate'] < 50): return 6 sub1['PROGRESS'] = sub1.apply (lambda row: PROGRESS (row),axis=1)

print ('printing a after PROGRESS variable,n=7') a = sub1.head (n=12) print(a)

#frequency distributions for primary and secondary ethinciity variables print ('counts for Employ Rate') c10 = sub1['employrate'].value_counts(sort=True) print(c10)

print ('percentages for Employ Rate') p10 = sub1['employrate'].value_counts(sort=True, normalize=True) print (p10)

print ('counts for Female Employ Rate') c11 = sub1['femaleemployrate'].value_counts(sort=True) print(c11)

print ('percentages for Female Employ Rate') p11= sub1['femaleemployrate'].value_counts(sort=True, normalize=True) print (p11)

print ('counts for Internet Use Rate') c12 = sub1['internetuserate'].value_counts(sort=True) print(c12)

print ('percentages for Internet Use Rate') p12 = sub1['internetuserate'].value_counts(sort=True, normalize=True) print (p12)

print ('counts for Urban Rate') c13 = sub1['urbanrate'].value_counts(sort=True) #print(c13)

print ('percentages for Urban Rate') p13 = sub1['urbanrate'].value_counts(sort=True, normalize=True) #print (p13)

print ('counts for Armed Forces Rate') c14 = sub1['armedforcesrate'].value_counts(sort=True) #print(c14)

print ('percentages for Armed Forces Rate') p14 = sub1['armedforcesrate'].value_counts(sort=True, normalize=True) #print (p14)

print ('counts for number of Gender Equity') c15 = sub1['GENDEREQUITY'].value_counts(sort=True) print(c15)

print ('counts for PROGRESS CATEGORIES') c16 = sub1['PROGRESS'].value_counts(sort=True) print(c16)

print ('percentages for PROGRESS CATEGORIES') p16 = sub1['PROGRESS'].value_counts(sort=True, normalize=True) print (p16)

#basic scatterplot: Q->Q scat1 = seaborn.regplot(x="urbanrate", y="internetuserate", fit_reg=False, data=data) plt.xlabel('Urban Rate') plt.ylabel('Internet Use Rate') plt.title('Scatterplot for the Association Between Urban Rate and Internet Use Rate')

scat2 = seaborn.regplot(x="urbanrate", y="internetuserate", data=data) plt.xlabel('Urban Rate') plt.ylabel('Internet Use Rate') plt.title('Scatterplot for the Association Between Urban Rate and Internet Use Rate')

scat3 = seaborn.regplot(x="incomeperperson", y="internetuserate", data=data) plt.xlabel('Income per Person') plt.ylabel('Internet Use Rate') plt.title('Scatterplot for the Association Between Income per Person and Internet Use Rate')

scat4 = seaborn.regplot(x="incomeperperson", y="hivrate", data=data) plt.xlabel('Income per Person') plt.ylabel('HIV Rate') plt.title('Scatterplot for the Association Between Income per Person and HIV Rate')

# quartile split (use qcut function & ask for 4 groups - gives you quartile split) print ('Income per person - 4 categories - quartiles') data['INCOMEGRP4']=pandas.qcut(data.incomeperperson, 4, labels=["1=25th%tile","2=50%tile","3=75%tile","4=100%tile"]) c10 = data['INCOMEGRP4'].value_counts(sort=False, dropna=True) print(c10)

# bivariate bar graph C->Q seaborn.factorplot(x='INCOMEGRP4', y='hivrate', data=data, kind="bar", ci=None) plt.xlabel('income group') plt.ylabel('mean HIV rate')

print('PRINTING INCOMEGRP4 NOW') c11= data.groupby('INCOMEGRP4').size() print (c11) print('-------------------------------------------------') #result = data.sort(['INCOMEGRP4'], ascending=[1]) #print(result)

-----------------------------------------------------------------------------------

The Output from the second part of the python code is below:

-----------------------------------------------------------------------------------

printing a before PROGRESS variable,n=5 country incomeperperson alcconsumption armedforcesrate \ 9 Australia 25249.986060 10.21 0.486280 10 Austria 26692.984110 12.4 0.815580 15 Barbados 9243.587053 6.42 0.663956 17 Belgium 24496.048260 10.41 0.815648 32 Canada 25575.352620 10.2 0.342976 49 Czech Rep. 7381.312751 16.47 0.515706 50 Denmark 30532.277040 12.02 1.012373 59 Estonia 6238.537506 17.24 0.998428 63 Finland 27110.731590 13.1 1.177416 64 France 22878.466570 12.48 1.233780

breastcancerper100th co2emissions femaleemployrate hivrate \ 9 83.2 12970092667 54.599998 0.100000 10 70.5 4466084333 49.700001 0.300000 15 62.5 36160666.67 60.299999 1.400000 17 92 10897025333 41.700001 0.200000 32 84.3 24979045667 58.900002 0.200000 49 58.4 1776016000 47.599998 0.060000 50 88.7 3503877667 58.099998 0.200000 59 47.7 277170666.7 52.099998 1.200000 63 84.7 2420300667 53.400002 0.100000 64 91.9 33341634333 45.599998 0.400000

internetuserate lifeexpectancy oilperperson polityscore \ 9 75.895654 81.907 1.913026109 10 10 72.731576 80.854 1.548790966 10 15 70.028599 76.835 17 73.733934 80.009 8 32 81.338393 81.012 3.007355851 10 49 68.638133 77.685 0.876778335 8 50 88.770254 78.826 1.567527461 10 59 74.163040 74.825 9 63 86.898845 79.977 1.938654268 10 64 77.498619 81.539 1.328291411 9

relectricperperson suicideper100th employrate urbanrate GENDEREQUITY 9 2825.391095 8.470030125 61.500000 88.740000 -6.900002 10 2068.123309 13.09437 57.099998 67.160000 -7.399998 15 3.108602524 66.900002 39.840000 -6.600002 17 1920.962215 15.95385 48.599998 97.360000 -6.899998 32 4772.370648 10.10099 63.500000 80.400000 -4.599998 49 1438.780412 12.36798 56.000000 73.500000 -8.400002 50 1884.299342 8.973104 63.099998 86.680000 -5.000000 59 1411.230532 16.95924 56.500000 69.460000 -4.400002 63 4036.953993 16.23437 57.200001 63.300000 -3.799999 64 2539.753273 14.09153 51.200001 77.360000 -5.600002 printing a after PROGRESS variable,n=7 country incomeperperson alcconsumption armedforcesrate \ 9 Australia 25249.986060 10.21 0.486280 10 Austria 26692.984110 12.4 0.815580 15 Barbados 9243.587053 6.42 0.663956 17 Belgium 24496.048260 10.41 0.815648 32 Canada 25575.352620 10.2 0.342976 49 Czech Rep. 7381.312751 16.47 0.515706 50 Denmark 30532.277040 12.02 1.012373 59 Estonia 6238.537506 17.24 0.998428 63 Finland 27110.731590 13.1 1.177416 64 France 22878.466570 12.48 1.233780 69 Germany 25306.187190 12.14 0.575810 83 Hong Kong, China 35536.072470 nan

relectricperperson suicideper100th employrate urbanrate GENDEREQUITY \ 9 2825.391095 8.470030125 61.500000 88.740000 -6.900002 10 2068.123309 13.09437 57.099998 67.160000 -7.399998 15 3.108602524 66.900002 39.840000 -6.600002 17 1920.962215 15.95385 48.599998 97.360000 -6.899998 32 4772.370648 10.10099 63.500000 80.400000 -4.599998 49 1438.780412 12.36798 56.000000 73.500000 -8.400002 50 1884.299342 8.973104 63.099998 86.680000 -5.000000 59 1411.230532 16.95924 56.500000 69.460000 -4.400002 63 4036.953993 16.23437 57.200001 63.300000 -3.799999 64 2539.753273 14.09153 51.200001 77.360000 -5.600002 69 1693.891898 9.211085 53.500000 73.640000 -6.700001 83 1468.640784 59.000000 100.000000 -7.400002

PROGRESS 9 3 10 4 15 3 17 4 32 3 49 4 50 3 59 3 63 3 64 4 69 4 83 3 counts for Employ Rate 53.500000 2 65.000000 2 48.599998 1 62.299999 1 66.900002 1 63.500000 1 56.000000 1 56.500000 1 59.000000 1 73.599998 1 47.299999 1 57.200001 1 59.900002 1 52.500000 1 57.099998 1 53.099998 1 60.700001 1 58.900002 1 51.200001 1 62.400002 1 61.299999 1 51.299999 1 53.400002 1 48.700001 1 56.799999 1 55.900002 1 63.099998 1 64.300003 1 59.299999 1 57.299999 1 61.500000 1 Name: employrate, dtype: int64 percentages for Employ Rate 53.500000 0.060606 65.000000 0.060606 48.599998 0.030303 62.299999 0.030303 66.900002 0.030303 63.500000 0.030303 56.000000 0.030303 56.500000 0.030303 59.000000 0.030303 73.599998 0.030303 47.299999 0.030303 57.200001 0.030303 59.900002 0.030303 52.500000 0.030303 57.099998 0.030303 53.099998 0.030303 60.700001 0.030303 58.900002 0.030303 51.200001 0.030303 62.400002 0.030303 61.299999 0.030303 51.299999 0.030303 53.400002 0.030303 48.700001 0.030303 56.799999 0.030303 55.900002 0.030303 63.099998 0.030303 64.300003 0.030303 59.299999 0.030303 57.299999 0.030303 61.500000 0.030303 Name: employrate, dtype: float64 counts for Female Employ Rate 41.700001 2 47.599998 1 45.599998 1 58.900002 1 48.000000 1 51.599998 1 57.000000 1 54.299999 1 56.000000 1 69.599998 1 60.299999 1 49.700001 1 53.400002 1 56.700001 1 54.599998 1 45.299999 1 40.299999 1 50.700001 1 53.099998 1 49.400002 1 46.400002 1 58.299999 1 52.099998 1 45.900002 1 46.200001 1 48.799999 1 60.900002 1 58.099998 1 42.099998 1 51.299999 1 46.799999 1 51.000000 1 Name: femaleemployrate, dtype: int64 percentages for Female Employ Rate 41.700001 0.060606 47.599998 0.030303 45.599998 0.030303 58.900002 0.030303 48.000000 0.030303 51.599998 0.030303 57.000000 0.030303 54.299999 0.030303 56.000000 0.030303 69.599998 0.030303 60.299999 0.030303 49.700001 0.030303 53.400002 0.030303 56.700001 0.030303 54.599998 0.030303 45.299999 0.030303 40.299999 0.030303 50.700001 0.030303 53.099998 0.030303 49.400002 0.030303 46.400002 0.030303 58.299999 0.030303 52.099998 0.030303 45.900002 0.030303 46.200001 0.030303 48.799999 0.030303 60.900002 0.030303 58.099998 0.030303 42.099998 0.030303 51.299999 0.030303 46.799999 0.030303 51.000000 0.030303 Name: femaleemployrate, dtype: float64 counts for Internet Use Rate 90.016190 1 93.277508 1 62.811900 1 72.731576 1 74.247572 1 82.515928 1 81.338393 1 69.770394 1 70.028599 1 71.849124 1 83.002584 1 75.895654 1 65.387786 1 73.733934 1 84.731705 1 79.889777 1 82.526898 1 90.079527 1 82.166660 1 62.471230 1 65.808554 1 68.638133 1 86.898845 1 71.131707 1 65.163251 1 74.163040 1 77.498619 1 95.638113 1 69.339971 1 71.514724 1 88.770254 1 90.703555 1 77.638535 1 Name: internetuserate, dtype: int64 percentages for Internet Use Rate 90.016190 0.030303 93.277508 0.030303 62.811900 0.030303 72.731576 0.030303 74.247572 0.030303 82.515928 0.030303 81.338393 0.030303 69.770394 0.030303 70.028599 0.030303 71.849124 0.030303 83.002584 0.030303 75.895654 0.030303 65.387786 0.030303 73.733934 0.030303 84.731705 0.030303 79.889777 0.030303 82.526898 0.030303 90.079527 0.030303 82.166660 0.030303 62.471230 0.030303 65.808554 0.030303 68.638133 0.030303 86.898845 0.030303 71.131707 0.030303 65.163251 0.030303 74.163040 0.030303 77.498619 0.030303 95.638113 0.030303 69.339971 0.030303 71.514724 0.030303 88.770254 0.030303 90.703555 0.030303 77.638535 0.030303 Name: internetuserate, dtype: float64 counts for Urban Rate percentages for Urban Rate counts for Armed Forces Rate percentages for Armed Forces Rate counts for number of Gender Equity -7.000000 2 -4.000000 2 -6.700001 2 -6.600002 2 -5.600002 1 -7.399998 1 -11.099998 1 -6.500000 1 -6.299999 1 -8.400002 1 -7.300003 1 -4.899998 1 -6.099998 1 -6.900002 1 -7.599998 1 -8.100002 1 -8.900002 1 -6.200001 1 -7.400002 1 -10.799999 1 -3.799999 1 -10.900002 1 -11.100002 1 -6.899998 1 -4.099998 1 -4.299999 1 -4.599998 1 -4.400002 1 -5.000000 1 Name: GENDEREQUITY, dtype: int64 counts for PROGRESS CATEGORIES 3 17 4 12 6 3 5 1 Name: PROGRESS, dtype: int64 percentages for PROGRESS CATEGORIES 3 0.515152 4 0.363636 6 0.090909 5 0.030303 Name: PROGRESS, dtype: float64 Income per person - 4 categories - quartiles 1=25th%tile 48 2=50%tile 47 3=75%tile 47 4=100%tile 48 Name: INCOMEGRP4, dtype: int64 PRINTING INCOMEGRP4 NOW INCOMEGRP4 1=25th%tile 48 2=50%tile 47 3=75%tile 47 4=100%tile 48 dtype: int64

------------------------------------------------------------------------------------

Summary: There is a lot of information in the output. Just to capture the main points: (a) lower middle-income democratic countries are skewed towards high democratic scores [mode of 90, mean of 68.9 and standard deviation of 20] ; (b) income per person [also referred to as per capita income] seems to correlate with Internet use [i.e, more internet use is linked to higher per capita incomes] in both the scatter plot and histogram plot; (c) higher Income per person is correlated with lower HIV rate as clearly seen in the histogram, but the scatter plot for income per person versus HIV rate seems to imply the opposite [i.e, lower HIV rate is correlated with lower per capita income, but this result is suspect and likely due to variability in HIV rate in middle-income countries]; and (d) Income per person and urban rate is very weakly correlated for lower incomes, with tremendous variability in urban rates for the same Income group [see scatter plot], but there is a clear correlation between income per person and higher urban rates for middle to high income countries, as indicated in the histogram plot.

------------------------------END OF ASSIGNMENT 4--------------------------

Sudhir’s Assignment 3

Background: In this assignment, I follow the program provided in the class notes. I have broken up the program code and output into two halves. In both parts of this assignment, I use the Gapminder Data Set. In the first half, I have made a detailed analysis of highly democractic lower middle-income countries. I define highly democratic countries to be the ones that have polity scores greater than 6. In the Gapminder Data set, polity scores range from -10 for highly authoritarian countries to +10 for highly democratic countries. Also, lower middle-income countries have per capita income between $1025 and $4036. I also introduce new variables (”Democratic Score” and “Governance Score”). Here is the python code and the output. In the second half of the assignment, I look at gender equity by tapping into the information contained in the female employment rate versus the general employment rate.

The First Half of the Program Code For Assignment 3 is below:

--------------------------------------------------------------------------------

# -*- coding: utf-8 -*- """ Created on Sun Aug 30 10:50:43 2015; modified on Sat Jun 17, 2017

@author: ldierker; modified by Sudhir """ import pandas import numpy # any additional libraries would be imported here data = pandas.read_csv('SudhirCodeBook.csv', low_memory=False) print ('_____________________________________________________') print ('OUTPUT PRINTOUT STARTS HERE') print ('Total Number of Countries') print (len(data)) #number of observations (rows) print ('Total Number of Variables') print (len(data.columns)) # number of variables (columns) #setting variables you will be working with to numeric print ('The Variables in this Database:') print('(1) country, (2) incomeperperson, (3) polityscore') print('(4) femaleemployeerate, (5) internetuserate, and (6) urbanrate') data['incomeperperson'] = data['incomeperperson'].convert_objects(convert_numeric=True) data['polityscore'] = data['polityscore'].convert_objects(convert_numeric=True) data['femaleemployrate'] = data['femaleemployrate'].convert_objects(convert_numeric=True) data['internetuserate'] = data['internetuserate'].convert_objects(convert_numeric=True) data['urbanrate'] = data['urbanrate'].convert_objects(convert_numeric=True)

#counts and percentages (i.e. frequency distributions) for each variable) print ('Examples of Sorting by Frequency and by Values') print ('List Polity Scores for Entire Data Set, Sorted by Frequency') c2 = data['polityscore'].value_counts(sort=True) print(c2) print ('List Polity Scores for Entire Data Set, sorted by Values') ct2= data.groupby('polityscore').size().sort_index(ascending=True) print (ct2) ######################################################################################## # REST OF ASSIGNMENT FOCUSES ON HIGHLY DEMOCRATIC (Polity Score > 6 ) # LOWER MIDDLE-INCOME COUNTRIES ($1025 < GNI < $4036) # #For the current 2017 fiscal year, low-income economies are defined as those with a #GNI per capita, calculated using the World Bank Atlas method, of $1,025 or less in 2015; #lower middle-income economies are those with a GNI per capita between $1,026 and $4,035; #upper middle-income economies are those with a GNI per capita between $4,036 and $12,475; #high-income economies are those with a GNI per capita of $12,476 or more. #https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups # Polity scores range from -10 for undemocratic to +10 for highly democratic countries. # # REST OF ASSIGNMENT FOCUSES ON HIGHLY DEMOCRATIC (Polity Score > 6 ) # LOWER MIDDLE-INCOME COUNTRIES ($1025 < GNI < $4036) ########################################################################################

print ('_____________________________________________________') print('Rest of Assignment will Focus on Highly Democratic Lower Mid-Income Countries') print('Highly Democratic Countries Have Polity Scores > 6 ') print('Lower Middle-Income Countries have $1025 < Per Capital Income < $4036 ') print ('_____________________________________________________') # Sub-sampling entire data set to only pick up Highly Democratic lower middle-income countries sub1=data[(data['incomeperperson']>1025) & (data['incomeperperson']<4036) & (data['polityscore']> 6)] #make a copy of my new subsetted data sub2 = sub1.copy() # frequency distritions on new sub2 data frame # First Fill in Blanks with NaN. Then print out original counts print ('Original Counts for FemaleEmployRate For Lower Democratic Lower Mid-Income Countries, Sorted by Values') c5 = sub2['femaleemployrate'].value_counts(sort=True,dropna=False).sort_index(ascending=True) print(c5)

sub2['femaleemployrate']=sub2['femaleemployrate'].replace(-9.0, numpy.nan) sub2['femaleemployrate'].fillna(-99.0,inplace=True) print ('New Counts for FemaleEmployRate For Democratic Lower Mid-Income Countries, Sorted by Values') c55 = sub2['femaleemployrate'].value_counts(sort=True,dropna=False).sort_index(ascending=True) print(c55) #sub2['femaleemployeerate']=sub2['femaleemployrate'].replace(-99.0,numpy.nan) #c2=sub2['femaleemployrate'].value_counts(sort=True,dropna=False) #print(c2) #chk2=sub2['femaleemployrate'].value_counts(sort=True,dropna=True) #print(chk2) ds2=sub2['femaleemployrate'].describe() print('Statistics for Female Employment Rate') print(ds2) print('____________________________________________') print('I will now delete data points for which the female Employment Rate is Missing or NaN' ) sub1=data[(data['incomeperperson']>1025) & (data['incomeperperson']<4036) & (data['polityscore']> 6) & (data['femaleemployrate']>0)] sub2 = sub1.copy() recode1={7:0,8:33,9:66,10:100} print('Introducing Democratic Score, which is mapped from Polity Score') print('Polity Score Ranges from 7 to 10, and Democratic Score Ranges from 0 to 100') sub2['democraticscore']=sub2['polityscore'].map(recode1)

print('Introducing New Variable: Governance Score') print('GOVERNANCE SCORE=Democratic Score+Female Employment Rate+Urban Rate') sub2['GOVERNANCESCORE']=sub2['democraticscore'] + sub2['femaleemployrate']+sub2['urbanrate'] c3=sub2['GOVERNANCESCORE'].value_counts().sort_index(ascending=True) print ('GOVERNANCE SCORES') print(c3) sub2['INCOMEGROUP']=pandas.qcut(sub2.incomeperperson, 5, labels=["1=0-20","2=20-40","3=40-60","4=60-80","5=80-100"]) c4 = sub2['INCOMEGROUP'].value_counts(sort=False, dropna=True) print(c4) print('I break up the Lower Middle-Income group of countries into four sub-groups') print('The four sub-groups have per capital income: $1025-$1750,$1750-$2500,$2500-$3250,$3250-$4036') print('INCOME-4 categories: 1025-1750, 1750-2500, 2500-3250, 3250-4036') sub2['INCOMEGROUP']=pandas.cut(sub2.incomeperperson,[1025,1750,2500,3250,4036]) c5=sub2['INCOMEGROUP'].value_counts(sort=True,dropna=True) print(c5) print('printing crosstabs') print(pandas.crosstab(sub2['INCOMEGROUP'],sub2['incomeperperson']))

print('____________________________________________')

print ('Counts for Income Per Person for Democratic Lower Middle-Income Countries, Sorted by Values') c6 = sub2['incomeperperson'].value_counts().sort_index(ascending=True) print(c6) print ('Percentages for Income Per Person for Democratic Lower Middle-Income Countries, Sorted by Values') p6 = sub2['incomeperperson'].value_counts(normalize=True).sort_index(ascending=True) * 100 print (p6) print ('Counts for Female Employ Rate for Democratic Lower Middle-Income Countries, Sorted by Values') c7 = sub2['femaleemployrate'].value_counts().sort_index(ascending=True) print(c7) print ('Percentages for Female Employ Rate for Democratic Lower Middle-Income Countries, Sorted by Values') p7 = sub2['femaleemployrate'].value_counts(normalize=True).sort_index(ascending=True) * 100 print (p7) print ('Counts for Polity Scores for Democratic Lower Middle-Income Countries, Sorted by Values') c8 = sub2['polityscore'].value_counts().sort_index(ascending=True) print (c8) print ('Percentages for Polity Scores for Democratic Lower Middle-Income Countries, Sorted by Values') p8 = sub2['polityscore'].value_counts(normalize=True).sort_index(ascending=True) * 100 print (p8) print ('OUTPUT PRINTOUT ENDS HERE') print ('_____________________________________________________') #upper-case all DataFrame column names - place afer code for loading data aboave data.columns = map(str.upper, data.columns) # bug fix for display formats to avoid run time errors - put after code for loading data above pandas.set_option('display.float_format', lambda x:'%f'%x) print('END OF OUTPUT FROM FIRST HALF OF PYTHON CODE FOR ASSIGNMENT 3') print('_______________________________________________________________')

Here is the Output for the first half of the python code

------------------------------------------------------------------------------------

runfile('C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/WEEK3/SudhirAssignment31.8.py', wdir='C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/WEEK3') _____________________________________________________ OUTPUT PRINTOUT STARTS HERE Total Number of Countries 213 Total Number of Variables 6 The Variables in this Database: (1) country, (2) incomeperperson, (3) polityscore (4) femaleemployeerate, (5) internetuserate, and (6) urbanrate Examples of Sorting by Frequency and by Values List Polity Scores for Entire Data Set, Sorted by Frequency 10.000000 33 8.000000 19 9.000000 15 7.000000 13 -7.000000 12 6.000000 10 5.000000 7 -4.000000 6 0.000000 6 -3.000000 6 -2.000000 5 -1.000000 4 -9.000000 4 4.000000 4 1.000000 3 -6.000000 3 2.000000 3 -8.000000 2 3.000000 2 -5.000000 2 -10.000000 2 Name: polityscore, dtype: int64 List Polity Scores for Entire Data Set, sorted by Values polityscore -10.000000 2 -9.000000 4 -8.000000 2 -7.000000 12 -6.000000 3 -5.000000 2 -4.000000 6 -3.000000 6 -2.000000 5 -1.000000 4 0.000000 6 1.000000 3 2.000000 3 3.000000 2 4.000000 4 5.000000 7 6.000000 10 7.000000 13 8.000000 19 9.000000 15 10.000000 33 dtype: int64 _____________________________________________________ Rest of Assignment will Focus on Highly Democratic Lower Mid-Income Countries Highly Democratic Countries Have Polity Scores > 6 Lower Middle-Income Countries have $1025 < Per Capital Income < $4036 _____________________________________________________ Original Counts for FemaleEmployRate For Lower Democratic Lower Mid-Income Countries, Sorted by Values 26.799999 1 34.200001 1 34.299999 1 42.099998 2 43.799999 1 44.000000 1 44.099998 1 44.799999 1 46.799999 1 47.500000 1 49.400002 1 51.299999 1 54.900002 1 59.799999 1 61.599998 1 65.300003 1 nan 2 Name: femaleemployrate, dtype: int64 New Counts for FemaleEmployRate For Democratic Lower Mid-Income Countries, Sorted by Values -99.000000 2 26.799999 1 34.200001 1 34.299999 1 42.099998 2 43.799999 1 44.000000 1 44.099998 1 44.799999 1 46.799999 1 47.500000 1 49.400002 1 51.299999 1 54.900002 1 59.799999 1 61.599998 1 65.300003 1 Name: femaleemployrate, dtype: int64 Statistics for Female Employment Rate count 19.000000 mean 31.305263 std 46.883040 min -99.000000 25% 38.199999 50% 44.099998 75% 50.350000 max 65.300003 Name: femaleemployrate, dtype: float64 ____________________________________________ I will now delete data points for which the female Employment Rate is Missing or NaN Introducing Democratic Score, which is mapped from Polity Score Polity Score Ranges from 7 to 10, and Democratic Score Ranges from 0 to 100 Introducing New Variable: Governance Score GOVERNANCE SCORE=Democratic Score+Female Employment Rate+Urban Rate GOVERNANCE SCORES 82.080001 1 102.259999 1 117.380002 1 125.580000 1 127.179998 1 128.559998 1 129.400002 1 138.499999 1 144.719999 1 154.819998 1 158.600003 1 159.699999 1 161.039999 1 164.039999 1 166.800000 1 179.199998 1 197.199999 1 Name: GOVERNANCESCORE, dtype: int64 1=0-20 4 2=20-40 3 3=40-60 3 4=60-80 3 5=80-100 4 Name: INCOMEGROUP, dtype: int64 I break up the Lower Middle-Income group of countries into four sub-groups The four sub-groups have per capital income: $1025-$1750,$1750-$2500,$2500-$3250,$3250-$4036 INCOME-4 categories: 1025-1750, 1750-2500, 2500-3250, 3250-4036 (1025, 1750] 7 (2500, 3250] 5 (1750, 2500] 3 (3250, 4036] 2 Name: INCOMEGROUP, dtype: int64 printing crosstabs incomeperperson 1036.830725 1143.831514 1144.102193 1232.794137 \ INCOMEGROUP (1025, 1750] 1 1 1 1 (1750, 2500] 0 0 0 0 (2500, 3250] 0 0 0 0 (3250, 4036] 0 0 0 0

incomeperperson 1383.401869 1392.411829 1621.177078 1860.753895 \ INCOMEGROUP (1025, 1750] 1 1 1 0 (1750, 2500] 0 0 0 1 (2500, 3250] 0 0 0 0 (3250, 4036] 0 0 0 0

incomeperperson 1914.996551 2221.185664 2549.558474 2557.433638 \ INCOMEGROUP (1025, 1750] 0 0 0 0 (1750, 2500] 1 1 0 0 (2500, 3250] 0 0 1 1 (3250, 4036] 0 0 0 0

incomeperperson 2636.787800 3180.430612 3233.423780 3665.348369 \ INCOMEGROUP (1025, 1750] 0 0 0 0 (1750, 2500] 0 0 0 0 (2500, 3250] 1 1 1 0 (3250, 4036] 0 0 0 1

incomeperperson 3745.649852 INCOMEGROUP (1025, 1750] 0 (1750, 2500] 0 (2500, 3250] 0 (3250, 4036] 1 ____________________________________________ Counts for Income Per Person for Democratic Lower Middle-Income Countries, Sorted by Values C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/WEEK3/SudhirAssignment31.8.py:21: FutureWarning: convert_objects is deprecated. Use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric. data['incomeperperson'] = data['incomeperperson'].convert_objects(convert_numeric=True) C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/WEEK3/SudhirAssignment31.8.py:22: FutureWarning: convert_objects is deprecated. Use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric. data['polityscore'] = data['polityscore'].convert_objects(convert_numeric=True) C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/WEEK3/SudhirAssignment31.8.py:23: FutureWarning: convert_objects is deprecated. Use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric. data['femaleemployrate'] = data['femaleemployrate'].convert_objects(convert_numeric=True) C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/WEEK3/SudhirAssignment31.8.py:24: FutureWarning: convert_objects is deprecated. Use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric. data['internetuserate'] = data['internetuserate'].convert_objects(convert_numeric=True) C:/Users/sudhir/Desktop/PERSONAL.BIZ.WORK.JUL2017/COURSERA/DataAnalysisInterPret/WEEK3/SudhirAssignment31.8.py:25: FutureWarning: convert_objects is deprecated. Use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric. data['urbanrate'] = data['urbanrate'].convert_objects(convert_numeric=True) 1036.830725 1 1143.831514 1 1144.102193 1 1232.794137 1 1383.401869 1 1392.411829 1 1621.177078 1 1860.753895 1 1914.996551 1 2221.185664 1 2549.558474 1 2557.433638 1 2636.787800 1 3180.430612 1 3233.423780 1 3665.348369 1 3745.649852 1 Name: incomeperperson, dtype: int64 Percentages for Income Per Person for Democratic Lower Middle-Income Countries, Sorted by Values 1036.830725 5.882353 1143.831514 5.882353 1144.102193 5.882353 1232.794137 5.882353 1383.401869 5.882353 1392.411829 5.882353 1621.177078 5.882353 1860.753895 5.882353 1914.996551 5.882353 2221.185664 5.882353 2549.558474 5.882353 2557.433638 5.882353 2636.787800 5.882353 3180.430612 5.882353 3233.423780 5.882353 3665.348369 5.882353 3745.649852 5.882353 Name: incomeperperson, dtype: float64 Counts for Female Employ Rate for Democratic Lower Middle-Income Countries, Sorted by Values 26.799999 1 34.200001 1 34.299999 1 42.099998 2 43.799999 1 44.000000 1 44.099998 1 44.799999 1 46.799999 1 47.500000 1 49.400002 1 51.299999 1 54.900002 1 59.799999 1 61.599998 1 65.300003 1 Name: femaleemployrate, dtype: int64 Percentages for Female Employ Rate for Democratic Lower Middle-Income Countries, Sorted by Values 26.799999 5.882353 34.200001 5.882353 34.299999 5.882353 42.099998 11.764706 43.799999 5.882353 44.000000 5.882353 44.099998 5.882353 44.799999 5.882353 46.799999 5.882353 47.500000 5.882353 49.400002 5.882353 51.299999 5.882353 54.900002 5.882353 59.799999 5.882353 61.599998 5.882353 65.300003 5.882353 Name: femaleemployrate, dtype: float64 Counts for Polity Scores for Democratic Lower Middle-Income Countries, Sorted by Values 7.000000 4 8.000000 6 9.000000 7 Name: polityscore, dtype: int64 Percentages for Polity Scores for Democratic Lower Middle-Income Countries, Sorted by Values 7.000000 23.529412 8.000000 35.294118 9.000000 41.176471 Name: polityscore, dtype: float64 OUTPUT PRINTOUT ENDS HERE _____________________________________________________ END OF OUTPUT FROM FIRST HALF OF PYTHON CODE FOR ASSIGNMENT 3 _______________________________________________________________

The second half of the Python Code for Assignment 3 is below:

-------------------------------------------------------------------------

# -*- coding: utf-8 -*- """ Created on Sat Jun 17 23:50:31 2017

@author: sudhir """

#ADDHEALTH EXAMPLE

import pandas import numpy

data = pandas.read_csv('gapminder.csv', low_memory=False)

#making individual PROGRESS variables numeric data['employrate'] = data['employrate'].convert_objects(convert_numeric=True) data['femaleemployrate'] = data['femaleemployrate'].convert_objects(convert_numeric=True) data['internetuserate'] = data['internetuserate'].convert_objects(convert_numeric=True) data['urbanrate'] = data['urbanrate'].convert_objects(convert_numeric=True) data['armedforcesrate'] = data['armedforcesrate'].convert_objects(convert_numeric=True)

#Set missing data to NAN data['employrate']=data['employrate'].replace('', numpy.nan)

data['femaleemployrate']=data['femaleemployrate'].replace('', numpy.nan)

data['internetuserate']=data['internetuserate'].replace('', numpy.nan)

data['urbanrate']=data['urbanrate'].replace('', numpy.nan)

data['armedforcesrate']=data['armedforcesrate'].replace('', numpy.nan)

#count of number of PROGRESS categories endorsed, GENDEREQUITY data['GENDEREQUITY']=data['femaleemployrate'] - data['employrate'] print('GENDEREQUITY')