Proteins and Data Blog @proteinsareawesome-blog - Tumblr Blog

K-means cluster analysis: assignment 4

This week’s assignment involves running a k-means cluster analysis. Cluster analysis is an unsupervised machine learning method that partitions the observations in a data set into a smaller set of clusters where each observation belongs to only one cluster. The goal of cluster analysis is to group, or cluster, observations into subsets based on their similarity of responses on multiple variables. Clustering variables should be primarily quantitative variables, but binary variables may also be included.

1) Code can be found here: https://github.com/fsee/DataScienceClass4/blob/master/Assignment4.py

2) I will use the same data set as in week 3 for the Lasso regression to identify observations that cluster together using K-means cluster analysis. The observations are computational parameters that pertain to the computational protein design problem.

a) Using the elbow method, I identified the number of clusters to divide my data set into as 2. The lecturer points out that this selection is highly subjective but it seems that 2 is the smallest cluster number with the largest drop in variance/ average distance within the clusters:

b) The number of observations in the data set is too large to meaningfully visualize here, that is why I applied canonical discrimination analysis to reduce the number of variables in the data set to a linear combination that can be expressed as the most important= canonical variables:

The blue and red cluster seem to separate fairly well with minimal visual overlap but some inner-cluster variance.

c) Create a unique identifier variable

When comparing the means of the clustering variables by cluster, the means are not very different between the two clusters as compared to the example shown in the lecture. The values are very similar for most clustering variables.

d) Analysis of variance: on the Tryp_ec50_y experimental variable

The ANOVA analysis shows that the difference on the two clusters is statistically significant on the Tryp_ec50_y observation variable.

The means and standard deviations reflect this accordingly:

means for Tryp_ec50 by cluster Tryp_ec50_y cluster 0 -0.400811 1 0.416393 standard deviations for Tryp_ec50 by cluster Tryp_ec50_y cluster 0 0.954523 1 0.892508

The Tukey test shows that both clusters differ significantly in Tryp_ec50_y

Multiple Comparison of Means - Tukey HSD,FWER=0.05 =========================================== group1 group2 meandiff lower upper reject ------------------------------------------- 0 1 0.8172 0.7418 0.8926 True -------------------------------------------

#data analysis #coursera #k-means clustering #protein engineering

Lasso Regression Analysis Assignment

This week’s assignment involves running a lasso regression analysis. Lasso regression analysis is a shrinkage and variable selection method for linear regression models. The goal of lasso regression is to obtain the subset of predictors that minimizes prediction error for a quantitative response variable. The lasso does this by imposing a constraint on the model parameters that causes regression coefficients for some variables to shrink toward zero. Variables with a regression coefficient equal to zero after the shrinkage process are excluded from the model. Variables with non-zero regression coefficients variables are most strongly associated with the response variable. Explanatory variables can be either quantitative, categorical or both.

1) Here is my code: https://github.com/fsee/DataScienceClass4/blob/master/Assignment3.py

2) I am using total_score, which is a measure of in silico protein energy as the target variable to be predicted and various computational parameters as predictors:

{'rotamer_boltz_core_min': 0.0, 'degree_core_SASA': 0.0, 'fa_intra_rep': 0.0, 'exposed_hydrophobics': 0.0, 'buried_np_per_res': 0.0, 'hbond_sc': 0.0, 'hphob_sc_degree': 0.0, 'fa_sol': 0.0, 'res_count_ALA': 0.0, 'hbond_bb_sc': 0.0, 'netcharge': 0.0, 'cavity_volume': 0.0, 'ss_sc': 0.0, 'contact_core_SCN': 0.0, 'two_core_each': 0.0, 'fa_atr_per_res': 0.0, 'fa_elec': 0.0, 'packstat': 0.0, 'ref': 0.0, 'hydrophobicity': 0.0, 'hphob_sc_contacts': 0.0, 'abego_res_profile': 0.0, 'score_per_res': 4.3595478714475773, 'yhh_planarity': 0.0, 'n_hydrophobic': 0.0, 'fa_dun': 0.0, 'abego_res_profile_penalty': 0.0, 'fa_rep': 0.0, 'n_res': 0.0, 'largest_hphob_cluster': 0.0, 'p_aa_pp': 0.0, 'unsat_hbond': 0.0, 'contact_core_SASA': 0.0, 'hbond_sr_bb': 0.0, 'n_hydrophobic_noA': 0.0, 'buried_np': 0.0, 'one_core_each': 0.0, 'score_per_residue': 0.0, 'hbond_lr_bb': 0.0, 'buried_over_exposed': 0.0, 'buried_minus_exposed': 0.0, 'fa_atr': 0.0, 'buried_minus_exposed_per_res': 0.0, 'fa_rep_per_res': 0.0, 'percent_core_SASA': 0.0, 'rama': 0.0, 'contact_all': 0.0, 'degree_core_SCN': 0.0, 'omega': 0.0, 'res_count_core_SASA': 0.0, 'exposed_np_per_res': 0.0, 'pro_close': 0.0, 'dslf_fa13': 0.0, 'total_hydrophobic': 0.0, 'holes': 0.0, 'n_charged': 0.0, 'degree': 0.0, 'res_count_core_SCN': 0.0, 'n_hphob_clusters': 0.0, 'ss_contributes_core': 0.0, 'mismatch_probability': 0.0, 'percent_core_SCN': 0.0}

As you can see from the score_per_residues has the strongest correlation coefficients and is also the only predictor for total_score from this set of predictors.

When visualizing the progression of the regression coefficients throughout the model selection process, we see that there is only one coefficient that follows a straight line:

When visualizing the mean squared error on each fold, the mean squared error decreases linearly with the log(alpha) and reaches zero at 11 folds.

training data MSE 1.51914509201e-16 test data MSE 1.51890754342e-16 training data R-square 1.0 test data R-square 1.0

While the training and test MSE are in a similar range, and the R-square values are 100% it seems that this data set is not suitable for application of the Lasso regression since it only extracts one single variable out of all predictors that has the strongest association with the target.

I tested what happens when taking out the score_per_residue variable of the predictor pool but none of the other variable predict total_score from this data set.

#machine learning #data ana #lasso regression analysis

Machine Learning for Data Analysis: Assignment 2, Random Forests

The second assignment deals with Random Forests. Random forests are predictive models that allow for a data driven exploration of many explanatory variables in predicting a response or target variable. Random forests provide importance scores for each explanatory variable and also allow you to evaluate any increases in correct classification with the growing of smaller and larger number of trees.

1) My code can be found here: https://github.com/fsee/DataScienceClass4/blob/master/Assignment2.py

2) In this class I am trying to identify the best numeric explanatory variables for my categorical response variable Keq_cat, which represent experimentally measured protein affinity.

Analyzing the importance scores of all explanatory variables of interest, it is striking that most values lie in the range of 0.015-0.026 with no single variable being significantly more predictive than the others:

[ 0.01468628 0.02507908 0.02183499 0.01518011 0.01714734 0.02053914 0.01373848 0.01440824 0.0159825 0.01495651 0.02104426 0.01926355 0.01935499 0.02173912 0.02246309 0.02986529 0.01440262 0.01505287 0.01465393 0.01495517 0.02602115 0.02098841 0.02630557 0.02014262 0.01383168 0.01003434 0.0153252 0.01457605 0.0206847 0.01448661 0.0204876 0.01468704 0.02642043 0.01795576 0.01767197 0.01977737 0.01828355 0.01570475 0.02586849 0.01330683 0.01788548 0.02113189 0.02185395 0.02190741 0.0177671 0.02030823 0.01292998 0.01882512 0.01916932 0.01504427 0.01113858 0.01618605 0.0182557 0.01021534 0.00995965 0.00851426]

3) Evaluating the number of decision trees needed to arrive at a correct classification it becomes clear that the maximum accuracy level lies at 98.5% and can be achieved after 1-5 decision tree generations, as visualized by this plot:

It is striking that the accuracy of this model seems to be so high despite the poor quality of the input data set. I suspect this is due to the fact that only a small fraction of the data can be categorized as positive working Keq_cat results and thus predicting that Keq_cat =0 will almost always be the right prediction.

#data science #machine learning #coursera #protein design

Machine Learning for Data Analysis: Assignment 1, Decision Trees

This week’s assignment involves decision trees, and more specifically, classification trees. Decision trees are predictive models that allow for a data driven exploration of nonlinear relationships and interactions among many explanatory variables in predicting a response or target variable. When the response variable is categorical (two levels), the model is a called a classification tree. Explanatory variables can be either quantitative, categorical or both. Decision trees create segmentations or subgroups in the data, by applying a series of simple rules or criteria over and over again which choose variable constellations that best predict the response (i.e. target) variable.

Pretty cool stuff. I will use my data set from the previous class about protein-protein interactions to test which computationally calculated variables best predict my real world experimental outcome.

My code can be found here: https://github.com/fsee/DataScienceClass4/blob/master/Assignment1.py

Decision tree analysis was performed to test nonlinear relationships among a series of explanatory variables and a binary, categorical response variable, Keq_cat. All possible separations (categorical) or cut points (quantitative) are tested.

The following explanatory variables were used to construct the classification tree with Keq_cat as the response variable: 'ddg', 'ddg_fa_atr', 'ddg_fa_atr_norepack', 'ddg_fa_atr_norepack_per_1000sasa', 'ddg_fa_atr_per_1000sasa', 'ddg_norepack', 'ddg_norepack_per_1000sasa', 'ddg_per_1000sasa', 'dslf_fa13', 'fa_atr', 'fa_dun', 'fa_elec', 'fa_intra_rep', 'fa_intra_sol_xover4', 'fa_rep', 'fa_sol', 'fxn_np_chain1_buried_approx', 'fxn_np_chain2_buried_approx', 'hbond_bb_sc', 'hbond_lr_bb', 'hbond_sc', 'hbond_sr_bb', 'interface_averageDegree_chA', 'interface_averageDegree_chB', 'interface_buried_sasa', 'interface_contactcount', 'interface_fxn_hydrophobic', 'interface_holes', 'interface_hydrophobic_sasa', 'interface_polar_sasa', 'interface_sc', 'interface_sc_int_area', 'interface_unsat_hbond', 'interface_unsat_hbond2', 'lk_ball_wtd', 'np_chain1_exposed_approx', 'np_chain2_exposed_approx', 'omega', 'p_aa_pp', 'pro_close', 'rama_prepro', 'ref', 'yhh_planarity', 'worstfrag', 'worst6frags', 'sumfrags', 'totalfrag', 'avg_best_frag', 'abego_res_profile', 'abego_res_profile_penalty', 'relax_rmsd_avg', 'relax_energy_avg', 'MinAbinitioDistance', 'MinAbinitioEnergy', 'last_opt', 'FFF_pass'

The variable hbond_sr_bb is the first variable that splits the classification tree into two groups: 2 observations that have a hbond_sc_bb value of > -105.6 and 4436 observations that fall into the category <= -105.6.

While the accuracy score of the classification graph is very high with 0.968581081081 and the confusion matrix suggests a high true positive and true negative rate,

[[2858 45] [ 48 9]]

the classification tree seems to be highly overfitted because the overall sample is divided into two groups of which one holds the vast majority of the observations. This classification tree does not seem to do a good job of prediction the outcome of the Keq_cat variable. It should be noted that the data set itself is highly biased and there it not too much differentiating information that is expected to be distracted from this data set.

#data science #machine learning #decision trees #coursera

Regression Modeling in Practice: Test a Logistic Regression Model

Assignment:

Write a blog entry that summarize in a few sentences 1) what you found, making sure you discuss the results for the associations between all of your explanatory variables and your response variable. Make sure to include statistical results (odds ratios, p-values, and 95% confidence intervals for the odds ratios) in your summary. 2) Report whether or not your results supported your hypothesis for the association between your primary explanatory variable and your response variable. 3) Discuss whether or not there was evidence of confounding for the association between your primary explanatory and the response variable (Hint: adding additional explanatory variables to your model one at a time will make it easier to identify which of the variables are confounding variables).

The code for this assignment can be found here:

https://github.com/fsee/DataScienceClass3/blob/master/Assignment4.py

Hypothesis: ddg_per_1000sasa is significantly associated with Keq_cat.

ddG_per_1000sasa is an thermodynamic term that describes the strength of an interaction between two proteins normalized by surface area of interaction and Keq_cat is the collapsed experimental variable that describes real-world success while 0 means no success and 1 means success of binding of the two proteins to each other.

Logistic Regression with Keq_cat as categorical response variable and ddg_per_1000sasa as explanatory variable:

The variable ddg_per_1000sasa and Keq_cat are significantly associated with a p-value=0.006 and a 95% significance interval of 0.869026 to 0.976990 and an odds ratio of 0.921428.

Odds Ratios: the is the probability of an event occurring in one group compared to the probability of the event occurring in another group. An odds ratio of 1 means that there is an equal probability in both groups of the event occurring.

Intercept 0.002045 ddg_per_1000sasa 0.921428

Lower CI Upper CI OR Intercept 0.000446 0.009367 0.002045 ddg_per_1000sasa 0.869026 0.976990 0.921428

An odds ratio of 0.92 can be interpreted that as ddg_per_1000sasa increases, the outcome of two proteins interacting with each other becomes less likely. This would be in agreement with our hypothesis as we hypothesized that negative ddg_per_1000sasa values are significantly associated with an outcome =1 in our Keq_cat variable.

Other confounding variables such as interface shape complementarity (OR=0.158, OR interval = 0.001650 - 15.081597, p-value = 0.427) were tested but not significantly associated with Keq_cat as a response variable.

#protein design #data science #logistic regression #coursera #data management

Regression Modeling in Practice: Test a Multiple Regression Model

Instructions

This week's assignment is to test a multiple regression model.

I would like to figure out which regression model best describes the relationship between protein protein interactions per solvent accessible surface area, ddg_per_1000sasa, and interface shape complementarity, interface_sc. I will look at ddg_fa_atr as a confounding variable.

My code can be found here: https://github.com/fsee/DataScienceClass3/blob/master/Assignment3.py

Summary

Note: this analysis does not actually make physical sense but was simply performed to satisfy the requirements of the data science course.

1) multiple regression analysis

I found that interface_sc to the power of three correlates better with ddg_per_1000sasa than ddg_per_1000sasa to the power of one. Adding ddg_fa_atr as a confounding variable further improves the correlation fit.

Linear regression: (see last week’s assignment)

Multiple Regression Model

After adjusting for potential confounding factors such as ddg_fa_atr, which is the attractive component of the interaction free energy, interface_sc (interface shape complementarity) was significantly positively associated with ddg_per_1000sasa (Beta= 161.5262, p= 0.000). Adding ddg_fa_atr as a confounding explanatory variable (Beta -0.0002, p = 0.000) improves the model fit by 0.2 %.

2) These results support my hypothesis that the interface_sc and ddg_per_1000sasa are associated and that ddg_fa_atr can be considered a small but significant confounding variable.

3) It seems that ddg_fa_atr can be characterized as a confounding variable since it does slightly increase the model fit based on the R-squared metric: 0.723 vs. 0.725. However, the improvement in fit is approximately 0.2 %, which means that there might be other confounding variable that could account for a larger portion of the change in variability of ddg_per_1000sasa.

4) Performance of regression diagnostics to identify potential specification (generation of regression model) errors.

a) q-q plot: to visualize that the residuals from our regression plot are normally distributed:

The qq plot shows that the residuals are normally distributed within 2 standarized residuals as can be seen by the straight line. There are some extreme outliers that are not captured by our regression model that do not fall within this 2 standarized residual interval. This suggests that the fit to the model is relatively poor and could be improved. This is most

b) standardized residuals for all observations

The residuals become larger with increasing ddg_fa_atr scores and peak around 0 and then become smaller again. This would suggest that the model does not capture the data very well.

c) leverage plot

It is evident from the leverage plot that most observations have zero to very little leverage for the final regression model but factors that most strongly influence the mode have extremely high standardized residues of ~ 15.

#data science #proteins #protein engineering #protein design

Regression Modeling in Practice: Test a Basic Linear Regression Model

This week's assignment asks you to test a basic linear regression model for the association between your primary explanatory variable and a response variable, and to create a blog entry describing your results.

Data preparation for this assignment:

https://github.com/fsee/DataScienceClass3/blob/master/Assignment2.py

1) If your explanatory variable is categorical with more than two categories, you will need to collapse it down to two categories, or subset your data to select observations from 2 categories (next week you'll learn how to analyze categorical explanatory variable with more than 2 categories).

2) If your response variable is categorical, you will need to identify a quantitative variable in the data set that you can use as a response variable for this assignment. Variables with response scales with 4-5 values that represent a change in magnitude (for example, "strongly disagree to strongly agree", "never to often") can be considered quantitative for the assignment.

The assignment:

1) If you have a categorical explanatory variable, make sure one of your categories is coded "0" and generate a frequency table for this variable to check your coding. If you have a quantitative explanatory variable, center it so that the mean = 0 (or really close to 0) by subtracting the mean, and then calculate the mean to check your centering.

2) Test a linear regression model and summarize the results in a couple of sentences. Make sure to include statistical results (regression coefficients and p-values) in your summary.

WHAT TO SUBMIT: Create a blog entry where you 1) post your program and output, and 2) post a frequency table for your (recoded) categorical explanatory variable or report the mean for your centered explanatory variable. 3) Write a few sentences describing the results of your linear regression analysis.

1) My code for this assignment can be found here:

2) I will use interface shape complementarity as my quantitative explanatory variable, which I centered, so that the mean of the distribution =0. I will then use the quantitative response variable ddG/ 1000 SASA (which is the computational Gibb’s free interaction energy per 1000 square Angstrom between two proteins ) to perform a linear regression analysis:

Pearson coefficient: -0.31760975665006264 p-value: 4.8970235233811167e-173

There is a negative correlation between the protein interface shape complementarity and the ddG energy per 1000x solvent accessible surface area of ~-0.32 with a p value that is well below our alpha cutoff of 0.05. This analysis suggests that we can safely reject the null hypothesis and conclude that the interface shape complementarity is associated with ddG energy per 1000x solvent accessible surface area.

The explanatory variable protein interface shape complementarity was centered around 0 by subtracting the mean of interface_sc of all observations from each single observation. The new summary of my data set for protein interface_sc_centered is the following and clearly shows a new mean value that is almost 0.

count 7.398000e+03 mean -8.235880e-17 std 4.161515e-02 min -1.432997e-01 25% -2.929968e-02 50% 1.700324e-03 75% 2.970032e-02 max 1.207003e-01 Name: interface_sc_centered, dtype: float64

The linear regression analysis provides the following information:

We can conclude the following linear relationship between the two analyzed variables:

ddG_per_1000sasa = -.0039 *interface_sc - 0.0971

According to the R-squared value of 0.101, this model accounts for 10.1% of the variability that we can observe in the response variable.

This linear regression model suggests that the ddG_per_1000sasa only changes by -0.0039 units for every unit change in interface_sc and is likely not to change dramatically with a change in interface_sc.

#data science #proteins #linear regression

Regression Modeling in Practice: Assignment 1

It’s important to know how the data set you’re working with was generated, and how to describe your data management steps to others. This week’s assignment is to submit a blog entry in which you describe 1) your sample, 2) the data collection procedure, and 3) a measures section describing your variables and how you managed them to address your own research question.

1) Description of my sample

Background on protein engineering and computational protein design

Proteins are amazing molecular machines that play crucial roles in human health and disease. The human body contains more than 100,000 types of proteins. After water, proteins make up most of the weight of the human body. And proteins are the targets of almost all pharmacological drugs.

The field of protein engineering and computational protein design attempts to understand how the three dimensional structure of a protein aids its biological function and how we can manipulate both the structure and consequently the function of existing and novel proteins to develop better therapeutics, renewable energy resources, or smart materials.

The computational algorithm that designs proteins from scratch is imperfect and does not capture the real world properties of proteins in its entirety. My data set contains both metrics from the computational optimization function that designed the proteins of interest on the computer as well as data that describes how well the computer models reflect reality.

a) Describe the study population (who or what was studied).

Protein structures

b) Report the level of analysis studied (individual, group, or aggregate).

Individual protein structures

c) Report the number of observations in the data set.

n= 7153

d) Describe your data analytic sample (the sample you are using for your analyses).

I will use the entire data set for analysis.

2) The data collection procedure

The unique identifiers (n=7153) in my data set are protein complexes that consist of protein A and protein B. Protein A is a naturally occurring protein that is targeted by a completely novel protein B. The computational method that describes this protein complex evaluates many different metrics that have been found to be important descriptors of these proteins in the real world in previous studies. An additional part of the data set comes from a biochemical wet lab experiment in which all protein complexes were treated in exactly the same way to test whether the newly designed protein B forms its desired structure in reality and binds protein A.

In summary, the data collection procedure is partially a stochastic algorithm that describes the protein complex in silico and biochemical laboratory experiments that describe the same complex in vitro.

a) Report the study design that generated that data (for example: data reporting, surveys, observation, experiment).

The study comes from computational and real world observations. I would call it an experiment as a wet lab scientist but it is not experimental data that follows the definition that we just learned in the course: aka data that was influenced by the researcher while collecting the data.

b) Describe the original purpose of the data collection.

Research about how to better design novel proteins on the computer

c) Describe how the data were collected.

Computational modeling and wet lab experimental observations

d) Report when the data were collected.

A few months ago

e) Report where the data were collected.

In a biochemistry lab

3) Variable management

My research question is: Which computational variables are good predictors for real world success?

The quantitative explanatory variables I will use describe both protein B as an individual unit as well as protein A and B in complex with each other. Examples of these variables are the computational energy of folding and the shape complementarity between protein A and protein B. The response variable of experimental success is a categorical variable that describes whether the designed protein B exhibits similar properties in vitro as our in silico model would suggest.

There are a lot of lurking variables that could potentially make this analysis hard. For instance, we are using an experimental high-throughput method as a proxy for the proper formation of protein B but do not directly measure the fit and deviation to and from our designed model. One problem with this type of analysis might be false positives in which our experimental results lead us to believe that our computational parameters were accurate while they are not.

a) Describe what your explanatory and response variables measured.

Quantitative explanatory variables are numerous computational protein metrics

b) Describe the response scales for your explanatory and response variables.

The categorical response variable is a metric for real world success in the wet lab

c) Describe how you managed your explanatory and response variables.

I grouped the data set by response variable to determine statistically significant differences between the different categories.

#protein design #coursera #data analysis #linear regression #protein engineering #biochemistry #proteins

Data Analysis Tools: Assignment 4

The final assignment of the Data Analysis Tools deals with testing a potential moderator. When testing a potential moderator, we are asking the question whether there is an association between two constructs for different subgroups within the sample.

I will test whether a country’s average income can be a moderator variable for the relationship between the female to male employment rate and life expectancy using the gapminder data set.

1) My code can be found here:

https://github.com/fsee/DataScienceClass2/blob/master/Assignment4.py

2) This code generates the following output:

association between female to male employment ratio and life expectancy for LOW income countries (0.0027047009848168361, 0.98643701949869023)

association between female to male employment ratio and life expectancy for MIDDLE income countries (-0.17817757827434305, 0.10925721326852184)

association between female to male employment ratio and life expectancy for HIGH income countries (0.53637414790735216, 0.00025077753920340912)

Summary: There is a slight positive and negative correlation between the female to male employment ratio for low and middle income countries, respectively, and a stronger positive correlation between the female to male employment ratio and life expectancy for high income countries. The p values of these Pearson correlations suggest that only for the high income countries are the two compared variables statistically significantly associated while the relationship is not significant for middle and low income countries.

The corresponding scatter graphs illustrate this relationship graphically:

#data analysis #data science #coursera #gapminder

Data Analysis Tools: Assignment 3

In the third week of the Data Analysis Tools class, I will examine the correlation coefficient between the female to male employment rate and the average life expectancy of a country based on the GapMinder data set.

In Class 1 Assignment 4, I already visualized the relationship between these two variables using a scatter plot. The scatter plot showed no clear trend between the female to male employment ratio and the average life expectancy of the countries investigated.

1) The code for the correlation coefficient analysis can be found here:

https://github.com/fsee/DataScienceClass2/blob/master/Assignment3.py

2) This graph shows the scatterplot of the association beween the female to male employment ratio vs. life expectancy:

association between female to male employment ratio and life expectancy

R = -0.12849694519197702 and p= 0.089204159694071111

The Pearson correlation coefficient of R = -0.12849694519197702 tells us there is a slight negative correlation between the female to male employment ratio and the average life expectancy of a country, however, the p value of p= 0.089204159694071111, which is larger than 0.05 suggests that the relationship is not statistically significant. In conclusion, we cannot reject the null hypothesis that the female to male employment ratio and the average life expectancy of a country are not associated.

#pearson coefficient #data analysis #data visualization #data science #gapminder #coursera

Data Analysis Tools: Assignment 2

A Chi-Square Test of Independence compares frequencies of one categorical variable for different values of a second categorical variable. The null hypothesis is that the relative proportions of one variable are independent of the second variable; in other words, the proportions of one variable are the same for different values of the second variable. The alternate hypothesis is that the relative proportions of one variable are associated with the second variable. Since the GapMinder data set only includes quantitative variables, I decided to categorize the female to male employment ratio into two categories: less than 65% (=0 ) and more than 65% (=1) female to male employment.

1) My code can be found here:

https://github.com/fsee/DataScienceClass2/blob/master/Assignment2.py

2) The female to male employment rate was categorized into two categories: less or equal to than an average of 65% and more than 65% as the response variable. The explanatory variables are the 6 different continents.

This table represents the contingency table of observed counts:

continent f_m_rate_binned <65 >65

AF 17 34 AS 24 25 EU 4 35 NoAm 11 10 OC 1 5 SA 6 6

continent f_m_rate_binned <65 >65

AF 0.269841 0.295652 AS 0.380952 0.217391 EU 0.063492 0.304348 NoAm 0.174603 0.086957 OC 0.015873 0.043478 SA 0.095238 0.052174

The chi-square analysis suggests that the p-value of 0.001, which is less than the Bonferroni adjustment for 15 pairwise comparisons of 0.003. This suggests that we should reject the null hypothesis that states that the the explanatory variable “continent” is not associated with the response variable “frequency of binned female to male employment ratio”.

chi-square value, p value, expected counts

(19.517010486309509, 0.0015391951359215172, 5,

array([[ 18.0505618 , 32.9494382 ],

[ 17.34269663, 31.65730337], [ 13.80337079, 25.19662921], [ 7.43258427, 13.56741573], [ 2.12359551, 3.87640449], [ 4.24719101, 7.75280899]]))

The post hoc analysis should show which pairwise comparisons are statistically relevant, which means that we can reject the null hypothesis and accept the alternate hypothesis that there is a correlation between the continent category and the binned female to male employment ratio category.

However, the new data frame “sub_cont_mean” still contained the information about more than just the two compared continents, which is an issue that I could not troubleshoot and I will ask Stackoverflow.com for help.

Any ideas are welcome. This is an example output so far:

Pairwise comparison of: NoAm and EU f_m_employ_ratio continent f_m_rate_binned 1 69.357488 EU >65 10 77.054269 EU >65 12 83.724143 NoAm >65 15 82.040811 NoAm >65 16 83.505145 EU >65 17 75.135142 EU >65 18 51.871657 NoAm <65 23 73.473687 EU >65 27 80.190473 EU >65 32 86.490459 NoAm >65 44 52.010375 NoAm <65 46 71.897819 EU >65 47 63.982432 NoAm <65 48 76.681616 EU >65 49 73.913039 EU >65 50 85.315712 EU >65 53 62.403697 NoAm <65 56 62.569829 NoAm <65 59 85.550077 EU >65 63 87.540986 EU >65 64 80.281683 EU >65 69 77.740862 EU >65 72 61.563519 EU <65 75 79.958249 NoAm >65 77 54.054053 NoAm <65 81 46.335080 NoAm <65 82 43.622451 NoAm <65 84 74.217311 EU >65 85 89.690721 EU >65 90 74.127904 EU >65 92 62.807012 EU <65 93 68.940492 NoAm >65 104 80.604138 EU >65 110 85.017424 EU >65 111 75.122754 EU >65 113 62.325574 EU <65 119 47.401577 EU <65 121 81.623935 NoAm >65 124 51.968499 NoAm <65 126 96.888885 EU >65 136 79.502196 EU >65 137 78.631047 NoAm >65 140 44.117648 NoAm <65 144 88.133145 EU >65 148 56.955385 NoAm <65 153 76.130192 EU >65 154 78.328173 EU >65 155 68.924293 NoAm >65 158 79.347824 EU >65 159 84.326019 EU >65 170 73.001770 EU >65 174 73.658531 EU >65 175 79.166667 EU >65 179 65.876779 EU >65 184 87.635240 EU >65 185 79.608932 EU >65 194 69.655172 NoAm >65 200 83.164984 EU >65 202 81.068700 EU >65 203 81.632655 NoAm >65 f_m_rate_binned <65 >65 continent AF 0 0 AS 0 0 EU 4 35 NoAm 11 10 OC 0 0 SA 0 0 f_m_rate_binned <65 >65 continent AF 0.000000 0.000000 AS 0.000000 0.000000 EU 0.266667 0.777778 NoAm 0.733333 0.222222 OC 0.000000 0.000000 SA 0.000000 0.000000

#chi square #data analysis #data tool #coursera #data management #gapminder

Data Analysis Tools: Assignment 1

In the second class of the data science specialization, we I will learn more about the statistical tools that are needed to properly analyze multi-variate data sets.

Here I will perform a ANOVA (= analysis of variance) on the GapMinder data set. Analysis of variance assesses whether the means of two or more groups are statistically different from each other. This analysis is appropriate whenever you want to compare the means (quantitative variables) of groups (categorical variables). The null hypothesis is that there is no difference in the mean of the quantitative variable across groups (categorical variable), while the alternative is that there is a difference.

Since the GapMinder data set does not contain categorical variables, I will categorize all countries in the data set into continents groups as the categorized independent variable and will assess whether the dependent variable female to male employment ratio is statistically significantly different among the different continental groups.

The null hypothesis is: there is no difference in the mean female to male employment rate across continents.

The alternate hypothesis is: there is a difference in the mean female to male employment rate across continents.

My code can be found here: https://github.com/fsee/DataScienceClass2/blob/master/Assignment1.py

Here is the output of the F statistic and associated P-value:

Based on the P-value of 0.00952, which is < 0.05, we can reject the null hypothesis and accept the alternate hypothesis there is a difference in the mean female to male employment rate across continents. However, a significant ANOVA does not give us information about which continent groups are different from one another.

Here are the average values of the female to male employment ratio and their standard deviations:

Mean of female to male employment ratio grouped by continent

continent f_m_employ_ratio

AF 72.265396

AS 60.220208

EU 77.342409

NoAm 65.310113

OC 73.171496

SA 66.527351

Stdev of female to male employment ratio grouped by continent

continent f_m_employ_ratio

AF 20.413902

AS 24.520950

EU 9.074391

NoAm 14.259248

OC 16.262748

SA 10.254428

At this point I need to perform a post hoc test in order to avoid excessive type 1 error (error of falsely accepting the null hypothesis). They are designed to perform comparisons between the value of means while preventing inflation of type 1 error.

Here is the post hoc test that will give information about which pairwise continent group comparisons are statistically different from each other:

Tukey Post hoc analysis of female to male employment ratio grouped by continent

Multiple Comparison of Means - Tukey HSD,FWER=0.05 ============================================== group1 group2 meandiff lower upper reject ---------------------------------------------- AF AS -12.0452 -22.7568 -1.3336 True AF EU 5.077 -6.3135 16.4675 False AF NoAm -6.9553 -20.8391 6.9286 False AF OC 0.9061 -22.2047 24.0169 False AF SA -5.738 -22.9184 11.4423 False AS EU 17.1222 5.6314 28.613 True AS NoAm 5.0899 -8.8763 19.0562 False AS OC 12.9513 -10.2091 36.1117 False AS SA 6.3071 -10.9399 23.5542 False EU NoAm -12.0323 -26.5258 2.4612 False EU OC -4.1709 -27.653 19.3112 False EU SA -10.8151 -28.4917 6.8616 False NoAm OC 7.8614 -16.9263 32.649 False NoAm SA 1.2172 -18.1601 20.5946 False OC SA -6.6441 -33.4179 20.1296 False ----------------------------------------------

The post hoc analysis shows that we should only reject the null hypothesis for the pairwise comparison between Africa and Asia as well as Asia and Europe.

#data science #statistics #anova #coursera

Data Management and Visualization Completion

I am happy to report that I finished my first class of the Data Science Specialization offered by Wesleyan University on Coursera.

Here is my certificate as proof!

I will soon start the second class on Data Analysis Tools.

#data management #data visualization #data science #coursera #wesleyan #success

Data Management and Visualization: Assignment 4

My program can be found here: https://github.com/fsee/DataScienceClass1/blob/master/Assignment4.py

For this course, I have analyzed the GapMinder data set in order to determine whether female employment rate of a country and life expectancy are associated. I have considered the following variables from the GapMinder data set: female employment rate, overall employment rate, and life expectancy. For assignment 3, I have further created a secondary variable, which is the ratio of the female to male employment rate of a country (estimating binary gender distributions in all countries) and generated frequency distribution tables summarizing countries by continents.

For assignment 4 STEP 1, I plotted the four variables of interest in univariate graphs: In the univariate graphs, the independent or explanatory variable is always the country for which the data was collected. This variable is categorical by nature. The dependent and response variable is the measured/calculated quantity of female employment rate, overall employment rate, life expectancy, and female:male employment rate of that country.

Variable by Country

femaleemployrate employrate lifeexpectancy f_m_employ_ratio count 178.000000 178.000000 191.000000 178.000000 mean 47.549438 58.635955 69.753524 68.885106 std 14.625743 10.519454 9.708621 19.436830 min 11.300000 32.000000 47.794000 19.871793 25% 38.725000 51.225000 64.447000 57.395756 50% 47.549999 58.699999 73.131000 74.172607 75% 55.875000 64.975000 76.593000 82.219064 max 83.300003 83.199997 83.394000 114.484671

The count for each variables differs because some data is missing for certain countries.

Female Employment Rate Univariate Distribution

Unimodal distribution: The mean female employment rate is ~48 with a spread of +- 15 % across all assessed countries, which means that approximately half of the women contribute to the active measured workforce.

Overall Employment Rate Univariate Distribution

Unimodal distribution that peaks at ~59 with a spread of +- 11 %.

Life Expectancy Univariate Distribution

The average life expectancy across all countries is 70 years with a spread of +- 10 years. The distribution is unimodal but skewed heavily to the right side.

Female To Male Employment Rate Univariate Distribution

The mean female to male employment rate lies around 68 with a spread of +- 19 %, which means that for every three men in the workforce, there are two women on average. The data assumes a bimodal distribution in which one lobe peaks around 30% and the second one around 80%.

If we group the data by continent, this is is what the data look like:

Female Employment Rate By Continent

Overall Employment Rate By Continent

Life Expectancy By Continent

Female to Male Employment Rate By Continent

For assignment 4 STEP 2, I plotted a scatter graph of the female to male employment rate as the independent/ explanatory variable and the life expectancy of all countries as the dependent/ response variable:

Association between Female to Male Employment Rate and Life Expectancy

There seems to be no clear trend when analyzing the association between Female to Male Employment Rate and Life Expectancy. This was to be expected from the earlier inspection of the frequency distributions as some African countries had a higher participation of women than men in the workforce yet Africa as a continent shows the lowest life expectancy.

Improvement suggestions for the future:

- Learn how to label, customize, and generate plots using the seaborn package. - write code with more bite-size functions

#data science #data visualization #coursera #gapminder

Data Management and Visualization: Assignment 3

1) My program can be found here:

https://github.com/fsee/DataScienceClass1/blob/master/Assignment3.py

2) These are the data management decisions that I have made:

Classify all missing data as NaN in python so that it will not be used for further downstream data analysis

Create a secondary variable that is the ratio of the female employment rate of a country (defined as the percentage of women in the population in the workforce) divided by the male employment rate of the country (defined by the overall percentage of the male population that is employed). The GapMinder data set only provides information on the female employment rate of a country and the overall employment rate of the country. Based on the assumption that the human sex ration of binary genders is approximately equal to 1, we can estimate the female to male employment rate to be:Female Employment Rate / Male Employment Rate = Female Employment Rate / (2* Overall Employment Rate - Female Employment Rate)

I then added continent information to the data set so that I could group each country into continent groups to find out how the female to male employment ratio differs by geographic location. In order to achieve this I installed and imported geonamescache to my program. The Geonamescache package contains country-specific information about the language, population size, continent, etc. of a country. I extracted my own dictionary that maps each country in my GapMinder Data set onto one of the seven continents: 'AF’= Africa, ‘NA’=North America, 'OC’=Oceania, 'SA’=South America, 'AN’=Antartica, 'EU’=Europe, 'AS’=Asia.

This is my program output that displays four variables (female employment rate, overall employment rate, life expectancy, and female to male employment rate) as frequency tables:

Female Employment Rate Interval Frequency

min-max range 11.30000019 83.30000305 Variable Bins Frequency (11, 16] 0.014085 (16, 21] 0.023474 (21, 26] 0.032864 (26, 31] 0.028169 (31, 36] 0.056338 (36, 41] 0.103286 (41, 46] 0.122066 (46, 51] 0.136150 (51, 56] 0.117371 (56, 61] 0.075117 (61, 66] 0.032864 (66, 71] 0.042254 (71, 76] 0.014085 (76, 81] 0.028169 (81, 86] 0.009390 dtype: float64

Overall Employment Rate Interval Frequency

min-max range 32.0 83.19999695 Variable Bins Frequency (32, 37] 0.004695 (37, 42] 0.037559 (42, 47] 0.075117 (47, 52] 0.098592 (52, 57] 0.136150 (57, 62] 0.192488 (62, 67] 0.136150 (67, 72] 0.061033 (72, 77] 0.042254 (77, 82] 0.032864 (82, 87] 0.014085 dtype: float64

Life Expectancy Interval Frequency

min-max range 47.794 83.394 Variable Bins Frequency (47, 52] 0.084507 (52, 57] 0.051643 (57, 62] 0.056338 (62, 67] 0.070423 (67, 72] 0.122066 (72, 77] 0.314554 (77, 82] 0.183099 (82, 87] 0.014085 dtype: float64

Female To Male Employment Rate Interval Frequency

min-max range 19.8717931673 114.484670555 Variable Bins Frequency (19, 24] 0.028169 (24, 29] 0.009390 (29, 34] 0.028169 (34, 39] 0.018779 (39, 44] 0.018779 (44, 49] 0.042254 (49, 54] 0.032864 (54, 59] 0.037559 (59, 64] 0.070423 (64, 69] 0.065728 (69, 74] 0.061033 (74, 79] 0.117371 (79, 84] 0.131455 (84, 89] 0.084507 (89, 94] 0.042254 (94, 99] 0.028169 (99, 104] 0.014085 (104, 109] 0.000000 (109, 114] 0.000000 dtype: float64

this is my program output that groups the female to male employment rate by continent and shows frequency tables for the same bin range and size:

Female to male employment rate ordered by continent

overall min-max range 19.8717931673 114.484670555 Female to male employment rate distrib for OC Variable Bins Frequency (19, 24] 0.000000 (24, 29] 0.000000 (29, 34] 0.000000 (34, 39] 0.000000 (39, 44] 0.000000 (44, 49] 0.050000 (49, 54] 0.000000 (54, 59] 0.000000 (59, 64] 0.000000 (64, 69] 0.100000 (69, 74] 0.000000 (74, 79] 0.000000 (79, 84] 0.100000 (84, 89] 0.000000 (89, 94] 0.000000 (94, 99] 0.050000 (99, 104] 0.000000 (104, 109] 0.000000 (109, 114] 0.000000 dtype: float64

Female to male employment rate distrib for AS

Variable Bins Frequency (19, 24] 0.122449 (24, 29] 0.020408 (29, 34] 0.081633 (34, 39] 0.061224 (39, 44] 0.040816 (44, 49] 0.020408 (49, 54] 0.020408 (54, 59] 0.081633 (59, 64] 0.020408 (64, 69] 0.061224 (69, 74] 0.040816 (74, 79] 0.122449 (79, 84] 0.102041 (84, 89] 0.122449 (89, 94] 0.040816 (94, 99] 0.020408 (99, 104] 0.020408 (104, 109] 0.000000 (109, 114] 0.000000 dtype: float64

Female to male employment rate distrib for AF

Variable Bins Frequency (19, 24] 0.000000 (24, 29] 0.018519 (29, 34] 0.037037 (34, 39] 0.018519 (39, 44] 0.018519 (44, 49] 0.074074 (49, 54] 0.018519 (54, 59] 0.018519 (59, 64] 0.111111 (64, 69] 0.074074 (69, 74] 0.055556 (74, 79] 0.111111 (79, 84] 0.092593 (84, 89] 0.074074 (89, 94] 0.111111 (94, 99] 0.055556 (99, 104] 0.037037 (104, 109] 0.000000 (109, 114] 0.000000 dtype: float64

Female to male employment rate distrib for SA

Variable Bins Frequency (19, 24] 0.000000 (24, 29] 0.000000 (29, 34] 0.000000 (34, 39] 0.000000 (39, 44] 0.000000 (44, 49] 0.000000 (49, 54] 0.166667 (54, 59] 0.083333 (59, 64] 0.166667 (64, 69] 0.166667 (69, 74] 0.083333 (74, 79] 0.250000 (79, 84] 0.083333 (84, 89] 0.000000 (89, 94] 0.000000 (94, 99] 0.000000 (99, 104] 0.000000 (104, 109] 0.000000 (109, 114] 0.000000 dtype: float64

Female to male employment rate distrib for NA

Variable Bins Frequency (19, 24] 0.000000 (24, 29] 0.000000 (29, 34] 0.000000 (34, 39] 0.000000 (39, 44] 0.032258 (44, 49] 0.064516 (49, 54] 0.096774 (54, 59] 0.064516 (59, 64] 0.096774 (64, 69] 0.064516 (69, 74] 0.032258 (74, 79] 0.032258 (79, 84] 0.161290 (84, 89] 0.032258 (89, 94] 0.000000 (94, 99] 0.000000 (99, 104] 0.000000 (104, 109] 0.000000 (109, 114] 0.000000 dtype: float64

Female to male employment rate distrib for EU

Variable Bins Frequency (19, 24] 0.000000 (24, 29] 0.000000 (29, 34] 0.000000 (34, 39] 0.000000 (39, 44] 0.000000 (44, 49] 0.021277 (49, 54] 0.000000 (54, 59] 0.000000 (59, 64] 0.063830 (64, 69] 0.021277 (69, 74] 0.127660 (74, 79] 0.191489 (79, 84] 0.212766 (84, 89] 0.148936 (89, 94] 0.021277 (94, 99] 0.021277 (99, 104] 0.000000 (104, 109] 0.000000 (109, 114] 0.000000 dtype: float64

Most of the female to male employment frequency distributions lie between 39% and 84%. However, I noticed that there are three countries in this data set that have a slightly larger female working force than other countries with are largely dominated by a male working force. I was curious which countries these were, here is the result:

Countries in which more women than men work

29 Burundi 131 Mozambique 160 Rwanda Name: country, dtype: object

This is what I have learned this week:

- Any data frame can be added to any other data frame even if they do not have the same number of rows, be careful when merging raw data with frequency distributions, for instance

- a set() is a built-in Python function that outputs the unique subset of a list and gets rid of multiples that might exist in a list

- importing data dictionaries from the web e.g. country and continent information, curate them to match the unique identifiers of the GapMinder data set and use this additional information to gain geographic knowledge from the provided data

- good data analysis requires careful deliberation before anything gets implemented: suggestion for the future: Think and write the summary before impl

#data management #data visualization #data science

Data Management and Visualization: Assignment 2

1) My program can be found here:

https://github.com/fsee/DataScienceClass1/commit/a0b02fc18073ee2b3274fe433992fcb65 d1d6c36DataScienceClass1/Assignment2.py

2) this is my program output that displays three of your variables as frequency tables

3) a few sentences describing your frequency distributions in terms of the values the variables take, how often they take them, the presence of missing data, etc.

Since all the values for the data that I am looking at are unique, I took the liberty of binning the overall employment rate, the female employment rate, and the life expectancy of the investigated countries into 5% bins by starting from the minimum value that each variable (column) takes and creating 5% intervals from there.

While the overall employment rate of the countries investigated ranges from ~32% to 83.2%, the female employment rate ranges from 11.2% to 83.3%. The average life expectancies of all countries that were captured by the GapMinder data set ranged from 47 to 82 years. The data set contains missing data and partially duplicate information for each country: in the case of missing data, the country will not be taken into account in the frequency distribution calculation; in the case of duplicate data, the country will be incorrectly accounted for twice.

The tables clearly show the frequencies for each interval for each of the three investigated variables.

Future questions that arise from this that might be of interest to analyze:

1. Is the ratio of the female employment rate to the overall employment rate similar in every country? If not, how does the ratio differ by continent/ world region?

2. Is the female and/or overall employment rate directly associated with life expectancy?

3. Which are the countries at the extremes of the frequency distribution? Are they the same for all three variables?

Note: I am excited to learn more about the pandas and numpy libraries and how to a) print the frequency distribution tables including the variable name b) format the tables for a more appealing appearance c) plot and visualize the data for easier analysis

#data management #data visualization #coursera #learn python

Data Management and Visualization: Assignment 1

After looking through the available Codebooks I determined that I am particularly interested in learning how the female employment rate is associated with the overall employment rate of a country. To that end, I selected the data from the GapMinder set and specifically determined the following variables to the significant for my analysis: female employment rate, overall employment rate, and income per person. The unique identifier for this data set is each country for which these data are available.

As my second topic of interest, I am curious of whether the female employment rate in any given country are associated with life expectancy. To that end, I will add the life expectancy variable to my codebook.

Based on data from the department of labor (http://www.dol.gov/wb/factsheets/Qf- ESWM07.htm ; search term “women employment and general employment rate”) I hypothesize that the female employment rate is associated with the overall employment rate as the female employment rate is part of the overall employment rate but the rate does not change at the same rate.

For the second question, based on a public health report that reports an increase in life expectancy with female employment (Rogot, E., Sorlie, P. D. & Johnson, N. J. Life expectancy by employment status, income, and education in the National Longitudinal Mortality Study. Public Health Rep. Wash. DC 1974 107, 457–461 (1992); search term: women employment and life expectancy), I hypothesize that life expectancy is positively associated with female employment rate.

#datamanagement data visualization coursera learnpyt

Trending Blogs

Recently Viewed Blogs

Proteins and Data Blog