Data

Hi there. I am learning new things, especially data science.

[briefexpertfun-blog]

Milestone Assignment 3: Preliminary Results

Descriptive Statistics:

The first test used was ANOVA.

In the table below you can see one example of the descriptive statistics for the response variable- Ethnic group and Drug abuse behavior like categorical predictors. The one question is now( and variable ): Your drug use is a problem for you?

The average ethnic groups were between  groups 3.5, or 4-6 that are Mexican National, Puerto Rican and South American(sd= between 0.00, 2.94, 0.00, 1.73 and 3.21- from the first to the fifth group ), and for the mean of groups ( from 5, 3.6, 5, 4.9  into 3.4 ) where the ethnic groups are:

(1)  Central American  

(2)  Cuban

(3)  Dominican

(4)  Mexican National

(5)  Puerto Rican

(6)  South American

(7)  Other (specify)

(8) Mexican American

and  drug using is coded like this:

(1) Disagree Strongly (2)Disagree (3)Uncertain (4)Agree (5)Agree Strongly.

The groups of drug using: 2 ( Disagree ) and  5 ( Agree Strongly ) are in the middle with the mean 3.6 and 3.4. And for the groups 1, 3, 4 is the mean approximately in a value of 5.00.

P-value is greater than 0.05 ( > 0.6342 ) which means that there is not any correlation with the race and  drug abuse.

Another test which was conducted:

CHI SQUARE test of statistics. We know that this test is used either as a fit test ( determines if a sample data matches a population ) or we run a chi-square test of independence for comparing two variables in a contingency table to see if they are related. Used was the chi-square test of independence In a more general sense, it tests to see whether distributions of categorical variables differ from each another.  

Here was used another explanatory variable TCU001  Use more drugs than intended or for longer amount of time. And we must perform post hoc test known as the BONFERRONI Adjustment.

 According to the table above the outcome was that the probability or the p- value is greater than the value 0,05 in all events.

( Chi-Square= 0.4632, Likelihood Ratio Chi-Square= 0.2865, Mantel-Haenszel Chi-Square= 0.3963 ) which means that again there is not any correlation between the ethnic groups and using drugs. If it is really true, we will find the answer on this question in the later examples and analysis. And the image above is not a full picture and one right answer.

Bivariate Analyses:

First I used ANOVA/DUNCAN analysis.

Scatter plots and bar charts revealed that the association between the ethnic group response variable and too for categorical predictors like using drugs are not in coherence. Although most of the answers were yes- positive.

Ethnic groups were not associated ( F Value =0.72 , p>0.4002 ), and for the questions the answers were like this:

Received residential treatment in prison:

1- yes ( 68% )

0- no ( 32%),

Received residential treatment after release from prison:

0- no ( 50%)

1- yes (14%)

2- Still in Prison (36%).

Your drug use is a problem for you:

1 (11%)Disagree Strongly

2 (10%)Disagree

3 (5%)Uncertain

4 (37%)Agree

5 (38%)Agree Strongly.

Use more drugs than intended or for longer amount of time:

1- yes (89%)

0- no (11%)

Unable to cut down:

1- yes (72%)

0- no (28%)

In the Scatter plot below is:

In this Explanatory variable ( Use more drugs than intended or for longer amount of time ) and in a response variable Ethnic groups is clear that only two categories- Puerto Rican and Other answered no and the same is true for yes (Puerto Rican and Other).

According to the answers above and in the scatter plot below we can see that 89% of the people had some problem in using drugs and association between the races are not obvious ( below is one example of the Scatter plot, though it is not fully completed ), we can partly see that 1- yes is for the group with the positive association and 0 for none. And 0- negative is too for only Puerto Rican and Other categories( points 5 and 7).

Again, Ethnic groups were not- significantly associated with the number of steps involved in the process (F Value =0.72, p= 0.4002 ) and the drugs using. For example the table TCU001 is for categorical: Use more drugs than intended or for longer amount of time still with the p-value> 0.4002 – no association.

Other bar graphs show us that in most cases there is not any association in using drugs:

RTPRRC Received residential treatment in prison or 1- yes (68% ) 0- no ( 32% )

RTARRC Received residential treatment after release from prison  still 0- no ( 50%)

1- yes (14%)

2- Still in Prison (36%)

CESI001 Your drug use is a problem for you showed us again:

(1)Disagree Strongly (2)Disagree (3)Uncertain (4)Agree (5)Agree Strongly

with the most answers 4-  Agree and 5- Agree Strongly that is 36 respectively 38%.

1 and 2 are for disagree strongly or disagree(11, 10%).

TCU002 is Unable to cut down 1- yes (72%) 0- no (28%).

Although the most answers were positive: for example drug use is a problem or Unable to cut down, still p- value is unbelievably high.

There is again no association between the race and using drugs( p-value= 0.4002 ).

Another test, PROC GLM- general linear model  that I used is for a basic linear regression. I was adding all of the categorical explanatory variables.  

 ( RTPRRC  Received residential treatment in prison

    RTARRC  Received residential treatment after release from prison

    CESI001  Your drug use is a problem for you

    TCU001   Use more drugs than intended or for longer amount of time

    TCU002   Unable to cut down )

As you can see below:

only explanatory variable Received residential treatment after release from prison is in the correlation with the categorical response variable( R-Square=0.199, The F statistic is 11.87 and the P- value is very small ( p- value = 0.0011 ). Considerably less than our alpha level of .05 which tells us that we can reject the null hypothesis and conclude that only this one Received residential treatment after release from prison is significantly associated with Ethnic groups.          

Other p-values for the other explanatory variables were not associated ( p- value is always  greater  than the alpha level 0.05 ).

In The GLM Procedure Means are 5.05 for no, negative association and mean 4.52 is for yes, positive association. StDev in the first case is 1.43 and second 2.22. Min values are equal for both of the variables- O and Max value is 7.00 too for both ones.

Once again there is no association for any of the races and for the most of the variables.

In the last scatter plot is graphically explained that the ethnic groups are significantly similar in both cases- both have the same values, negative or positive ( only groups 5 and 7 ) Puerto Rican and Other.

Where a negative correlation is visible from the left point- values 5, nearly 6 (  South American did not touch ) and the skewness is to the lower right near the point 4 ( Mexican National, again did not touch ). One positive value is too at the bottom  on the right, which is in this case Spending lots of time getting, using, recovering from drug use plus the groups 5 and 7- Puerto Rican and Other.

[briefexpertfun-blog]

Data Analysis-capstone Milestone Assignment 2: Methods

In this study was used this data set:

Criminal Justice Drug Abuse Treatment Studies (CJDATS):Step 'Nout, 2002-2006 [UnitedStates].

Sample:

In the Criminal Justice Drug Abuse Treatment Studies the target population is parolees with pre-incarceration substance use disorders who are at moderate-to-high-risk of recidivism.

Inclusion criteria are:

(a) at least 18 years of age;

(b) English speaking;

(c) Probable drug dependence immediately prior to incarceration as determined by a score of 3 or higher on the TCU Drug Screen II (Knight, Simpson, & Hiller, 2002) or mandated drug treatment; (d) Substance use treatment as a mandated or recommended condition of parole;

(e) Moderate-to-high-risk of drug relapse and=or recidivism as determined by a Lifestyle Criminality Screening Form (LCSF) score of or greater (Walters & McDonough, 1998), or a history of two or more prior episodes of drug abuse treatment or drug-related convictions.

Exclusion criteria are: (a) psychotic symptoms on a SCID screener (First, 2002); and (b) Correctional or supervision conditions that prohibit them from participating in the study, including failure to leave prison on parole or probation; mandate to a special parole caseload; or transfer to a non-participating supervision office.

Characteristics of Participants:

Recruitment for the Step’n Out study began in March 2005 and continued until September 2007. The refusal rate has been approximately 10% with another 10% of contacted subjects failing to meet eligibility criteria. Across sites, 406 participants have been randomized. Subjects are approximately 83% male, mean age is 34 #8.9 years, and traditional racial minorities comprise the majority enrolled.

Measures:

The ethnic group ( exactly- What ethnic group do you consider yourself? ) is the response variable

and it was measured for our main focus. This are mainly the groups which are the most drug-dependent, or addicted:

Predictors included:

Received residential treatment in prison,

Received residential treatment after release from prison

Your drug use is a problem for you

Use more drugs than intended or for longer amount of time

Unable to cut down

( CHETHN What ethnic group do you consider yourself?

RTPRRC Received residential treatment in prison

RTARRC Received residential treatment after release from prison

CESI001 Your drug use is a problem for you

TCU001 Use more drugs than intended or for longer amount of time

TCU002 Unable to cut down )

The records were used to determine these factors:

if there are any treatments in prison or not and after releasing it,

if withdrawal when not taking drug,

if the troubles were so long after non using drugs, or:

there is any dependance on these groups.

Analyses:

The distributions for the predictors and the ethnic group response variable were evaluated by examining frequency tables for categorical variables and calculating the mean, standard deviation and minimum and maximum values for categorical variables. Because my all- response and explanatory variables are categorical.

Scatter plots and box plots were also examined, and

ANOVA test is a way to find out if our experiment results are significant.

We can figure out if we need to reject the null hypothesis or accept the alternate hypothesis. One-way or two-way refers to the number of independent variables in the Analysis of Variance test. One-way has one independent variable and two-way has two independent variables (can have multiple levels). The one-way analysis of variance (ANOVA) was used to determine whether there are any statistically significant differences between the means of independent (unrelated) groups.

CHI SQUARE test was used for the chi-square statistic and distribution for independence

- compares two variables in a contingency table to see if they are related. In a more general sense, it tests to see whether distributions of categorical variables differ from each another to identify the subset of variables that best predicted ethnic group.

ANOVA/DUNCAN was used: When we run Analysis of Variance, the results will tell us if there is a difference in means. However, it won’t pinpoint which means are different. Duncan’s Multiple Range test (DMRT) is a post hoc test to measure specific differences between pairs of means.

PROC GLM- general linear model that I used is for a basic linear regression. I was adding all of the categorical explanatory variables for identifying the groups that are in accordance with the response variable or not.   

[briefexpertfun-blog]

( Data Analysis and Interpretation Capstone Week 1) Title and Introduction to the Research Question: The  relationship between the ethnic group and drug abuse behavior

In this study was used this data set:

Criminal Justice Drug Abuse Treatment Studies (CJDATS):Step 'Nout, 2002-2006 [UnitedStates].

I want to find some related factors as an answer on this simple question:

Is there any stronger relationship between the ethnic group and drug abuse behavior?

I want to find the answers on that stuff and later I can add the subsequent outcomes within that area.

After that I will too add the following supplemented questions, like for example: Received residential treatment after release from prison or: Using more drugs than intended or for longer amount of time and

Spending lots of time getting, using, recovering from drug use?

Source ( for all the Capstone project ):

PETER D. FRIEDMANN , ELIZABETH C. KATZ , ANNE G. RHODES , FAYE S. TAXMAN , DANIEL J. O'CONNELL , LINDA K. FRISMAN , WILLIAM M. BURDON , BENNETT W. FLETCHER , MARK D. LITT , JENNIFER CLARKE & STEVEN S. MARTIN

To cite this article: PETER D. FRIEDMANN , ELIZABETH C. KATZ , ANNE G. RHODES , FAYE S. TAXMAN , DANIEL J. O'CONNELL , LINDA K. FRISMAN , WILLIAM M. BURDON , BENNETT W. FLETCHER , MARK D. LITT , JENNIFER CLARKE & STEVEN S. MARTIN (2008) Collaborative Behavioral Management for Drug-Involved Parolees: Rationale and Design of the Step'n Out Study, Journal of Offender Rehabilitation

[briefexpertfun-blog]

K-mean cluster analysis  - Week 4 Machine Learning/ SAS

libname mydata "/courses/d1406ae5ba27fe300" access=readonly;

*DATA MANAGEMENT;

data clust;

set mydata.gapminder;

* create a unique identifier to merge cluster assignment variable with 

the main data set;

country=_n_;

keep country co2emissions femaleemployrate hivrate internetuserate lifeexpectancy oilperperson polityscore 

relectricperperson suicideper100th employrate urbanrate incomeperperson;

 * delete observations with missing data;

 if cmiss(of _all_) then delete;

 run;

ods graphics on;

* Split data randomly into test and training data;

proc surveyselect data=clust out=traintest seed = 123

   samprate=0.7 method=srs outall;

run;   

data clus_train;

set traintest;

if selected=1;

run;

data clus_test;

set traintest;

if selected=0;

run;

* standardize the clustering variables to have a mean of 0 and standard deviation of 1;

proc standard data=clus_train out=clustvar mean=0 std=1; 

var co2emissions femaleemployrate hivrate internetuserate lifeexpectancy oilperperson polityscore 

relectricperperson suicideper100th employrate urbanrate; 

run; 

%macro kmean(K);

proc fastclus data=clustvar out=outdata&K. outstat=cluststat&K. maxclusters= &K. maxiter=300;

var co2emissions femaleemployrate hivrate internetuserate lifeexpectancy oilperperson polityscore 

relectricperperson suicideper100th employrate urbanrate;

run;

%mend;

%kmean(1);

%kmean(2);

%kmean(3);

%kmean(4);

%kmean(5);

%kmean(6);

%kmean(7);

%kmean(8);

%kmean(9);

* extract r-square values from each cluster solution and then merge them to plot elbow curve;

data clus1;

set cluststat1;

nclust=1;

if _type_='RSQ';

keep nclust over_all;

run;

data clus2;

set cluststat2;

nclust=2;

if _type_='RSQ';

keep nclust over_all;

run;

data clus3;

set cluststat3;

nclust=3;

if _type_='RSQ';

keep nclust over_all;

run;

data clus4;

set cluststat4;

nclust=4;

if _type_='RSQ';

keep nclust over_all;

run;

data clus5;

set cluststat5;

nclust=5;

if _type_='RSQ';

keep nclust over_all;

run;

data clus6;

set cluststat6;

nclust=6;

if _type_='RSQ';

keep nclust over_all;

run;

data clus7;

set cluststat7;

nclust=7;

if _type_='RSQ';

keep nclust over_all;

run;

data clus8;

set cluststat8;

nclust=8;

if _type_='RSQ';

keep nclust over_all;

run;

data clus9;

set cluststat9;

nclust=9;

if _type_='RSQ';

keep nclust over_all;

run;

data clusrsquare;

set clus1 clus2 clus3 clus4 clus5 clus6 clus7 clus8 clus9;

run;

* plot elbow curve using r-square values;

symbol1 color=blue interpol=join;

proc gplot data=clusrsquare;

 plot over_all*nclust;

 run;

*further examine cluster solution for the number of clusters suggested by the elbow curve

* plot clusters for 4 cluster solution;

proc candisc data=outdata4 out=clustcan;

class cluster;

var co2emissions femaleemployrate hivrate internetuserate lifeexpectancy oilperperson polityscore 

relectricperperson suicideper100th employrate urbanrate;

run;

proc sgplot data=clustcan;

scatter y=can2 x=can1 / group=cluster;

run;

* validate clusters on incomeperperson;

* first merge clustering variable and assignment data with incomeperperson data;

data incomeperperson_data;

set clus_train;

keep country incomeperperson;

run;

proc sort data=outdata4;

by country;

run;

proc sort data=incomeperperson_data;

by country;

run;

data merged;

merge outdata4 incomeperperson_data;

by country;

run;

proc sort data=merged;

by cluster;

run;

proc means data=merged;

var incomeperperson;

by cluster;

run;

proc anova data=merged;

class cluster;

model incomeperperson = cluster;

means cluster/tukey;

run;

I used the variables from the gapminder datas. The clustering variables are:

co2emissions femaleemployrate hivrate internetuserate lifeexpectancy oilperperson polityscore relectricperperson suicideper100th employrate urbanrate,                                                                                                          plus an incomeperperson which I'm going to use to externally validate the clusters. A country is an unique identifier to merge cluster assignment variable with the main data set.

After running the code this plot shows the increase in the proportion advantage in the clustering variables( explained by each of the cluster solutions ). We start with the K equals 1R squared which is zero because there's no clustering yet. Then we can see that the two cluster solution accounts for about 22% of the variance. The R-square value increases as more clusters are specified. What we're looking for here is a bend in the elbow that kind of shows where the R-square value might be leveling off. Sets the adding more cluster doesn't increase the r-square is much. There appears to be a couple of bends in the line at 2 clusters, 4 clusters, 6 and at 8 clusters. Figuring out the best solution, we should further examine the results for the 2, 4, 6 and 8 cluster solutions( To see whether the clusters overlap ). And whether there are significant differences between the clusters on our external validation variable, incomeperperson.

The cluster analysis in the scatter plot:

There are only two canonical observations by the cluster they were assigned to in the cluster analysis. The observations in clusters 4 and 1 are densely packed, meaning they are highly correlated/ each other, but it is not possible find the right way for other clusters, which are missing.

The means on the clustering variables show that compared to the other clusters, I think incomeperperson in cluster 1 had moderate levels on a clustering variable. They had a relatively low likelihood of using lifeexpectancy or urbanrate, but moderate levels of relectricperperson and internetuserate. They also appeared to have fairly low levels of lifeexpectancy, hivrate, and too urbanrate. Other models within this field are in the tables.

The box plots showed the mean for incomeperperson by cluster. The clusters 1 and 4 are in high values, the most troubled groups in the cluster 2 and 3 had the lowest incomeperperson, but they are still in the middle- in the approximate exact mean of the value.

The Tukey test shows that the clusters differed significantly in the mean income per persone with the exception of clusters 2 and 3, which did not differ significantly from each other.

[yvesmulkers]

Visual Marketing and the Future of Geolocation

Visual Marketing and the Future of Geolocation“8 Habits of Highly Successful Entrepreneurs”May 6, 2016Join us for a free, live webinar covering social media strategies to increase sales. Tune in 5/17 at 9 a.m. PT. Register Now »Even when it’s armed with tools to target a specific consumer gr ..

Read More at:

http://bit.ly/2fWgb3y

[yvesmulkers]

Big Data Analytics: Finding Healthcare in a New Frontier

It’s the new millennium, and instilled therewith came the progressive cycle of technological development, bringing in the age of Big Data analytics and secure cloud resources. The past decade has brought on breath-taking advances which coordinate the way we routinely store mass amounts of informat ..

Read More at:

http://bit.ly/2yHwNzs

[briefexpertfun-blog]

Lasso Regression

libname mydata "/courses/d1406ae5ba27fe300" access=readonly;

DATA new; set mydata.gapminder; 

 keep country co2emissions femaleemployrate hivrate internetuserate lifeexpectancy oilperperson polityscore relectricperperson suicideper100th employrate urbanrate; 

*delete observations with missing data; 

if cmiss(of _ all_) then delete; run;

ods graphics on; 

 * Split data randomly into test and training data;

proc surveyselect data=new out=traintest seed = 123 samprate=0.7 method=srs outall; run;

ods graphics on;

Split data randomly into test and training data; proc surveyselect data=new out=traintest seed = 123 samprate=0.7 method=srs outall; run;

lasso multiple regression with lars algorithm k=10 fold validation; proc glmselect data=traintest plots=all seed=123; partition ROLE=selected(train='1' test='0'); 

class country; 

model co2emissions = country femaleemployrate hivrate internetuserate lifeexpectancy oilperperson polityscore relectricperperson suicideper100th employrate urbanrate/selection=lar(choose=cv stop=none) cvmethod=random(10); 

run;

A lasso regression analysis was conducted to identify a subset of variables from a pool of 11 categorical and quantitative predictor variables that best predicted a quantitative response variable measuring co2emissions. Categorical predictors which are included: only one- country ( 56 countries altogether ) to improve interpretability of the selected model with fewer predictors. 

Quantitative predictor variables include femaleemployrate, hivrate, internetuserate, lifeexpectancy, oil consumption per person, polityscore ( democracy ), residential electricity consumption, suicide per 100 000 people, employrate, and urbanrate. All predictor variables were standardized to have a mean of zero and a standard deviation of one. Data were randomly split into a training set that included 70% of the observations (N=40) and a test set that included 30% of the observations (N=16). The least angle regression algorithm with k=10 fold cross validation was used to estimate the lasso regression model in the training set, and the model was validated using the test set. The change in the cross validation average (mean) squared error at each step was used to identify the best subset of predictor variables.

The table for LAR selection information  shows the steps in the analysis and the variable that is entered at each step. Of the 11 predictor variables, 10 were retained in the selected model. During the estimation process femaleemployrate_56,53,46, and internetuserate were most strongly associated with co2 emissions, followed by femaleemployrate_45,32  and may too with hivrate- 0.2. The best predictor of all is: femaleemployrate_56.

All of the predictors were nearly positively associated with co2 emissions according to Average Square Error ( ASE and Test ASE table) and mainly femaleemployrate, internetuserate, hivrate were positively associated with co2 emissions. These variables accounted for 39.5% of the variance in the co2 emissions response variable.

[yvesmulkers]

Machine learning could help revolutionize cancer diagnosis

Machine learning is a subfield of computer science, that grew out of the quest for artificial intelligence. It is so pervasive in today’s world that you probably use it often in daily life, without even realising it. Machine learning has given us self-driving cars, effective web search, recommend ..

Read More at:

http://bit.ly/2yy66NQ

[briefexpertfun-blog]

Running a Random Forest               ( Week 3 Machine Learning SAS )

LIBNAME mydata "/courses/d1406ae5ba27fe300 " access=readonly;

DATA new; set mydata.addhealth_pds;

PROC SORT; BY AID;

proc hpforest;

target H1TS4/level=nominal;

input H1GH1 H1GH1A H1TS1 H1TS3 H1TS5 H1TS6 /level=nominal;

  input age H1DA3 H1DA4 H1DA5 H1DA6 H1GH2 H1GH10

  H1GH12 H1GH13 H1GH14  H1GH17 /level=interval;

RUN;

Random forest analysis was performed in order to evaluate the importance of a series of explanatory variables in predicting a binary, categorical response variable/H1TS4. The following explanatory variables ( categorical and quantitative ) were included as possible contributors to a random forest evaluating The problems of being overweight (my target or response variable):

How is your health, Difficulty using your hands,  Foods you shouldn’t eat, Smoking, Drinking, and Drug abuse. Then Poor appetite, Watching television or videos, or playing video games, Going roller-blading, roller-skating, skate-boarding, or bicycling, play an active sport, such as baseball, softball, basketball, Doing exercise, such as jogging, walking, karate, Having a headache and Feeling sick.

From the Model Information table in the output there are these values:

Variables to Try: 4( exlanatory variables randomly selected ), Maximum Trees:100, In performing the Bagging process: 60% samples, SPLIT criteria: GINI index.  Number of  observations read is the same as a number of observations used: 6504.

The explanatory variables with the highest relative importance scores are:       the foods you should and shouldn’t eat,  smoking, drinking and  drug abuse with the tables H1TS1, H1TS3, H1TS5, H1TS6. Missclasification rate is 0.405, so we can calculate the accuracy: 100-40.5. So we can tell that the accuracy of the random forest is 59.5%, with the subsequent growing of multiple trees rather than a single tree, adding little to the overall accuracy of the model ( and suggesting that interpretation of a single decision tree may be appropriate ).

[briefexpertfun-blog]

Running a Classification Tree (Week 1 Machine Learning)

LIBNAME mydata "/courses/d1406ae5ba27fe300 " access=readonly;

DATA new; set mydata.addhealth_pds;

/* Datamanagement for Being overweight */

IF H1TS4=8 THEN H1TS4=.;

IF H1TS4=6 THEN H1TS4=.;

IF H1TS4=. THEN H1TS4=9;

if H1TS4=1 then H1TS4=1;

else H1TS4=2;

ods graphics on;

proc hpsplit assignmissing=popular seed=13039;

class H1TS4 H1GH1 H1GH1A H1TS1 H1TS3 H1TS5 H1TS6

  H1DA3 H1DA4 H1DA5  H1DA6  H1GH2  H1GH10;

model H1TS4 =AGE H1GH1 H1GH1A H1TS1 H1TS3 H1TS5 H1TS6  H1GH17

  H1DA3 H1DA4 H1DA5 H1DA6 H1GH2 H1GH10

  H1GH12 H1GH13 H1GH14  H1GH17;

grow entropy;

prune costcomplexity;

RUN;

In the example above/ below was performed Decision tree analysis in order to test nonlinear relationships among a series of explanatory variables and a binary, categorical response variable. All possible separations (categorical) or cut points (quantitative) are tested. For the analyses, from a few I chose an option entropy for the split criterion/ to grow the tree. And a cost complexity algorithm was used for pruning the full tree into a final subtree.

The following explanatory variables were included as possible contributors to a classification tree model evaluating The problems of being overweight ( H1TS4- for a response variable), age, How is your health, Difficulty using your hands, Foods you shouldn’t eat, Smoking, Drinking, and Drug abuse. Then Poor appetite, Watching television or videos, or play video games, Going roller-blading, roller-skating, skate-boarding, or bicycling, play an active sport, such as baseball, softball, basketball, Doing exercise, such as jogging, walking, karate, Having a headache and Feeling sick.

The HP SPLIT Procedure:

Number of Leaves Before Pruning: 231 leaves

Number of Leaves After Pruning: 12 leaves

Model Event Level: 1 which means that is yes for the target variable             The problems of being overweight.

Number of Observations Read: 6504

Number of Observations Used: 6504

Proc HPSLIT creates a plot of the cross validated Average standard Error/ASE or Average missclasification rate based on number of leaves each of the trees generated on the training sample. Vertical reference line represents the tree with the number of leaves that has the lowest cross validated ASE. It is 8 for this line. Horizontal reference line represents the average standard error plus one standard error for the complexity parameter. 1-SE is used for pruning the cost complexity. Minimal average missclasification rate is 0.315. Parameter is 0.0010.

The pruning plot for a smaller tree with the subtree started at node =0 is presented with splits for these tables: H1TS1/ The foods you should and shouldn’t eat, H1TS3/ Smoking and Age. Model-based confusion matrix shows the performance for the final classification tree. The model classified correctly 87% : one minus error rate (1- 0.13) for the people who have some problems in overweight. And 43% (1- 0.57) is for those who did not. People who eat non healthy food, are more likely prone to smoking in age >= 15 ( exactly in the age >=15.732).

From the Model-Based Fit Statistics for Selected Tree also Sensitivity with 0.870 value and Specificity with 0.43 value is clearly right the statement above:

 The total model classified 31% of the sample correctly, 87% experimenters (Sensitivity) and 43% of age above 15/ smokers (Specificity). Moreover it was graphically plotted in the last plot ROC Curve for H1TS4.

[thumbsupviz]

A scale is one of the elements that separates a visualization from a picture. Not always easy, especially from an extreme perspective, or if the background of an image is 200 miles away from the foreground.

David McClure elegantly solves this problem by including familiar (to North Americans, at least) objects on a photograph taken from orbit. The solouhette of a six-foot-tall man corresponds to the scale in the foreground (the Agena Target Vehicle), and an outline of Manhattan gives a sense of the size of Isla San Jose and the Baja California Coast.

-RS

[briefexpertfun-blog]

Logistic Regression Model/ Week 4

/* H1GH1A Do you have difficulty using your hands, arms, legs, or feet because of a 

   physical condition that has lasted for the past twelve months or more?

   H1GH1 In general, how is your health?  Would you say ... 

   H1TS4 The problems of being overweight - Response VARIABLE */

LIBNAME mydata "/courses/d1406ae5ba27fe300 " access=readonly;

DATA new; set mydata.addhealth_pds;

LABEL H1GH1A="Difficulty of limbs"

      H1GH1= "General Health"

      H1TS4= "Being overweight"      

/* Datamanagement for Difficulty of limbs */

      IF H1GH1A=8 THEN H1GH1A=.;

      IF H1GH1A=7 THEN H1GH1A=.;

      IF H1GH1A=6 THEN H1GH1A=.;

      IF H1GH1A=. THEN H1GH1A=9;

      if H1GH1A=1 then H1GH1A=1;

      else H1GH1A=0;   

/* Datamanagement for Being overweight- response var collapsed to two categories:0 and 1 */

      IF H1TS4=8 THEN H1TS4=.;

      IF H1TS4=6 THEN H1TS4=.;

      IF H1TS4=. THEN H1TS4=9;

      if H1TS4=1 then H1TS4=1;

      else H1TS4=0;   

/* Datamanagement for General Health collapsed to two categories:0 and 1 */    

      IF H1GH1=8 THEN H1GH1=.;

      IF H1GH1=6 THEN H1GH1=.;

      IF H1GH1=. THEN H1GH1=9;

      IF H1GH1=1 or H1GH1=2 THEN H1GH1=0;

      IF H1GH1=3 or H1GH1=4 THEN H1GH1=1;

      IF H1GH1=5 THEN H1GH1=2;

      /* Frequency for Explanatory Variables */    

PROC FREQ; TABLES H1GH1 H1GH1A;

* bar graph with 2 categories coded 0 and 1;

PROC GCHART; vbar H1GH1/discrete 

type=mean sumvar=H1TS4;

run;

* centering quantitative number of gen.health;

* print mean;

PROC MEANS;

var H1GH1;

run;

* centering (subtract mean);

data new2;

set new;

H1GH1_c = H1GH1 - 2.1097786;

run;

* check coding;

PROC MEANS;

var H1GH1_c;

run;

*adding 5 category In general, how is your health?  Would you say...;

PROC GLM; 

class H1GH1 (ref="0"); 

model H1TS4= H1DA1 H1GH1A H1GH1_c H1DA3 H1DA7

H1GH1/solution;

run;

* change the reference group to non-Health good;

PROC GLM; 

class H1GH1 (ref="1"); 

model H1TS4=H1DA1 H1GH1A H1GH1_c H1DA3 H1DA7

H1GH1/solution;

run;

****************************************************************************************************

LOGISTIC REGRESSION

****************************************************************************************************;

Proc logistic descending; model H1TS4= H1TS3;

run;

* adding headache;

Proc logistic descending; model H1TS4= H1GH2 H1TS3;

run;

Like in a Multiple Regression I have chosen the same variables in the Linear regression for some Explanatory variables:

H1GH1A- Do you have difficulty using your hands, arms, legs, or feet because of a physical condition that has lasted for the past twelve months or more?

H1GH1- In general, how is your health? Would you say ...

Categorical response variable: H1TS4 The problems of being overweight.

An Explanatory variable-

I collapsed H1GH1 down to two categories and the same I did with the Categorical response variable H1TS4 with its 5 categories. These are: excellent, very good, good, fair, poor. Now they are coded like this:

0- excellent, very good,

1- good, fair,

2- poor.

9- missing values.

Categorical response variable: H1TS4 is collapsed down to two categories 0 and 1.

Other variables are added below.

One question is if there are any relationship for a General Health associated with a Being overweight and other quantitative variables.

As we can see a table for parameter estimates, there are these regression coeficients

for a categorical General Health variable:

H1GH1 0 is our reference group for , In general, how is your health with categories excellent and very good.- regression coeficient= 0, no estimate of the standard error of p-value

H1GH1 9 is for any of missing values- regression coeficient= 0, no estimate of the standard error of p-value.

Other dummy codes are:

H1GH1 1 -.086 with the p-value 0.0003

H1GH1 2 0.17 with p-value 0.0994

Groups H1GH1 1 and H1GH1 2 compare health categories for the excellent or similar general health group. Within that only H1GH1 1 is associated. P-value is less than our alpha level 0.05, which stands for category good/ fair health.

Then we can see that H1GH1_c is highly associated, p- value is 0.0003. Also the table H1DA7 is too associated with the p- value 0.0027. This is for- During the past week, how many times did you just hang out with friends?

More quantitatives:

H1TS3 is for a category smoking- estimate is 1.2108 with in alpha level and p- value <.0001 and the finding is that there is a significant regression according to our response variable H1TS4- overweight.

Linear equation is like Being overweight= -0.7279+1.2108( smoking ).

Odds ratio/ is not linear point/ according to that means that people who smoke are 3.4 times more likely to have overweight than those who don´t smoke. Confidence interval for odds ratio is from 2.776 to 4.057 % for 95% confidence limit. But this is only a sample of a population.Now it is between 95 and 100%.

If we compare both samples- smoking H1TS3 and H1GH2 How often have you had a headache?

Again only smoking is independently associated with this finding.

Estimate for headacheh is -0.0102 and p- value 0.7645. No association.

Estimate for smoking is 1.2116 and p- value 0.7645 <.0001. There is the association.  

People who smoke are 3.4 more likely to have overweight than people without smoking after controlling for some headache. Group of people with a headache are not 0.990 times likely to have overweight.

Probability modeled is H1TS4=1. We know that 1 means the presence and number 0 is for an absence. So for our response variable is outcome- yes, there is a strong association between the General Health and Being overweight.

[briefexpertfun-blog]

Multiple Regression  Model/ Week 3

/* H1GH1A Do you have difficulty using your hands, arms, legs, or feet because of a   physical condition that has lasted for the past twelve months or more?   H1GH1 In general, how is your health?  Would you say ...   H1TS4 The problems of being overweight - Response VAR */

LIBNAME mydata "/courses/d1406ae5ba27fe300 " access=readonly; DATA new; set mydata.addhealth_pds; LABEL H1GH1A="Difficulty of limbs"      H1GH1= "General Health"      H1TS4= "Being overweight"

/* Datamanagement for Difficulty of limbs */      IF H1GH1A=8 THEN H1GH1A=.;      IF H1GH1A=7 THEN H1GH1A=.;      IF H1GH1A=6 THEN H1GH1A=.;      IF H1GH1A=. THEN H1GH1A=9;      if H1GH1A=1 then H1GH1A=1;      else H1GH1A=0;   /* Datamanagement for Being overweight */      IF H1TS4=8 THEN H1TS4=.;      IF H1TS4=6 THEN H1TS4=.;      IF H1TS4=. THEN H1TS4=9; /* Datamanagement for General Health */          IF H1GH1=8 THEN H1GH1=.;      IF H1GH1=6 THEN H1GH1=.;      IF H1GH1=. THEN H1GH1=9;

/* Frequency for Explanatory Variables */     PROC FREQ; TABLES H1GH1 H1GH1;

* bar graph with 2 categories coded 0 and 1; PROC GCHART; vbar H1GH1A/discrete type=mean sumvar=H1TS4; run;

* centering quantitative number of gen.health; * print mean; PROC MEANS; var H1GH1; run;

* centering (subtract mean); data new2; set new; H1GH1_c = H1GH1 - 2.1097786; run;

* check coding; PROC MEANS; var H1GH1_c; run;

*  multiple regression model with centered General Health; PROC GLM; model H1TS4=H1GH1A H1GH1_c/solution; RUN;

* regression model with H1DA1: During the past week, how many times did you do work around the house,   such as cleaning, cooking, laundry, yardwork, or caring for a pet? ; PROC  GLM; model H1TS4=H1DA1 H1GH1_c/solution; run;

* adding ,Difficulty of limbs´; PROC GLM; model H1TS4=H1DA1 H1GH1A/solution; run;

* adding more explanatory variables; PROC  GLM; model H1TS4=H1DA1 H1GH1A H1GH1_c H1DA3 H1DA7/solution; run;

* confidence intervals for parameter estimates; PROC  GLM; model H1TS4=H1DA1 H1GH1A H1GH1_c H1DA3 H1DA7/solution clparm; run;

* POLYNOMIAL REGRESSION; * scatterplot with linear regression line Beingoverweight/ H1TS4 response variable; proc sgplot;  reg x=H1GH1 y=H1TS4 / lineattrs=(color=blue thickness=2) clm;  yaxis label="General Health";  xaxis label="Being overweight"; run; * scatterplot with linear and quadratic regression line; proc sgplot;  reg x=H1GH1 y=H1TS4 / lineattrs=(color=blue thickness=2) degree=1 clm;  reg x=H1GH1 y=H1TS4 / lineattrs=(color=green thickness=2) degree=2 clm;  yaxis label="General Health";  xaxis label="Being overweight"; run;

* request regression diagnostic plots; PROC glm PLOTS(unpack)=all; model H1TS4=H1DA1 H1GH1A H1GH1_c H1DA3 H1DA7/solution clparm; output residual=res student=stdres out=results; run;

* plot of standardized residuals for each observation; proc gplot; label stdres="Standardized Residual" aid="AID"; plot stdres*aid/vref=0; run;

* using proc reg to get a partial regression plot; * calculate quadratic terms; data partial; set new2; H1GH2=H1GH1_c*H1GH1_c; run;

*partial regression plot; PROC reg plots=partial; model H1TS4=H1GH1 H1GH2 H1DA7/partial; run;

1. Explanatory variables:

H1GH1A Do you have difficulty using your hands, arms, legs, or feet because of a physical condition that has lasted for the past twelve months or more?

H1GH1 In general, how is your health?  Would you say ...

Categorical response variable: H1TS4 The problems of being overweight

Other variables are added below.

One question is if a General Health is associated with a Being overweight and other quantitative variables.

First we must find the mean, which is 2.1097786 and center a quantitative variable ( H1GH1_c = H1GH1 – 2.1097786 ).

 A centered mean for var H1GH1_c is -2.214022E-9. R-square is 0.012839 which means that there is only 13% in a variability. F-value is 0.66 and p- value for H1GH1A/Difficulty of limbs is 0.4169 ( in the second case is still 0.1795) which is greater than .005. We know that we cannot reject the null hypothesis and we can say that The Difficulty of limbs is not associated with the H1TS4 /Being overweight.

A table/ H1GH1_c shows F-value 83.89 and p-value <.0001 so that we  we can reject the null hypothesis and say that the General Health/ H1GH1_c is significantly associated with the H1TS4 /Being overweight.

Intercept is 0.6215270004 which is only an estimate.

2. and 3. point: 

After we were adding H1DA1/ During the past week, how many times did you do work around the house, such as cleaning, cooking, laundry, yardwork, or caring for a pet?

The tables show  higher R-square 0.016626 – 17% in variability, F-value is 26.84. P- value is less than 0.0001 and we again reject the null hypothesis and say that the work around the house is significantly associated with Being overweight.

Then adding more explanatory variables- H1DA3 H1DA7/ During the past week, how many times did you watch television or videos, or play video games? and  During the past week, how many times did you just hang out with friends?

- I found that all the p-values are still less than  0.0001 and still is valid that these variables are significantly associated with Being overweight. Variability is higher- 23%  for R-square 0.022841.

- Estimates say that numbers are higher positive and p is less than .0001.

- Only p- value 0.1660 is greater than 0.05 for  H1GH1A/Difficulty of limbs: no association.

- One Estimate is negative for H1GH1A/Difficulty of limbs: no association with a value -.1087568616.

95% Confidence Limits are shown in the table. All are positive, only Difficulty of limbs has the range from -.2626560731 to 0.0451423499 for the confidence limit.

So only one variable-  Difficulty of limbs is not associated  with the Being overweight. The estimates were nearly the same as confidence limits and we know that this is true for all the population, not only for one or two samples.

4. If a model is correctly specified, then the residuals/ error terms are not correlated with the explanatory variables.

Intercept is the value of the response variable when all the explanatory variables are held constant at a value of zero.R-square is really small with 0.022841, only 2% variability.

( Intercept is 0.3904020402 and estimates are from 0.32 and confidence limit is higher to the 0.094 )

Residuals are the residual values with mean of 0 and stdev of 1.

In the first graphs we see that most of the residuals fall within one standard deviation of the mean. Only category 8 – Overweight and General health have some higher points as an outliers. In the graph- standardized residual are most of the residuals within -1.1, or positive 1. But there are several residuals that are above positive 11 and 13. There is may an evidence for unacceptable level error- poor model for observed data.

Outliers are observations with residuals greater than two or less than negative two. On the last plot there aren´t any outliers.

[briefexpertfun-blog]

Linear Regression Basics/ Repaired

I did some correction in Week 2  Linear Regression Basics. I didn´t know one basic thing- how to  collapse a categorical explanatory variable down to two categories, which is only a simple step in if else statement / * Datamanagement for Difficulty of limbs */:

/* H1GH1A Do you have difficulty using your hands, arms, legs, or feet because of a   physical condition that has lasted for the past twelve months or more?     H1GH1 In general, how is your health?  Would you say ...   H1PL8 Physic Activity */

LIBNAME mydata "/courses/d1406ae5ba27fe300 " access=readonly; DATA new; set mydata.addhealth_pds; LABEL H1GH1="General Health"      H1PL8="Physic Activity"      H1GH1A="Difficulty of limbs"

/* Datamanagement for General Health */          IF H1GH1=8 THEN H1GH1=.;      IF H1GH1=6 THEN H1GH1=.;      IF H1GH1=. THEN H1GH1=9;     /* Datamanagement for Physic Activity */      IF H1PL8=8 THEN H1PL8=.;      IF H1PL8=6 THEN H1PL8=.;      IF H1PL8=. THEN H1PL8=9; /* Datamanagement for Difficulty of limbs */      IF H1GH1A=8 THEN H1GH1A=.;      IF H1GH1A=7 THEN H1GH1A=.;      IF H1GH1A=6 THEN H1GH1A=.;      IF H1GH1A=. THEN H1GH1A=9;      if H1GH1A=1 then H1GH1A=1;      else H1GH1A=0;

* categorical explanatory variable; PROC GLM; model H1GH1=H1GH1A /solution; RUN;

* means for H1GH1A; PROC SORT; by H1GH1A; /* Frequency for Explanatory Variables */     PROC FREQ; TABLES H1PL8 H1GH1A; /* Pearson Correlation */     PROC CORR; VAR H1GH1 H1PL8 H1GH1A; PROC means; var H1GH1;by H1GH1A; run;

* bar graph; PROC GCHART; vbar H1GH1A/discrete type=mean sumvar=H1GH1; run;

[briefexpertfun-blog]

Week 2 Basic Linear Regression Model

/*  H1GH1- In general, how is your health?  Would you say ...

   H1TS2- The importance of exercise */

LIBNAME mydata "/courses/d1406ae5ba27fe300 " access=readonly;

DATA new; set mydata.addhealth_pds;

PROC SORT; by AID;

* categorical explanatory variable;

PROC GLM; model H1GH1=H1TS2 /solution;

RUN;

* means for H1TS2;

PROC SORT; by H1TS2;

PROC means; var H1GH1;by H1TS2;

run;

* bar graph;

PROC GCHART; vbar H1TS2/discrete type=mean sumvar=H1GH1;

run;

For this study I  used an explanatory categorical variable and a categorical response variable:

( H1TS2 )- The importance of exercise/  learned about this in a class at school.

Values- 0 no, 1 yes, 6 refused, 8 don´t know.  The most answers for yes are 5646 and number 844 is for no. Refused and don’t know are insignificant numbers with 5- 9 respondents.

( H1GH1 )- In general, how is your health?  Would you say ...  

with the values- 1 excellent, 2 very good, 3 good,  4 fair, 5 poor, 6 refused, 8 don´t know.  The  first sequence is for an answer very good with 2608 respondents, second place is category  excellent with the nb 1847.  And good is on the third place with number 1605. The number 408 is for response- fair. Other answers are not significant with only a few numerals.

GLM Procedure:

Number of Observations Read and Number of Observations Used are the same: 6504.

F- value is 39.45 and p-value is less than 0.0001, which means that we can reject the null hypothesis and conclude that the LEARNED-IMPORTANCE OF EXERCISE is significantly associated with a General Health.

Estimates/ beta waves are:

Data sub one bay is 0.18 and data sub zero value is  1.94.

Our equation for the best fit line of this graph is:

General health = 1.94 + 0.18 * Importance of exercise

Pr greater than the absolute value of T is p-value for our explanatory variables association with their response variable. Both values are the same-  less than 0.0001. If we remember it can be the same according to Pearson correlation. The  p-value is nearly 0.000 which is really small.

R- square is 0.006030 which we know as a proportion of  variance in the response variable that can be explained by the explanatory variable.

We know that this model accounts for about really small amount- 0.06% variability in the response variable- General health.

The explanatory categorical variable- The importance of exercise:

Only the coefficients for the mean are different in the category 6 and 8/ refused, don´t know ( mean values: 5 and 4.4 ).

The means are nearly the same- 2.17 is for the category 0 which is for the answer NO and 2.09 is for  the answer YES according to our bar graph.

One more, with the p-value less than 0.0001 we know that is less than 0.05 we can say that The learned importance of exercise is significantly associated with the General Health.

[briefexpertfun-blog]

Add Health/ wk1 Regression modeling

Sample:

This sample is from The National Longitudinal Study of Adolescent to Adult Health (Add Health).

This is a longitudinal study of a nationally representative sample of adolescents in grades 7-12 in the US during the 1994-95 school year. The group was further followed into young adulthood, the most recent into 2008 when the sample was aged 24-32.

Ethnic: The numbers of respondents in these samples:

1,038 blacks from well-educated families (with a parent with a college degree)

334 Chinese

450 Cuban

437 Puerto Rican

In addition, the main sample contains more than 1,500 Mexican-Americans and significant numbers of Nicaraguans, Japanese, South Koreans, Filipinos, and Vietnamese.

Add Health combines longitudinal survey data on respondents' social, economic, psychological, and physical well-being with contextual data on the family, neighborhood, community, school,  friendships, peer groups, and romantic relationships.

Add Health data include group of participants in the categories:  

- approximately 90,000 students in grades 7 through 12,  adolescents (18 to 26 years old),

and respondents 24 to 32 years old. There are variety of variables, categories and studies. I have chosen this one: Overweight.

Most of the respondents-  3381 are about the right weight and 1808 are slightly overweight.

238 number of people  are very overweight. Nearly quarter into half of them does not train( 1607 who not marked excersise vs 2851 respondents who do the exercise for losing weight).

Procedure:

Add Health Waves data collection took place in 1994/1995, and included both an in-school questionnaire and in-home interview. Based on self-reported data from the In-School Questionnaire, four supplementary ethnic-group samples were drawn.

The in-school questionnaire was administered to more than 90,000 students in grades 7 through 12, and gathered information on social and demographic characteristics of adolescent respondents, education and occupation of parents, household structure, expectations for the future, etc. 

Then Add Health data collection recommenced in1996, and included almost 15,000 follow-up in-home interviews with a group of adolescents.

In 2001/2002, data were collected through in-home interviews with 15,170 respondents (now 18 to 26 years old), as well as interviews with their partners.

Another in-home interviews were conducted in 2008 and 2009 when the original the respondents were 24 to 32 years old.

All necessary is in this link: http://www.icpsr.umich.edu 

Measures:

Exercise activities,  diet and overweight was assessed in the National Longitudinal Study of Adolescent to Adult Health (Add Health), 1994-2008 [Public Use] and gives us a variety of pattern s for daily activities, overivews for weight, etc. and additional stats.

That was conducted or funded by United States Department of Health and Human Services, National Institutes of Health, and many other great institutions.

Our focus of interest is a group of people  in terms of weight:

Current overview of weight is gained through: frequency of excercise or diet ( During the past seven days, which of the following things did you do in order to lose weight or to keep from gaining weight? ), or the exact question for losing weight ( Are you trying to lose weight, gain weight, or stay the same weight? ) and many more if we need.

Entire datasets and all the files with pdf-s are publically available here:

http://www.icpsr.umich.edu/icpsrweb/DSDR/studies/21600#bibcite  

[briefexpertfun-blog]

Moderator in the ANOVA

I tried to figured some relationship in ANOVA for:  Language spoken in your home does any impact and correlation on  the Birth date? And for this purpose it may be interesting:   In what country were you born?

/* Language spoken in your home- H1GI10, In what country were you born- H1GI12, Birth date- H1GI1Y */ You can see the code below:

There are a lot of observations for that many countries, something look like this:¨

This is only an abbreviation from these findings.

Probabilities are like these ones from p- values for some countries: 0., 0.633, 0,  0.4041,  0.2929,  0.5053,  0.6610, 0,  <.0001,  0.4904, 0, 0, 0,  0.9141,  0.3328, ..., 0. 

Above it is may the main focal point what I thought/ but I can´t prove it, when p- value is missing in a lot of cases/, but after aforementioned stuff which is incomplete, because that are many countries I can say that  that is not any significant finding for the country where you were born and date of born according to p-value, which is almost greater than .005.