First steps in DS Ocean

Hi there! The idea of this blog is to share, summarize and revise problems I am facing when learning statistics, machine learning algorithms and other mathematical and computer science interesting things. I would like to highlight that I am not an expert, so whenever you see something incorrect just let me know! Cheers, szarki9

[szarki9]

changing blog address

Hi guuuys!

Ive been working to move my super cool blog to new website address, and to new portal, as tumblr is awesome, but I need more expanded options haha.

So please be welcome to visit me on: https://szarki9.github.io/ !!!

Cheeers,

szarki9

[szarki9]

deep learning - what is dad

Hola chicos,

this weekend I will be a participant of a hackathon!!! Yey, right? Honestly, I am super excited but I feel a bit unprepared. Hackathon challenge is going to be about image processing so there is a lot to learn about it. I’m not sure whether the problem will be given as supervised or unsupervised but what I do know – from my brief research, that I need to be acquainted with deep learning so today I want to focus on that.

Question for today friends: what deep learning is about?

The concept of deep learning has over 50 years as for now, but until the second decade of the 21st century, there was not enough data to use it and not strong enough processors to perform it. The main idea behind it is to process or analyze the problem as the human brain does, which is process it simultaneously within numerous neurons connected with each other. And from that objective programming systems and structures (often machine learning processes) operating on that idea are called neural networks. So deep learning is a part of machine learning which is a part of artificial intelligence.

Following question arises: what is a neural network?

This is a enormous number of connected with each other and working at the same time processors. Each processor has it’s own memory and task to do, which often might be a machine learning process. The network learns something simple and basic at the beginning of the hierarchy and then passes information about it’s discovery to the next level. There, basic information is combined into something more complicated and so on, until information is processed through all hierarchy and through all processors.

Difference between Machine Learning and Deep Learning:

As previously stated, deep learning is a part of machine learning. But machine learning always needs to have a human operator for providing testing examples and manually amend mistakes. What is more machine learning is very time-consuming and it only depends on human creativity when figuring out how to make a model better.

On the contrary, deep learning is mostly perform as unsupervised learning. Raw data are provided for the model and no more human participation is needed, as computer thinks independently and learns. In deep learning, we are not using linear logic, as in most machine learning problems, but layers allow the model to learn and they reorganize after every new experience.

This is all for now, thanks!!

xoxo,

szarki9

[szarki9]

multiple regression - Q&A 3

Hello again chicos and chicas,

So we went through almost all the important stuff for multiple linear regression, but I have one more for you and then I think we might go further into developing beautiful word of analysis and statistics 😊. So last for now, but not least of course:

Model fit verification

So as in simple linear regression we compute RSE and R2 for our model – here link to my post explaining that.

R2 close to 1 will again indicate that the model explains a lot. But what we need to remember, that R­­2 will always increase when more variables will be added to the model, even if they are not so significant. In order to see that better, we shall consider examples, so we will, but in one of the following posts.

Okay, what about RSE? So formula for that is in general given by: (RSS/(n-p-1))1/2, which in simple linear regression case simplifies to the formula given previously. And again, deciding what are the outcomes of RSE requires knowledge about the model itself and its values.

That was quick. There are many things about the model that needs to be considered except that, but for the introduction theory, I would say that is it and further important details shall be discovered on ‘living’ examples.

Thank you and see you soon,

xoxo,

szarki9

[szarki9]

multiple regression - Q&A 2

Hi there,

it’s me again! Another question you might want to consider when performing multiple linear regression is…

How to decide which variables are important and which are not?

So after checking that there are some variables that are important and related to an outcome by using F-statistics we would like to determine which one of those might be. The concept of choosing what kind of variables is called variable selection. In general, we want to compare models that have a different set of variables and there are several criteria to conclude model fit, but I will come back to that someday. But what I would like to bring on today is selecting which models to verify, as all the possible models for p predictor is 2p in total. We have three approaches to that:

1. Forward selection – at the beginning we start with a null model, where are no predictors, just an intercept. Then we fit p simple linear regressions and pick that that results in the lowest RSS. Then we have our model with one selected variable and then we check p-1 two-variable models and again we select the one that has the lowest RSS. We might continue that until we will decide that the result is satisfactory.

2. Backward selection – here we start with all the variables in the model and we remove the variable with the largest p-value (as this variable is least statistically significant). Then we perform our model for p-1 variables and then again, we remove variable with the largest p-value. We continue until the results are satisfactory or e.g. all of the variables have p-value at some point.

3. Mixed selection – a combination of both listed before. We start with a model that has no variables and we add a variable that provides the best fit. We add variables one by one and simultaneously we check p-value for variables in our new models, and if p-value for the certain variable will rise above some fixed level we remove that variable for the model. We continue to execute these steps until all variables have p-value low enough and all variables that are not included in the model have a large p-value when they are in it.

Thank you for all the support!

xoxo,

szarki9

[szarki9]

multiple regression - Q&A 1

Hi again on these first days of December!

As promised last time, there are several questions needed to be answered regarding multiple linear regression described in my previous post. Let me start with:

How to determine whether there is a relationship between the response and the predictors?

In order to verify that, we will use F-statistic with the null hypothesis: H0: β1 = β2 = … = βn = 0 and the alternative hypothesis will be: at least one of coefficients is non-zero.

Hope you remember TSS used in R­­² statistics, so the formula for F is as follows: F=[(TSS-RSS)/p]/[RSS/(n-p-1)], where (!) p – number of predictors and n – number of observations in our sample. When to reject the null hypothesis and when not? When n is large, F-statistics that is just a little larger than 1 might still provide evidence to reject the null hypothesis. In contrast, a larger F-statistics is needed to reject H0 if n is small. As in the previously described statistic, we might also look into p-value for that one and then decide what to do with hypothesis.

In general, F-tests are used for comparing two samples and their variances, as in our case, for a sample of predicted values and sample of given data. When errors have normal distribution, the F-statistics follows F-distribution.

Okay, but we might want to check whether just set of coefficients equals zero and whether just set is important in our model. In that case we will consider null hypothesis for a subset of q coefficients, where variables chosen for exclusion are at the end: H0: βn-q+1 = βn-q+2 = … = βn = 0

Then we will be considering a model that uses all the variables except those last q. RRS for that model will be assigned to RRS0, and F-statistics will be given by: F=[(RSS0-RSS)/q]/[RSS/(n-p-1)].

Todo, wait for more!

hugs&kisses,

szarki9

[szarki9]

multiple linear regression ahoi!

Hola chicos,

Last time I was preparing myself for a job interview, so that's why I focused mostly on R, but I am back with statistical learning material!

So we have discussed linear regression models for simple regression, then t-statistics which serves to determine whether there is a relationship between variables and after that we went through the measurement of model quality. So all of that was designed for one variable regression and today I would like to discuss multiple linear regression .

First of all, what is the point of having multiple regression? Let’s consider an example, where we want to predict how variables such as lecture attendance, number of the book read, exercises done and IQ are predicting the result which is exam score at e.g. Statistics. Having described by me in previous post tools – simple linear regression, we can try to predict the impact of each of the variables for the score. But as all of us students know, success on the exam consists of diverse approaches and combination of mentioned above factors. So here comes multiple linear regression, which will be given by the equation:

Y = β0 + β1X1 + β2X2 + … + βnXn + ε, where Xp is a single predictor variable.

How to estimate coefficients?

As in linear regression, we will estimate parameters using the least-squares approach and we minimize the sum of squared residuals. RSS formula below:

Okay, todo for today. In the next post we will ponder about questions that are needed to be answered for multiple regression such as model fit and determination whether there is a relationship between response and predictors. So see you soon!

xoxo,

szarki9

[szarki9]

R intro part 3

Oh god hi,

that is going to be my last post about an intro to R. This one will be more theoretical and I will try to cover the data cleaning process, approach to variables and sampling when performing studies. So let me start with… :)

DATA CLEANING

1.Exploring raw data: class, dim, names, str, glimpse – dplyr library, summary, head, tail, hist, plot.

2.Tidy data rules: A dataset is a collection of values, each value belongs to a variable and an observation, a variable contains all values measured on the same unit across attributes. 

Symptoms of messy data: Multiple values stored in one column, a single observational unit is stored in multiple tables, column headers are values, not variable names.

Tidyr: gather when columns are not variables and we want to collapse them into key-value pairs, spread takes key-value pairs and spreads them across multiple columns, separate separate one columns into multiple columns, unite takes multiple columns and pastes them together.

3.Preparing data for analysis: type conversions using as.*(), using a combination of y,m,d,h,m,s to parse char to date, strings manipulation thanks to stringr: str_trim, str_pad, toupper, tolower, str_replace, str_detect.

Special values in R: NaN, Inf, #N/A.

is.na, any(is.na), complete.cases – see which rows have no missing cases, na.omit – remove all rows with any missing values, identifying outliers: summary, hist, or boxplot.

DATA in R

Types of variables:

numerical – numerical values, might be continuous (often measured, like weight) or discrete (often counted, as a number of cars).

categorical – limited numbers of distinct categories, might be ordinal (finite number of values within a given range, often measured, like socio economic status („low income”, „middle income”, „high income”)) or simply categorical (like whether you are a vegan or not).

table, droplevels drops empty levels from the data frame.

Discretize variable: converting a numerical variable to a categorical variable based on the searching criteria, e.g. adding a new variable to dataset where 1 corresponds to value bigger than average income and 0 corresponds to value lower than average income, we can do it using mutate from dplyr library.

Observational studies and experiments:

Observational study: collect data in a way that does not directly interfere with how the data arise, only correlation can be inferred.

Experiment study: randomly assign subjects to various treatments, causation can be inferred.

Example: drinking beetroot juice and physical capacity

So in the observational study, we will select two groups of people, the ones that drink beetroot juice and the ones that does not and then we will compare the physical capacity results for both groups.

In the experiment study, we will sample a group people of the population and then we will split them into a group that will drink beetroot juice and the ones that won't (decision of drinking is imposed by the researchers) and then after a while, we will compare the result of physical capacity for both of the groups.

When we will find a difference in the observational study, we cannot assign the result only to beetroot juice as they might be other variables responsible for the result and e.g. people drinking the juice might be also the ones that practice more or eat more healthy in general.

Also in the experimental study, when we select a random sample out of the population, these additional variables that we did not take under account in the observational study will be equally distributed with that random sample so when we will find a difference in the physical capacity we might do a causal statement that drinking beetroot roots affect physical capacity.

Random sampling and random assignments:

Random sampling – subjects are being selected for a study, and they are selected randomly, helps generalizability results.

Random assignments – only in experimental studies, where we assign subjects to various treatments, helps infer causation from results.

Simpson’s paradox

Explanatory variables – x and response variable – y, might be multivariate relationship.

Example: study results as a response and number of books read, exercises done, lecture attendance as a explanatory variables.

By Simpson’s paradox, we call the incident when the inclusion of the third variable in the analysis might change previously found the outcome and relationship between the explanatory variable and a response.

Example: The study shows that more men then women were accepted to universities, but when looking into each of the departments, women were usually accepted at a rate higher than men. Why is that? Because women tend to apply for the departments that have harder admission criteria and they admit fewer people.

SAMPLING STRATEGIES:

Simple random sampling: we randomly select a sample from the population so each case is likely to have similar attributes. Example: taking a random sample of dog owners.

Stratified sampling: we divide the population into a homogenous group called strata, and then we randomly sample from each of the strata. Example: if we want to have people from each of the education classes, we divide them into strata for each class and then randomly sample from each of the education classes.

Cluster sampling: we divide a population into clusters, then randomly select a few clusters and then sample all observation within these clusters, clusters are heterogeneous within themselves, and each cluster is similar to the other. Example: An NGO wants to create a sample of girls across 5 neighboring towns to provide education. Using single-stage cluster sampling, the NGO can randomly select towns (clusters) to form a sample and extend help to the girls deprived of education in those towns.

Multistage sampling: we divide a population to clusters, randomly sample a few clusters and then we randomly sample observations from those clusters.

Sampling in R: sample_n from dplyr.

PRINCIPLES OF EXPERIMENTAL DESIGN:

1.Control: compare the treatment of interest to a control group

2.Randomize: randomly assign subjects to treatments

3.Replicate: collect a sufficient large sample within a study, or replicate the entire study

4.Block: account for the potential effect of confunding variables (first we group subjects into blocks based on these variables and then we randomize within each block to treatment groups)

Okay, that one was long one :D. Hope it is clear, have a good night all!

xoxo,

szarki9

[szarki9]

R intro part 2

Hi there again,

part 2 of R intro summary :)

other handy functions:

lapply - returns a list of the same length as X, each element of which is the result of applying FUN to the corresponding element of X.

sapply – returns vector, matrix or array of output from applying FUN to elements of X, more user-friendly then lapply.

vapply – tries to generate named array, has a pre-specified type of return value, so it can be safer (and sometimes faster) to use

seq, rep, sort, rev – reverse the elements, append – merge vectors, is.* – check the class of R object, as.* – convert an R object from one class to another, unlist – flatten list to a vectors

REGULAR EXPRESSIONS! – is a sequence of characters that define a search pattern.

grepl - true if a pattern is found, grep - vector of indices of the character strings that contain the pattern, in sub and gsub you can specify a replacement argument, sub only first match, gsub all the matches.

Date and POSIXct objects –

dyplr: arrange – in descending or ascending order, filter , mutate – adding new column out of the rest

ggplot2: ggplot, facet_wrap – wraps a 1d sequence of panels into 2d, expand_limits

different plots with ggplot2: + geom_col -bar plot, geom_point - points, geom_line, geom_histogram, geom_boxplot

LOADING DATA INTO R

Utils package:

read.csv, read.delim – from txt files, read.table, which.min, which.max – returns an index of min or max value in the column, in read.delim for example you can add column names and column classes

readr library:

read_csv, read_tsv – tsv files, read_delim

collectors – are used to pass information about how to interpret values in a column; col_integer, col_factors

data.table library:

fread – same as read.table, but extremely fast and easy

readxl library:

excel_sheets – prints out names of the sheets in excel, read_excel – imports excels as tbl_df, tbl, data.frame,

gdata library:

read.xls – converting excel files to csv and then reading csv files using read.csv,

XLConnect library:

loadWorkbook – a bridge between Excel file and R session, getSheets – list of sheets, readWorksheet – importing the sheet into a data frame, createSheet – create a new sheet, writeWorksheet – adding data frames to a sheet, saveWorkbook – store adapter excel file, renameSheet, removeSheet

DBI library:

dbConnect – MySQLConnection, dbListTables – list of the tables in db, dbReadTable, dbGetQuery – using query to get data from a database table, dbDisconnect

dbSendQuery, then dbFetch – fetching results of executing a query, gives the ability to fetch the query’s result in chunks rather than all at once, dbClearResult – frees the memory

Importing flat files from the web: using readr library, read_csv or read_tsv with url address.

Downloading files from the web: readxl and gdata libraries, read.xls(gdata) and then download.file or read_excel (readr).

Downloading .RData files, download.file first and then load it into the workspace with load.

httr library:

GET – to get request from the web, in result we get the response object, that provides easy access to the status code, content-type, and actual content, using content we can extract the content, and we can define what object we want to retrieve: raw object, R object (list) or a character vector.

JSON files: first GET, then content as text but we can also use jsonlite library and use fromJSON to convert character data into a list, we can pass an object as an argument or URL, we can convert data to JSON file using toJSON, prettify makes JSON files pretty and minify makes in as concise as possible.

haven library:

read_sas for SAS

read_dta for STATA, columns are imported as a labeled vector and in order to change it for R format we need to use as_factor function and then we can convert it for the wanted data type.

read_sav or read_por for SPSS, also labeled class and we need to change it to other standard R class.

foreign library:

Simple functions to import STATA data and SPSS - read.dta for STATA, read.spss for SPSS.

thanks, wait for more!

szarki9

[szarki9]

R intro part 1

Hola, hola.

As another part of my studying experience I want to sum up my weekend entertainment, which is learning R. This sum up is going to be in the form of a note with links to documentation.

First of all, let me tell you what R is.

R is a programming language (interpreted language, which is a language that for most of the implementations execute instructions without previously compiling a program), created in the 90s for data analysis and as a teaching aid for learning statistics. As it was an open-source project, R easily became popular and even now many users are engaged into developing R and in working on new packages and libraries. Thanks to that, R has a lot of useful functions and is used for statistical calculations, data visualization, data analysis and data science.

Data types, objects:

Basic object type in R is a vector. Elements of the vector have the same data type, one of the following: numeric, integer, character or logical. Except vectors we have matrix, data frame, and list.

Matrix – all of the columns must have the same type and same length.

Data Frames – used for storing data tables, and it is a list of vectors with equal length, columns might have different data type.

List – object that contains element of different types.

Factors – used to represent categorical data, are stored as integers and have labels associated with these unique integers, R sorts levels in alphabetical order.

Arrays – object that can store data in more than two dimensions.

Tibble - is a data frame providing a nicer printing method, useful when working with large data sets.

Useful functions:

class() - will return in a class of the object, names() - will assign names to values of the vector/colnames and rownames accordingly for matrices, cbind() – adding new vector to matrix or merging matrices, by columns ,rbind() – combines vector, martix, data frame by row, levels(), summary(), str() - display the structure of an arbitrary R object, head().

LIBRARIES worth to know:

ggplot2 – data visualization package and „most elegant and aesthetically pleasing graphics framework available in R”.

dplyr - is a grammar of data manipulation, providing a consistent set of verbs that help you solve the most common data manipulation challenges (like select, mutate, filter, arrange, %>% !pipelines!).

readr – provide a fast and friendly way to read rectangular data (like csv, tsv, and fwf) (read_csv, read_tsv, read_delim etc.)

readxl – makes it easy to get data out of Excel and into R, has no external dependencies and is designed to work with tabular data (read_excel, excel_sheets etc.)

XLConnect – comprehensive and cross-platform R package for manipulating Microsoft Excel files from within R.

DBI – a database interface definition for communication between R and relational database management systems.

httr – useful tools for working with HTTP organized by HTTP verbs (GET(), POST(), etc). Configuration functions make it easy to control additional request components (authenticate(), add_headers() and so on).

jsonlite – a fast JSON parser and generator optimized for statistical data and the web.

haven – enables R to read and write various data formats used by other statistical packages (SAS, SPSS, STATA).

foreign – another package to read and write data formats such as SAS, SPSS, STATA.

tidyr - the goal of tidyr is to help you create tidy data, functions such as gather, spread, separate, unite, extract.

lubridate – provides easy and fast parsing of date-times, simple functions to get and set components of a date-time help to handle time zones (y, m, d, h, m, s)

stringr – provide a cohesive set of functions designed to make working with strings as easy as possible (str_detect, str_replace).

tidyverse - set of packages that work in harmony because they share common data representations and API design, such as ggplot2, dplyr, tidyr, readr, purrr, tibble, haven, stringr.

Todo for now, will get to you with more soon!

szarki9

[szarki9]

How to measure the quality of our model?

In the previous post, I have described how we can test the existence of the coefficient standing by the variable. So right now we will assume that relationship connecting these two – X and Y exists and we will think about how to measure the quality of our simple linear regression model.

Residual Standard Error

Remember RSS (residual sum of squares)? RSE is given by the formula: (square of  RSS divided by n-2) and it is an estimate of the standard deviation of ε (error term) and it is the average (and absolute) amount that the response will deviate from the true regression line. But the determination of whether the RSE is big or not depends on the complexity of the problem statement and the data that we have and what as a whole we examine and only after analysing that we can say whether RSE is large or not.

R² statistics

The first advantage of the R² statistics is that their values belong between 0 and 1, as R² is a proportion of explained variance. The formula for R² statistics is:(TSS-RSS)/TSS , where TSS in the total sum of squares (sum of squares of differences between each Y value and the mean of Y, RSS as above). TSS measures the total variance for Y, so R² measures the proportion of the variability of Y that can be explained using X. Closeness to 1 indicates that a large proportion of the variablity in the response has been explained, and on the contrary 0 means that the linear model might be wrong.

To sum up, the determination of whether the model is well suited for our data might be still a matter of its’ application and we need to look over each case individually as there is no clear and one way to decide it.

 szarki9

[szarki9]

t-statistics and how to find some relationship?

But how to check whether we can apply linear regression and whether there is a relationship between X and Y or not? This is the reason I did mention hypothesis testing in one of the posts before. So we will test the existence of the relationship using hypothesis testing.

Let’s take the null hypothesis H: There is no relationship between X and Y, and the alternative hypothesis K: There is some relationship between X and Y. This will lead to test H: β = 0 and K: β != 0. We need to compute t-statistics here, which measures the number of standard deviations that β is away from 0. In order to measure that we need to divide our estimated parameter β by its standard error*. Then we compare the results with t-student distribution for n-2 degrees of freedom and then we have basics to reject or not the null hypothesis. Rather an easy one right? 

*standard error (SE) - tells us the average amount that this estimated parameter differs from the actual value of the parameter.

Others interesting facts about t-student distribution:

So, t-student distribution, from what I have found out, is used for hypothesis testing or for estimation of uncertainty of measurement (<- that is our case guys).

Other crucial to know fact is that when we have more than 30 degrees of freedom, t-distribution is very close to normal distribution so we can use normal distribution in our work instead of t. Look at the picture below to see how t-student looks like for different degrees of freedom.

Photo borrowed from: link.

thanks again, see you soon!

xoxo,

szarki9

[szarki9]

Linear Regression and que te pasa

So one of the many algorithms used in statistical learning is linear regression. It is uncomplicated, the concept is rather understandable and linear regression might be useful in many prediction problems. We might either try to estimate some wanted value out of one variable or out of more variables. And here we have two types of linear regression – simple linear regression and multiple linear regression. Let me go for you with simple linear regression firstly.

As I have mentioned, in simple linear regression we are looking for a linear relationship (as the name suggested) between variable and measured output, and that relationship supposed to approximate the output. Like in the linear function, we assume that our output Y, might be represented using X variable in the following pattern: Y = βx + α (1), where α, β will be named coefficients or parameters of that function. The problem itself concludes to find those coefficients that will produce the line as close as it might be to the provided data. How to do it?

Minimizing least squares criterion is a popular way to do so. In general, it comes to find the minimum of residual sum of squares (RSS), which is like the name indicate the sum of squared residuals. Residual is the difference between the output from our linear function (the one that we assume is approximately correct for our data) and the actual data gathered. If you would like to look into actual formulas I highly recommend the book: „An Introduction to Statistical Learning”.

But in general, we assume that true relationship between variables and the output is given with the formula Y = f(X) + ε, and for a linear regression it will take the form of: Y = βx + α + ε (2), where ε is an error term with zero-mean. The model given by (2) characterizes population regression line, which is the best approximation for the relationship between Y and X, and model given by (1) characterizes least squares line. What is the difference between these two lines?

So the first one, least squares line depends on the data set that is given, which means that it might be slightly different for different random samples we are working with. The second one, the population regression line is the actual line that represents the relationship that we are looking for and it is rarely known. But like when estimating parameters in statistics, an average of least squares line should be the best approximation of population regression line.

Addition clarification question: what is the difference between residual and error term?

As above described, residual for a point is the difference between model value and observed data value, whereas error term of the observed value is a deviation of the observed value from (unobserved) true value.

It is long enough so far, so in the next post I will go with other interesting questions regarding this subject.

xoxo,

szarki9

[szarki9]

inferential statistics vol 2

Hello there,

Before we will go with linear regression I would like to bring in a couple of definitions. Remember the post about inferential statistics? I have written that using inferential statistics we can estimate values of particular parameters etc. And we can estimate either point values or interval ones. So let us start with the first question:

What confidence interval is and why it is important?

By confidence interval (CI) we mean an interval when the probability of achieving the value of the parameter will be greater than or equal to a given number (which we call the level of confidence).

The confidence level represents the proportion of possible confidence intervals that contain true value of the unknown parameter. In order to appoint CI for a population we use data from a specific sample so the level of confidence gives us a level of certainty that we apply to the population, the higher the confidence level the more certain you are that results are accurate.

For example (to point the general idea) after taking sample of kids in the age of 10, with 95% confidence level (means when we will repeat the survey or study the 95% of the results will be also in that interval) we might say they are 128-153 tall (and this interval is CI).

Let’s consider another example (imaginary data). We want to measure the average time spend on playing board games in Poland. We have a sample of 1000 random people and weekly time they spend on games. We count a mean for that sample and this is 1:56 hours. Then thanks to statistical calculations with a 95% level of confidence we might say that true time spend on board games in Poland belongs between 1:43 and 2:20 hrs per week. So here we need to use a level of confidence in order to emphasize that we cannot be sure for 100% with a given sample of 1000 out of 38 mln people in Poland that average time belongs between these two exact numbers. Statistics is knowledge about estimating values so this is why confidence interval and confidence level are so important.

Statistical tests, what, how and why?

Thanks to statistics we may test or verify the hypothesis and we might do that using statistical tests. We always consider two hypotheses, null hypothesis and alternate hypothesis. The null hypothesis is a commonly accepted fact and it proposes that no statistical significance exists in a set of given observations and the opposite to it is the alternate hypothesis.

By statistical test we define a function that has two return values, one is 1 when we reject the null hypothesis and 0 when we do not have the basis to reject it. When we decide whether the value of the function is 1 or 0? In general, we count particular statistic* for particular test and we check whether the value of that statistic belongs to critical region** or not. If it belongs then we take 1 as a value of a statistical test and 0 when not.

*By statistic (singular) we define a measurable function of a sample that does not depend on the measured parameter. I.e. mean or median is a statistic.

**By critical region we define the region of values that corresponds to the rejection of the null hypothesis at some chosen probability level.

But there is another way of testing a hypothesis. We can count the p-value. P-value is a probability of obtaining statistic at least as extreme as the one that we have just observed in a world when the null hypothesis is true. For me, that becomes quite understandable but here is a good video explaining it in terms of puppies: https://www.youtube.com/watch?time_continue=2&v=9jW9G8MO4PQ.

Let us consider a null hypothesis: people spend on average about 1:30 hrs per week on playing board games, an alternate hypothesis will be that this time is lower. P-value will answer a question: if an average time spend on board games in Poland is 1:30 per week or more, how surprising will it be in real life to spend it less? How ridiculous is our null hypothesis? If it is highly ridiculous we should reject it. 

I recommend reading the post I have drawn my inspiration from: post.

Btw, I would like to highlight it one more time that I am not an expert so with my posts I simply explain how I see these things and how I do interpret them :D. So if there is something that you do not agree with feel free to share it!

 Thanks,

xoxo,

szarki9

[szarki9]

supervised learning vs unsupervised learning

Good hmm night? guyz,

today's topic - statistical learning -  was developed out of inferential statistics and was the first concept out of what the term machine learning was created. Basically, we can say that this is the same when we consider ML as mathematic tools, algorithms and statistical models used to perform complicated tasks without previously specified formulas and without direct interference from a user. But as there are many ML definitions and from what I have learned - many people understand it in a bit different way, it may be safer to say, that statistical learning was something that has evolved into ML. 

But what statistical learning is exactly?

Statistical learning is a study about understanding data. Having given data set and using statistical learning tools we can either predict values of wanted variables or look for a connection between them or do both - that depends on what we want to know and what we can get out of the gathered data. As the title says - we recognize a distinction on supervised and unsupervised statistical learning. From what depends on that differentiation? Let us consider the following examples (first two are from the book ”An Introduction to Statistical Learning” - a recommendation from my side guyz):

1. Assume that we collect a set of data on the top 500 firms in Europe. For each firm, we record profit, number of employees, industry and the CEO salary. We are interested in understanding which factors affect CEO salary. 

In that case, we are looking for a connection/inference between variables and we want to establish which of them are relevant when we define salary. But what we also know is the CEO's salaries itself for each of 500 companies. This type of instance when we know the output of variables we will consider as supervised learning.

2. We are interested in predicting the % change in the USD/Euro exchange rate in relation to the weekly changes in the world stock markets. Hence we collect weekly data for all of 2018. For each week we record the % change in the USD/Euro, the % change in the US market, the % change in the British market and the % change in the German market.

This one, as stated in the problem statement is a prediction case. Having 4 variables and data from 52 weeks we would like to predict the % change in the USD/Euro exchange rate. Again, we have historical data from the previous year, so the output variable (which is % change) is known and hence again we will use supervised learning tools to solve that.

3. Let us have a bunch of photos of 6 people but without information about who is on which one and you want to divide this dataset into 6 piles, each with the photos of one individual.

Here we want our algorithm to classify by itself who is who and we do not provide at the beginning photos of everyone with a label and their name. When we do not know the output (here: who is who and what characteristics define each one of them) we apply unsupervised learning tools, as we want the algorithm to find characteristics that will distinct people (in that case).

As I have mentioned under the examples, supervised and unsupervised learning use different tools/models/techniques to solve the problems that are given. The main distinction between these two types is the knowledge about output variables and prior knowledge about labels (i.e. in photo cases). Supervised learning might be used to find inferences or to predict values. The tools that are most common here are linear regression, logistic regression, classification, etc. On the other hand, we also might use unsupervised learning to find inferences using techniques such as clustering or association. But before we decide what we are going to use, we must understand what are data all about and what is the use of them from a business perspective. But - another but hehe, I guess that understanding data is the first obvious step before all the data related work.

Hope you enjoyed, next time I will try briefly explain supervised learning techniques and que te pasa with a regression.

xoxo,

szarki9

[szarki9]

statistics vol 2 - inferential statistics intro

Hi there,

For the last two weeks, I have been researching what is the best and most achievable way (from my perspective ofc) to prepare yourself to work as a data specialist. I decided to start with the interview questions as I think they can give you quite a good overview of the skills that you need to have at the beginning and after them, you can start developing skills in more advanced way or make the DS portfolio.

There are a lot of question that you might be asked from fields such as statistics, machine learning, data quality or data cleansing (including use of most common python libraries), SQL query optimization and so on. I will start with statistics – that is the most basic knowledge that you should have and will be used everywhere further.

So let us start with the types of statistics and what is a statistic in general?

There are two of them, descriptive statistics and inferential statistics. I have described the first one widely a couple of posts before. But just to recall, the descriptive statistics is used to see tendencies (such as average, mean, std, etc.) in given data set and visualize them (using charts).

Inferential statistics, on the other hand, is much more complex, uses high mathematical solutions and works on random samples in order to make inferences about a population. For example, having a random sample of people and their salaries you can test whether a statement that men earn more than women can be rejected as a hypothesis or not. We can also estimate an interval for the value of specific parameters (mean or standard deviation). Given that, inferential statistics can be divided into two: estimation (point estimation or interval estimation) and hypothesis testing. I will go with specifics of each of the statistics in later posts, but in the end I would like to highlight the fundamental difference about them which is that you cannot make predictions out of info described within methods from descriptive statistics because they just characterize this specific data set that is given and these characterizations are always precise numbers. With inferential statistics output of the considered problems is rather a range of potential numbers with a degree of confidence.

Hope it clarifies a little, see you soon

xoxo

szarki9

[szarki9]

Lecture 1: Introduction to Statistics | Lecture Videos | Statistics for Applications | Mathematics | MIT OpenCourseWare

Wanna just share some videos from MIT, I did not have a chance to go through all of them, but from what I have watched I would totally recommend :)

[szarki9]

statistics vol 1 - descriptive intro

Hola hola mis amigos,

Hope you are having great Saturday. Mine is partly with statistics so I would like to share with you some things that I have learned so far and their application. I am rather pragmatic so this is why I am paying attention to the application every time :D. As much as I like theoretical approach and mind exercises, being able to use the things you know is a great experience and I am always very excited when it happens to me hah. Okay after this brief introduction let’s focus on today’s topic: statistics!

Statistics is a mathematics discipline that provides tools to interpret observations about some quality/qualities described within a random sample. We might consider the distribution of this sample and some characteristics that will highlight for us some interesting values. For example, we can consider a sample of n random people where all of them are football players and we can analyze their height or weight or BMI and their goal stats and see whether can we say something about the dependence between those qualities or not.

Most basic field of the statistic is descriptive statistics. In general, we have some data that we want to visualize or we want to extract from them some characteristics. Visualizing data can be done by bar charts, pie charts or any time of charts – but this is trivial, all of you know that already :D. What I want to mention more widely is characteristics for that data and random samples. So we have a couple of types of characteristics:

Measures of central tendency (measures of location)

They respond to the question what is the most typical value for that sample? And we consider as a central tendency measure:

Arithmetic Mean, pros: equal worth for each value, lack of promotion, cons: lack of resistance for outliers. Median, pros: resistance for outliers, cons: not taking under consideration values a bit further from the median. Mode – the most common appearing observation. Truncated Mean – arithmetic mean but without k-observations from each side from the ordered sample. Winsorized Mean – arithmetic mean but we replace k-observations from each side with k observation for the left side and n-k observation for the right side. Weighted Arithmetic Mean, Geometric Mean (the n-th root of the product of each of n values from the sample), Harmonic Mean (a multiplicative inverse of the arithmetic mean of the multiplicative inverse of the values).

And others such as quantiles (quartiles – division by 4, deciles – division by 10 and percentiles – division by 100)

Measures of dispersion

They respond to the question of how typical is a typical value for that sample?

Range – just distance between last and first value in the ordered sample. Interquartile Range – which is IQR=Q3-Q1, the difference between third and first quartile, pros: resistance for outliers,  Quarter Deviation is an IQR/2. Variance for random sample – tells how averagely data are inclined from the mean (expected value). Standard Deviation -  square root of the variance, Coefficient of Variation – the ratio of the standard deviation to the mean, shows the extent of the variability in relation to the mean of the population, MAD (Median Absolute Deviation) – this is median of the absolute deviations from the data’s median, pros: more resilient to outliers then standard deviation.

Measures of the shape

Skewness – there are different formulas, but in general it relates to the symmetry of the distribution and distribution of a random variable about its mean. Kurtosis -  similar to skewness, it is a descriptor of the shape of a probability distribution, it helps to see how far a distribution deviates from normal.

Btw I am still working on the approach to this super, knowledge-sharing posts so if you have any thoughts on that I will appreciate them! Thanks, guyz,

xoxo,

szarki9