AI and Data

[aianddata]

10 Smart R Programming Tips to become Better R Programmer

Coding is the process by which a programmer converts tasks from human-readable logic to machine-readable language. The reason behind coding being so popular is that there are so many ways to do the same thing that programmers don’t know the right choice anymore.

As a result, each programmer has his/her own style in writing implementations to the same part of an algorithm.

Writing code can sometimes be the most difficult and time-consuming part of any project. If the code is written in such a way that it is hard to change or requires a lot of work for every small update, then the investments will keep on piling up and more and more issues will crop up as the project progresses.

A good and well-written code is reusable, efficient and written cleverly by a smart programmer. This is what differentiates programmers from each other.

So, here are some tips to becoming a SMART coder:

Table of contents:

Writing codes for Programmer, Developer, and Even for A Layman

Knowing how to improve the code

Writing robust code

When to use shortcuts and when not to use

Reduce effort through code reuse

Write  planned out code

Active memory management

Remove redundant tasks

Learn to adapt

Peer review

1. Writing Codes for Programmer, Developer, and Even for A Layman

Though codes are primarily written for the machine to understand. They should be structured and well organized for other developers or for any layman to understand. In reality, codes should be written for all the three.

Those who keep this fact in mind are one step ahead of other coders while those who are able to make sure everyone can understand their code are miles ahead than their struggling friends.

Good programmers always document their codes and make use of IDE. I will use R language to explain the concept. Using IDE such as Rstudio makes it easier to write code quickly.

The main advantage available in almost all IDE is the auto-completion feature which suggests the function or command when part of it is written.

IDE is also known to suggest the syntax of the selected functions which saves time. Rstudio IDE environment also displays environment variables alongside with some basic details of each variable.

Documentation is another ability which differentiates good programmers from the rest.

Let’s look at this viewpoint using an example. Say you read the following code:

Code snippet 1

# Code snippet 1

a=16

b=a/2

c=(a+b)/2

Code snippet 2

# Code snippet 2

# store the max memory size

a=16

# taking half of the maximum memory as the minimum memory

b=a/2

# taking mean of maximum and minimum memory as the recommended memory

c=(a+b)/2

Code snippet 3

# Code snippet 3

# store the max memory size

max_mem=16

# taking half of the maximum memory as the minimum memory

min_mem=max_mem/2

# taking mean of maximum and minimum memory as the recommended memory

mean_mem=(max_mem+min_mem)/2

The difference in documentation is highlighted in these three code snippets and this is just a simple demonstration of code understandability.

The first code is difficult to understand. It just sets the values of three variables. There are no comments and the variable names do not explain anything.

The second code snippet explains that ‘a’ is the maximum memory, ‘b’ is the minimum memory and ‘c’ is the mean of the two.

Without the comments in code snippet 2, no one can understand whether the calculation for ‘c’ is correct or not.

The third code is a step further with the variables representing what is stored in them.

The third code is the easiest to understand even though all the three codes perform similar tasks. Moreover, when the variables are used elsewhere, the variables used in the third snippet are self-explanatory and will not require a programmer to search in the code for what they store until an error occurs in the code.

2. Knowing how to Improve

R has multiple ways to achieve a task. Each of the possibilities comes from using more memory, faster execution or different algorithm/logic.

Whenever possible, good programmers make this choice wisely.

R has the feature to execute code in parallel. Lengthy tasks such as fitting models can be executed in parallel, resulting in time-saving. Other tasks can also be executed faster based on the logic and packages used.

As an illustration, the following code snippets reflects the same task, one with sqldf package and another with dplyr package.

These practices are foundational not only for efficient programming, but also for building scalable AI and machine learning solutions.

Using sqldf version

# Using sqldf version

install.packages(“sqldf”)

library(sqldf)

Out_df=sqldf(“select * from table_a left outer join table_b on table_a.var_x=table_b.var_x”)

Using dplyr version

# Using dplyr version

install.packages(“dplyr”)

library(dplyr)

Out_df=left_join(table_a,table_b)

I personally prefer the dplyr version whenever possible. However, there are some differences between the outputs.

The dplyr version will look at all variables with the same name and join using them. If there is more than one such variable, I need to use them by field. Moreover, left join using dplyr will not keep both copies of the variable used to join tables whereas sqldf does.

One advantage of sqldf is that sqldf is not case sensitive and can easily join tables even if the variable names in the two tables are completely different. However, it is slower than dplyr.

3. Writing Robust Code

While writing code, you can make the code simple but situation specific or write a generic code. One such way in which programmers write simple but situation-specific code is by ‘Hard Coding’.

It is the term given to fixing values of variables and is never recommended.

For example, dividing the sum of all salaries in a 50,000-row salary data by 50,000 rather than dividing the sum of that sum with the number of rows may seem to make the same sense but have a different meaning in programming.

If the data changes with the change in the number of rows, the number 50,000 needs to be searched and updated. If the programmer misses making the small change, all the work goes down the drain. On the other hand, the latter approach automatically does the task and is a robust method.

Another popular programming issue quite specific to languages such as R is Code Portability. Codes running on one computer may not work on another because the other computer does not have some packages installed or has outdated packages.

Such cases can be handled by checking for installed packages first and then installing them. These tasks can be collectively called as robust programming and make the code error free.

Using an illustration for checking and installing/updating h2o package.

# If h2o package is already loaded, unload it and uninstall

if (“package:h2o” %in% search()) { detach(“package:h2o”, unload=TRUE) }

# Checking

if (“h2o” %in% rownames(installed.packages())) { remove.packages(“h2o”) }

# Next, we download packages that H2O depends on.

# methods

if (! (“methods” %in% rownames(installed.packages()))) { install.packages(“methods”) }

# statmod

if (! (“statmod” %in% rownames(installed.packages()))) { install.packages(“statmod”) }

# stats

if (! (“stats” %in% rownames(installed.packages()))) { install.packages(“stats”) }

# graphics

if (! (“graphics” %in% rownames(installed.packages()))) { install.packages(“graphics”) }

# Rcurl

if (! (“RCurl” %in% rownames(installed.packages()))) { install.packages(“RCurl”) }

# jsonlite

if (! (“jsonlite” %in% rownames(installed.packages()))) { install.packages(“jsonlite”) }

# tools

if (! (“tools” %in% rownames(installed.packages()))) { install.packages(“tools”) }

# utils

if (! (“utils” %in% rownames(installed.packages()))) { install.packages(“utils”) }

# Finally install and load h2o package

install.packages(“h20”)

library(h2o)

4. When to Use Shortcuts and When Not to

Using shortcuts may be tempting in the pursuit of writing code swiftly but the right practice is to know when to use them.

For instance, shortcut keys are something which is really helpful and can always be used. Using Ctrl+L in windows clears the console output screen, Using Ctrl+Shift+C in windows comments and un-comments all selected lines of code in one go are my favorite shortcuts in Rstudio.

Another shortcut is writing code for fixing code temporarily or writing faulty fixes which are not desired.

Here are some of the examples of faulty fixes.

This code changes a particular column name without checking its existing name

# This code changes a particular column name without checking its existing name

colnames(data_f)[5]=”new_name”

This removes certain columns using a number. This may remove important ones and code may give the error if the number of columns less than 10 in this case.

# This removes certain columns using a number. This may remove important ones and code may give error if the number of columns are less than 10 in this case

data_f=data_f[,1:4,6:10]

This converts a value to numeric without checking if it actually has all numbers. If the value does not contain numbers, it may produce NAs by coercion

# This converts a value to numeric without checking if it actually has all numbers. If the value does not contain numbers, it may produce NAs by coercion

Num_val=”123″

The following converts Num_val to 123 correctly

# The following converts Num_val to 123 correctly

Num_val=as.numeric(Num_val)

char_val=”A_Name”

The following issues a warning and converts Num_val to NA as it is not a number

# The following issues a warning and converts Num_val to NA as it is not a number

char_val=as.numeric(char_val)

5. Reduce Effort Through Code Reuse

When you start writing a code, you don’t need to waste time if a particular piece of logic has already been written for you. Better known as “Code Re-use”, you can always use your own code you previously wrote or even google to reach out the large R community.

Don’t be afraid to search. Looking up for already implemented solutions online is very helpful in learning the methods prevalent for similar situations and the pros and cons associated with them.

Even when it becomes necessary to reinvent the wheel, the existing solutions can serve as a benchmark to test your new solution. An equally important part of writing code is to make your own code reusable.

Here are two snippets which highlight reusability.

Code which needs to be edited before resuing it

# Code which needs to be edited before reusing it

for(i in 1:501)             {

df[,i]=as.numeric(df[,i])

}

Code which can be reused with lesser editing

# Code which can be reused with lesser editing

for(i in 1:ncol(df))      {

df[,i]=as.numeric(df[,i])

}

6. Write  Planned Out Code

Writing code on the fly may be a cool-to-have skill but not helpful for writing efficient codes. Coding is most efficient when you know what you are writing.

Always plan and write your logic on a piece of paper before implementing it. Inculcating the habit of adding tabs and spaces and basic formatting as you code is another time-saving skill for a good programmer.

For instance, every new ‘if’, ‘for’ or ‘while’ statement can be followed by tabs so that indentation is clearly visible. Although optional, such actions separate out blocks of code and helpful in identifying breakpoints as well as debugging.

A more rigorous but helpful approach is to write code using functions and modules and explaining every section with examples in comments or printing progress inside loops and conditions. Ultimately it all depends on the programmer how he/she chooses to document and log in the code.

7. Active Memory Management

Adding memory handling code is like handling a double-edged sword. It may not be useful for small-scale programs due to a slowdown in execution speed but nevertheless a great skill to have for writing scalable code.

In Rstudio, removing variables and frames when they are no longer required with the rm() function, garbage collection using gc() command and selecting the relevant features and data for proceeding are ways to manage memory.

Adjusting RAM usage with memory.limit() and setting parallel processing are also tasks for managing your memory usage. Remember! Memory management goes hand in hand with data backup.

It only takes a few seconds create and store copies of data. It should be done to ensure that data loss does not occur if backtracking is required.

Have a look at this example snippet which stores the master data and then frees up memory.

# dividing master dataset into train and test with ratio 7:3

library(dplyr)

train<-sample_frac(master_data, 0.7

train_ind<-as.numeric(rownames(train))

test<-master_data[-train_ind,]

# saving backup of master_data and removing unneeded data

write.csv(master_data,”master_data.csv”)

rm(master_data)

rm(train_ind)

gc()<span style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif; font-size: 16px; background-color: #ffffff;"> </span>

8. Remove Redundant Tasks

Sometimes programmers do some tasks repeatedly or forget to remove program code without knowing it.

Writing separate iterations for each data manipulation step, leaving libraries loaded even after they are no longer required, not removing features until the last moment, multiple joins and queries,etc. are some examples of redundancy lurking in your code.

While these happen somewhat naturally as more and more changes are made and new logic is added. It is a good practice to look at existing code and adjust your new lines to save runtime.

Redundancy can slow your code so much that removing it can do wonders in execution speed.

# Redundant code

# Takes about 0.5 seconds for iris data   

for(i in 1:ncol(df))      {

            df[,i]=as.numeric(df[,i])

}

for(i in 1:ncol(df))      {

            #storing missing values per column in mis vector

            mis[i]=length(which(is.na(df[,i])))

}

#Better implementation (implementations faster than the one below also exist)

#Gives a similar output but takes about 0.3 seconds for iris data - 35% improvement

for(i in 1:ncol(df))      {

            df[,i]=as.numeric(df[,i])

            #storing missing values per column in mis vector

          mis[i]=length(which(is.na(df[,i])))

}

9. Learn to Adapt

No matter how good a programmer you are, you can always be better! This tip is not related to typical coding practices but teamwork. Sharing and understanding codes from peers, Reading codes online (such as from repositories).

setting yourself up to date with books and blogs and learning about new technologies and packages which are released for R are some ways to learn.

Being flexible and adaptive to new methods and keeping yourself up to date with what’s happening in the analytics industry today can help you in avoiding becoming obsolete with old practices.

10. Peer Review

The code you write may be straightforward for you but very complex for everyone else. How will you know that? The only way is to know what others think about it.

Code review is thus the last but not the least in terms of importance for better coding. Ask people to go through your code and be open to suggested edits. You may come across situations when some code you thought is written beautifully can be replaced with more efficient code.

Code review is a process which helps both the coder and reviewer as it is a way of helping each other to improve and move forward.

The Path is Not So Difficult: Conclusion

Becoming a good programmer is no easy feat but becoming better at programming as you progress is possible. Though it will take time, persevering to add strong programming habits will make you a strong member in every team’s arsenal.

These tips are just the beginning and there may be more ways to improve. The knowledge to always keep improving will take you forward and let you taste the sweet results of being a hi-tech programmer.

In the rapidly changing analytics world, staying with the latest tools and techniques is a priority and being good at R programming can be a prime factor towards your progress in your analytics career.

So go out there and make yourself acquainted with the techniques of becoming better at R programming.

This article was originally published at Perceptive Analytics.

Perceptive Analytics partners with businesses to unlock value in data and drive innovation. With two decades of experience, we’ve delivered results for 100+ clients worldwide. Our expertise includes Tableau development services, Chatbot Consulting, and Power BI development services.

[aianddata]

Learn Generalized Linear Models (GLM) using R

In this article, we aim to discuss various GLMs that are widely used in the industry. We focus on: a) log-linear regression b) interpreting log-transformations and c) binary logistic regression.

Generalized Linear Model (GLM) helps represent the dependent variable as a linear combination of independent variables. Simple linear regression is the traditional form of GLM. Simple linear regression works well when the dependent variable is normally distributed. The assumption of normally distributed dependent variable is often violated in real situations. For example, consider a case where dependent variable can take only positive values and has fat tail. The dependent variable is number of coffee sold and the independent variable is the temperature.

Let's assume that we have modeled a linear relationship between the variables. The expected number of coffee sold decreases by 10 units as temperature increases by 1 degree. The problem with this kind of model is that it can give meaningless results. There will be situation when a increase of 1 degree in temperature would force the model to output negative number for number of coffee sold. GLM comes in handy in these types of situations. GLM is widely used to model situations where the independent variable has arbitrary distributions i.e. distributions other than normal distribution.  The basic intuition behind GLM is to not model dependent variable as a linear combination of independent variable but model a function of dependent variable as a linear combination of dependent variable. This function used to transform independent variable is known as link function. In the above example the distribution of number of coffee sold will not be normal but poisson and the log transformation (log will be the link function in this case) of the variable before regression would lead to a logical model. The ability of GLM to transform data with arbitrary distribution to fit a meaningful linear model makes it a powerful tool.

In this article, we aim to discuss various GLMs that are widely used in the industry. We focus on: a) log-linear regression b) interpreting log-transformations and c) binary logistic regression. We also review the underlying distributions and the applicable link functions. However, we start the article with a brief discussion on the traditional form of GLM, simple linear regression. Along with the detailed explanation of the above model, we provide the steps and the commented R script to implement the modeling technique on R statistical software.  For the purpose of illustration on R, we use sample datasets. We hope that you find the article useful.

Linear Regression

Linear regression is the most basic form of GLM. Linear regression models a linear relationship between the dependent variable, without any transformation, and the independent variable. The model assumes that the variables are normally distributed. It is represent in the form Yi= α+ βXi [Eq. 1]. The coefficients are computed using the Ordinary Least Square (OLS) method.  For a detailed explanation on linear regression and OLS, please refer to our earlier article at building regression models support vector regression. The article provides explanation and an example of linear regression. We hope that now you are comfortable with the idea of linear regression.

Now, with the objective of showing limitation of linear regression, we will implement the linear model on R using a not so perfect case. The data consists of two variables - Temperature and Coca-Cola sales in a university campus. Please click here to download. Let us visualize the data and fit a linear model to predict sales of coca cola based on given temperature.

The R code is below:

## Prepare scatter plot #Read data from .csv file data = read.csv("Cola.csv", header = T) head(data) #Scatter Plot plot(data, main = "Scatter Plot")

Figure 1 visualizes the data to provide us with a better understanding of the relationship between Temperature and Coca-Cola sales. We observe that sales increase exponentially with increase in temperature.## Add best-fit line to the scatter plot #Install Package install.packages("hydroGOF") library("hydroGOF") #Fit linear model using OLS model = lm(Cola ~ Temperature, data) #Overlay best-fit line on scatter plot abline(model) #Calculate RMSE PredCola = predict(model, data) RMSE = rmse(PredCola, data$Cola)

The relationship between Temperature and Cola Sales is represented in Equation [2]. The Root Mean Square Error for the model is pretty high at 241.49. The values of cola sales can be obtained by plugging the temperature in the equation.

[2]

Figure 2 shows the best-fit line as per the simple linear regression. The fit is bad and leads to absurd predictions. As per the model the Cola sales will be negative for temperature below 10 units. There are two ways to deal with the situation. First, fit a non-linear model. Second, transform the data to fit a linear model. In the next section we will discuss the second method.

Log-Linear Regression

Log-linear regression becomes a useful tool when the dependent and independent variable follow an exponential relationship. It means that Y does not change linearly with a unit change in X but Y changes by a constant percentage with unit change in X.  For example, the amount due, in case of compound interest, follows an exponential relationship with time, T. As T increases by one unit, the amount due increases by a certain percentage i.e. interest rate. Another example can be that of expected salary and education. Expected salary does not follow a linear relationship with level of education. It grows exponentially with level of education. Such growth models depict a variety of real life situations and can be modeled using log-linear regression.  Apart from exponential relationship, log transformation on dependent variable is also used when dependent variable follows: a) log-normal distribution - log-normal distribution is distribution of a random variable whose log follows normal distribution. Thus, taking log of a log-normal random variable makes the variable normally distributed and fit for linear regression.  b) Poisson distribution - Poisson distribution is the distribution of random variable that results from a Poisson experiment. For example, the number of successes or failures in a time period T follows Poisson distribution.

In this article we focus on the exponential relationship, which is expressed asY=a(b)X[Eq. 2].  In this case, a log transformation would make the relationship linear. We can representlog(Y)as a linear combination on Xs. Taking log on both sides in the equation, we get log(Y)=log(a)+log⁡(b)X. Now we can estimate the model using OLS. Please note that this equation is very similar to Eq. 1,log(a)andlog(b)are equivalent toαandβrespectively. Now we will look into interpretation of log linear models.*log(a)is the constant term andlog(b)is the growth rate with which Y grows for a unit change in X. A negative value oflog(b)*would indicate that Y decreases by a certain percentage for unit increase in X. Now we will implement the model on R using the Coca-Cola sales data.

The R code is below.

## Fitting Log-linear model # Transform the dependent variable data$LCola = log(data$Cola, base = exp(1)) #Scatter Plot plot(LCola ~ Temperature, data  = data , main = "Scatter Plot") #Fit the best line in log-linear model model1 = lm(LCola ~ Temperature, data) abline(model1) #Calculate RMSE PredCola1 = predict(model1, data) RMSE = rmse(PredCola1, data$LCola)

Figure 3 displays the best-fit line using log-linear regression. We can look at it as a two-step process i.e. data transformation by taking log on both sides and then using simple linear regression on the transformed data. The computed model is as follows:

[3]

The Cola sales can be predicted by plugging the values of temperature in Equation [3]. We observe that the fit has greatly improved over the simple linear regression. The RMSE for the transformed model is 0.24 only. Please note that log-linear regression has also solved the issue of absurd negative values for cola sales. For no value of temperature we get a negative value of cola sales. A simple log transformation helps us to deal with the absurdity. In the next section we will discuss other log transformations that come handy in various situations.

Interpreting Log Transformations

Log transformations of dependent and independent data is an easy way to handle non-linear relationships. The transformation helps to analyze non-linear relationships using linear models. We have discussed the log-linear regression. There are two more variants – a) linear–log regression –the independent variables are log transformed and b) log-log regression – both the dependent and independent variables are transformed. The table below displays the equations and interpretation for each of the models.

Binary Logistic Regression

Binary logistic regression is used when the dependent variable is categorical and takes values - 0 and 1. Unlike simple linear regression, where conditional distribution of dependent variable is normal, in logistic regression the conditional distribution of dependent variable is Bernoulli. In Bernoulli distribution the variable can only take two values – 0 and 1 with certain probabilities.

Lets understand with the help of an example. Let us assume that in football the ability to convert a penalty depends on number of hours of practice by the shooter.  We can represent a successful penalty by 1 and an unsuccessful penalty by 0. The data looks as follows:

The binary logistic regression model will output the probability of successful penalty shoot based on the hours of practice. The logistic regression uses logistic function to model the relationship. Logistic function allows to model the relationship in form of probabilities as it takes values between 0 and 1. It is represented as follows:

[4]

A positive value (negative value) of β1 would indicate that probability of Y=1 increases (decreases) as X increases. Logistic regression is one of the widely used model of class prediction. The multinomial logistic regression extends the binary model to deal with problems involving multiple classes. For example, whether a person will redeem coupon A, coupon B or coupon C. Now we will implement the logistic regression model in R. The sample data consists of two variables – success/ failure in penalty shoot out represent 1/0 and hours of practice.

The R code is follows:

## Prepare scatter plot #Read data from .csv file data1 = read.csv("Penalty.csv", header = T) head(data1) #Scatter Plot plot(data1, main = "Scatter Plot")

We can observe that the dependent variable can take only two values – 1 and 0. As the number of practice hours increases the efficiency of player increases. Now we will prepare a model using logistic regression to predict the probability of a success or failure based on the practice hours.

The R code is as follows:

## Fitting Logistic regression model fit = glm(Outcome ~ Practice, family = binomial(link = "logit"), data = data1) #Plot probabilities plot(data1, main ="Scatter Plot") curve(predict(fit,data.frame(Practice = x), type = "resp"), add = TRUE)  points(data1$Practice,fitted(fit),pch=20)

Figure 5 displays the probability values obtained from the logistic regression. We can see that the model does a good job. The probability of success increases with increase in practice hours. The model is represented in equation [5]. The probability values can be obtained by plugging in the number of practice hours.

[5]

Conclusion

In this article we learned about Generalized Linear Model (GLM). Simple linear regression is the most basic form of GLM. Advance form of GLM helps to deal with non-normal distributions and non-linear relationships in a simple manner. We focus on log-linear regression and binary logistic regression. Log-linear regression is useful when the relation between dependent and independent variable is non-linear. It also provides a quick fix when dependent variable follows log-normal or Poisson distribution.

Further, we discussed the basic concepts of binary logistic regression. Binary logistic regression is beneficial when the dependent variable follows Bernoulli distribution, i.e. can take only values of 0 and 1. We also provide equations and interpretation for various log transformations that are used with regression models.

Along with the theoretical explanation, we share the R codes, so that you can implement the model on R. For better understanding, we display the results along with the codes.

We hope you find the article is useful.

This article was originally published by Perceptive Analytics. At Perceptive Analytics, we specialize in helping businesses unlock the full potential of their data through Tableau. With over 20 years of experience, we've earned our place among top Tableau Consulting Companies, supporting clients from Fortune 500 firms to fast-growing startups. Whether you need expert Tableau consulting or a dedicated Power BI consultant to build impactful dashboards, we deliver solutions that turn data into strategic decisions.

[aianddata]

Performing Nonlinear Least Square and Nonlinear Regressions in R

Linear regression is a basic tool. It works on the assumption that there exists a linear relationship between the dependent and independent variable, also known as the explanatory variables and output. However, not all problems have such a linear relationship. In fact, many of the problems we see today are nonlinear in nature. A very basic example is our own decision making process which involves deciding an outcome based on various questions. For example, when we decide to have dinner, our thought process is not linear. It is based a combination of our tastes, our budget, our past experiences with a restaurant, alternatives available, weather conditions etc. There can be other simple nonlinear cases such as quadratic or exponential dependencies which are not too difficult to imagine. This is how non-linear regression came into practice — a powerful alternative to linear regression for nonlinear situations. Similar to linear regression, nonlinear regression draws a line through the set of available data points in such a way that the line fits to the data with the only difference that the line is not a straight line or in other words, not linear.

Non-linear Regression — An Illustration

In R, we have lm() function for linear regression while nonlinear regression is supported by nls() function which is an abbreviation for nonlinear least squares function. To apply nonlinear regression, it is very important to know the relationship between the variables. Looking at the data, one should be able to determine the generalized equation of the model which will fit the data. This model is then specified as the ‘formula’ parameter in nls() function. The function then determines the coefficients of the parameters in the model. Let’s try linear and nonlinear regression models on an exponential data. I will use the runif() function to generate an exponential set of values for y. Here I will use x as a sequence from 0 to 100.

I will also use a set.seed() function so that the values are regenerated for you.

#set a seed value

set.seed(23)

#Generate x as 100 integers using seq function

x<-seq(0,100,1)

#Generate y as a*e^(bx)+c

y<-runif(1,0,20)*exp(runif(1,0.005,0.075)*x)+runif(101,0,5)

#How does our data look like? Lets plot it

plot(x,y)

This seems a fairly smooth non-linear plot. To illustrate the difference between linear and nonlinear models, let’s fit them both:

#Linear model

lin_mod=lm(y~x)

#Plotting the model

plot(x,y)

abline(lin_mod)

There is little overlap between the actual values and the fitted plot. Now let’s try the nonlinear model and specify the formula

nonlin_mod=nls(y~a*exp(b*x),start=list(a=13,b=0.1)) #a is the starting value and b is the exponential start

#This new plot can be made by using the lines() function

plot(x,y)

lines(x,predict(nonlin_mod),col=”red”)

This is a much better fit and clearly passes through most of the data. For more clarity, we will now calculate the errors for both the models

#Error calculation

error <- lin_mod$residuals

lm_error <- sqrt(mean(error²)) #5.960544

error2=y-predict(nonlin_mod)

nlm_error <- sqrt(mean(error2²)) #1.527064

The linear model has more than twice the error than that of nonlinear one. This shows that the nonlinear model fits better for nonlinear data.

Understanding the nls() function

There are a few parameters that the nls() function requires. I used two parameters to define the model in the above illustration — the formula and the start parameters. Nonlinear function requires us to look at the data first and estimate the model to fit in. This estimated model is specified as the formula parameter. We can also specify the coefficients as variables to be estimated. The next step involves specifying the start parameter. This parameter specifies the starting values of the coefficients we used in the formula. Here we have ‘a’ and ‘b’ as the coefficients. I took ‘a’ as the nearest integer to minimum value of y (which is approximately 13.19) and ‘b’ as the increment for the exponent. Using these values, the nls() function determines the optimal values of ‘a’ and ‘b’. It is very important to set the right starting parameter values otherwise the model may give us absurd results or even fail. Let’s see what are the estimated values of ‘a’ and ‘b’ for this dataset:

nonlin_mod

Nonlinear regression model

model: y ~ a * exp(b * x)

data: parent.frame()

a b

13.60391 0.01911

residual sum-of-squares: 235.5

Number of iterations to convergence: 15

Achieved convergence tolerance: 4.975e-07

The values for ‘a’ and ‘b’ estimated for this model are 13.60391 and 0.01911 respectively which are very close to those we provided as starting values. This shows that that the model estimated by the nls() function is y=13.60391*e^(0.01911*x). Further, the estimated values of ‘a’ and ‘b’ are very close to the starting values we provided. These results will remain the same if we keep ‘b’ as 0.01 or even 0.001 or keep ‘a’ as 10 or 100 or 1000. As long as the model is able to converge at the optimal estimation, some approximation is admissible. However, if the values of ‘a’ and ‘b’ are completely out of range, say 1 and 1, we get an error as the model fails. The right set of starting values need to be estimated by looking at the data before implementing the model.

Self-Starting Functions

The problem arises when one is beginning with nonlinear functions and does not know what value should be estimated for the parameters. To illustrate this problem, I will now use a non-linear dataset available in R. The Puromycin data shows the concentration and reaction rate for enzymatic reaction of Puromycin antibiotic. I will plot the data to understand the data and estimate the formula equation

attach(Puromycin)

plot(Puromycin$conc,Puromycin$rate)

This data is specific to biological reactions and can be estimated using the famous enzyme kinetics equation known as the Michaelis-Menten equation. For this, we will separate the dataset based on whether the state is “treated” or “untreated” and define a function for the equation

#Define a function to apply Michaelis-Menten equation

mm=function(conc,vmax,k) vmax*conc/(k+conc)

#Use the nls data over the first subset of treated data. I will set the starting values as 50 and 0.05

mm1=nls(rate~mm(conc,vmax,k),data=Puromycin,start=c(vmax=50,k=0.05),subset=state==”treated”)

#Use a similar function for the second subset of untreated data

mm2=nls(rate~mm(conc,vmax,k),data=Puromycin,start=c(vmax=50,k=0.05),subset=state==”untreated”)

Both the models, mm1 and mm2 make good estimations of the data and fit the model. However, it is hard to estimate the starting values looking at the plot of Puromycin conc. vs rate. The Puromycin concentration vs rate plot suggested that the minimum conc. on the x- axis is around 0.01 and the maximum rate (vmax) on the y-axis is around 200 yet I purposely used values which are very different from these estimations so that the model will fit while converging slowly. In this case, I am taking a risk on the estimation ability of the model. This is where “self-starting” functions come into the picture. As the name suggests, a self-starting function does not need a starting value for the parameters and do estimate themselves. We can rewrite the above two functions using the SSmicmen function which is a self starting function for Michaelis-Menten equation. The new models are:

mm3=nls(rate~SSmicmen(conc,vmax,k),data=Puromycin,subset=state==”treated”)

mm4=nls(rate~SSmicmen(conc,vmax,k),data=Puromycin,subset=state==”untreated”)

Let us compare the corresponding models by calling the model variables in R. We will first look at the models with state=”treated” which are mm1 and mm3 and compare the vmax and k values. We will then compare the models with state=”untreated” which are mm2 and mm4.:

#Print the model summary and estimated parameters for mm1

mm1

Nonlinear regression model

model: rate ~ mm(conc, vmax, k)

data: Puromycin

vmax k

212.68369 0.06412

residual sum-of-squares: 1195

Number of iterations to convergence: 7

Achieved convergence tolerance: 2.703e-06

#Print the model summary and estimated parameters for mm3

mm3

Nonlinear regression model

model: rate ~ SSmicmen(conc, vmax, k)

data: Puromycin

vmax k

212.68371 0.06412

residual sum-of-squares: 1195

Number of iterations to convergence: 0

Achieved convergence tolerance: 1.937e-06

#Print the model summary and estimated parameters for mm2

mm2

Nonlinear regression model

model: rate ~ mm(conc, vmax, k)

data: Puromycin

vmax k

160.28001 0.04771

residual sum-of-squares: 859.6

Number of iterations to convergence: 7

Achieved convergence tolerance: 2.039e-06

#Print the model summary and estimated parameters for mm4

mm4

Nonlinear regression model

model: rate ~ SSmicmen(conc, vmax, k)

data: Puromycin

vmax k

160.28012 0.04771

residual sum-of-squares: 859.6

Number of iterations to convergence: 5

Achieved convergence tolerance: 3.942e-06

The corresponding models have estimated the same coefficients up to the third decimal. This shows that self-starting functions fairly well in place of functions where I need to define the start parameters. The big limitation of estimating the starting parameters can be avoided using the self-starting functions. R has many self-starting functions available. A list of the same can be obtained by using the apropos function:

apropos(“^SS”)

[1] “SSasymp” “SSasympOff” “SSasympOrig” “SSbiexp” “SSD”

[6] “SSfol” “SSfpl” “SSgompertz” “SSlogis” “SSmicmen”

[11] “SSweibull”

With the exception of the SSD function, there are 10 self-starting functions here in R. The final step in the model. I have given a brief description of what all these functions are defined for (in alphabetical order)

SSasymp asymptotic regression models

SSasympOff asymptotic regression models with an offset

SSasympOrig asymptotic regression models through the origin

SSbiexp biexponential models

SSfol first-order compartment models

SSfpl four-parameter logistic models

SSgompertz Gompertz growth models

SSlogis logistic models

SSmicmen Michaelis–Menten models

SSweibull Weibull growth curve models

Goodness of Fit

As an additional verification step, I will also check the goodness of fit of the model. This can be done by looking that the correlation between the values predicted by the model and the actual y values.

#Goodness of fit for first nonlinear function

cor(y,predict(nonlin_mod)) #0.9976462

#Goodness of fit for treated values of Puromycin function

cor(subset(Puromycin$rate,state==”treated”),predict(mm3)) #0.9817072

cor(subset(Puromycin$rate,state==”treated”),predict(mm1)) #0.9817072

#Goodness of fit for untreated values of Puromycin function

cor(subset(Puromycin$rate,state==”untreated”),predict(mm2)) #0.9699776

cor(subset(Puromycin$rate,state==”untreated”),predict(mm4)) #0.9699777

All our models have a high correlation value which indicates that the values are very close to each other and accurate. The corresponding model summary and estimation parameters also show the same observation.

Summary

Regression is a fundamental technique to estimate the relationships among variables and nonlinear regression is a handy technique if that relationship is nonlinear. It is similar to linear regression and provides a powerful method to fit a nonlinear curve based on the estimated formula while minimizing the error using nonlinear least squares method. There are a variety of other nonlinear models available such as SVM and Decision trees. Nonlinear regression is a robust technique over such models because it provides a parametric equation to explain the data. As the models becomes complex, nonlinear regression becomes less accurate over the data. This article gives an overview of the basics of nonlinear regression and understand the concepts by application of the concepts in R. Here is the complete R code used in the article.

#set a seed value

set.seed(23)

#Generate x as 100 integers using seq function

x<-seq(0,100,1)

#Generate y as a*e^(bx)+c

y<-runif(1,0,20)*exp(runif(1,0.005,0.075)*x)+runif(101,0,5)

#How does our data look like? Lets plot it

plot(x,y)

#Linear model

lin_mod=lm(y~x)

#Plotting the model

plot(x,y)

abline(lin_mod)

nonlin_mod=nls(y~a*exp(b*x),start=list(a=13,b=0.1)) #a is the starting value and b is the exponential start

#This new plot can be made by using the lines() function

plot(x,y)

lines(x,predict(nonlin_mod),col=”red”)

#Error calculation

error <- lin_mod$residuals

lm_error <- sqrt(mean(error²)) #5.960544

error2=y-predict(nonlin_mod)

nlm_error <- sqrt(mean(error2²)) #1.527064

nonlin_mod

attach(Puromycin)

plot(Puromycin$conc,Puromycin$rate)

#Define a function to apply Michaelis-Menten equation

mm=function(conc,vmax,k) vmax*conc/(k+conc)

#Use the nls data over the first subset of treated data. I will set the starting values as 50 and 0.05

mm1=nls(rate~mm(conc,vmax,k),data=Puromycin,start=c(vmax=50,k=0.05),subset=state==”treated”)

#Use a similar function for the second subset of untreated data

mm2=nls(rate~mm(conc,vmax,k),data=Puromycin,start=c(vmax=50,k=0.05),subset=state==”untreated”)

mm3=nls(rate~SSmicmen(conc,vmax,k),data=Puromycin,subset=state==”treated”)

mm4=nls(rate~SSmicmen(conc,vmax,k),data=Puromycin,subset=state==”untreated”)

#Print the model summary and estimated parameters for mm1

mm1

#Print the model summary and estimated parameters for mm3

mm3

#Print the model summary and estimated parameters for mm2

mm2

#Print the model summary and estimated parameters for mm4

mm4

#Print the names of all functions in R which start with SS

apropos(“^SS”)

#Goodness of fit for first nonlinear function

cor(y,predict(nonlin_mod)) #0.9976462

#Goodness of fit for treated values of Puromycin function

cor(subset(Puromycin$rate,state==”treated”),predict(mm3)) #0.9817072

cor(subset(Puromycin$rate,state==”treated”),predict(mm1)) #0.9817072

#Goodness of fit for untreated values of Puromycin function

cor(subset(Puromycin$rate,state==”untreated”),predict(mm2)) #0.9699776

cor(subset(Puromycin$rate,state==”untreated”),predict(mm4)) #0.9699777

This article was originally published at Perceptive Analytics. At Perceptive Analytics, our mission is simple: “to enable businesses unlock value in data.” With 20+ years of experience, we’ve served over 100 clients—including Fortune 500 firms—by delivering actionable insights and scalable analytics solutions. Recognized among top Tableau Consulting Companies, we offer expert Tableau consulting and Power BI consultant services to help businesses build powerful dashboards and make confident, data-driven decisions.

[aianddata]

Discriminant Analysis: Statistics All The Way

Discriminant analysis is used when the variable to be predicted is categorical in nature. This analysis requires that the way to define data points to the respective categories is known which makes it different from cluster analysis where the classification criteria is not know. It works by calculating a score based on all the predictor variables and based on the values of the score, a corresponding class is selected. Hence, the name discriminant analysis which, in simple terms, discriminates data points and classifies them into classes or categories based on analysis of the predictor variables. This article delves into the linear discriminant analysis function in R and delivers in-depth explanation of the process and concepts. Before we move further, let us look at the assumptions of discriminant analysis which are quite similar to MANOVA.

Since we are dealing with multiple features, one of the first assumptions that the technique makes is the assumption of multivariate normality that means the features are normally distributed when separated for each class. This also implies that the technique is susceptible to possible outliers and is also sensitive to the group sizes. If there is an imbalance between the group sizes and one of the groups is too small or too large, the technique suffers when classifying data points into that ‘outlier’ class

The second assumption is about homoscedasticity. This states that the variance of the features is same across all the classes of the predictor feature

We also assume that the features are sampled randomly

The final assumption is about the absence of multicollinearity. If the variables are correlated with each other, the predictive ability will decrease.

Though the discriminant analysis can discriminate features non-linearly as well, linear discriminant analysis is a simpler and more popular methodology. We have normally distributed conditional probability functions for each class. If y is the class to be predicted with two values, 1 and 2 and x is the combined set of all the predictor features, we can assume a threshold value T such that the value which comes as a result of linear combination of features of x belongs to class 1 if it is less than T and belongs to class 2 otherwise. Mathematically,

(x−μ1)TΣ1−1(x−μ1)+ln|Σ1|−(x−μ2)TΣ2−1(x−μ2)−ln|Σ2|T

Where (μ1,Σ1) and(μ2, Σ2) are the respective means and variances of x for class 1 and class 2. We sometimes simplify our calculations by assuming equal variances of the two classes to get a simplified version    w.x>c where c is the threshold and w is the weight combined with x.

Let’s understand Fisher’s LDA which is one of the most popular variants of LDA    Fisher’s Linear Discriminant analysis – How and when to use it?

Fisher’s linear discriminant finds out a linear combination of features that can be used to discriminate between the target variable classes. In Fisher’s LDA, we take the separation by the ratio of the variance between the classes to the variance within the classes. To understand it in a different way, it is the interclass variance to intraclass variance ratio

S= ?2between/?2within = (w⋅(μ2−μ1))2/ wT(Σ1+Σ2)w

Fisher’s LDA maximizes this ratio and has a lot of applications. One of the recent applications involve classification of speech and audio. Other past usages include face recognition where Fisher’s LDA is used to create Fisher’s Faces and combined with PCA technique to get eigenfaces. Fisher’s LDA also finds usages in earth science, biomedical science, bankruptcy problems and finance along with in marketing. That’s all on the theoretical aspect of LDA. Let’s understand using an example in R.

LDA Classification example in R

R has a MASS package which has the lda() function. For dataset, we will use the iris dataset and try to classify the classes.

#Load the library containing lda() function

library(MASS)

#Store the dataset

dataset=iris

Before running the lda() function, let’s start with the help documentation of lda()

#Help Documentation

?lda

The description for lda() is minimalistic and simple. We are interested in the details section of the documentation which describes the process which the function uses. As the documentation mentions — the lda() function also tries to detect if the within-class covariance matrix is singular. We can also define a tolerance such that if any variable has within-group variance less than tol² it will stop and report the variable as a constant.  Another possible adjustment is the prior probabilities. The prior parameter in lda() function is used to specify the prior probabilities. If not specified, the function calculates the prior probabilities to be the same as the distribution of classes in the data. These prior probabilities also affect the rotation of the linear discriminants.    Let us proceed with performing linear discriminant analysis over the iris dataset.

#Perform LDA over the data

lda_iris=lda(Species~.,data=dataset)

#Prior Probabilities and coefficients of Linear discriminants

lda_iris

Call:

lda(Species ~ ., data = dataset)

Prior probabilities of groups:

setosa versicolor virginica

0.3333333 0.3333333 0.3333333

Group means:

Sepal.Length Sepal.Width Petal.Length Petal.Width

setosa 5.006 3.428 1.462 0.246

versicolor 5.936 2.770 4.260 1.326

virginica 6.588 2.974 5.552 2.026

Coefficients of linear discriminants:

LD1 LD2

Sepal.Length 0.8293776 0.02410215

Sepal.Width 1.5344731 2.16452123

Petal.Length -2.2012117 -0.93192121

Petal.Width -2.8104603 2.83918785

Proportion of trace:

LD1 LD2

0.9912 0.0088

#Check the accuracy of our analysis

Predictions=predict(lda_iris,dataset)

table(Predictions$class, dataset$Species)

setosa versicolor virginica

setosa 50 0 0

versicolor 0 48 1

virginica 0 2 49

With LDA, we are able to classify all but 3 data points correctly in iris dataset. This is probably because the iris data is linearly separable. How do we know whether a data is linearly separable or not? We use the pairs function to see the scatter plots of data and see if they are separable

#Check how easily we can linearly separate the iris dataset

As we can see, one of the classes is completely separate while the other two are somewhat overlapping. However, LDA is still able to distinguish between the two. A better version of using lda is lda() with CV. This can be done by passing the CV=TRUE in the lda function.

#LDA with CV

lda_cv_iris=lda(Species~.,data=dataset,CV=TRUE)

#The predictions are already generated in lda_cv_iris

table(lda_cv_iris$class, dataset$Species)

I didn’t generate the summary for the model as it will also produce all the predictions. As we already know from the summary of lda_iris, the function first calculates the prior probabilities of the classes in the dataset unless provided specifically. The iris dataset had 50 data points for each class hence the prior probabilities are calculated to be 0.33 each. It then makes the necessary calculations which involves means of each class and overall variance and gets the linear discriminant. The function also scales the value of the linear discriminants so that the mean is zero and variance is one. The final value, proportion of trace that we get is the percentage separation that each of the discriminant achieves. Thus, the first linear discriminant is enough and achieves about 99% of the separation.    As a final step, we will plot the linear discriminants and visually see the difference in distinguishing ability. The ldahist() function helps make the separator plot. For the data into the ldahist() function, we can use the x[,1] for the first linear discriminant and x[,2] for the second linear discriminant and so on

#Plot the predictions - first linear discriminant

The data points are almost completely separated by the first linear discriminant and that is why we see the three classes in different ranges of values. To further our understanding, we also see the second linear discriminant.

#Plot the predictions - second linear discriminant

From the plot of the second linear discriminant, we see that we can hardly differentiate between the three groups hence the proportion of trace values.

Everything is not linear — quadratic discriminant analysis

MASS package also contains the qda() function which stands for quadratic discriminant analysis. The idea is simple — if the data can be discriminated using a quadratic function, we can use qda() instead of lda(). The rest of the nuances are the same for qda() as were in lda()

#QDA

qda_iris=qda(Species~.,data=dataset)

qda_iris

Call:

qda(Species ~ ., data = dataset)

Prior probabilities of groups:

setosa versicolor virginica

0.3333333 0.3333333 0.3333333

Group means:

Sepal.Length Sepal.Width Petal.Length Petal.Width

setosa 5.006 3.428 1.462 0.246

versicolor 5.936 2.770 4.260 1.326

virginica 6.588 2.974 5.552 2.026

#Check the accuracy of our analysis of qda

Predictions_qda=predict(qda_iris,dataset)

table(Predictions_qda$class, dataset$Species)

setosa versicolor virginica

setosa 50 0 0

versicolor 0 48 1

virginica 0 2 49

Since the data has a linear relation, the qda function also applies the same statistics and returns similar results.

Conclusion: Evaluating LDA and QDA

Even though LDA is a tough problem to understand, its implementation in R is simple. As a final step, we will look into another package- the klaR package which helps to create an exploratory graph for LDA or QDA.  The package contains the partimat() function which takes a similar input as the lda() function but returns a plot instead of the model. The function stands for partition matrix and plots the ability of the features to partition the target class taking combinations of two at a time.

#Using the klarR package

# install.packages("klaR")

library(klaR)

Our data has four features so we have 4C2 =6 combinations to classify our data. The plot show how different classes are defined based on the two features on x-axis and y-axis. As a summary, it is important to know that one should look at the data first to know whether the data seems to be linearly separable (or quadratically separable in case of qda) before selecting the technique. Since LDA makes some assumptions about the data, we also need to preprocess the data and perform univariate analysis to see if the normality assumption holds for each class of the data. In the absence of normality, that is, in case there is a violation of the normality condition, one can still proceed with LDA or QDA but the results will not be appropriate and will lack in accuracy. We also need to analyze whether the features are related to each other and some of them need to be omitted from our analysis. The rest is up to the lda() function to calculate and make predictions on. Here is the entire code used in this article:

#Load the library containing lda() function

library(MASS)

#Store the dataset

dataset=iris

#Help Documentation

?lda

#Perform LDA over the data

lda_iris=lda(Species~.,data=dataset)

#Prior Probabilities and coefficients of Linear discriminants

lda_iris

#Check the accuracy of our analysis

Predictions=predict(lda_iris,dataset)

table(Predictions$class, dataset$Species)

#Check how easily we can linearly separate the iris dataset

pairs(dataset)

#LDA with CV

lda_cv_iris=lda(Species~.,data=dataset,CV=TRUE)

#The predictions are already generated in lda_cv_iris

table(lda_cv_iris$class, dataset$Species)

#Plot the predictions - first linear discriminant

ldahist(data = Predictions$x[,1], g=Species)

#Plot the predictions - second linear discriminant

ldahist(data = Predictions$x[,2], g=Species)

#QDA

qda_iris=qda(Species~.,data=dataset)

qda_iris

#Check the accuracy of our analysis of qda

Predictions_qda=predict(qda_iris,dataset)

table(Predictions_qda$class, dataset$Species)

#Using the klarR packagew

# install.packages("klaR")

library(klaR)

partimat(Species~.,data=dataset,method="lda")

Ensuring model assumptions hold and results are valid can be complex. That’s where AI consulting services help you solve this issue — from data preparation to model tuning.

Published by Perceptive Analytics. We help businesses get clarity from chaos. Over the past 20 years, we’ve delivered solutions to more than 100 clients. Our key services include Snowflake Consulting, Power BI Consulting, and Tableau Consulting Services, all built to empower better business decisions.

[aianddata]

How Crucial is R for Mastering Data Science?

Why Data Science?

The Digital age is here. The world’s most valuable resource is no longer oil, but data (the Economist). Smartphones and the internet have made data abundant, ubiquitous and far more valuable.

One of the most important aspects of data science is that its results are applicable to almost any sector like healthcare, e-commerce, travel and education among others. A sound understanding of the implications of data science can help sectors identify and quantify their challenges and address them in an effective manner. For instance, retail giant Target identifying pregnant women based on products they purchased, or the recommendation engines being used by Amazon, Netflix and the likes. Data Science enables all that and more.

A lot of people are baffled by the number of tools available for data science and what tools to start with. The one I recommend is R. Let’s discuss why.

Why R?

R is a modern implementation of S Language which started as a research project at Bell Labs in 1975. R was developed for data analysis, statistical modeling, simulation and graphics. However, with the powerful features it offers, its uses go beyond the generic data analysis and statistics. It is therefore more of general purpose language than a domain specific language (DSL).

R is Not Just a Statistics Package, It’s a Language

R is not just a statistics package which allows you to use certain predefined functionalities, it is rather a language through which you can develop and execute functionalities according to your own need. Take a look at this interview of Joe Cheng — “…people say that one of the differences between say Python or Julia and R is that R is a DSL for stats, whereas these other things are general purpose languages. R is not a DSL. It’s a language for writing DSLs, which is something that’s altogether more powerful.”

R is Designed to Operate the Way that Problems are Thought About

One of the goals of R is that the language should imitate the way that people think about real-world problems. For instance, take the concept of vectorization in R. Suppose we want to change a field denoting time in minutes to one in seconds. The command to do this in R would be -time.sec <- time.min*60

The vector time.min may contain hundreds and thousands of numbers. Humans tend to think of the situation — rather than as a collection of individual numbers. R makes the statement more human by hiding the series of multiplications as in any other logical programming language like C/C++, Java.

R is Both Flexible and Powerful

R is also known for its flexibility. Being open source, one can easily obtain the source code of a piece of functionality and tweak it according to one’s own needs in no time. Vectorization, as discussed above, gives R an edge over other languages too. One would have to write loops to execute the same thing in a low level language that R can do in a single line of code. Functions in R are treated as first class objects, which is another example of R’s flexibility. With R you can do an awful lot of things. From making a web application to generating mesmerizing harmonographs, you can implement a wide variety of concepts with R.R also allows incorporating C and C++ functionalities from within R to powerfully complement your R code. So you can mix tools — choosing the best one for a specific job.

R also allows incorporating C and C++ functionalities from within R to powerfully complement your R code. So you can mix tools — choosing the best one for a specific job. Rcpp package helps connect C++ to R. Several other packages also use low level compiled languages like C++ in their implementation. For instance ReadXL package surpasses it’s counterparts for importing Excel files in R, by leveraging capabilities from C++.

Let’s add a few more reasons to choose R over other similar languages.

R Package Ecosystem

A large part of R’s success is due to the ecosystem of open-sourced packages that add functionality other than what the core installation of R (base R) offers. There are about 10,000 packages on CRAN (as of 2017) and more on gitHub. Users can make their own packages and publish it on CRAN or more easily on gitHub.Amazing community

Amazing Community

R has an amazing community online, committed to improving data analytics. Various platforms offer online community support and discussion portals. Others sources include exploration and starter scripts on competition platforms where one can observe great deal of knowledge dissipation.

Functions as First-Class Objects

R treats functions as first class objects. Therefore, users can create functions, pass them as arguments, and have them returned as the result of other computations. For instance, functions like mean, sd or a user defined function can be treated as data itself and be used in other function arguments.

Data Structures in R

When programming in C or C++ and the likes, the data type of every object must be specified by the user. This allows the compiler to perform type-specific optimization. But, it is inefficient to type and produces fragile code. Essentially, there is a trade-off between processor run time and developer’s thinking time. R data types allow programmers to use the data in a natural form without having to put it into a particular predefined structure.

Graphics

Graphics can undoubtedly said to be central to data science. Humans perceive visual information far better than numbers. It is easy to produce publication-quality graphs in R. A very famous package for developing elegant graphics is ggplot by Hadley Wikham.

End Notes

Hope this article helps you in your journey with Data Science. Once you’re comfortable with R, you would be competent enough to proceed further confidently, adding more tools to your skillset — Python, D3, Tableau, and others. Choosing which tool to take up is dependent on a variety of factors, such as

Choosing which tool to take up is dependent on a variety of factors, such as background in programming or statistics, or what you want as your output, or simply availability. There cannot be one universal answer to this. However, with R, you can execute a wide variety of tasks and this set only keeps increasing as CRAN and other repositories keep adding packages. R is also the first choice of academicians to publish implemented techniques as R-packages. The flexibility and ease of doing things in R appeal to the mass, which reflects clearly in results of KDnuggets’ 18th annual poll of data science software usage and several other similar surveys. For organizations seeking guidance in selecting the right tools and applying them effectively, partnering with AI Consulting Services can accelerate their data science journey.

Of course, mastering R or any other tool should not be the absolute goal. Keep in mind that languages and tools are just a means to execute. One should be thorough with techniques and methodologies and at the same time up to speed with the latest tools at disposal.

Published by Perceptive Analytics. We help businesses get clarity from chaos. Over the past 20 years, we’ve delivered solutions to more than 100 clients. Our key services include Snowflake Consulting, Power BI Consultants, and Tableau Consulting Services, all built to empower better business decisions.

[aianddata]

Comprehensive Tutorial on Data Importing in R

Data importing into R is a task in itself. There seems to be a different importing function for almost every type of file format. One can agree it becomes quite confusing and frustrating to mix up between those functions and yet again their arguments. But once you get to know what packages and commands to use for which file types, it is fairly simple. To help you with this, here is a comprehensive tutorial for your reference, covering these commands to import files into your R session.

So, next time you find yourself searching ‘how to load xyz type of file into R’, you’d know where to look.

We will discuss importing the most common file types such as .TXT, .CSV, .JSON, Excel worksheets, SAS/SPSS/STATA datasets, Matlab files, etc. with examples and code. We will also see how to use ODBC connections to import tables from relational databases directly into R. You are also in for an amazing importing hack which I find very useful for some quick and dirty adhoc analysis.

Let’s dive in!!

How to prepare workspace environment before data importing

Generally, files to be loaded for a particular task are stored in a single directory. Further, this directory can be assigned as the “working directory” inside your R environment for ease of importing. To know the location of your current working directory use -getwd()

You could bring your files to the location returned by getwd. It could also be that you want to change the working directory to a location of your choice. Then you must use -setwd(“<physical path to directory>”)setwd(“<physical path to directory>”)

R would now know which folder you’re working in. This makes it easier to import and export files using relative path rather than the long absolute path to file location.

Your workspace environment would often be filled with data and values from the last R session. It is often required to clean the environment before proceeding afresh. You can do it usingrm(list=ls())

It’s better to start with a clean environment. An alternative can be — not saving the workspace on ending the R session. Objects from the previous R sessions can lead to errors which are often very hard to debug. The above command gets rid of all the variables, functions and objects in your current environment. My advice — use it wisely.

Yayy, you’re finally done with the setup phase!

However, do keep in mind that the set up phase is important for eliminating majority of kinks you would come across in subsequent phases of your project. Now, it’s time to get to business.

Loading TXT/CSV/JSON/other separator files into R

Reading text files

Data in text files is of a form where values are separated by a delimiter such as tab, or comma or semicolon. Your file will look similar to the below format –

Category V1 V2 A 3 2 B 5 6 B 2 3 A 4 8 A 7 3

Above is an example of a tab delimited file. Similarly, a comma or semicolon could replace the tab separator as shown in the next section. Flat files like these can be read using the function read.table() –df <- read.table(“<path/file.txt>”, header = FALSE) (Note that you could skip writing the path in the above example if your file is in the working directory)

Read.table is the most common way of bringing simple files into your workspace. It is also very flexible. For instance, to import files delimited by characters other than tab, you can pass the sep argument indicating which character is the separator.

Reading CSV files

A CSV file is where entries are separated by a comma (,) or a semi-colon (;). Your file will look like this –

Category V1 V2 A 3 2 B 5 6 B 2 3 A 4 8 A 7 3

To load files like this, you can use read.csv() or read.csv2() function. For (,) separated files use read.csv, and for (;) separated files use read.csv2. Remember, files of both kind can also be read using read.table() function. read.csv() and read.csv2() functions are essentially just wrapper functions around read.table(). Look at their source codes:> read.csv function (file, header = TRUE, sep = ",", quote = "\"", dec = ".", fill = TRUE, comment.char = "", ...) read.table(file = file, header = header, sep = sep, quote = quote, dec = dec, fill = fill, comment.char = comment.char, ...)read.csv2 function (file, header = TRUE, sep = ";", quote = "\"", dec = ",", fill = TRUE, comment.char = "", ...) read.table(file = file, header = header, sep = sep, quote = quote, dec = dec, fill = fill, comment.char = comment.char, ...)

You’ll find the read.csv and read.csv2 are just wrapper functions around read.table with few changes in their default arguments.

Essentially,df <- read.table(“<path/comma_file.csv>”, sep = “,”) # is same as: df <- read.csv(“<path/comma_file.csv>”)

anddf <- read.table(“<path/semi_colon_file.csv>”, sep = “;”) # is same as: df <- read.csv2(“<path/ semi_colon_file.csv>”)

Similarly, for other separators, you can use either read.table or read.delim() with appropriate ‘sep’ argument.

Copy and paste from files directly into R

This is a quick and dirty hack that can save you a lot of time and effort. Select the data and copy it and run this command to bring it into R –df <- read.table(“clipboard”)

Sometimes when your data won’t be in the appropriate format, it wouldn’t be read correctly. But, this is the best compromise when you want to do some ad-hoc analysis on the fly.

Packages

We’ve seen how to import files through basic R commands. You can also do this by loading in packages and using the packages’ functions. Furthermore, almost all complicated file types need respective packages for getting imported to R.

Some Workspace Prepping (yet again)

To use the import functions in a package, you first need to install your package and load it in your environment. Simply run the following two commands: install.packages(“<name of the package>”) library(“<name of the package>”)

Reading JSON files

For reading JSON files into R, you’ll need a package called ‘rjson’. Run the commands to download and install the package into R – install.packages(“rjson”)And load it into the environment by library(rjson)

Now, you can use the fromJSON() function to read JSON data.#1 Import data from json file in your working directory JsonData <- fromJSON(file= “<inputJsonFile.json>”) #2 Import data from json file on a specific URL JsonData <- fromJSON(file= “<URL to your JSON file>”)

Tip: The input JSON file is stored in JsonData object in R environment as a list. You can view the contents by printing the object JsonData. Also you can convert it into a data frame by running the below commandJsonDF <- as.data.frame(JsonDF <- as.data.frame(JsonData) print(JsonData) #prints final data.frame

Importing XML data and data from HTML tables

Mostly while working with data on the World Wide Web, you’ll have to deal with HTML and XML formats.

Both XML and HTML data can be imported easily by the XML package in R. It offers many approaches for both reading and creating XML (and HTML) documents (including DTDs), both local and accessible via HTTP or FTP.

First, install and load the package, just like demonstrated above. Use the xmlParse() function to parse the XML file as shown in the code snippet –# Activate the libraries library(XML) library(RCurl)# Read XML file by URL xmlData <- xmlTreeParse(“<URL of XML data>”)# Read XML file in working directory XmlData <- xmlTreeParse(“<inputXMLfile.xml>”)

Tip: Again the xmlTreeParse() function returns a list into XmlData object. The same data can be read as a more usable data frame format by running below code –xmldataframe <- xmlToDataFrame(“<inputXMLfile.xml>”) print(xmldataframe)

Another useful function from the XML package allows reading data from one or more HTML tables. The readHTMLTable function and its methods provide robust methods for extracting data from HTML tables from an HTML document from a file, or (http: or ftp:) URL, or an already parsed document of htmlParse() function (XML package). One can also specify a specific <table> node in the HTML document to be read.# Read the HTML table HtmlData <- readHTMLTable(getURL(”<url to file>”))

Reading entire Excel workbooks

There are many packages to load excels into R — XLConnect, XLSX, gdata. A recent addition to these is the Hadley Wickham’s 2016 ReadXL package.

ReadXL surpasses it’s counterparts in both speed — leveraging R’s established RapidXML C++ library and ease of loading excel documents — since external code libraries e.g.: java jdk ; ActiveState PERL are not required.

Begin by installing and loading the package. Use read_excel function with default arguments to read in the first tab of your excel sheet. Remember your file must be in the current working directory, otherwise you need to write the path here instead of filename.df <- read_excel(“<inputExcelFile.xlsx>”)

You can read different sheets of your excel workbook into R by using the ‘sheet’ argument.read_excel(“<inputExcelFile.xlsx >”,sheet=”Sheet 3”)

With the sheet indexing starting at 1, you can also use indices to load in the corresponding tab with the following code:read_excel(“<inputExcelFile.xlsx >”,sheet=3)

You can also read data from excel spreadsheets using RODBC package which will be covered in the last section of the article.

Reading data from foreign statistical software

Sometimes, one would want to be able to shift between tools. R is very flexible in this aspect. It allows users to import (as well as export) data from foreign statistical software packages such as SAS, SPSS, Stata, Matlab, Octave. In this section, we will see how to do that.

Importing SAS, SPSS and Stata Files

There are a few contender packages available for importing SAS datasets — foreign, sas7bdat, haven. Among these, Haven package from Hadley Wickham seems the best option in terms of speed. It imports foreign statistical formats into R via the embedded ‘ReadStat’ C library.Install and load the package into your environment and run the following code to import from external software packages –Install and load the package into your environment and run the following code to import from external software packages – # Read a SAS dataset in your working directory read_sas(“InputSAS.sas7bdat”)

You can use similar functions of Haven package to read SPSS and Stata files as well.# Read a SPSS dataset in your working directory read_sav(“InputSPSS.sav”) # Read a Stata dataset in your working directory read_dta(“InputStata.dta”)

Importing Matlab and Octave Files

Use the R.matlab package to import MATLAB files into R. It consists of methods readMat() and writeMat() for reading and writing MAT files. The package also provides methods for controlling MATLAB (v6 or newer) via R and sending and retrieving data between R and MATLAB.#Install and load R.matlab package install.packages(“R.matlab”) library(R.matlab)# Read mat file data <- readMat("<Path to input file>")

For Octave files, you will need the foreign package. Use the read.octave() function to import Octave text data into R:library(foreign) data <- read.octave("<Path to input file>")

Other Relational Databases

RODBC package provides an ODBC database interface. It allows access to relational databases (including Microsoft Access and Microsoft SQL Server) through the ODBC interface. The primary functions are given below.odbcConnect(dsn, uid="", pwd="") odbcConnect establishes a connection to the specified DSN sqlFetch(channel, sqtable) sqlFetch by default retrieves the entire contents of the table sqlQuery(channel, query) sqlQuery sends the SQL statement query to the server, using connection channel returned by odbcConnect, and retrieves (some or all of) the results sqlSave(channel, mydf, tablename = sqtable, append = FALSE) sqlSave saves the data frame dat in the table tablename. If the table exists and has the appropriate structure it is used, or else it is created anew. sqlClear(channel, sqtable) & sqlDrop(channel, sqtable) sqlClear deletes all the rows of the table sqtable, similar to TRUNCATE TABLE sql command; sqlDrop removes the table sqtable (if permitted), similar to DROP TABLE sql command. odbcClose(channel) odbcClose closes connections to ODBC databases, helps clean up and free resources. Another function odbcCloseAll closes all open channels. Channels are closed at the end of an R session, and may also be closed by garbage collection if no object refers to them anymore.

Let’s take up an example.

The following code imports 2 tables Table1 and Table2 from a DBMS into R Dataframe objects SqlData1 and SqlData2 using above functions.install.packages(“RODBC”) library(RODBC) con <-odbcConnect(“dsn”, uid=”userID”, pwd=”123”) SqlData1 <- sqlFetch(con,”Table1”) SqlData2 <- sqlQuery(con, “select * from Table2”) odbcClose(con)

Tips for making data import easier in R

Some points to keep in mind while importing files in R–

First row is usually reserved for column headers or names of the variables. In case you don’t have any column headers or names, R will automatically assign them for you once you import the file

Note that the column names should be unique, so remove duplicates

Also note that R is case sensitive

Short names are preferred over longer names

Follow the R naming conventions:

Avoid blank spaces

Avoid names with special symbols: ?, $, *, +, #, (, ), -, /, }, {, |, >, < etc. Instead, use underscore (var_name) or dot (var.name) or camel case (varName)

Variable names should not start with a number

Blank entries or missing values in the data should be replaced by NA

Remove any comments in the file to avoid unnecessary NA’s and extraneous columns

Follow one style of writing code. It helps on code consistency and maintains readability. You can find some style guides here

End Notes

Loading data into R is just a small step in your exciting data exploration, manipulation and visualization journey. To take the next step — like building predictive models or deploying AI solutions — check out our AI Consulting Services. We learnt how to import CSV, TXT, HTML, JSON files, as well as data sets from external statistical packages like SAS, SPSS, STATA and Matlab. Lastly, we discussed connecting to relational databases with ODBC in R. Note that there are always more than one ways to do a task in R (..almost always). I have tried to list the common commands with efficient counterparts if any, to the best of my knowledge. Please do share in comments if you feel there is a better way to do the discussed tasks.

Hope this article makes your importing tasks easy’R!!

This article was originally published on Perceptive Analytics Our mission is “to enable businesses unlock value in data”. We do many activities to do that. Helping you solve a tough problem is just one of them. For over 20 years, we’ve partnered with more than 100 clients — from Fortune 500 companies to mid-sized firms — to solve complex data analytics challenges. Our services include Talend Consulting, Looker Consulting and Snowflake Consulting

[aianddata]

Anomaly Detection in R

The World of Anomalies

Imagine you are a credit card selling company and you know about a particular customer who makes a purchase of 25$ every week. You guessed this purchase is his fixed weekly rations but one day, this customer makes a different purchase of 700$. This development will not just startle you but also compel you to talk to the customer and find out the reason so you can approve the transaction. This is because, the behavior of the customer had become fixed and the change was so different that it was not expected. Hence we call this event as an anomaly.

Detecting such anomalies is not just critical in BFSI but also a common offering by any AI consulting company that helps clients leverage advanced analytics for fraud detection and transaction monitoring. Tools like Power BI, combined with AI models, help teams visualize anomalies in real-time.

Anomalies are hard to detect because they can also be real phenomenon. Let’s say that the customer in the example above made the usual purchases while he was living alone and will be starting his family this week. This will mean that this should be the first of his future purchases of similar magnitude or he is throwing a party this week and this was a one-time large purchase. In all these cases, the customer will be classified as making an ‘abnormal’ choice. We as the credit card seller need to know which of these cases are genuine and which are mistakes which can be corrected if they reconfirm the same with the customer. The usefulness of detecting such anomalies are very useful especially in BFSI industry with the primary use in credit card transactions. Such anomalies can be signs of fraud or theft. Someone making multiple transactions of small amounts from the same credit card, making one very large transaction which is a few order of magnitudes larger than the average, making transactions from an unfamiliar location are such examples that can caused by fraudsters and must be caught. With the popularity of adoption, let’s study the ways we can detect anomalies.

Detecting The Pattern To Find Anomalies

Many artificial intelligence consulting services deploy such anomaly detection algorithms to help businesses uncover operational inefficiencies, customer behavior shifts, or suspicious transactions. Anomalies are essentially the outliers in our data. If something happens regularly, it is not an anomaly but a trend. Things which happen once or twice and are deviant from the usual behavior, whether continuously or with lags are all anomalies. So it all boils down to the definition of outliers for our data. R provides a lot of packages with different approaches to anomaly detection. We will use the AnomalyDetection package in R to understand the concept of anomalies using one such method. However, the package needs to be installed specially from github. This requires the install_github() function in devtools package. We will also use the Rcpp package which helps us to integrate R with C++ functions. Another github package to be used in this article is the wikipedia trend package which contains the API to access wikipedia and create data for anomaly detection analysis.

The package is capable of identifying outliers in the presence of seasonality and trend in the data. The package uses an algorithm known as Seasonal Hybrid ESD algorithm which finds outliers globally as well as locally in time series or a vector of data. The package has a lot of features, some of which include visualization graphs, type of anomalies (positive or negative) and specifying the window of interest.#Install the devtools package then github packages install.packages("devtools") install.packages("Rcpp") library(devtools) install_github("petermeissner/wikipediatrend") install_github("twitter/AnomalyDetection") #Loading the libraries library(Rcpp) library(wikipediatrend) library(AnomalyDetection)

The first step is data preparation. We will use the page views on wikipedia page marked on fifa data starting from date 18th March 2013. (link: https://en.wikipedia.org/wiki/FIFA). The wp_trend function gives us the access statistics for the page with the ability to filter data from within the function. We will use this data to model day wise page views and understand anomalies in the pattern of those view numbers#Download wikipedia webpage "fifa" fifa_data_wikipedia = wp_trend("fifa", from="2013-03-18", lang = "en")

This gives us a dataset of about 1022 observations and 8 columns. Looking at the data reveals some redundant information captured#First_look fifa_data_wikipedia project language article access agent granularity date views 197 wikipedia en Fifa all-access all-agents daily 2016-01-13 116 546 wikipedia en Fifa all-access all-agents daily 2016-12-27 64 660 wikipedia en Fifa all-access all-agents daily 2017-04-20 100 395 wikipedia en Fifa all-access all-agents daily 2016-07-29 70 257 wikipedia en Fifa all-access all-agents daily 2016-03-13 75 831 wikipedia en Fifa all-access all-agents daily 2017-10-08 194 229 wikipedia en Fifa all-access all-agents daily 2016-02-14 84 393 wikipedia en Fifa all-access all-agents daily 2016-07-27 140 293 wikipedia en Fifa all-access all-agents daily 2016-04-18 105 420 wikipedia en Fifa all-access all-agents daily 2016-08-23 757

We see that project, language, article, access, agent and granularity appear to be same for all rows and are irrelevant for us. We are only concerned with date and views as the features to work on. Let’s plot the views against date#Plotting data library(ggplot2) ggplot(fifa_data_wikipedia, aes(x=date, y=views, color=views)) + geom_line()

We see some huge spikes at different intervals. There are a lot of anomalies in this data. Before we process them further, let’s keep only the relevant columns.# Keep only date & page views and discard all other variables columns_to_keep=c("date","views") fifa_data_wikipedia=fifa_data_wikipedia[,columns_to_keep]

We will now perform anomaly detection using Seasonal Hybrid ESD Test. The technique maps data as a series and captures seasonality while pointing out data which does not follow the seasonality pattern. The AnomalyDetectionTs() function finds the anomalies in the data. It will basically narrow down all the peaks keeping in mind that not more than 10% of data can be anomalies (by default). We can also reduce this number by changing the max_anoms parameter in the data. We can also specify which kind of anomalies are to be identified using the direction parameter. Here, we are going to specify only positive direction anomalies to be identified. That means that sudden dips in the data are not considered.#Apply anomaly detection and plot the results anomalies = AnomalyDetectionTs(fifa_data_wikipedia, direction="pos", plot=TRUE) anomalies$plot

Our data has 5.68% anomalies in positive direction if we take a level of significance (alpha) to be 95%. Since we had a total of 1022 observations, 5.68% of the number is about 58 observations. We can look at the specific dates which are pointed out by the algorithm.# Look at the anomaly dates anomalies$anoms timestamp anoms 1 2015-07-01 269 2 2015-07-02 233 3 2015-07-04 198 4 2015-07-05 330 5 2015-07-06 582 6 2015-07-07 276 7 2015-07-08 211 8 2015-07-09 250 9 2015-07-10 198 10 2015-07-20 315 11 2015-07-21 209 12 2015-07-25 202 13 2015-07-26 217 14 2015-09-18 278 15 2015-09-25 234 16 2015-09-26 199 17 2015-10-03 196 18 2015-10-07 242 19 2015-10-08 419 20 2015-10-09 240 21 2015-10-11 204 22 2015-10-12 223 23 2015-10-13 237 24 2015-10-18 204 25 2015-10-28 213 26 2015-12-03 225 27 2015-12-21 376 28 2015-12-22 212 29 2016-02-24 240 30 2016-02-26 826 31 2016-02-27 516 32 2016-02-29 199 33 2016-04-04 330 34 2016-05-13 217 35 2016-05-14 186 36 2016-06-10 196 37 2016-06-11 200 38 2016-06-12 258 39 2016-06-13 245 40 2016-06-14 204 41 2016-06-22 232 42 2016-06-27 273 43 2016-06-28 212 44 2016-07-10 221 45 2016-07-11 233 46 2016-08-22 214 47 2016-08-23 757 48 2016-08-24 244 49 2016-09-18 250 50 2016-09-19 346 51 2017-01-10 237 52 2017-03-29 392 53 2017-06-03 333 54 2017-06-21 365 55 2017-10-08 194 56 2017-10-09 208 57 2017-10-11 251 58 2017-10-14 373

We have the exact dates and the anomaly values for each date. In a typical anomaly detection process, each of these dates are looked case by case and the reason for anomalies is identified. For instance, the page views can be higher on these dates if there had been fifa matches or page updates on these particular days. Another reason could be big news about fifa players. However, if page views on any of the dates does not correspond to any special event, then those days are true anomalies and should be flagged. In other situations such as credit card transactions, such anomalies can indicate fraud and quick action must be taken on identification.

The ‘Anomaly Way’

Anomalies are a kind of outlier so SH-ESD (Seasonal Hybrid ESD) is not the only way to detect anomalies. Moreover, ‘AnomalyDetection’ is not the only package we will look upon. Let’s try the anomalize package which is available in CRAN. However, it is always recommended to update the package using github as the owners keep the most recent package versions there and it takes time and testing for the changes to move into standard repositories such as CRAN. We will first install the package from CRAN so that the dependencies are also installed then update the package using devtools#Installing anomalize install.packages('anomalize') #Update from github library(devtools) install_github("business-science/anomalize") #Load the package library(anomalize) # We will also use tidyverse package for processing and coindeskr to get bitcoin data library(tidyverse) library(coindeskr)

I am also using the tidyverse package (Link) and coindeskr package (Link). The coindeskr package is used to download the bitcoin data and tidyverse is used for speedy data processing. We will now download bitcoin data from 1st January 2017#Get bitcoin data from 1st January 2017 bitcoin_data <- get_historic_price(start = "2017-01-01")

This data indicates the price per date. Let’s convert it into a time series#Convert bitcoin data to a time series bitcoin_data_ts = bitcoin_data %>% rownames_to_column() %>% as.tibble() %>% mutate(date = as.Date(rowname)) %>% select(-one_of('rowname'))

In the time series conversion, we are actually converting the data to a tibble_df which the package requires. We could have alternatively converted the data into tibbletime object. Since it is a time series now, we should also see the seasonality and trend patterns in the data. It is important to remove them so that anomaly detection is not affected. We will now decompose the series. We will also plot the series

#Decompose data using time_decompose() function in anomalize package. We will use stl method which extracts seasonalitybitcoin_data_ts %>% time_decompose(Price, method = "stl", frequency = "auto", trend = "auto") %>% anomalize(remainder, method = "gesd", alpha = 0.05, max_anoms = 0.1) %>% plot_anomaly_decomposition() Converting from tbl_df to tbl_time. Auto-index message: index = date frequency = 7 days trend = 90.5 days

We have some beautiful plots with the first plot being overall observed data, second being season, third being trend and the final plot analyzed for anomalies. The red points indicate anomalies according to the anomalize function. However, we are not looking for this plot. We only want the anomalies plot with trend and seasonality removed. Let’s plot the data again with recomposed data. This can be done by setting the time_recomposed() function

#Plot the data again by recomposing databitcoin_data_ts %>% time_decompose(Price) %>% anomalize(remainder) %>% time_recompose() %>% plot_anomalies(time_recomposed = TRUE, ncol = 3, alpha_dots = 0.5) Converting from tbl_df to tbl_time. Auto-index message: index = date frequency = 7 days trend = 90.5 days

This is a better plot and shows the anomalies. We all know how bitcoin prices shot up in 2018. The grey portion explains the expected trend. Let’s see what these red points are.#Extract the anomalies anomalies=bitcoin_data_ts %>% time_decompose(Price) %>% anomalize(remainder) %>% time_recompose() %>% filter(anomaly == 'Yes')

Now the anomalies dataset consists of the data points which were identified as anomalies by the algorithm

Looking to do more with your data? Check out our services : Microsoft Power BI Consultant | Power BI Consulting | Power BI Expert | Power BI Consultant | Tableau Consultants | Tableau Consulting | Tableau Expert | Tableau Contractor | Tableau Freelance Developer | Tableau Developer | Talend Consultant | Looker Consulting | Chatbot Consulting | Snowflake Consulting | AI Consulting Conclusion: Are You An Anomaly?

We have twitter’s anomaly detection package based on Seasonal Hybrid ESD (SH-ESD) as well as CRAN’s anomaly detection package based on factor analysis, Mahalanobis distance, Horn’s parallel analysis or Principal component analysis. We also have TsOutliers package and anomalize packages in R. There are a lot more packages than one could find in R. They all have the same concept but differ in the underlying algorithm which they use to detect anomaly. Hence, one can get a general idea from all such packages: anomalies are data points which do not follow the general trend or do not lie under the expected behavior of the rest of the data. The next question which is raised is the criteria for a data point to be following expected behavior. The rest of the data points are all anomalies. One can also have varying types of anomalies such as direction based anomalies as described by the anomaly detection package (positive or negative) or anomalies not following events such as matches in fifa data. One can similarly pitch in another logic for anomaly classification and treat them accordingly. These techniques are increasingly implemented by artificial intelligence consulting companies to support clients in everything from finance to healthcare.

Here is the entire code used in this article#Install the devtools package then github packages install.packages("devtools") install.packages("Rcpp") library(devtools) install_github("petermeissner/wikipediatrend") install_github("twitter/AnomalyDetection") #Loading the libraries library(Rcpp) library(wikipediatrend) library(AnomalyDetection) # Download wikipedia webpage "fifa" fifa_data_wikipedia = wp_trend("fifa", from="2013-03-18", lang = "en") #First_look fifa_data_wikipedia # Plotting data library(ggplot2) ggplot(fifa_data_wikipedia, aes(x=date, y=views, color=views)) + geom_line() # Keep only date & page views and discard all other variables columns_to_keep=c("date","views") fifa_data_wikipedia=fifa_data_wikipedia[,columns_to_keep] #Apply anomaly detection and plot the results anomalies = AnomalyDetectionTs(fifa_data_wikipedia, direction="pos", plot=TRUE) anomalies$plot # Look at the anomaly dates anomalies$anoms #Installing anomalize install.packages('anomalize') #Update from github library(devtools) install_github("business-science/anomalize") #Load the package library(anomalize) # We will also use tidyverse package for processing and coindeskr to get bitcoin data library(tidyverse) library(coindeskr) #Get bitcoin data from 1st January 2017 bitcoin_data = get_historic_price(start = "2017-01-01") #Convert bitcoin data to a time series bitcoin_data_ts = bitcoin_data %>% rownames_to_column() %>% as.tibble() %>% mutate(date = as.Date(rowname)) %>% select(-one_of('rowname')) #Decompose data using time_decompose() function in anomalize package. We will use stl method which extracts seasonality bitcoin_data_ts %>% time_decompose(Price, method = "stl", frequency = "auto", trend = "auto") %>% anomalize(remainder, method = "gesd", alpha = 0.05, max_anoms = 0.1) %>% plot_anomaly_decomposition() #Plot the data again by recomposing data bitcoin_data_ts %>% time_decompose(Price) %>% anomalize(remainder) %>% time_recompose() %>% plot_anomalies(time_recomposed = TRUE, ncol = 3, alpha_dots = 0.5) #Extract the anomalies anomalies=bitcoin_data_ts %>% time_decompose(Price) %>% anomalize(remainder) %>% time_recompose() %>% filter(anomaly == 'Yes')

[aianddata]

Forget Departmental Stores; Superstores are the Trend!

In addition to that from 1997 to 2007 numbers of warehouse club and super stores have increased by 178% while Departmental stores have decreased by 18%. The facts that numbers of Super stores are increasing and sales of Departmental stores are decreasing indicate that Warehouse clubs and Super stores are replacing Departmental stores.

Alcohol No Longer a Luxury!

Sports Habits Die Hard

From the above graph, it is also evident that sales growth is never negative indicating that Sports goods stores sales have never declined majorly. In addition to that, during recession sports goods stores sales has grown more than GDP in 2008 and maintained the sales i.e., 0% growth till 2009 end. This implies that even during recession people tend to keep up with their sports habits even though they may not take new ones.

No to Exclusive stores; Welcome Family Stores

The market share composition of clothing industry show that Family clothing stores market share has increased (from 44% to 66%) while the market share of women’s and men’s clothing stores has decreased (from 42% to 28% and 14% to 6% respectively) — a change in preference. The Compound Annual Growth Rate (CAGR) of Men’s clothing stores, Women’s Clothing stores and Family Clothing stores are -1.5%, 0.83% and 5.42% respectively. The facts that increasing market share and high CAGR implies that people are shifting their preference from exclusive stores to Family clothing stores.

Men’s clothing stores are majorly affected by family stores than women’s clothing stores. According to US Census data Men’s Clothing stores sales have decreased from $10 billion to $7 billion from 1992–2010 while women’s clothing stores sales have increased from $31 billion to $36 billion during the same period. In addition to that the growth rate trend of men’s clothing stores is decreasing while that of women’s clothing stores is increasing. This shows that men’s clothing stores with decreasing sales and sales growth are largely affected by family stores than women’s clothing stores. But something interesting is happening with women’s clothing stores sales, even though sales are growing its market share is decreasing, it means family stores are not replacing women’s clothing they are just growing more than women’s clothing stores while on the other hand in men’s clothing stores case family stores are replacing their sales.

The retail landscape is evolving rapidly, and businesses need to adapt fast. Retail analytics services can help decode these market shifts and guide smarter decisions.

Looking to do more with your data? Check out our services : Microsoft Power BI Consultant | Power BI Consulting | Power BI Expert | Power BI Consultant | Tableau Consultants | Tableau Consulting | Tableau Expert | Tableau Contractor | Tableau Freelance Developer | Tableau Developer | Talend Consultant | Looker Consulting | Chatbot Consulting | Snowflake Consulting | AI Consulting

This was originally published on Perceptive Analytics

[aianddata]

Tableau Sales Dashboard Performance

Business heads often use KPI tracking dashboards that provide a quick overview of their company’s performance and well-being. A KPI tracking dashboard collects, groups, organizes and visualizes the company’s important metrics either in a horizontal or vertical manner. The dashboard provides a quick overview of business performance and expected growth.

An effective and visually engaging way of presenting the main figures in a dashboard is to build a KPI belt by combining text, visual cues and icons. By using KPI dashboards, organizations can access their success indicators in real time and make better informed decisions that support long-term goals.

What is a KPI?

KPIs  (i.e.Key Performance Indicators) are also known as performance metrics, performance ratios or business indicators. A Key Performance Indicator is a measurable value that demonstrates how effectively a company is achieving key business objectives.

A sales tracking dashboard provides a complete visual overview of the company’s sales performance by year, quarter or month. Additional information such as the number of new leads and the value of deals can also be incorporated.

Example of KPIs on a Sales Dashboard:

Number of New Customers and Leads

Churn Rate (i.e. how many people stop using the product or service)

Revenue Growth Rate

Comparison to Previous Periods

Most Recent Transactions

QTD (quarter to date) Sales

Profit Rate

State Wise Performance

Average Revenue for Each Customer

Bringing It All Together with Dashboards and Stories

An essential element of Tableau’s value is delivered via dashboards. Well-designed dashboards are visually engaging and draw in the user to play with the information. Dashboards can facilitate details-on-demand that enable the information consumer to understand what, who, when, where, how and perhaps even why something has changed. This is where Tableau development services come into play, enabling customized solutions tailored to business-specific metrics.

Best Practices to Create a Simple and Effective Dashboard to Observe Sales Performance KPIs

A well-framed KPI dashboard instantly highlights problem areas. The greatest value of a modern business dashboard lies in its ability to provide real-time information about a company’s sales performance. As a result, business leaders, as well as project teams, are able to make informed and goal-oriented decisions, acting on actual data instead of gut feelings. The choice of chart types on a dashboard should highlight KPIs effectively. If you’re looking to implement these practices in your organization, working with Tableau experts ensures your dashboards follow industry standards and drive actionable insights.

Bad Practices Examples in a Sales Dashboard:

A sales report displaying 12 months of history for twenty products; 12 × 20 = 240 data points.

Multiple data points do not enable the information consumer to effectively discern trends and outliers as easily as a time-series chart comprised of the same information

The quality of the data won’t matter if the dashboard takes five minutes to load

The dashboard fails to convey important information quickly

The pie chart has too many slices, and performing precise comparisons of each product sub-category is difficult

The cross-tab at the bottom requires that the user scroll to see all the data

Now, we will focus on thebest practices to create an effective dashboardto convey the most important sales information. Tableau is designed to supply the appropriate graphics and chart types by default via the “Show me” option.

I. Choose the Right Chart Types 

With respect to sales performance, we can use the following charts to show the avg. sales, profits, losses and other measures.

Bar chartsto compare numerical data across categories to show sales quantity, sales expense, sales revenue, top products and sales channel etc. This chart represents sales by region.

Line chartsto illustrate sales or revenue trends in data over a period of time:

AHighlight tableallows us to apply conditional formatting (a color scheme in either a continuous or stepped array of colors from highest to lowest) to a view.

UseScatter plotsorscatter graphsto investigate the relationship between different variables or to observe outliers in data. Example: sales vs profit:

UseHistogramsto see the data distribution across groups or to display the shape of the sales distribution:

Advanced Chart Types:

UseBullet graphsto track progress against a goal, a historical sales performance or other pre-assigned thresholds:

TheDual-line chart(or dual-axis chart), is an extension of the line chart and allows for more than one measure to be represented with two different axis ranges. Example: revenue vs. expense

ThePareto chartis the most important chart in a sales analysis. The Pareto principle is also known as 80-20 rule; i.e roughly 80% of the effects come from 20% of the causes.

When performing a sales analysis, this rule is used for detecting the 80% of total sales derived from 20% of the products.

UseBox plotsto display the distribution of data through their quartiles and to observe the major data outliers

Tableau Sales Dashboard

Here is a Tableau dashboard comprised of the aforementioned charts. This interactive dashboard enables the consumer to understand sales information by trend, region, profit and top products.

II. Use Actions to filter instead of Quick Filters

Using actions in place of Quick Filters provides a number of benefits. First, the dashboard will load more quickly. Using too many Quick Filters or trying to filter a very large dimension set can slow the load time because Tableau must scan the data to build the filters. The more quick filters enabled on the dashboard, the longer it will take the dashboard to load.

 III. Build Cascading Dashboard Designs to Improve Load Speed

By creating a series of four-panel, four cascading dashboards the load speed was improved dramatically and the understandability of the information presented was greatly enhanced. The top-level dashboard provided a summary view, but included filter actions in each of the visualizations that allowed the executive to see data for different regions, products, and sales teams.

IV. Remove All Non-Data-Ink

Remove any text, lines, or shading that doesn’t provide actionable information. Remove redundant facts. Eliminate anything that doesn’t help the audience understand the story contained in the data.

V. Create More Descriptive Titles for Each Data Pane

Adding more descriptive data object titles will make it easier for the audience to interpret the dashboard. For example:

Bullet Graph—Sales vs. Budget by Product

Sparkline—Sales Trend

Cross-tab—Summary by Product Type

Scatter Plot—Sales vs. Marketing Expense

VI. Ensure That Each Worksheet Object Fits Its Entire View

When possible, change the graphs fit from “Normal” to “Entire View” so that all data can be displayed at once.

VII. Adding Dynamic Title Content

There is an option to use dynamic content and titles within Tableau. Titles can be customized in a dynamic way so that when a filter option is selected, the title and content will change to reflect the selected value. A dynamic title expresses the current content. For example: if the dashboard title is “Sales 2013” and the user has selected year 2014 from the filter, the title will update to “Sales 2014”.

VIII. Trend Lines and Reference Lines

Visualizing granular data sometimes results in random-looking plots. Trend lines help users interpret data by fitting a straight or curved line that best represents the pattern contained within detailed data plots. Reference lines help to compare the actual plot against targets or to create statistical analyses of the deviation contained in the plot; or the range of values based on fixed or calculated numbers.

 IX. Using Maps to Improve Insight

Seeing the data displayed on a map can provide new insights. If an internet connection is not available, Tableau allows a change to locally-rendered offline maps. If the data includes geographic information, we can very easily create a map visualization.

This map represents sales by state. The red color represents negative numbers and the green color represents positive numbers.

X. Developing an Ad Hoc Analysis Environment

Tableau facilitates ad hoc analysis in three ways:

Generating new data with forecasts

Designing flexible views using parameters

Changing or creating designs in Tableau Server

 XI. Using Filters Wisely

Filters generally improve performance in Tableau. For example, when using a dimension filter to view only the West region, a query is passed to the underlying data source, resulting in information returned for only that region. We can see the sales performance of the particular region in the dashboard. By reducing the amount of data returned, performance improves.

Enhance Visualizations Using Colors, Labels etc.

I. Using colors:

Color is a vital way of understanding and categorizing what we see. We can use color to tell a story about the data, to categorize, to order and to display quantity. Color helps with distinguishing the dimensions. Bright colors pop at us, and light colors recede into the background. We can use color to focus attention on the most relevant parts of the data visualization. We choose color to highlight some elements over others, and use it to convey a message.

Red is used to denote smaller values, and blue or green is used to denote higher values. Red is often seen as a warning color to show the loss or any negative number whereas blue or green is seen as a positive result to show profit and other positive values.

Without colors:

II. Using Labels:

Enable labels to call out marks of interest and to make the view more understandable. Data labels enable comprehension of exact data point values. In Tableau, we can turn on mark labels for marks, selected marks, highlighted marks, minimum and maximum values, or only the line ends.

Without labels:

With labels:

Using Tableau to enhance KPI values

The user-friendly interface allows non-technical users to quickly and easily create customized dashboards. Tableau can connect to nearly any data repository, from MS Excel to Hadoop clusters. As mentioned above, using colors and labels, we can enhance visualization and enhance KPI values. Here are some additional ways by which we can enhance the values especially with Tableau features.

I. Allow for Interactivity

Playing, exploring, and experimenting with the charts is what keeps users engaged. Interactive dashboards enable the audiences to perform basic analytical tasks such as filtering views, drilling down and examining underlying data – all with little training.

II. Custom Shapes to Show KPIs

Tableau shapes and controls can be found in the marks card to the right of the visualization window. There are plenty of options built into Tableau that can be found in the shape palette.

Custom shapes are very powerful when telling a story with visualizations in dashboards and reports. We can create unlimited shape combinations to show mark points and create custom formatting. Below is an example that illustrates how we can represent the sales or profit values with a symbolic presentation.

Here green arrows indicate good sales progress and red arrows indicate a fall in Year over Year Sales by Category

III. Creating Calculated Fields

Calculated fields can be used to create new dimensions such as segments, or new measures such as ratios. There are many reasons to create calculated fields in Tableau. Here are just a few:

Segmentation of data in new ways on the fly

Adding a new dimension or a new measure before making it a permanent field in the underlying data

Filtering out unwanted results for better analyses

Using the power of parameters, putting the choice in the hands of end users

Calculating ratios across many different variables in Tableau, saving valuable database processing and storage resources

IV. Data-Driven Alerts

With version 10.3, Tableau has introduced a very useful feature: Data-Driven Alerts. We may want to use alerts to notify users or to remind that a certain filter is on and want to be alerted somehow if performance is ever higher or lower than expected. Adding alerts to dashboards can help elicit necessary action by the information consumer. This is an example of a data driven alert that we can set while displaying a dashboard or worksheet.

In a Tableau Server dashboard, we can set up automatic mail notifications to a set of recipients when a certain value reaches a specific threshold.

Summary

For an enterprise, a dashboard is a visual tool to help track, monitor and analyze information about the organization. The aim is to enable better decision making. To ensure these dashboards effectively align with sales and revenue goals, tableau professional services provide tailored visual analytics solutions.

A key feature of sales dashboards in Tableau is interactivity. Dashboards are not simply a set of reports on a page; they should tell a story about the business. In order to facilitate the decision-making process, interactivity is an important part of assisting the decision-maker to get to the heart of the analysis as quickly as possible.

Looking to do more with your data? Check out our services : Microsoft Power BI Consultant | Power BI Consulting Tableau Consultants Talend Consultant | Looker Consulting | Chatbot Consulting | Snowflake Consulting | AI Consulting

[aianddata]

A Beginner’s Guide to Channel Attribution Modeling in Marketing

Introduction

In a typical ‘from think to buy’ customer journey, a customer goes through multiple touch points before zeroing in on the final product to buy. This is even more prominent in the case of e-commerce sales. It is relatively easier to track which are the different touch points the customer has encountered before making the final purchase.

As marketing moves more and more towards the consumer driven side of things, identifying the right channels to target customers has become critical for companies. This helps companies optimise their marketing spend and target the right customers in the right places.

More often than not, companies usually invest in the last channel which customers encounter before making the final purchase. However, this may not always be the right approach. There are multiple channels preceding that channel which eventually drive the customer conversion. The underlying concept to study this behavior is known as ‘multi-channel attribution modeling.’

In this article, we look at what channel attribution is and how it ties into the concept of Markov chains. We’ll also take a case study of an e-commerce company to understand how this concept works, both theoretically and practically (using R).

Table of Contents

What is Channel Attribution?

Markov Chains

Removal Effect

Case Study of an E-Commerce Company

Implementation in R

What is Channel Attribution?

Google Analytics offers a standard set of rules for attribution modeling. As per Google, “An attribution model is the rule, or set of rules, that determines how credit for sales and conversions is assigned to touchpoints in conversion paths. For example, the Last Interaction model in Analytics assigns 100% credit to the final touchpoints (i.e., clicks) that immediately precede sales or conversions. In contrast, the First Interaction model assigns 100% credit to touchpoints that initiate conversion paths.”

We will see the last interaction model and first interaction model later in this article. Before that, let’s take a small example and understand channel attribution a little further. Let’s say we have a transition diagram as shown below:

In the above scenario, a customer can either start their journey through channel ‘C1’ or channel ‘C2’. The probability of starting with either C1 or C2 is 50% (or 0.5) each. Let’s calculate the overall probability of conversion first and then go further to see the effect of each of the channels.

P(conversion) = P(C1 -> C2 -> C3 -> Conversion) + P(C2 -> C3 -> Conversion)

= 0.5*0.5*1*0.6 + 0.5*1*0.6 = 0.15 + 0.3 = 0.45

Markov Chains

Markov chains is a process which maps the movement and gives a probability distribution, for moving from one state to another state. A Markov Chain is defined by three properties:

State space – set of all the states in which process could potentially exist

Transition operator –the probability of moving from one state to other state

Current state probability distribution – probability distribution of being in any one of the states at the start of the process

We know the stages through which we can pass, the probability of moving from each of the paths and we know the current state. This looks similar to Markov chains, doesn’t it?

Removal Effect

This is, in fact, an application of a Markov chains. We will come back to this later; let’s stick to our example for now. If we were to figure out what is the contribution of channel 1 in our customer’s journey from start to end conversion, we will use the principle of removal effect. Removal effect principle says that if we want to find the contribution of each channel in the customer journey, we can do so by removing each channel and see how many conversions are happening without that channel being in place.

For example, let’s assume we have to calculate the contribution of channel C1. We will remove the channel C1 from the model and see how many conversions are happening without C1 in the picture, viz-a-viz total conversion when all the channels are intact. Let’s calculate for channel C1:

P(Conversion after removing C1) = P(C2 -> C3 -> Convert)

= 0.5*1*0.6

= 0.3

30% customer interactions can be converted without channel C1 being in place; while with C1 intact, 45% interactions can be converted. So, the removal effect of C1 is

0.3/0.45 = 0.666.

The removal effect of C2 and C3 is 1 (you may try calculating it, but think intuitively. If we were to remove either C2 or C3, will we be able to complete any conversion?).

This is a very useful application of Markov chains. In the above case, all the channels – C1, C2, C3 (at different stages) – are called transition states; while the probability of moving from one channel to another channel is called transition probability.

Customer journey, which is a sequence of channels, can be considered as a chain in a directed Markov graph where each vertex is a state (channel/touch-point), and each edge represents transition probability of moving from one state to another. Since the probability of reaching a state depends only on the previous state, it can be considered as a memory-less Markov chain.

Case Study of an E-Commerce Company

Let’s take a real-life case study and see how we can implement channel attribution modeling.

An e-commerce company conducted a survey and collected data from its customers. This can be considered as representative population. In the survey, the company collected data about the various touch points where customers visit before finally purchasing the product on its website.

In total, there are 19 channels where customers can encounter the product or the product advertisement. After the 19 channels, there are three more cases:

#20 – customer has decided which device to buy;

#21 – customer has made the final purchase, and;

#22 – customer hasn’t decided yet.

The overall categories of channels are as below:CategoryChannel Website (1,2,3) Company’s website or competitor’s website Research Reports (4,5,6,7,8) Industry Advisory Research Reports Online/Reviews (9,10) Organic Searches, Forums Price Comparison (11) Aggregators Friends (12,13) Social Network Expert (14) Expert online or offline Retail Stores (15,16,17) Physical Stores Misc. (18,19) Others such as Promotional Campaigns at various location

Now, we need to help the e-commerce company in identifying the right strategy for investing in marketing channels. Which channels should be focused on? Which channels should the company invest in? We’ll figure this out using R in the following section. To complement such advanced analytics, companies often engage AI consulting firms to integrate predictive capabilities into their business processes.

Implementation using R

Let’s move ahead and try the implementation in R and check the results. You can download the dataset here and follow along as we go. #Install the libraries install.packages("ChannelAttribution") install.packages("ggplot2") install.packages("reshape") install.packages("dplyr") install.packages("plyr") install.packages("reshape2") install.packages("markovchain") install.packages("plotly") #Load the libraries library("ChannelAttribution") library("ggplot2") library("reshape") library("dplyr") library("plyr") library("reshape2") library("markovchain") library("plotly") #Read the data into R > channel = read.csv("Channel_attribution.csv", header = T) > head(channel)

Output:R05A.01R05A.02R05A.03R05A.04…..R05A.18R05A.19R05A.20 16 4 3 5   NA NA NA 2 1 9 10   NA NA NA 9 13 20 16   NA NA NA 8 15 20 21   NA NA NA 16 9 13 20   NA NA NA 1 11 8 4   NA NA NA

We will do some data processing to bring it to a stage where we can use it as an input in the model. Then, we will identify which customer journeys have gone to the final conversion (in our case, all the journeys have reached final conversion state).

We will create a variable ‘path’ in a specific format which can be fed as an input to the model. Also, we will find out the total occurrences of each path using the ‘dplyr’ package. > for(row in 1:nrow(channel)) {   if(21 %in% channel[row,]){channel$convert[row] = 1} } > column = colnames(channel) > channel$path = do.call(paste, c(channel[column], sep = " > ")) > head(channel$path) [1] "16 > 4 > 3 > 5 > 10 > 8 > 6 > 8 > 13 > 20 > 21 > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > 1"     [2] "2 > 1 > 9 > 10 > 1 > 4 > 3 > 21 > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > 1"     [3] "9 > 13 > 20 > 16 > 15 > 21 > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > 1" [4] "8 > 15 > 20 > 21 > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > 1" [5] "16 > 9 > 13 > 20 > 21 > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > 1" [6] "1 > 11 > 8 > 4 > 9 > 21 > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > NA > 1" > for(row in 1:nrow(channel)) {   channel$path[row] = strsplit(channel$path[row], " > 21")[[1]][1] } > channel_fin = channel[,c(23,22)] > channel_fin = ddply(channel_fin,~path,summarise, conversion= sum(convert)) > head(channel_fin)

Output:pathconversion 1 > 1 > 1 > 20 1 1 > 1 > 12 > 12 1 1 > 1 > 14 > 13 > 12 > 20 1 1 > 1 > 3 > 13 > 3 > 20 1 1 > 1 > 3 > 17 > 17 1 > 1 > 6 > 1 > 12 > 20 > 12 1 > Data = channel_fin > head(Data)

Output:pathconversion 1 > 1 > 1 > 20 1 1 > 1 > 12 > 12 1 1 > 1 > 14 > 13 > 12 > 20 1 1 > 1 > 3 > 13 > 3 > 20 1 1 > 1 > 3 > 17 > 17 1 1 > 1 > 6 > 1 > 12 > 20 > 12 1

Now, we will create a heuristic model and a Markov model, combine the two, and then check the final results. > H <- heuristic_models(Data, 'path', 'conversion', var_value='conversion') > H

Output:channel_namefirst_touch_conversions…..linear_touch_conversionslinear_touch_value 1 130   73.773661 73.773661 20 0   473.998171 473.998171 12 75   76.127863 76.127863 14 34   56.335744 56.335744 13 320   204.039552 204.039552 3 168   117.609677 117.609677 17 31   76.583847 76.583847 6 50   54.707124 54.707124 8 56   53.677862 53.677862 10 547   211.822393 211.822393 11 66   107.109048 107.109048 16 111   156.049086 156.049086 2 199   94.111668 94.111668 4 231   250.784033 250.784033 7 26   33.435991 33.435991 5 62   74.900402 74.900402 9 250   194.07169 194.07169 15 22   65.159225 65.159225 18 4   5.026587 5.026587 19 10   12.676375 12.676375 > M <- markov_model(Data, 'path', 'conversion', var_value='conversion', order = 1)> M

Output:channel_nametotal_conversiontotal_conversion_value 1 82.482961 82.482961 20 432.40615 432.40615 12 83.942587 83.942587 14 63.08676 63.08676 13 195.751556 195.751556 3 122.973752 122.973752 17 83.866724 83.866724 6 63.280828 63.280828 8 61.016115 61.016115 10 209.035208 209.035208 11 118.563707 118.563707 16 158.692238 158.692238 2 98.067199 98.067199 4 223.709091 223.709091 7 41.919248 41.919248 5 81.865473 81.865473 9 179.483376 179.483376 15 70.360777 70.360777 18 5.950827 5.950827 19 15.545424 15.545424

Before going further, let’s first understand what a few of the terms we’ve seen above mean.

First Touch Conversion: The conversion happening through the channel when that channel is the first touch point for a customer. 100% credit is given to the first touch point.

Last Touch Conversion: The conversion happening through the channel when that channel is the last touch point for a customer. 100% credit is given to the last touch point.

Linear Touch Conversion: All channels/touch points are given equal credit in the conversion.

Getting back to the R code, let’s merge the two models and represent the output in a visually appealing manner which is easier to understand. # Merges the two data frames on the "channel_name" column. R <- merge(H, M, by='channel_name') # Select only relevant columns R1 <- R[, (colnames(R) %in %c('channel_name', 'first_touch_conversions', 'last_touch_conversions', 'linear_touch_conversions', 'total_conversion'))] # Transforms the dataset into a data frame that ggplot2 can use to plot the outcomes R1 <- melt(R1, id='channel_name') # Plot the total conversions ggplot(R1, aes(channel_name, value, fill = variable)) +   geom_bar(stat='identity', position='dodge') +   ggtitle('TOTAL CONVERSIONS') +   theme(axis.title.x = element_text(vjust = -2)) +   theme(axis.title.y = element_text(vjust = +2)) +   theme(title = element_text(size = 16)) +   theme(plot.title=element_text(size = 20)) +   ylab("")

The scenario is clearly visible from the above graph. From the first touch conversion perspective, channel 10, channel 13, channel 2, channel 4 and channel 9 are quite important; while from the last touch perspective, channel 20 is the most important (in our case, it should be because the customer has decided which product to buy). In terms of linear touch conversion, channel 20, channel 4 and channel 9 are coming out to be important. From the total conversions perspective, channel 10, 13, 20, 4 and 9 are quite important.

End Notes

In the above chart we have been able to figure out which are the important channels for us to focus on and which can be discarded or ignored. This case gives us a very good insight into the application of Markov chain models in the customer analytics space. E-commerce companies can now confidently create their marketing strategy and distribute their marketing budget using data driven insights. Our team of experienced AI consulting firms can assist you in transforming these insights into impactful business outcomes.

Read more: Tableau Sales Dashboard Performance | Anomaly Detection | Text Mining Tips | Machine Learning with SVM