AWay to Data

Docendo discimus. Yerzhan Karatay's blog.

[awaytodata]

My Review on Confident Data Skills (2018) by Kirill Eremenko

Recently, I was invited to become an ambassador of Yandex.Praktikum, I have joyfully accepted the invitation, and because I love writing I joined a blogging marathon among a subgroup of ambassadors, sadly, it didn’t get supported by many but I prepared this review. A part of it was written 2 years ago, my highlights from the book have been kept publicly and privately on goodreads, so I decided to elaborate the review by adding them to the first version.

Source

I learnt a lot from Kirill as I also took some of his courses previously (R A-Z, Python A-Z), watched streams and even a DataScienceGO conference. So it was a pleasure to get to know him better in the introduction chapter where he shares his background story, it was really surprising that he grew up in South Africa, for example.

Confident Data Skills is a great introduction guide to understand what it means to be a data analyst and data scientist, how to become one, what they do and what their actual task is, which is to ensure that stakeholders understand the outcome of analysis, the required following actions and their costs or as he put: being a data scientist ultimately means being the storyteller of information. As much as I like writing code, I enjoy writing in general, my code writing skills were poorer 2 years ago, so this quote alone brought me hope back in that time.

The author explains well that we need analytics everywhere with case studies within Maslow’s Hierarchy of Need.

As a nota bene: the even better news is that data will not deny you your interests. If you’re only playing at being the connoisseur but still enjoy a trashy zombie movie once the curtains are drawn, your data will continue to keep that secret enthusiasm of yours fed.

He also seeks to clarify general misconceptions one may have.

The machines are not telling us what we want; they are making connections for us that we could not possibly have known.

The book will guide you on how an analysis is performed (the whole process generally), some analysis techniques, intuition behind them, statistics and personal tips on which tools to use in different cases, and visualization techniques. The data science process

Identify the question

Prepare the data

Analyze the data

Visualize the data

Present your findings

He has convinced me to appreciate the value of graphics as I had appreciated the value of numbers more than plots (which is probably a weird thing to claim considering that I’ve worked as a graphic designer, yet, numbers are important, there’d be no graphs, hah!), so it was in a way eye opening cause a great analyst should not underestimate the importance of presentation skills and it’s more than writing code, as one should aim to make plots as clear and intuitive as possible. It was one the plot twists of my life.

Carrying out BI might seem unsophisticated, … but dismiss it at your peril; I don’t know a single executive who would genuinely prefer to read a 50-page report over a visual representation of the problem and its solution. If visualized well, BI dashboards will engage and persuade your audience to make the changes that you suggest.

One will also find tips on where to start developing his or her skils and the career, and what one could expect in a job interview. I particularly enjoyed an example with questions like how many people have red cars in Australia where it’s important to show that one can reason to get to the answer instead of demonstrating the knowledge of facts, if one even possesses it.

[awaytodata]

SWIPE

View this post on Instagram

A post shared by Erjan Karatay 🌍 (@yerzh_n) on Aug 11, 2020 at 9:27am PDT

For Russian speakers:

Яндекс.Практикум — сервис онлайн-образования от Яндекса

Здесь учатся, получают навыки и опыт, создают новое и меняют свою жизнь.

For US citizens:

Practicum by Yandex is an online education service

Here students acquire skills and experience, broaden their horizons, and change their lives.

[awaytodata]

I Took 2 Udemy Courses On Process Flowcharts and Process Mapping

... and it was a long time ago but here is what I managed to make on my own with what I learnt in the first course from Jeremy Aschenbrenner, I sent these to him by email and got some feedback. I wanted to learn the common rules of designing flowcharts and learnt a lot more. 

Here’s one of the flowcharts which was a course task to design Pay at the Pump Process.

I was digitizing an old book at the time of taking the course, it hasn’t been digitized, and I did it for my private use with my phone, one doesn’t even need a scanner, so I decided to document the process. I used Office Lens to scan the pictures (with text recognition tools, namely OCR) and then corrected the mistakes. Higher level flowcharts keep breaking down in more and more details, so in Level 0 one sees a very general process.

[awaytodata]

How many followers would Lin Dan have, had he an Instagram profile?

What if...? Since Instagram and other social media are not accessible in China, its top athletes have to use local solutions. Lin Dan is one of the China's top marketable athletes that has become a two-time Olympic, five-time World, as well as a six-time All England champion. Marketers could use his image cause badminton is the second most popular sport in the world, but it seems like it is not being used to its full potential, which is why I am curious.

Data and Model’s Assumptions

I’ve made a linear regression model that tries to answer that question theoretically based on the number of followers other badminton players have considering their age, home country population, and career earnings from Badminton World Federation’s profiles, assuming that earnings correlate with fame the players get. For instance, badminton is big in Indonesia and young sportsmen are followed by a million people but the official profile of retired Taufik Hidayat only has under 190k followers currently, so one can assume that age correlates rather negatively, which I would say because older people seem to be reluctant to start and maintain a social media profile. I also assume that the number of followers positively correlates with a home country’s population, thus, the model assumes that Instagram is accessible in China.

I used the list of countries by population (United Nations) to merge it with previously gathered data of badminton players which includes country, career earnings, birth year and followers.

That is of course considering they had registered their instagram profiles at around the same time and kept them public, cause time is of importance to accumulate followers.

Regression Analysis Results

linearmodel <- lm(Followers~Population2017+CareerEarnings+Age, data=model)

summary(linearmodel)

Residual standard error: 360300 on 44 degrees of freedom

Multiple R-squared: 0.2604, Adjusted R-squared: 0.21 F-statistic: 5.164 on 3 and 44 DF,  p-value: 0.003813

p-value suggests that the model is statistically significant as it is smaller than any common level of significance, so I can assume that prediction results are explained by the model. 

Now I also prepared a test sample:

Running predict(linearmodel,prediction) results in 

Chen Long: 878529.1 Shi Yuqi: 768120.9 Lin Dan: 596571.1 Taufik Hidayat: NA Son Wan Ho: 166731.7 Lee Yong Dae: 300816.6

Interpretation of Results

Son Wan Ho and Lee Yong Dae are from Korea which explains the reason why they would have less followers even when their career earnings are comparable to Chinese players, age here works in favor of Shi Yuqi and against Lin Dan, who would have 600k followers approximately and whose number is comparable to Lee Chong Wei’s 529k at the time of data collection, while Chen Long’s career earnings would make him the most followed player in the sample, as he’s a 30-year-old 1-time Olympic and 2-time world champion. 

You may ask ‘why do Chinese players not have much more followers here?’, because Indian players are there in the dataset with comparable population but their players do not get as much attention as Indonesians get from their supporters.

Now, Taufik Hidayat's BWF Profile doesn't show his career earnings, and it is quite absurd to calculate money that he earned based on Indonesia’s population, his instagram followers and age but let’s do it for fun:

CareerEarn <- lm(CareerEarnings~Population2017+Followers+Age, data=model) TaufikHCareerEarn <- predict(CareerEarn, prediction[4,]) TaufikHCareerEarn

606796.1

[awaytodata]

Top 20 most followed Badminton players on Instagram in 2019

According to the data that I have collected (April, 2019), Jonatan Christie (21) has the biggest number of followers on Instagram among badminton players, most likely because the last year, 2018, he appeared to be the first Indonesian sportsman to claim Asian Games men’s singles gold title in the last 12 years. Not to mention that badminton is Indonesia’s national sport.

Position / Player/Profile (Country) / Category / Followers 01. Jonatan Christie (INA) / MS / 1.9m 02. Saina Nehwal (IND) / WS / 1.4m  03. PV Sindhu (IND) / WS / 1.3 m 04. Kevin Sanjaya (INA) / MD / 1.1m 05. Anthony Ginting (INA) / MS / 772k 06. Marcus Gideon (INA) / MD / 561k 07. Lee Chong Wei (MAS) / MS / 533k 08. Ratchanok Intanon (THA) / WS / 476k 09. Rian Ardianto (INA) / MD / 407k 10. Fajar Alfian (INA) / MD / 395k 11. Carolina Marin (ESP) / WS / 337k 12. Jwala Gutta (IND) / XD / 334k 13. Greysia Polii (INA) / WD / 301k 14. Goh Liu Ying (MAS) / XD / 299k 15. Viktor Axelsen (DEN) / MS / 270k 16. Tai Tzu Ying (TPE) / WS / 267k 17. Hendra Setiawan (INA) / MD / 187k 18. Debby Susanto (INA) / XD / 149k 19. Mohammad Ahsan (INA) / MD / 140k 20. Kidambi Srikanth (IND) / MS / 129k

The number of social media followers doesn’t mean that these are the most popular players in the world cause, well, China’s not participating, so I tried to make a linear regression model to predict the number of followers top Chinese players would have had, had they access to Instagram.

[awaytodata]

Tutorial: How to display any List of Geolocations on a Map?

Let's say you need to display a number of different locations on a map. I would like to visualize where the Airbnb rentals under $30 in Berlin are located. Besides listings data from InsideAirbnb, I am going to need RStudio packages (data.table, tidyr, and R.utils), Google Earth Pro (download and install it, if you don’t have it yet), and online CSV to KML Converter. If your data’s ready, then you can just skip the first part.

Getting data in shape (R code)

setwd(”...”) #setting the working directory to find the listings file library(data.table) #to read big files quick install.packages("R.utils") #to read .gz files right away Berlin <- fread("Berlinlistings.csv.gz") #the filename has been changed colnames(Berlin)  which(colnames(Berlin)=="name") #to find the column to export which(colnames(Berlin)=="latitude") which(colnames(Berlin)=="longitude") which(colnames(Berlin)=="is_location_exact") which(colnames(Berlin)=="price") Berlinlocation <- Berlin[,c(5,49,50,51,61)] str(Berlinlocation) library(tidyr) Berlinlocation <- separate(Berlinlocation,col=price,into=c("sign","chprice")) Berlinlocation <- data.frame(Berlinlocation, price = as.numeric(Berlinlocation$chprice)) Berlinlocation <- Berlinlocation[,c(-5,-6)] #delete odd columns berlinlocation <- subset(Berlinlocation, price <= 30 & is_location_exact == "t") write.csv(berlinlocation[,-4],"berlinlocation.csv") #save without the is_location_exact column

functions used: fread (data.table) to read big files, separate (tidyr) to delete the $ sign before the price in order to subset the data with needed filters (exact location and price below $30).

As a result, I got the .csv file that contains the 4 selected columns: name, price, geolocation coordinates (latitude, longitude) and 4495 rows/listings. 

Creating a KML file (from .csv) to visualize

Here’s an example of what a file should be like, the columns’ order doesn’t matter.

The next step is to convert the .csv file into a .kml (Keyhole Markup Language) file with free CSV to KML converter. After choosing .csv file, I take price as Name Field, that will allow me to see price on the map, the price column number is 5 (column E) in .csv, and what happens when I click on a data point? We’ll see its description, therefore the description field is going to be the name column of a listing in the data, it comes second (column B), so one will be able to see prices before the names. 

Highlighting locations from the dataset on the map

Now the final step is to launch the newly generated and downloaded KML file on Google Earth.

You get to know price and name of an apartment when you click on its location.

One can conclude that density of Airbnb listings under $30 in East Berlin is higher than in the city’s Western part.

If you want to launch the files, you can get them here.

P.S. Obviously, it doesn’t matter whether you need to highlight the points on a city, country, or cross-country level as long as you have the coordinates and Google Earth Pro. Have fun and let me know if it helped! It will make me happy.

[awaytodata]

Tutorial: How to import multiple (large) data files in R at once?

I had a project where I had to import around 100 different files 30 times (from 30 folders), in order to do that, I used a function called fread from data.table package (a faster alternative to read.csv that can work with big files), dirLister (a free and simple tool), and sheets.google.com (any Excel-ish software will do).

So, let’s assume we’ve got a folder with the following files:

I suggest to download and install dirLister, after launching the DirLister.exe file, navigate to the folder in Input tab.

Now switch to the Output tab, and choose where you want to save the filenames file, I chose Plain text (.txt) as an Output format and the same folder, then pressed Set as default options and Start.

The file with a list of folder file names will open (if you tick “Open outpute file/folder after list generation”), otherwise check the Output folder.

You can now copy the generated list in a Google Sheets file, where you are going to need a function called CONCATENATE.

id,name,host_id,host_name,neighbourhood_group,neighbourhood,latitude,longitude,room_type,price,minimum_nights,number_of_reviews,last_review,reviews_per_month,calculated_host_listings_count,availability_365

Considering these are the column names in the .csv files, let’s say we only want to select columns id, host_id, price, and availability_365, so the respective column numbers will be: 1, 3, 9, 16.  

NOTE: I’d advise to only import the columns that you need for your analysis as there are limits to either R or RAM. The rule is simple, the less data you load, the more files you can import.

So I pasted the list in column C, filled the first 3 cells in column A, and then filled the first row, used CONCATENATE in column F to connect all the other columns in one cell, then stretched the cells to the end of column C. 

=CONCATENATE(A1:D1)

Open R and write the following: install.packages("data.table") #if you don’t have the package preinstalled library(data.table) setwd("C://Users//ww//Documents//Data//InsideAirbnb") You can copy and paste column F from Google Sheets to R now. Here’s what I got:

file01<- fread("GermanyBerlin_listings.csv", select=c(1,3,10,16)) file02<- fread("Italy_Florencelistings.csv", select=c(1,3,10,16)) file03<- fread("ItalyBergamo_listings.csv", select=c(1,3,10,16)) file04<- fread("ItalyBologna_listings.csv", select=c(1,3,10,16)) file05<- fread("ItalyMilan_listings.csv", select=c(1,3,10,16)) file06<- fread("ItalyNaples_listings.csv", select=c(1,3,10,16)) file07<- fread("ItalyPuglia_listings.csv", select=c(1,3,10,16)) file08<- fread("ItalyRome_listings.csv", select=c(1,3,10,16)) file09<- fread("ItalySicily_listings.csv", select=c(1,3,10,16)) file10<- fread("ItalyVenice_listings.csv", select=c(1,3,10,16)) file11<- fread("SpainBarcelona_listings.csv", select=c(1,3,10,16)) file12<- fread("SpainEuskadi_listings.csv", select=c(1,3,10,16)) file13<- fread("SpainGirona_listings.csv", select=c(1,3,10,16)) file14<- fread("SpainMadrid_listings.csv", select=c(1,3,10,16)) file15<- fread("SpainMalaga_listings.csv", select=c(1,3,10,16)) file16<- fread("SpainMallorca_listings.csv", select=c(1,3,10,16)) file17<- fread("SpainMenorca_listings.csv", select=c(1,3,10,16)) file18<- fread("SpainSevilla_listings.csv", select=c(1,3,10,16)) file19<- fread("SpainValencia_listings.csv", select=c(1,3,10,16)) file20<- fread("TrentinoItaly_listings.csv", select=c(1,3,10,16)) file21<- fread("UKBristol_listings.csv", select=c(1,3,10,16)) file22<- fread("UKEdinburgh_listings.csv", select=c(1,3,10,16)) file23<- fread("UKGreaterManchester_listings.csv", select=c(1,3,10,16)) file24<- fread("UKLondon_listings.csv", select=c(1,3,10,16)) file25<- fread("UKManchester_listings.csv", select=c(1,3,10,15))

allfiles <- rbind(file01, file02, file03, file04, file05, file06, file07, file08, file09, file10, file11, file12, file13, file14, file15, file16, file17, file18, file19, file20, file21, file22, file23, file24, file25)

Use rbind then to unite all the files into one dataset. This is the way I did it for over 30 times, I hope that this semiautomatic method helped you and I would be happy to know of other more efficient ways.