Sim's Blog @simeonlobo - Tumblr Blog

Installing Key Data Science Libraries in R

I use the below script to set up my R environment with required Data Science libraries. Through trial and error, I've found that MacOS/X is my preferred O/S environment for Data Science. Some of the below libraries (like 'bigrf' used for Random Forests for larger datasets) are not available for Windows. Also for packages that I needed to compile, MacOS/X just worked every time. It is important to ensure that Java is installed on your system prior to running the below.

# STEP 1: INSTALL JAVA & POINT R TO IT # On a Mac add the following to your bash_profile $ export LD_LIBRARY_PATH=/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/jre/lib:/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/jre/lib/server # Run the below to create the relevant links $ sudo ln -s $(/usr/libexec/java_home)/jre/lib/server/libjvm.dylib /usr/local/lib # Point R to Java $ sudo R CMD javareconf # Install the pre-requisite rJava package install.packages('rJava', type='source') # STEP 2: PRE-MODELING STAGE PACKAGES # Data Visualisation install.packages("ggvis") install.packages("ggplot2") install.packages("googleVis") # Data Transformation install.packages("plyr") install.packages("data.table") # Missing Value Imputations install.packages("missForest") install.packages("missMDA") # Outlier Detection install.packages("outliers") install.packages("evir") # Feature Selection install.packages("features") install.packages("RRF") # Dimension Reduction install.packages("FactoMineR") install.packages("CCP") # STEP 3: MODELING STAGE PACKAGES # Continuous regression install.packages("car") install.packages("randomForest") # Ordinal regression install.packages("rminer") install.packages("CORElearn") # Classification install.packages("caret") install.packages("devtools") library(devtools) install_github("bigrf", repo='aloysius-lim/bigrf') #install.packages("~/Downloads/bigrf_0.1-11.tar.gz", repos = NULL, type = "source") #LINUX # Clustering install.packages("cba") install.packages("Rankcluster") # Time Series install.packages("forecast") install.packages("ltsa") # Survival install.packages("survival") install.packages("BaSTA") # STEP 3: POST MODELING STAGE PACKAGES # General Model Validation install.packages("lsmeans") install.packages("comparison") # Regression Validation install.packages("regtest") install.packages("ACD") # Classification Validation install.packages("binomTools") install.packages("Daim") # Clustering Validation install.packages("clusteval") install.packages("sigclust") # ROC Analysis install.packages("pROC") install.packages("timeROC") # STEP 4: OTHER USEFUL PACKAGES # Improve Performance install.packages("Rcpp") install.packages("parallel") # Work with Web install.packages("XML") install.packages("jsonlite") install.packages("httr") # Report Results install.packages("shiny") install.packages("rmarkdown") # Text Mining install.packages("tm") install.packages("twitteR") # Database install.packages("sqldf") # Install unixodbc first before RODBC #On a Mac run: brew install unixodbc install.packages("RODBC") install.packages("RMongo") # Miscellaneous install.packages("swirl") install.packages("reshape2") install.packages("qcc") install.packages("qdap") # STEP 5: RUN INVENTORY OF INSTALLED PACKAGES # Get a full list of installed packages write.csv(installed.packages(), file = "InstalledPackages.csv")

#R #Data Science

Reading Large Datasets into R for Analysis

R can appear to run slow while processing larger data sets. Some tips on dealing with the issue of preprocessing Big Data in R are listed below: TIP 1 - Set the colclasses Argument While reading in large datasets using read.table(), R can seem to lag as it attempts to infer the class of each column in the background by scanning the dataset. This can slow down the process of reading in the dataset, especially with Big Data. The colclasses argument in read.table() must be set for large datasets. Use the below code to figure out the classes for each of your columns. This only uses the first 100 rows to infer the column class:

initial <- read.table("datatable.txt", nrows=100) classes <- sapply(initial, class) tabAll <- read.table("datatable.txt", colClasses=classes)

TIP 2 - Set the nrows Argument Set the nrows argument in read.table(). If you're on a Mac or Linux use the command line utility wc to get an estimate of the number of lines/rows in your input file. This doesn't have to exact, just close enough. TIP 3 - Set the comment.char Argument If there are no comment lines, then set the comment.char argument to be blank. TIP 4 - System Contingency When processing large datasets ensure that there is no resource contention on the same machine when using the RServer client, or in a distributed environment if you've chosen to use RStudio Server or Microsoft R Server. TIP 5 - Making Decisions about RAM Restrictions If the size of the dataset is larger than the memory available on your computer then you will not be able to process the data effectively in R. A 64-bit system will obviously be able to perform better than a 32-bit one. There is a rough formula that can be used for calculating the amount of RAM required for R to hold the dataset in memory. The MAJOR assumption in the examples below is that all columns are of the numeric class and therefore use 8 bytes/numeric. I use this as a "gut feel" when planning the pre-processing of large datasets.

amount_of_RAM_in_GB <- (no_of_rows * no_of_columns * 8)/2^20/1024

Please note that the actual memory required by the computer to run effectively is roughly double the below required by R. Rough Calculations:

4GB RAM for 2.69M Rows, 200 Columns

16GB RAM for 4.3M Rows, 500 Columns

75GB RAM for 10M Rows, 1000 Columns

15TB RAM for 1B Rows, 2000 Columns

TIP 6 - Remove NA values Based on your use case you may make the decision to exclude missing values prior to developing algorithms for machine learning. I've followed an approach of extracting bad data (or column data with "NA") into a separate dataset that I may choose to use for analysis at a later stage. See the below example for extracting bad data and retaining clean data. This should work also a larger data set.

# Create dummy data for data frame a <- c(1, 2, 3, 4, NA) b <- c(6, 7, 8, NA, 10) c <- c(11, 12, NA, 14, 15) d <- c(16, NA, 18, 19, 20) e <- c(21, 22, 23, 24, 25) # Combine vectors to form a larger data frame df <- data.frame(a, b, c, d, e) # Append dataframes with row bind rdf <- rbind(df, df, df) # Create new data frame with only the clean data, i,e; no NA rdf_only_good_data <- na.omit(rdf) # Create a new data frame with only bad data, i.e.; only NA rdf_only_bad_data <- rdf[!complete.cases(rdf),] Output: > rdf a b c d e 1 1 6 11 16 21 2 2 7 12 NA 22 3 3 8 NA 18 23 4 4 NA 14 19 24 5 NA 10 15 20 25 6 1 6 11 16 21 7 2 7 12 NA 22 8 3 8 NA 18 23 9 4 NA 14 19 24 10 NA 10 15 20 25 11 1 6 11 16 21 12 2 7 12 NA 22 13 3 8 NA 18 23 14 4 NA 14 19 24 15 NA 10 15 20 25 > rdf_only_good_data a b c d e 1 1 6 11 16 21 6 1 6 11 16 21 11 1 6 11 16 21 > rdf_only_bad_data a b c d e 2 2 7 12 NA 22 3 3 8 NA 18 23 4 4 NA 14 19 24 5 NA 10 15 20 25 7 2 7 12 NA 22 8 3 8 NA 18 23 9 4 NA 14 19 24 10 NA 10 15 20 25 12 2 7 12 NA 22 13 3 8 NA 18 23 14 4 NA 14 19 24 15 NA 10 15 20 25

#R #DataScience #Analytics #BigData

Nice visual summary of Machine Learning Algorithms http://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/

#DataScience

#Predictions

The bots vendor landscape

#MachineLearning #DataScience

Bots are the next big thing!

#MachineLearning #DataScience

#DataScience #MachineLearning

The next iteration of the Amazon Prime Air looks pretty sturdy. It can cover 24 kms with a max speed of 88.5 kph. The video mentions a delivery SLA of 30 minutes.

#Innovation #Supply Chain #Drones

Interesting KickStarter project with the aim of delivering an economical USD $49 heads up display that anyone with a smartphone can use https://goo.gl/x3NjES

#Innovation #Mobility #DrivingApps

Yamaha’s Bicycle-powered Bamboo battery idea includes a sleek bicycle, battery and charging stand. This means the battery can be charged while hinging the bicycle on the charging stand and pedalling away.

Source: knstrct.com

#Innovation #Design

Interesting Indiegogo Project

#IoT #Internet of Everything #IoE #Mobility

futurescope

CREATIVITY VS. ROBOTS - The creativity economy and the future of employment

Nesta released a follow-up to their “Our Work Here is Done: Visions of a Robot Economy” report from 2014. This time they are looking at creativity in the UK and how creative jobs will be more resistant to automation in the future.

In ‘Creativity vs Robots’ we show that creativity is inversely related to computerisability: 87 per cent of highly creative workers are at low or no risk of automation, compared with 40 per cent of jobs in the UK workforce as a whole. At the regional level, we see that places with a higher proportion of the workforce in creative jobs, most obviously London, are also more immune to automation.

Such findings should not be surprising: they reflect the fact that machines can most successfully emulate humans when a problem is well specified in advance – that is, when performance can be straightforwardly quantified and evaluated – and when the work task environment is sufficiently simple to enable autonomous control. They will struggle when tasks are highly interpretive, geared at ‘products whose final form is not fully specified in advance’, and when work task environments are complex – a good description of most creative occupations.

A further new study for Nesta shows that creative occupations tend to be characterised by higher than average levels of life satisfaction, worthwhileness and happiness – but also higher levels of anxiety.

[Read the report (PDF)]

#Design #Creativity #Robots #Jobs