about Linux,Hadoop,Statistics,R and master thesis

[fatalzsq]

Solve GoAgent Socket Error

Warning: Listen on 127.0.0.1:8086 failed. Traceback (most recent call last):   File "./startup.py", line 8, in <module>     main()   File "<string>", line 1510, in _______   File "<string>", line 1475, in _93   File "<string>", line 1469, in _17   File "<string>", line 1449, in _36   File "<string>", line 1311, in __init__   File "/usr/lib/python2.7/SocketServer.py", line 408, in __init__     self.server_bind()   File "/usr/lib/python2.7/SocketServer.py", line 419, in server_bind     self.socket.bind(self.server_address)   File "/usr/lib/python2.7/socket.py", line 224, in meth     return getattr(self._sock,name)(*args) socket.error: [Errno 98] Address already in use

According to the error message, port 8086 is occupied. 

using command: lsof -i:8086

COMMAND PID         USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

Python  302 zsq   3u  IPv4 0xfd435e3061042831      0t0  TCP localhost:8086 (LISTEN)

Then kill the related PID and re-run GoAgent！

[fatalzsq]

Adding a Wordfile to UltraEdit for Syntax Highlighting

1.Download the language you need  from http://www.ultraedit.com/downloads/extras/wordfiles.html

2.copy the  uew file to worldfiles directory /Users/USERNAME/Library/Application Support/UltraEdit/wordfiles

[fatalzsq]

HDFS permisson problem:

Disabling the dfs permission.

By adding below property code to conf/hdfs-site.xml

 <property>     <name>dfs.permissions</name>     <value>false</value>   </property> Also can do this in CDH4 Manager HDFS CONFIGURATION.     

[fatalzsq]

My profile of Rhadoop Cluster

According to my previous post about RHadoop Installation on Cloudera QuickStart VM,my profile of real cluster is:

#/etc/profile export HADOOP_CMD=/opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop/bin/hadoop export HADOOP_STREAMING=/opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.6.0.jar export HADOOP_CONF_DIR=/etc/hadoop/conf export HADOOP_HOME=/opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop export JAVA_HOME=/usr/java/jdk1.6.0_31/ export JAVA_LIBRARY_PATH=/opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop/lib/native/ export HADOOP_LIBEXEC_DIR=/opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop/libexec/

BTW:using latest Rcpp(Rcpp_0.11.1.tar.gz) package from CRAN will report errors when installing rmr2. Install the older version(Rcpp_0.11.0.tar.gz) can be a simple solution to it.

[fatalzsq]

Antonio Piccolboni said in Rhadoop of Google Groups

The idea is, use the map to preprocess and split the data, then call rlm on the splits. So these are the good news. The bad news are if you need to solve one big problem. No natural split in the data exists or is desired and a single overall model is sought. Then you have to come up with your own algorithm, as in the random forest case. It makes use of a variety of existing packages, but it is a new algorithm. You can't just write rlm(some.formula, data = big.data.in.HDFS). It would be nice, but there is no shortcut, the algorithms will have to be written, that is true whether you are talking about R or Java or anything. Of course there may be a sudden breakthrough, I am just saying that it would be very big and unexpected. What rmr2 is hoping to achieve is making the writing of this new generation of algorithms more productive and more focused on the logic and less on the nitty-gritty.

[fatalzsq]

dnorm,pnorm,qnorm,rnorm

d  Density的缩写，表示密度函数。举个例子，标准正态分布x=0对应的值可以用dnorm(0)计算

p  Probability的缩写，表示概率函数。举个例子，标准正态分布从负无穷大到0的概率，可以用pnorm(0)计算

q  Quantile的缩写，表示分位函数。举个例子，如果知道标准正态分布从负无穷大到x的概率是0.9678，想要知道这个x的值，可以通过qnorm(0.9678)计算。

r  Random的缩写，表示随机函数。用于随机生成符合正态分布的数值，举个例子，如果想随机生成10个符合标准正态分布的函数，可以用rnorm(10)来获得。

[fatalzsq]

Our main goal was not simply to make it easy to run a mapreduce job but to make mapreduce jobs first class citizens of the R environment and to make it easy to create abstractions based on them. For instance, we wanted to be able to assign the result of a mapreduce job to a variable — and I mean the result, not some error code or diagnostics — and to create complex expressions including mapreduce jobs. We take the first step here by creating a function that is itself a job, can be chained with other jobs, executed in a loop etc.

——Mapreduce in R tutorial

[fatalzsq]

Understanding the Working Principal of MapReduce

Hadoop inside example for Word count：

file1:Hello World Bye Word

file2:Hello Hadoop GoodBye Hadoop

STEP1  file1--->Map1,file2--->Map2

map1:for each row set a pair of(k1,v1) key1 value1 0 Hello World Bye World map2:for each row set a pair of (k1,v1) key1 value1 0 Hello Hadoop GoodBye Hadoop

STEP2  Map1--->out1,Map2--->out2

map1: list(k2,v2) key2 value2 Hello 1 World 1 Bye 1 World 1 map2: list(k2,v2) key2 value2 Hello 1 Hadoop 1 GoodBye 1 Hadoop 1

STEP3  out1--->Combine1,out2--->Combine2

combine1: produce an intermediate list(k2,v2) add up the value when meet the same key key2 value2 Hello 1 World 2 Bye 1 combine2: produce an intermediate list(k2,v2) add up the value when meet the same key key2 value2 Hello 1 Hadoop 2 GoodBye 1

STEP4  Combine1 & Combine2--->Reduce

reduce: get the output from combine1 and combine2 new a list(k3,v3) add up the value when meet the same key key3 value3 Hello 2 World 2 Bye 1 Hadoop 2 GoodBye 1

The map and reduce function in Hadoop MapReduce have the following general form:

map:(K1,V1)→list(K2,V2) reduce:(K2,list(V2))→list(K3,V3)

If a combine function is used,then it's the same form as the reduce function (and is an implementation of Reducer), except its output types are the intermediate key and value types (K2 and V2), so they can feed the reduce function:

map:(K1,V1)→list(K2,V2) combine:(K2,list(V2))→list(K2,V2) reduce:(K2,list(V2))→list(K3,V3)

REFERENCE:Hadoop: The Definitive Guide

[fatalzsq]

Error while using package 'Rcmdr' on CentOS and R 3.0 (SOLVED)

I tried to load R Commander by using "library(Rcmdr)"

But it didn't work and reported the ERROR. 

Install the missing openGL libraries to fix the problem.

#run yum install in a terminal yum install mesa-libGL-devel mesa-libGLU-devel libpng-devel #install the package and load the GUI in R > install.packages("rgl") > library("Rcmdr")

[fatalzsq]

RHadoop Installation on Cloudera QuickStart VM

Download 3 packages (rmr,rhdfs,rhbase) for RHadoop from https://github.com/RevolutionAnalytics/RHadoop/wiki/Downloads

According 2 the installation manual rmr must be installed on all the hadoop nodes,  rhdfs to be installed on NameNode.

But i installed all the packages and modified /etc/profile on all the nodes in order to avoid errors...

#update the source to fedoraproject and get R installed rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm yum install R #run commands in R to install the following #packages related to rhdfs and rmr > install.packages("rJava") > install.packages("reshape2") > install.packages("Rcpp") > install.packages("iterators") > install.packages("itertools") > install.packages("digest") > install.packages("RJSONIO") > install.packages("functional") > install.packages("bitops") > install.packages("caTools")

  #find the working DIR for /bin/hadoop and #find the working DIR /contrib/streaming/hadoop-streaming*.jar locate /bin/hadoop locate  /contrib/streaming #add HADOOP_CMD path and HADOOP_STREAMING path #to the environment profile (vi /etc/profile) export HADOOP_CMD=/usr/lib/hadoop/bin/hadoop export HADOOP_STREAMING=/usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.4.0.jar export HADOOP_HOME=/usr/lib/hadoop export HADOOP_CONF_DIR=/etc/hadoop/conf #enable the modify without reboot source /etc/profile #install rmr and rhdfs in the terminal R CMD INSTALL "rmr2_3.0.0.tar.gz" R CMD INSTALL "rhdfs_1.0.8.tar.gz"

  #run commands in R > library(rhdfs) HADOOP_CMD=/usr/lib/hadoop/bin/hadoop Be sure to run hdfs.init() > hdfs.init() 14/02/17 09:29:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable #enter the DIR of /usr/lib/hadoop/lib/native #and copy libhadoop.so and libhadoop.so.1.0.0 to /usr/lib64 cd /usr/lib/hadoop/lib/native cp libhadoop.so /usr/lib64 cp libhadoop.so.1.0.0 /usr/lib64 #add JAVA_LIBRARY_PATH to profile to solve the problem #vi /etc/profile export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native #enable the ENV modify once more and re-config R java source /etc/profile R CMD javareconf #run commands in R and try 2 load rhdfs and rmr2 again #generally speaking the problem could be solved > library(rhdfs) > library(rmr2)

TROUBLESHOOTING !!!

#add HADOOP_LIBEXEC_DIR to /etc/profile #enable the setting！ export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec source /etc/profile

The example finally ran...

Checked the output！

[fatalzsq]

Cluster admin called me he monitored a high usage of CPUs. But i told him i never ran mapreduce in daytime...After checked the system logs, i found someone attempted to break in- -！They tried to crack through ssh,php scripts,GUI of DB administrator and etc...The system had to response these bad requests that was the cause of high CPU usage. 

Deploy Hadoop cluster in an intranet is much safer than expose them to internet！

[fatalzsq]

#copy logs from remote server scp root@ip_add:/var/log/secure* ./ #get my external ip curl ifconfig.me #view the accepted login cat secure|grep Accepted

[fatalzsq]

Some problems still need to be fixed...

[fatalzsq]

Ran an example on Hadoop

[fatalzsq]

Deploy easyHadoop on CentOS6.5 cluster (will be finished when i have time;)

1.Login as root and do some basic settings and checking

#check whether the runlevel is 3 cat /etc/inittab #view running services chkconfig --list #edit hostname for each server vi /etc/sysconfig/network #install git yum install git-core #clone project from github git clone git://github.com/zsqbb/easyhadoop.git

2.Install easyHadoop by guide (./easyhadoop/INSTALL)

#cd into easyhadoop DIR and run the python program python setup_centos_6.py #Open browser and access http://your_ip/ #After longin change the default user password

click 'Nodes'--->'Create  Node' or 'Create Nodes'. Then choose the correct OS,fill the IP and SSH Password,select the Role(s).Commonly a master included Namenode and Jobtracker,SecondaryNamenode is optional. Datanode and Tasktracker run on the slaves. Finally click Save button.

click 'Storage' to select position(s) for HDFS and then Save!

If can't select position for HDFS,try following commands to fix the problem. (Trouble-shooting Reference)

#make sure the firewall has been disabled chkconfig --list|grep iptables #run the command if the firewall is enabled,skip if not service iptables stop chkconfig iptables off #check whether the process is listening on 30050 port #if no process listening on 30050 #that means the NodeAgent.rpm wasn't installed correctly #remove the package and reinstall netstat -npl | grep 30050 cd /tmp rpm -e NodeAgent-1.2.1-1.x86_64 rpm -ivh NodeAgent-1.2.1-1.el6.x86_64.rpm

Follow the step by picture(Install-->Install-->Install) to install hadoop on each nodes.

TO BE CONTINUED...!!!

[fatalzsq]

Test easyHadoop on Ubuntu12.04 (To be continued...)

First step:Some basic settings

sudo passwd vi /etc/default/grub #change runlevel to 3 GRUB_CMDLINE_LINUX_DEFAULT="text" #update the grub configuration update-grub

Second step,LOGIN AS ROOT

Install Mysql5,Apache2,PHP5,phpMyAdmin

apt-get install mysql-server mysql-client apt-get install apache2 #install php5&its mod apt-get install php5 libapache2-mod-php5 #restart apache server /etc/init.d/apache2 restart #search php5 mods which need to be installed apt-cache search php5 #Install the following mods apt-get install php5-mysql php5-curl php5-gd php5-intl php-pear php5-imagick php5-imap php5-mcrypt php5-memcache php5-ming php5-ps php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl #restart apache server again /etc/init.d/apache2 restart apt-get install phpmyadmin #check http://ip/phpmyadmin/

Third step:easyHadoop Installation

Sign in https://github.com/ with your account (your email must be verified)

Press 'Fork' icon to fork from xianglei/easyhadoop (https://github.com/xianglei/easyhadoop)

Find 'HTTPS clone URL' and copy to clipboard

#install git apt-get install git #git clone+copied URL(from clipboard) git clone https://github.com/zsqbb/easyhadoop.git #run easyHadoop installation by python python exadoop/setup_ubuntu_12.04.py #login to mysql as root mysql -uroot -pxxx #create the easyhadoop database for system mysql> create database easyhadoop; mysql> exit; #import DB script mysql -uxxx -pxxx easyhadoop < easyhadoop.sql #Open browser and access http://your_ip/, the default user/pass : admin/123456

Fourth step:trouble shooting

Error message shown on browser as the following：

A Database Error Occurred

Unable to connect to your database server using the provided settings.

Filename: core/Loader.php

Line Number: 232

#manually update the system seaching index updatedb #forget it,actually no error in this file locate Loader.php #by google searching the trboule mostly is in the database.php #so locate the position of it locate database.php vi /var/www/application/config/database.php $db['default']['dbdriver'] = 'mysqli' #copy the following codes from http://ellislab.com/forums/viewthread/207916/ to the end of database.php,they can help you to find the bug

the debug report tell you there is no problem with connection between php and mysql

re-check the codes in database and find DB name was xadoop，but the DB name in mysql is easyhadoop

correct the error and restart service

OK！It works

[fatalzsq]

cook's distance

cook's distance 确是一个统计学词汇，用于诊断各种回归分析中是否存在异常数据。 SPSS专业统计软件对cook's distance 的解释。 Cook's distance.：A measure of how much the residuals of all cases would change if a particular case were excluded from the calculation of the regression coefficients. A large Cook's D indicates that excluding a case from computation of the regression statistics changes the coefficients substantially. 其意思就是，在你的数据资料中，如果某一条数据记录被排除在外，那么由此造成的回归系数变化有多大。显然，如果这个值过大，那么就表明这条数据对回归系数的计算产生了明显的影响，这条数据就是异常数据，需要好好考量是否在你的模型中使用这条数据。