Discover Top Posts Tagged with #cdh4

通过cloudera-manager来安装hadoop

本人笨，装的过程碰了不少东西，其他网站转载的文章也没具体写清楚，以下我实战了下总结下来。

1. ssh登陆要安装管理界面的机器，确定关闭防火墙和selinux，然后安装cloudera-manager-installer.bin

2. 修改host,并复制到所有节点

vim /etc/hosts

##内容--------------------------------

172.16.1.1x node1

172.16.1.2x node2

172.16.1.3x node3

127.0.0.1 localhost # 这是必须指定为localhost，且必须为第一个127.0.0.1的域

3. 打开管理界面 http://{{host}}:7180/

1）用来安装hadoop组件的帐号必须有ssh 且 root 权限

2）像我们的服务器都使用了key登陆，所以在安装时，必须为选用的帐号设置sudo权限且不需输入密码，以下操作是每一台节点机器必须进行

a. 用root操作，修改文件可写权限： chmod +w /etc/sudoers

b. vim /etc/sudoers 添加如： nic ALL=(ALL) NOPASSWD: ALL

c. 去除可写权限，chmod -w /etc/sudoers

3）给你安装hadoop的账户赋予这些文件的读权限和执行权限

chmod +r /bin/mktemp

chmod +x /bin/mktemp

chmod +r /usr/bin/tee

chmod +x /usr/bin/tee

chmod +r /usr/bin/tr

chmod +x /usr/bin/tr

4）随便进去一个目录wget下载各种hadoop组件安装包 (cloudera-manager所在机器如果不需要安装任何hadoop组件，则不需要下载和安装)

wget http://archive.cloudera.com/cm4/redhat/5/x86_64/cm/4.1.1/RPMS/x86_64/jdk-6u31-linux-amd64.rpm

wget http://archive.cloudera.com/cm4/redhat/5/x86_64/cm/4.1.2/RPMS/x86_64/cloudera-manager-agent-4.1.2-1.cm412.p0.428.x86_64.rpm

#hadoop #cdh4 #cloudera

Patching Wonderdog

For most of our processing, we are using Pig on top of Hadoop. To make things easier, CDH4 provides enough packages for us to get started fast.

Our views are stored in ElasticSearch, which means we need a way to read information from HDFS, process it in Pig and store the result in ElasticSearch. Luckily enough, the kind people at InfoChimps have a project on their github called Wonderdog.

Wonderdog contains an ElasticSearchStorage which we can use for reading and storing information from pig into ElasticSearch as we are used to.

Now for the catch.

We are using CDH4, which isn't supported by Wonderdog. Another downside is the fact that there is a nasty side-effect when you want to write multiple relations to ElasticSearch in a single pig script.

Since we really need this tool, we decided to patch wonderdog and commit the sources back to the community (https://github.com/calmera/wonderdog) Basically we did the following:

- Added CDH4/hadoop2 as a dependency to wonderdog - Upgraded the ElasticSearch dependency to 0.19.4 - Made the schema definitions which are stored in the UDFContext aware of the context signature.

The first two items allowed us to run Wonderdog on CDH4 while the last item allowed multiple calls to the ElasticSearchStorage within a single Pig Script.

#cdh4 #elasticsearch #wonderdog #infochimps

Connect to Hadoop Hive from within QlikView

This is an example to show how you can connect QlikView with Hadoop Hive by using the QlikView JDBC Connector. You need to install and activate the demo version from TIQ Solutions.

An easy way to start with Hadoop is the Cloudera Distribution Including Apache Hadoop (CDH) which is availble for download here: CDH4 (latest version is CDH4.0.1)

After starting the CentOS VM with Hadoop you can create the Beeswax for Hive examples (via web app Hue), which are two tables:

The Hive server service is already running on default port 10000. If not start the service:

/usr/bin/hive --service hiveserver

Don't forget to find out the IP address of your VM (call ifconfig).

Next steps are on the client side. Download Hive JDBC libraries and extract the files from the hive_jdbc-0.8.1.zip archive to a folder. Add all libraries in the shown order to the CLASSPATH with the config dialog of the QlikView JDBC Connector:

Now connect to the Hive instance from QlikView by chosing the JDBCConnector dll

and click Connect Button where you have to specify the the connection credentials:

Hadoop host IP address

Hive port (default is 10000)

JDBC driver class (as URL parameter, in bold)

Copy this connect string into the connect dialog window:

jdbc:hive://192.168.254.163:10000/default;

?connector.driverClass=org.apache.hadoop.hive.jdbc.HiveDriver;

Update: the URL parameter connector.driverClass is not needed anymore, set the JDBC driver classname in the config dialog sheet "Advanced Settings"

The dialog will past this custom connect statement into the script if you click OK:

CUSTOM CONNECT TO "Provider=JDBCConnector_x64.dll;jdbc:hive://192.168.254.163:10000/default;XUserId=ZCNDJPC;XPassword=aSBFNZC;";

Now you can use the table wizard to create your select statements (click select):

That's it. Everything is done so far. You can start loading you QlikView application now.

I have prepared a sample QlikView application QVHive.qvw you can download here.

#cdh4 #cloudera #hadoop #hive #jdbc #qlikview #bigdata