#job browser

[gethue-blog]

Secure your YARN cluster and access the jobs information safely

Hue can authenticate with Kerberos in YARN and guarantee than someone cannot access someone else’s MapReduce information.

As usual feel free to comment on the hue-user list or @gethue!

[gethue-blog]

Using Hadoop MR2 and YARN with an alternative Job Browser interface

Hue now defaults to using Yarn since version 3.

First, it is a bit simpler to configure Hue with MR2 than in MR1 as Hue does not need to use the Job Tracker plugin since Yarn provides a REST API. Yarn is also going to provide an equivalent of Job Tracker HA with YARN-149.

Here is how to configure the clusters in hue.ini. Mainly, if you are using a pseudo distributed cluster it will work by default. If not, you will just need to update all the localhost to the hostnames of the Resource Manager and History Server:

[hadoop] ... # Configuration for YARN (MR2) # ------------------------------------------------------------------------ [[yarn_clusters]] [[[default]]] # Enter the host on which you are running the ResourceManager resourcemanager_host=localhost # The port where the ResourceManager IPC listens on resourcemanager_port=8032 # Whether to submit jobs to this cluster submit_to=True # URL of the ResourceManager API resourcemanager_api_url=http://localhost:8088 # URL of the ProxyServer API proxy_api_url=http://localhost:8088 # URL of the HistoryServer API history_server_api_url=http://localhost:19888 # Configuration for MapReduce (MR1) # ------------------------------------------------------------------------ [[mapred_clusters]] [[[default]]] # Whether to submit jobs to this cluster submit_to=False

And that’s it! You can now look at jobs in Job Browser, get logs and submit jobs to Yarn!

As usual feel free to comment on the hue-user list or @gethue!

[gethue-blog]

JobTracker High Availability (HA) in MR1

When the Job Tracker goes down, Hue cannot display the Jobs in File Browser or submit to the correct cluster.

In MR1, Hadoop can support two Job Trackers, a master Job Tracker that can fail over to a standby Job Tracker and hence provide Job Tracker HA. Let's see how Hue 3.5 and CDH5beta1 (and probably CDH4.6) can take advantage of this.

Note: in MR1 Hue is using a plugin to communicate with the Job Tracker. This can be configured in CDH or Hadoop 0.23 / 1.2.0 (MAPREDUCE-461).

We configure two Job Trackers in the hue.ini:

[hadoop] ... [[mapred_clusters]] [[[default]]] # Enter the host on which you are running the Hadoop JobTracker jobtracker_host=host-1 # Whether to submit jobs to this cluster submit_to=True [[[ha-standby]]] # Enter the host on which you are running the Hadoop JobTracker jobtracker_host=host-2 # Whether to submit jobs to this cluster submit_to=True

And that's it! Hue will communicate with the available Job Tracker automatically!

Notice that in the case of Oozie jobs, Oozie will try to re-submit the job but will need a logical name (HUE-1631). To enable this in Hue, specify it in each MapReduce cluster, e.g.:

[hadoop] [[mapred_clusters]] [[[default]]] # JobTracker logical name. ## logical_name=MY_NAME

As usual feel free to comment on the hue-user list or @gethue!

[gethue-blog]

Hadoop tutorials: II.1. Prepare the data for analysis with Pig and Python UDF

Welcome to season 2 of the Hue video series. In this new chapter we are going to demonstrate how Hue can simplify Hadoop usage and lets you focus on the business and less about the underlying technology. In a real life scenario, we will use various Hadoop tools within the Hue UI and explore some data and extract some competitive advantage insights from it.

  Let’s go surf the Big Data wave, directly from your Browser!

We want to open a new restaurant. In order to optimize our future business we would like to learn more about the existing restaurants, which tastes are trending, what food eaters are looking for or are positive/negative about… In order to answer these questions, we are going to need some data.

Luckily, Yelp is providing some datasets of restaurants and reviews and we download them. What’s next? Let’s move the data into Hadoop and make it queryable!

Convert Json data with Pig

The current format is Json, which is easy to save but difficult to query as it consist in one big record for each row and requires a more sophisticated loader. We are also going to cleanup the data a bit in the process.

In order to do this in a scalable way, we are going to use the query tool Apache Pig and to make it easy, the Pig Editor in Hue. We explain two ways to do it.

All the code is available on the Hadoop Tutorial github.

Method 1: Pig JsonLoader/JsonStorage

Pig natively provides a JsonLoader. We load our data and map it to a schema, then explode the votes into 3 columns. Notice the clean-up of the text of the reviews.

Here is the script:

reviews = LOAD 'yelp_academic_dataset_review.json' USING JsonLoader('votes:map[],user_id:chararray,review_id:chararray,stars:int,date:chararray,text:chararray,type:chararray,business_id:chararray'); tabs = FOREACH reviews GENERATE (INT) votes#'funny', (INT) votes#'useful', (INT) votes#'cool', user_id, review_id, stars, REPLACE(REPLACE(text, '\n', ''), '\t', ''), date, type, business_id; STORE tabs INTO 'yelp_academic_dataset_review.tsv';

  Note: if the script fails with a ClassNotFound exception, you might need to logging as ‘oozie’ or ‘hdfs’ and upload /usr/lib/pig/lib/json-simple-1.1.jar into /user/oozie/share/lib/pig on HDFS with File Browser.

Method 2: Pig Python UDF

Let’s convert the business data to TSV with a great Pig features: Python UDF. We are going to process each row with with a UDF loading the Json records one by one and printing them with tabs as delimiter.

As Pig is currently using Jython 2.5 for executing Python UDF and there is no builtin json lib, we need to download jyson from http://downloads.xhaus.com/jyson/. Grab the jyson-1.0.2 version, extract it and upload jyson-1.0.2.jar to /user/oozie/share/lib/pig with FileBrowser.

We need to import our Python UDF into Pig. Open up the Pig Editor and upload a file resource named converter.py [link]. You can also create the file directly on HDFS with FileBrowser, then edit it and add this script:

from com.xhaus.jyson import JysonCodec as json @outputSchema("business:chararray") def tsvify(line): business_json = json.loads(line) business = map(unicode, business_json.values()) return '\t'.join(business).replace('\n', ' ').encode('utf-8')

Go to ‘Properties’, ‘Resource’ and specify the path to converter.py on HDFS.

You are then ready to type the following Pig script:

REGISTER 'converter.py' USING jython AS converter; reviews = LOAD '/user/romain/yelp/yelp_academic_dataset_business.json' AS (line:CHARARRAY); tsv = FOREACH reviews GENERATE converter.tsvify(line); STORE tsv INTO 'yelp_academic_dataset_business.tsv'

  What’s next?

Pig is a powerful tool for processing terabytes of data and Hue Pig Editor makes it easier to play around. Python UDF will become part of the editor when HUE-1136 is finished. In episode 3, we will see how to convert to even better formats.

In the next episode, let’s see how to query the data and learn more about the restaurant market!