Long Posts (Tech)

- iamjrp

[jumbojrp-blog]

Oozie SLA Monitoring

Introduction

Oozie workflow and coordinators allow SLA for

The maximum allowed time limit associated with when the job should start

By when the job should end

Duration of Run

Oozie can monitor the above states of a SLA-sensitive job and can send notification for SLA met or misses.

*Before oozie-4.x, client needs to query oozie SLA api, gets the data and has to calculate whether SLA met or misses.

Now, oozie provides 3 ways to get SLA information

SLA tab in oozie UI

JMS messages sent to a configured JMS provider for instantaneous tracking

RESTful API to query for SLA summary

Here we will discuss about RESTful API. Let's know a bit more about SLA, before going to implementation

So, there are 3 criteria of oozie SLA

Start Time

End Time

Duration

Depending on criteria, there can be 2 kinds of status , Event Status and SLA Status

Event Status

Start Time - START_MET, START_MISS

End Time - END_MET, END_MISS

Duration - DURATION_MET, DURATION_MISS

SLA Status

Not_Started <-- Job not yet begun

In_Process <-- Job started and is running, and SLAs are being tracked

Met <-- caused by an END_MET

Miss <-- caused by an END_MISS

Basically, SLA Status depends on END_MET or END_MISS as expected end-time is the most important criterion for majority of users.

For example:

For event status: START_MISS, END_MET, DURATION_MET

SLA Status would be END_MET

Here is the input in job.properties file: nominalTime=2017-07-09T16:47Z shouldStart=1 shouldEnd=5

Here is the response for such scenario and input

{ "slaSummaryList": [{ "nominalTime": 1499618820000, //GMT: Sunday, July 9, 2017 4:47:00 PM "jobStatus": "SUCCEEDED", "expectedEnd": 1499619120000, //GMT: Sunday, July 9, 2017 4:52:00 PM "appName": "simplejob", "actualEnd": 1499619043130, //GMT: Sunday, July 9, 2017 4:50:43.130 PM "actualDuration": 392, "expectedStart": 1499618880000, //GMT: Sunday, July 9, 2017 4:48:00 PM "expectedDuration": 240000, //4 min "durationDelay": -3, "slaStatus": "MET", "appType": "WORKFLOW_JOB", "slaAlertStatus": "Enabled", "eventStatus": "END_MET,DURATION_MET,START_MISS", "startDelay": 2, "id": "0000010-170708204706283-oozie-huse-W", "lastModified": 1499619052621, "user": "jrp", "actualStart": 1499619042738, //GMT: Sunday, July 9, 2017 4:50:42.738 PM "endDelay": -1 }] }

Also, when job doesn't end successfully i.e Failed/Error/Killed/Timeout, then also SLA Status would be Miss

Now you understood, when there will be SLA Met or Miss. To get this calculated SLA Status or to get notified in case of both SLA Met and Miss, we need to configure.

You can use JMS complient broker such as Apache ActiveMQ to send/receive messages. For more details, please check here https://oozie.apache.org/docs/4.1.0/AG_Install.html#Notifications_Configuration

Important

To utilize the service, you need to modify oozie-site.xml like below and restart oozie server

<property> <name>oozie.services.ext</name> <value> org.apache.oozie.service.EventHandlerService, org.apache.oozie.sla.service.SLAService </value> </property> <property> <name>oozie.service.EventHandlerService.event.listeners</name> <value> org.apache.oozie.sla.listener.SLAJobEventListener </value> </property>

*SLAJobEventListener - Processes job events and calculates SLA. Does not send any notifications (As I am targetting only getting calculated SLA, not notification)

Configuring SLA in application

To make your job SLA-sensitive job, you need to add sla tag in your workflow.xml.

for workflow schema before than "uri:oozie:workflow:0.5" ===> SLA schema version would be 0.1 i.e. xmlns:sla="uri:oozie:sla:0.1"

for new workflow schema i.e. xmlns="uri:oozie:workflow:0.5" ===> SLA schema version would be 0.2 i.e. xmlns:sla="uri:oozie:sla:0.2"

If you are using workflow version < 0.5 and sla schema version 0.2, you will get below error

Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a: Invalid content was found starting with element 'sla:info'. One of '{WC["uri:oozie:sla:0.1"]}' is expected.

So, in order to use new SLA schema, you will need to upgrade your workflow/coordinator schema to 0.5 i.e. 'uri:oozie:workflow:0.5'

To check schema for SLA 0.1 and 0.2, Please check here https://oozie.apache.org/docs/4.1.0/WorkflowFunctionalSpec.html#SLASchema

Sample Oozie Workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.4" xmlns:sla="uri:oozie:sla:0.1" name="simplejob"> <start to="hdfscommands"/> <action name="hdfscommands"> <fs> <delete path='${nameNode}/user/huser/oozie-simple/simplejob-dir'/> <mkdir path='${nameNode}/user/huser/oozie-simple/simplejob-dir'/> </fs> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}] </message> </kill> <end name="end"/> <sla:info> <!-- sla 0.2 --> <!-- <sla:nominal-time>${nominalTime}</sla:nominal-time> <sla:should-start>${shouldStart}</sla:should-start> <sla:should-end>${shouldEnd}</sla:should-end> <sla:max-duration>5</sla:max-duration> <sla:alert-events>start_miss,end_miss,duration_miss</sla:alert-events> <sla:alert-contact>${alertContact}</sla:alert-contact> <sla:notification-msg>${notificationMsg}</sla:notification-msg> --> <!-- sla 0.1 --> <sla:app-name>simplejob</sla:app-name> <sla:nominal-time>${nominalTime}</sla:nominal-time> <sla:should-start>${shouldStart}</sla:should-start> <sla:should-end>${shouldEnd}</sla:should-end> <sla:notification-msg>${notificationMsg}</sla:notification-msg> <sla:alert-contact>${alertContact}</sla:alert-contact> <sla:dev-contact>${devContact}</sla:dev-contact> <sla:qa-contact>${qaContact}</sla:qa-contact> <sla:se-contact>${seContact}</sla:se-contact> </sla:info> </workflow-app>

Sample job.properties

nameNode=hdfs://localhost:9000 jobTracker=localhost:8032 master=local[*] queueName=default oozie.use.system.libpath=true oozie.wf.application.path=${nameNode}/user/${user.name}/simplejob/ nominalTime=2017-07-09T17:36Z shouldStart=1 shouldEnd=5 notificationMsg=Notification for SLA [email protected] [email protected] [email protected] [email protected]

Important Tags

nominal-time: As the name suggests, this is the time relative to which your jobs' SLAs will be calculated.

should-start: Relative to nominal-time this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should start running to meet SLA. This is optional.

should-end: Relative to nominal-time this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should finish to meet SLA.

max-duration: This is the maximum amount of time (along with time-unit - MINUTES, HOURS, DAYS) your job is expected to run. This is optional.

alert-events: Specify the types of events for which Email alerts should be sent. Allowable values in this comma-separated list are start_miss, end_miss and duration_miss. *_met events can generally be deemed low priority and hence email alerting for these is not neccessary. However, note that this setting is only for alerts via email alerts and not via JMS messages, where all events send out notifications, and user can filter them using desired selectors. This is optional and only applicable when alert-contact is configured.

alert-contact: Specify a comma separated list of email addresses where you wish your alerts to be sent. This is optional and need not be configured if you just want to view your job SLA history in the UI and do not want to receive email alerts.

Accessing SLA Information

As I mentioned at the beginning, here we are discussing only accessing via RESTful API to query for SLA summary.

The Oozie Web Services API is a HTTP REST JSON API. All responses are in UTF-8 .There are two ways of accessing RESTful API of oozie

Command Line tool

OozieClient

To get calculated SLA information. Command Line is deprecated and will not give expected result. It will only returns sla informations passed in workflow by using below command

oozie sla -filter jobid=<jobID>

The only way to get the calculated SLA information is by using OozieClient, but there is a twist.

The webservices endpoint for sla is /v2/sla

You need to extend OozieClient class and implement for V2 version. Here is the CustomOozieClient.java present in github

https://github.com/jrphub/mapreduce/blob/master/src/main/java/com/sample/hadoop/oozie/CustomOozieClient.java

To implement more functionalities, please refer the Apache OozieClient.java here

https://github.com/apache/oozie/blob/branch-4.1/client/src/main/java/org/apache/oozie/client/OozieClient.java

[jumbojrp-blog]

Installing Oozie 4.3.0 on Hadoop 2.7.3

This weekend, I tried to configure oozie 4.3.0 and wanted to do some hands on submitting hadoop jobs to oozie. While installing oozie and to make it up, I struggled a lot with errors. So I thought to blog about it.

Pre-requisites

Java 1.7+

Hadoop 2.x (here, 2.7.3)

Maven3+

Unix box

Step1: Dist file

Get oozie tar.gz file from http://www-eu.apache.org/dist/oozie/4.3.0/ and extract it

cd $HOME tar -xvf oozie-4.3.0.tar.gz

Step2: Build Oozie

cd $HOME/oozie-4.3.0/bin ./mkdistro.sh -DskipTests

Step3: Server Installation

Copy the built binaries to the home directory as ‘oozie’

cd $HOME cp -R $HOME/oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0 .

Step 3.1: libext

Create libext directory inside oozie directory

cd $HOME/oozie mkdir libext

Note: ExtJS (2.2+) library (optional, to enable Oozie webconsole) But, The ExtJS library is not bundled with Oozie because it uses a different license :(

Now you need to put hadoop jars inside libext directory, else it will throw below error in oozie.log file

WARN ActionStartXCommand:523 - SERVER[data01.teg.io] USER[hadoop] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-161215143751620-oozie-hado-W] ACTION[0000000-161215143751620-oozie-hado-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]

So, let's put below jars inside libext directory

cp $HADOOP_HOME/share/hadoop/common/*.jar oozie/libext/ cp $HADOOP_HOME/share/hadoop/common/lib/*.jar oozie/libext/ cp $HADOOP_HOME/share/hadoop/hdfs/*.jar oozie/libext/ cp $HADOOP_HOME/share/hadoop/hdfs/lib/*.jar oozie/libext/ cp $HADOOP_HOME/share/hadoop/mapreduce/*.jar oozie/libext/ cp $HADOOP_HOME/share/hadoop/mapreduce/lib/*.jar oozie/libext/ cp $HADOOP_HOME/share/hadoop/yarn/*.jar oozie/libext/ cp $HADOOP_HOME/share/hadoop/yarn/lib/*.jar oozie/libext/

Step 3.2: Oozie Impersonate

To avoid impersonate error on oozie, modify core-site.xml like below

<!-- OOZIE --> <property><name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name><value>[OOZIE_SERVER_HOSTNAME]</value></property><property><name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name><value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value></property>

Assuming, my oozie user is huser and host is localhost and group is hadoop

<!-- OOZIE --> <property><name>hadoop.proxyuser.huser.hosts</name><value>localhost</value></property><property><name>hadoop.proxyuser.huser.groups</name><value>hadoop</value></property>

Note : You can use * in all values, in case of confusion

Step 3.3: Prepare the war

cd $HOME/oozie/bin ./oozie-setup.sh prepare-war

This will create oozie.war file inside oozie directory. If this war will be used further, you may face this error :

ERROR ActionStartXCommand:517 - SERVER[data01.teg.io] USER[hadoop] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-161220104605103-oozie-hado-W] ACTION[0000000-161220104605103-oozie-hado-W@mr-node] Error, java.lang.NoSuchFieldError: HADOOP_CLASSPATH

Why? because, The oozie compilation produced Hadoop 2.6.0 jars even when specifying Hadoop 2.7.3 with the option "-Dhadoop.version=2.7.3".

So, to avoid this error, copy the oozie.war file to a different directory

mkdir $HOME/oozie_war_dir cp $HOME/oozie/oozie.war $HOME/oozie_war_dir cd $HOME/oozie_war_dir jar -xvf oozie.war rm -f oozie.war/WEB-INF/lib/hadoop-*.jar rm -f oozie.war/WEB-INF/lib/hive-*.jar rm oozie.war jar -cvf oozie.war ./* cp oozie.war $HOME/oozie/

Then, regenerate the oozie.war binaries for oozie with a prepare-war

cd $HOME/oozie/bin ./oozie-setup.sh prepare-war

Step 3.4: Create sharelib on HDFS

cd $HOME/oozie/bin ./oozie-setup.sh sharelib create -fs hdfs://localhost:9000

Now, this sharelib set up may give you below error:

org.apache.oozie.service.ServiceException: E0104: Could not fully initialize service [org.apache.oozie.service.ShareLibService], Not able to cache sharelib. An Admin needs to install the sharelib with oozie-setup.sh and issue the 'oozie admin' CLI command to update the sharelib

To avoid this, modify oozie-site.xml like below

cd $HOME/oozie vi conf/oozie-site.xml

Add below property

<property> <name>oozie.service.HadoopAccessorService.hadoop.configurations</name> <value>*=/usr/local/hadoop/etc/hadoop/</value> </property>

The value should be your $HADOOP_HOME/etc/hadoop, where all hadoop configuration files are present.

Step 3.5 : Create Oozie DB

cd $HOME/oozie ./bin/ooziedb.sh create -sqlfile oozie.sql -run

Step 3.6 : Start Daemon

To start Oozie as a daemon use the following command:

./bin/oozied.sh start

To stop

./bin/oozied.sh stop

check logs for errors, if any

cd $HOME/oozie/logs tail -100f oozie.log

Use the following command to check the status of Oozie from command line:

$ ./bin/oozie admin -oozie http://localhost:11000/oozie -status System mode: NORMAL

Step 4: Client Installation

$ cd $ cp oozie/oozie-client-4.3.0.tar.gz . $ tar -xvf oozie-client-4.3.0.tar.gz $ mv oozie-client-3.3.2 oozie-client $ cd bin

Add $HOME/oozie-client/bin to PATH variable in .bashrc file and restart your terminal or do

source $HOME/.bashrc

For more details on set up, you can refer this URL https://oozie.apache.org/docs/4.3.0/DG_QuickStart.html

Now you can submit hadoop jobs to oozie in your terminal.

To run an example, you can follow this URL and set up your first example to run https://oozie.apache.org/docs/4.3.0/DG_Examples.html

You may face below error while running the map reduce example in above URL

java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

Solution:

Start mr-jobhistory-server.sh

cd $HADOOP_HOME/sbin ./mr-jobhistory-server.sh start historyserver

Another point to note about modifying job.properties file is :

nameNode=hdfs://localhost:9000 jobTracker=localhost:8032

in your case, this can be different, as I am using apache hadoop, you may be using cloudera/hdp/anything

To run spark job, I have tried running in local[], yarn-client and yarn-cluster as master, but succeeded in local[] only

Good luck and Keep Experimenting :)

[jumbojrp-blog]

HBase Fundamentals

An Overview

HBase is Open Source, Distributed, Versioned, Non-Relational, column-oriented Database, built on top of Hadoop File System, Horizontally Scalable, fault-tolerant and has easy Java API for client access.

It has below basic components. We will discuss one by one.

HBase Internals

Components

Region assignment, DDL (create, delete tables) operations are handled by the HBase Master process (HMaster).

Region servers serve data for reads and writes. When accessing data, clients communicate with HBase RegionServers directly.

HBase Tables are divided horizontally by row key range into “Regions.” A region contains all rows in the table between the region’s start key and end key. Regions are assigned to the nodes in the cluster, called “Region Servers,” and these serve data for reads and writes. A region server can serve about 1,000 regions.

Zookeeper, which is part of HDFS, maintains a live cluster state and provides server failure notification.

The ZooKeeper maintains ephemeral nodes for active sessions via heartbeats.

Each Region Server creates an ephemeral node. The HMaster monitors these nodes to discover available region servers, and it also monitors these nodes for server failures. HMasters compete with each other to create an ephemeral node. Zookeeper determines the first one and uses it to make sure that only one master is active. The active HMaster sends heartbeats to Zookeeper, and the inactive HMaster listens for notifications of the active HMaster failure.

Fault Tolerant

If a region server or the active HMaster fails to send a heartbeat, the session is expired and the corresponding ephemeral node is deleted. Listeners for updates will be notified of the deleted nodes. The active HMaster listens for region servers, and will recover region servers on failure. The Inactive HMaster listens for active HMaster failure, and if an active HMaster fails, the inactive HMaster becomes active.

There is a special HBase Catalog table called the META table, which holds the location of the regions in the cluster. ZooKeeper stores the location of the .META. table.

HBase Read

This is what happens the first time a client reads or writes to HBase:

The client gets the Region server that hosts the META table from ZooKeeper.

The client will query the .META. server to get the region server corresponding to the row key it wants to access. The client caches this information along with the .META. table location.

It will get the Row from the corresponding Region Server.

For future reads, the client uses the cache to retrieve the META location and previously read row keys. Over time, it does not need to query the META table, unless there is a miss because a region has moved; then it will re-query and update the cache.

HBase Meta Table

This META table is an HBase table that keeps a list of all regions in the system.

The .META. table is like a b tree.

The META table structure is as follows:

- Key: region start key,region id

- Values: RegionServer

Region Server Components

A Region Server runs on an HDFS data node and has the following components:

WAL: Write Ahead Log is a file on the distributed file system. The WAL is used to store new data that hasn't yet been persisted to permanent storage; it is used for recovery in the case of failure.

BlockCache: is the read cache. It stores frequently read data in memory. Least Recently Used data is evicted when full.

MemStore: is the write cache. It stores new data which has not yet been written to disk. It is sorted by key before writing to disk. There is one MemStore per column family per region.

Hfiles store the rows as sorted KeyValues on disk.

HBase Write

When the client issues a Put request

write the data to the write-ahead log, Edits are appended to the end of the WAL file that is stored on disk

Then it is placed in the MemStore and the put request acknowledgement returns to the client.

Region Flush

When the MemStore accumulates enough data, the entire sorted set is written to a new HFile in HDFS. This is a sequential write. It is very fast, as it avoids moving the disk drive head. HBase uses multiple HFiles per column family, which contain the actual cells, or KeyValue instances. These files are created over time as KeyValue edits sorted in the MemStores are flushed as files to disk.

Note that this is one reason why there is a limit to the number of column families in HBase. There is one MemStore per CF per region; when one is full, they all flush. It also saves the last written sequence number so the system knows what was persisted so far.

The highest sequence number is stored as a meta field in each HFile, to reflect where persisting has ended and where to continue. On region startup, the sequence number is read, and the highest is used as the sequence number for new edits.

HFile Structure

An HFile contains a multi-layered index which allows HBase to seek to the data without having to read the whole file. The multi-level index is like a b+tree:

Key value pairs are stored in increasing order

Indexes point by row key to the key value data in 64KB “blocks”

Each block has its own leaf-index

The last key of each block is put in the intermediate index

The root index points to the intermediate index

The trailer points to the meta blocks, and is written at the end of persisting the data to the file. The trailer also has information like bloom filters and time range info. Bloom filters help to skip files that do not contain a certain row key. The time range info is useful for skipping the file if it is not in the time range the read is looking for.

BloomFilters

A space-efficient Probabilistic Data Structure to test whether a StoreFile contains a specific row or row-col cell.

An empty Bloom filter is a bit array of m bits (m-array), all set to 0.

K independent hash functions , h0, h1, h2 … hk-1

Each element inserted will go through hash functions defined and the corresponding bit in m-array will be set to 1.

For example:

Let's say we want to insert an element x.

h0(x) = 1

h1(x) = 5

h2(x) = 13

Then

The bits corresponding to 1, 5, 13 will be set to 1.

Smilarly for other elements like y & z.

Suppose, we want to know the element w is present in our store or not. To know that, we derive has functions of the element w. We found that, result of two has functions point to the memory location of m-array having bit value 1, but the third hash function point to 0. That means the element w is not present in our store.

A query on bloom filter returns either the element is "possibly in set" or "definitely not in set.

Enabled on a Column Family, valid values :NONE, ROW,ROWCOL hbase> create 'mytable',{NAME => 'colfam1', BLOOMFILTER => 'ROWCOL'}

Alternatively, by using the setBloomFilterType method of HColumnDescriptor or using the HBase API.

More on Bloom filters here

HFile Index

HBase Read Merge

We have seen that the KeyValue cells corresponding to one row can be in multiple places, row cells already persisted are in Hfiles, recently updated cells are in the MemStore, and recently read cells are in the Block cache. So when you read a row, how does the system get the corresponding cells to return?

As discussed earlier, there may be many HFiles per MemStore, which means for a read, multiple files may have to be examined, which can affect the performance. This is called read amplification.

Minor compaction reduces the number of storage files by rewriting smaller files into fewer but larger ones, performing a merge sort.

Major compaction merges and rewrites all the HFiles in a region to one HFile per column family, and in the process, drops deleted or expired cells. This improves read performance; however, since major compaction rewrites all of the files, lots of disk I/O and network traffic might occur during the process. This is called write amplification.

Major compactions can be scheduled to run automatically. Due to write amplification, major compactions are usually scheduled for weekends or evenings.

Regions

Initially there is one region per table. When a region grows too large, it splits into two child regions.

Then the split is reported to the HMaster. For load balancing reasons, the HMaster may schedule for new regions to be moved off to other servers.

All writes and Reads are to/from the primary node. HDFS replicates the WAL and HFile blocks. HFile block replication happens automatically. HBase relies on HDFS to provide the data safety as it stores its files

So how does HBase recover the MemStore updates not persisted to HFiles?

Problems in HBase

WAL replay slow

Slow complex crash recovery

Major Compaction I/O storms

Storage Structure

A namespace is a logical grouping of tables analogous to a database in relation database systems. A namespace can be created, removed or altered.

#Create a namespace create_namespace 'my_ns' #create my_table in my_ns namespace create 'my_ns:my_table', 'fam' #drop namespace drop_namespace 'my_ns' #alter namespace alter_namespace 'my_ns', {METHOD => 'set', 'PROPERTY_NAME' => 'PROPERTY_VALUE'}

There are two predefined special namespaces:

hbase - system namespace, used to contain HBase internal tables

default - tables with no explicit specified namespace will automatically fall into this namespace

#namespace=foo and table qualifier=bar

create 'foo:bar', 'fam'

#namespace=default and table qualifier=bar

create 'bar', 'fam'

Basic Commands

$ ./bin/hbase shell hbase(main):001:0> create 'test', 'cf' hbase(main):002:0> list 'test' hbase(main):003:0> put 'test', 'row1', 'cf:a', 'value1' hbase(main):004:0> put 'test', 'row2', 'cf:b', 'value2' hbase(main):005:0> put 'test', 'row3', 'cf:c', 'value3' hbase(main):006:0> scan 'test' ROW COLUMN+CELL row1 column=cf:a, timestamp=1421762485768, value=value1 row2 column=cf:b, timestamp=1421762491785, value=value2 row3 column=cf:c, timestamp=1421762496210, value=value3 3 row(s) in 0.0230 seconds hbase(main):007:0> get 'test', 'row1' hbase(main):008:0> disable 'test' hbase(main):011:0> drop 'test' hbase(main):011:0> quit

Daemons

$ jps 15930 HRegionServer 16194 Jps 15838 HQuorumPeer 16010 HMaster

The HQuorumPeer process is a ZooKeeper instance which is controlled and started by HBase. If you use ZooKeeper this way, it is limited to one instance per cluster node, , and is appropriate for testing only. If ZooKeeper is run outside of HBase, the process is called QuorumPeer

The HMaster server controls the HBase cluster. You can start up to 9 backup HMaster servers, which makes 10 total HMasters, counting the primary.

The HRegionServer manages the data in its StoreFiles as directed by the HMaster. Generally, one HRegionServer runs per node in the cluster.You can run 99 additional RegionServers that are not a HMaster or backup HMaster, on a server

HBase has two run modes: standalone and distributed

In standalone mode (default), HBase does not use HDFS — it uses the local filesystem instead — and it runs all HBase daemons and a local ZooKeeper all up in the same JVM.

Distributed mode can be subdivided into Pseudo distributed (all daemons runs in a single node, can run against the local filesystem or it can run against an instance of HDFS) and Fully distributed (daemons are spread across all nodes in the cluster)

Pseudo-distributed mode means that HBase still runs completely on a single host, but each HBase daemon (HMaster, HRegionServer, and ZooKeeper) runs as a separate process.

Distributed Cluster Demo Architecture:

Node Name Master ZooKeeper RegionServer node-a.example.com yes yes no node-b.example.com backup yes yes node-c.example.com no yes yes

Keep Configuration In Sync Across the Cluster

When running in distributed mode, after you make an edit to an HBase configuration, make sure you copy the content of the conf/ directory to all nodes of the cluster. HBase will not do this for you.

Additional Points

Limits on Number of Files and Processes (ulimit -n) : 1024, can be raised more, like 10240

Each ColumnFamily has at least one StoreFile, and possibly more than six StoreFiles(HFiles) if the region is under load.

Potential Number of Open Files = (StoreFiles per ColumnFamily) x (regions per RegionServer)

HBase uses WAL to recover the memstore data that has not been flushed to disk in case of an RS failure. These WAL files should be configured to be slightly smaller than HDFS block (by default a HDFS block is 64Mb and a WAL file is ~60Mb).

HBase also has a limit on the number of WAL files, designed to ensure there’s never too much data that needs to be replayed during recovery.For example, with 16Gb RS heap, default memstore settings (0.4), and default WAL file size (~60Mb), 16Gb*0.4/60, the starting point for WAL file count is ~109.

A row in HBase consists of a row key and one or more columns with values associated with them. Rows are sorted alphabetically by the row key as they are stored. For this reason, the design of the row key is very important. The goal is to store data in such a way that related rows are near each other.

A new column qualifier (column_family:column_qualifier) can be added to an existing column family at any time.

HBase does not modify data in place, and so deletes are handled by creating new markers called tombstones. These tombstones, along with the dead values, are cleaned up on major compactions.

[jumbojrp-blog]

Building Test Framework for Restful Web Application

To be discussed

Configuring Integrated Test case for Restful Application using Grizzly2

Configuring Integrated Test case for servlet based Restful Web Application using External Test Container

Running IntegratedTest cases using Maven

Running Unit Test case using Mockito and Maven

Technology used

Jersey 2.8

Spring 4.0

Maven 2

Junit 4

Mockito

Tomcat

Let's start from basic a bit...

To create a maven-jersey2-spring4- REST application

mvn archetype:generate -DarchetypeArtifactId=jersey-quickstart-grizzly2 -DarchetypeGroupId=org.glassfish.jersey.archetypes -DinteractiveMode=false -DgroupId=com.sample -DartifactId=jersey2-spring4-service -Dpackage=com.sample -DarchetypeVersion=2.23.2

To create a maven-jersey2-spring4- REST WEB application

mvn archetype:generate -DarchetypeArtifactId=jersey-quickstart-webapp -DarchetypeGroupId=org.glassfish.jersey.archetypes -DinteractiveMode=false -DgroupId=com.example -DartifactId=webapp-jersey2-spring4-service -Dpackage=com.sample -DarchetypeVersion=2.23.2

Difference between Jersey1.x and Jersey2.x

Jersey1.x has package com.sun.jersey and Jersey2.x has package org.glassfish.jersey, because Jersey team is now part of another organization (Glassfish). There is a long list of difference mentioned here

for sample Jersey 1.x application, check this git hub repository

How Jersey Works?

Jersey framework basically uses ServletContainer servlet to intercept all the incoming requests. As we configure in our projects web.xml, that all the incoming rest request should be handled by that servlet. There is an init-param that is configured with the jersey servlet to find your REST service classes. REST service classes are not Servlet and they need NOT to extend the HttpServlet. These REST service classes are simple POJOs annotated to tell the jersey framework about different properties such as path, consumes, produces etc. When you return from your service method, jersey takes care of marshalling those objects in the defined 'PRODUCES' responseType and write it on the client stream. Here is a sample of jersey config in web.xml

Jersey 1.x

<servlet> <servlet-name>Jersey 1.x REST Tutorials</servlet-name> <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class> <init-param> <param-name>com.sun.jersey.config.property.packages</param-name> <param-value>your application package </param-value> </init-param> <load-on-startup>1</load-on-startup> </servlet>

Jersey 2.x

<servlet> <servlet-name>Jersey REST Tutorials</servlet-name> <servlet-class>org.glassfish.jersey.servlet.ServletContainer</servlet-class> <init-param> <param-name>jersey.config.server.provider.packages</param-name> <param-value>your application package</param-value> </init-param> <load-on-startup>1</load-on-startup> </servlet>

This is one good image explaining Servlet invocation in web application.

From the above picture, we understand all request/response are handled through servlet container. Jersey Test framework provides similar container (grizzly, in-memory, jdk, simple, jetty, external) for integration testing. Jersey Test also provides pre-configured client to access deployed application. We will use these concepts building our test framework for Restful web Application.

Configuring Integrated Test case for Restful Web Application using Grizzly2

Jersey provides 2 different test container factories based on Grizzly. The GrizzlyTestContainerFactory creates a container that can run as a light-weight, plain HTTP container. Almost all Jersey tests are using Grizzly HTTP test container factory. Second factory is GrizzlyWebTestContainerFactory that is Servlet-based and supports Servlet deployment context for tested applications. This factory can be useful when testing more complex Servlet-based application deployments.

<dependency> <groupId>org.glassfish.jersey.test-framework.providers</groupId> <artifactId>jersey-test-framework-provider-grizzly2</artifactId> <version>2.23.2</version> </dependency>

Check this github code repository for implementing a simple Restful application with jUnit Test cases using Grizzly2.

For Simple servlet based application, you have to use Grizzly2 web container, by overriding configureDeployment() and getTestContainerFactory(). The below is just a sample code to give a brief idea.

@RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(classes = AppContextInitializer.class) @WebAppConfiguration public class DataSourceWSTest extends JerseyTest { public ServletDeploymentContext configureDeployment() { return ServletDeploymentContext .forServlet(new ServletContainer(new ResourceConfig(MainController.class))).build(); } @Override protected TestContainerFactory getTestContainerFactory() { return new GrizzlyWebTestContainerFactory(); } @Override protected URI getBaseUri() { return UriBuilder.fromUri(super.getBaseUri()).path("applicationName").build(); }

Configuring Integrated Test Case for Servlet Based Restful Web Application

For complex servlet based web application, Jersey Test framework provided External Container support.

To use this, add below in your pom.xml

<dependency> <groupId>org.glassfish.jersey.test-framework.providers</groupId> <artifactId>jersey-test-framework-provider-external</artifactId> <version>2.23.2</version> </dependency>

Here is a sample code for Integration Test case using External Container

import java.net.URI; import javax.ws.rs.core.Application; import javax.ws.rs.core.UriBuilder; import org.glassfish.jersey.test.JerseyTest; import org.glassfish.jersey.test.TestProperties; import org.glassfish.jersey.test.external.ExternalTestContainerFactory; import org.glassfish.jersey.test.spi.TestContainerFactory; public class IntegrationTestHelper extends JerseyTest { /* Generated AUTH_KEY using username and password credential using postman client for authenticated application */ protected final String AUTH_KEY = "Basic bWFwX3VzZXI6bWFwX3VzZXI="; @Override protected Application configure() { setTestProperties(); return new Application(); // dummy Application instance for the test // framework - will not be used. } @Override protected TestContainerFactory getTestContainerFactory() { return new ExternalTestContainerFactory(); } @Override protected URI getBaseUri() { return UriBuilder.fromUri(super.getBaseUri()).path("fae-mdp-web") .build(); } protected void setTestProperties() { // Find first available port to run multiple test containers in parallel // to avoid TestContainerException: java.net.BindException: Address // already in use: bind forceSet(TestProperties.CONTAINER_PORT, "8080"); enable(TestProperties.LOG_TRAFFIC); enable(TestProperties.DUMP_ENTITY); } protected void preTestCase(String testcaseName) { System.out.println("START : " + testcaseName); } protected void postTestCase(String testcaseName) { System.out.println("END : " + testcaseName); } }

Your Test class

import static org.junit.Assert.assertEquals; import javax.ws.rs.core.Response; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import org.springframework.test.context.web.WebAppConfiguration; import com.saama.mdp.test.common.IntegrationTestHelper; import com.saama.mdp.web.conf.AppContextInitializer; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(classes = AppContextInitializer.class) @WebAppConfiguration public class DataSourceWSIT extends IntegrationTestHelper { private final String DATASOURCE_URI = "api/v1/dev/datasource/cname/"; @Test public void testGetById() { preTestCase("DataSourceWSIT.testGetById"); int id = 20; Response response = target(DATASOURCE_URI + id).request() .header("Authorization", AUTH_KEY).get(); assertEquals(200, response.getStatus()); postTestCase("DataSourceWSIT.testGetById"); } @Test public void testGetDataSources() { preTestCase("DataSourceWSIT.testGetDataSources"); Response response = target(DATASOURCE_URI).request() .header("Authorization", AUTH_KEY).get(); assertEquals(200, response.getStatus()); postTestCase("DataSourceWSIT.testGetDataSources"); } }

*In @ContextConfiguration, you can give the context xml file or the class name which configures the application

*for Servlet Context initialization in external container, You must use jersey 2.8 and also, add below in your pom.xml

<dependency> <groupId>org.glassfish.jersey.bundles.repackaged</groupId> <artifactId>jersey-guava</artifactId> <version>2.8</version> </dependency>

Running Integration Test

To run intgeration test cases, there are 2 ways 1. Using eclipse (Handy, but less popular) 2. Using Command line

To run in eclipse IDE, you need to run the test case in two steps:

Step1: Run the test case on "Run on server" on tomcat or any other application server

Step2: When Step1 is completed, Run the test case as "JUnit Test"

To run the test case on command or in Jenkin, we need to use embeded application server like Jetty or Tomcat. Here I will show using Tomcat as I had a very hard time to make my application up using Jetty.

To use embedded Tomcat application server, we need to use tomcat7-maven-plugin. You can use tomcat6-maven-plugin also.

Add below in your pom.xml

<plugin> <groupId>org.apache.tomcat.maven</groupId> <artifactId>tomcat7-maven-plugin</artifactId> <version>2.2</version> <configuration> <url>http://si-vm-201:8080/manager/text</url> <username>tomcat</username> <password>passwd</password> <skip>${skipITs}</skip> </configuration> <executions> <execution> <id>start-tomcat</id> <phase>pre-integration-test</phase> <goals> <goal>run</goal> </goals> <configuration> <fork>true</fork> </configuration> </execution> <execution> <id>stop-tomcat</id> <phase>post-integration-test</phase> <goals> <goal>shutdown</goal> </goals> </execution> </executions> </plugin>

For integration test case, we need to used another plugin called "maven-failsafe-plugin"

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>2.19.1</version> <configuration> <parallel>methods</parallel> <threadCount>10</threadCount> <skipITs>${skipITs}</skipITs> </configuration> <executions> <execution> <id>integration-test</id> <goals> <goal>integration-test</goal> </goals> </execution> <execution> <id>verify</id> <goals> <goal>verify</goal> </goals> </execution> </executions> </plugin>

Now let's understand, what we have done so far...

We have added failsafe plugin for running integration test case and tomcat plugin to be used as embedded application server where our application will be deployed and test case requests will be served.

When we do,

mvn clean install

in command line,

maven runs below life cycle phases in sequence

validate initialize generate-sources process-sources generate-resources process-resources compile process-classes generate-test-sources process-test-sources generate-test-resources process-test-resources test-compile process-test-classes test prepare-package package pre-integration-test integration-test post-integration-test verify install

This shows, install phase runs all goals like test (for unit test using surefire plugin), pre-integration-test (Tomcat start), integration-test(test cases), post-integration-test(tomcat shutdown), verify (checks test cases are passed or failed), install (build successfully or not)

Here just one command is sufficient to run integration test cases, where tomcat starts, application gets deployed, integration test cases are run, tomcat stops and build gets completed.

Naming your test cases:

For Unit Test cases:

*Test*.java *Test.java *TestCase.java

For Integration Test cases:

IT*.java *IT.java *ITCase.java

Running unit test cases with Mockito This is standard jUnit test cases irrespective of Restful Application.

To mock external sources, add this in your pom.xml

<dependency> <groupId>org.mockito</groupId> <artifactId>mockito-core</artifactId> <version>1.10.19</version> </dependency>

You need to use surefire plugin for running your unit test case.

Add below in your pom.xml

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.12.4</version> <configuration> <parallel>methods</parallel> <threadCount>10</threadCount> <skipTests>${skipTests}</skipTests> <excludes> <exclude>**/*IT.java</exclude> </excludes> </configuration>

Here, I have enabled, parallel processing of unit test cases up to 10, skipped integration test cases as we are targeting only unit test case in this goal.

mvn test

This command runs unit test cases using maven surefire plugin.

This is a simple article to learn using Mockito.

Enjoy!!!

Reference

https://www.petrikainulainen.net/programming/maven/integration-testing-with-maven/

http://cupofjava.de/blog/2013/02/05/integration-tests-with-maven-and-tomcat/

https://github.com/Kalimaha/maven-spring-jersey-example

https://github.com/thrawn01/Spring-Jersey-Hibernate-DbUnit-Example

https://github.com/janusdn/jersey2-spring4-grizzly2

https://github.com/alesaudate/kickstart-springjerseyhibernate

http://stackoverflow.com/questions/27430052/jersey-how-to-mock-service/27447345#27447345

http://stackoverflow.com/questions/27590896/how-to-use-mockito-for-testing-database-connection/27596173#27596173

http://stackoverflow.com/questions/10607168/unit-testing-with-spring-and-the-jersey-test-framework

https://blog.codecentric.de/en/2012/05/writing-lightweight-rest-integration-tests-with-the-jersey-test-framework/

[jumbojrp-blog]

Hadoop Q & A

What happens when two clients try to access the same file on the HDFS?

HDFS supports exclusive writes only. When the first client contacts the “Namenode” to open the file for writing, the “Namenode” grants a lease to the client to create this file. When the second client tries to open the same file for writing, the “Namenode” will notice that the lease for the file is already granted to another client, and will reject the open request for the second client.

What is speculative execution (also called backup tasks)? What problem does it solve?

In Hadoop during Speculative Execution a certain number of duplicate tasks are launched. On different slave node, multiple copies of same map or reduce task can be executed using Speculative Execution. In simple words, if a particular drive is taking long time to complete a task, Hadoop will create a duplicate task on another disk. Disk that finish the task first are retained and disks that do not finish first are killed.

What is the Bundle object in Oozie?

A set of co-ordinator objects without any condition is known as a Bundle object. The Oozie server create a bundle object by bundle applications with some specified conditions. Like kick-off time

What is the Time-Sensitive Data Loads in Pig?

Loading data is a key part of many businesses. Data comes in from outside of the database in text. XML, CSV, or some other arbitrary file format. The data then has to be processed into different formats and loaded into a database for later querying.

What is the Relational operators in pig?

The most salient characteristic about pig Latin as a language is it's relational operations. These operations define pig Latin as a data processing language. We'll quickly go over the more straightforward operations first, to acclimate ourselves to their style and syntax.

Explain “WAL” and “Hlog” in “HBase”?

“WAL” (Write Ahead Log) is similar to the “MySQL BIN” log; it records all the changes that occur in the data. It is a standard sequence file by Hadoop and stores “HLogkeys”. These keys consist of a sequential number as well as actual data and are used to replay not yet persisted data after a server crash. So, in case of server failure, the “WAL” works as the lifeline and retrieves the lost data.

For a job in Hadoop, is it possible to change the number of mappers to be created?

No, it is not possible to change the number of mappers to be created. The number of mappers is determined by the number of input splits.

What is the Lacalytics ?

This is a big marketing and analytics platform for mobile and web apps. It's developer is Localytics, in Boston. It supports cross platform and web-based applications . Localytics supports push messaging, business analytics, and acquisition campaigns management.

[jumbojrp-blog]

Understanding Secure Shell (SSH)

Secure Shell (SSH) provides Encryption and Authentication features, which enhance security. Other two important features are

SSH Tunneling

TCP Port forwarding

For authentication, two commonly used mechanism:

Password based

Key based (recommended)

SSH protocol also has different iterations and among them the most widely used versions are

SSH version 1

SSH version 2 (recommended)

When we discuss encryption and data security, there are two types of primarily used cryptographic systems.

Public Key cryptography(or sometimes called as asymmetric cryptography)

Secret key cryptography (or sometimes called as symmetric cryptography).

Most of the modern day security system's use these two types, in multiple ways to ensure security in communication.

Asymmetric encryption has a lot of overhead involved, and is a little bit time consuming to decrypt data using it. So Asymmetric encryption is only used to share the secret key, which will be used for symmetric encryption.

In order to establish SSH connection, we need to make a secure channel between server and client. First client authenticates the server as the connection is initiated by client. Then only, a secure symmetric channel is formed between them. This secure channel is used for authenticating client, sharing keys, passwords etc.

Let's go step by step:

Server discloses its identity to the client

This identity is a rsa public key of the server and it is mostly located in /etc/ssh directory .

if the client is communicating with server for the first time, the client will get the warning like this:

Are you sure you want to continue connecting (yes/no)?

After the first connection, this host (server) identity key will be saved in a file called "known_hosts" in client machine, so that in future client will not get such warning while connecting to the server.

The above key is a host identity, not a user identity. So, any client connecting to that server will get same host key as a server identity.

Server key

This key is generated each and every hour according to default configuration, mentioned in ssh configuration file (/etc/ssh/sshd_config)

Check bytes 8 random bytes which are also called checkbytes. It is necessary for the client to send these bytes in its reply to the server, during its next reply.

Encryption

According to the list of encryption algorithms supported by the server, the client simply creates a random symmetric key and sends that symmetric key to the server. This symmetric key will be used to encrypt as well as decrypt the whole communication during this session.

The client takes an additional care while sending this session key(symmetric key for this session) to the server, by doing a double encryption. The first encryption is done by the server host key(which was shared by the server), and the second encryption is done by the server key(which will keep on changing every one hour) This double encryption increases the security, because even if somebody has the host private key of the server (/etc/ssh/ssh_host_rsa_key), he will not be able to decrypt the message, because its still encrypted by the server key, which keeps on changing on an hourly basis. Along with this double encrypted session key, the client will also send the selected algorithm from the list of supported algorithm given by the server.

Authenticating the Server

In order to authenticate the server, the client needs to be sure, that the server was able to decrypt the session key. So after sending the session key(which is double encrypted with server key and server host key), the client waits for a confirmation message from the server. The confirmation from the server must be encrypted with the symmetric session key, which the client send. This step of waiting for a confirmation message is very important for the client, because the client has no other way to verify whether the server host key send, was from the intended server. Once the client receives a confirmation from the server, both of them can now start the communication with this symmetric encryption key.

Authenticating the Client

The client authentication happens over this encrypted channel. There are multiple methods that can be used to authenticate the client

Public Key

Password

Kerberos

Rhosts

Password Authentication: (Most commonly used)

Similar to login to local user using the correct password. The password transmitted by the client to the server is encrypted through the session symmetric key(which only the server and the client knows)

Public Key Authentication: (Strongest)

Client needs to create private (id_rsa) and public (id_rsa.pub) key by doing ssh-keygen

Share public key with the server(s) by adding the content of id_rsa.pub file to the server side authorized_keys file. This authorized_key file should be present in .ssh directory of server.

Server on receiving public key authentication request , will first generate a random 256 bit string as a challenge for the client, and encrypt it with the client public key, which is inside the authorized_keys file.

The client on receiving the challenge, will decrypt it with the private key(id_rsa).

Client combine challenge string with session key (which was previously negotiated and is being used for symmetric encryption.) and will generate md5 hash value and sent to the server.

Server will generate its own md5 hash value using challenge String and session key and compare the value with the value sent by the client. If it matches, client authentication succeeds.

Password-less Authentication

When we do ssh to remote server from client, it asks password of remote server machine. if this is the machine, you want to access often, you can configure ssh so that it won't ask password every time.

To do that, you just need to run only one command i.e. ssh-copy-id

ssh-copy-id appends the keys to the remote-host’s .ssh/authorized_key.

ssh-copy-id -i ~/.ssh/id_rsa.pub user@remotehost

This command asks for remote machine's password once.

After the above step, login to remote machine like below

ssh user@remotehost

It won't ask password any more.

SSH protocol version 2

SSH protocol version 2 is the default protocol used these days. Difference between SSH1 and SSH2

Diffie-Hellman key is used instead of the server key for sharing the session key in version 2 protocol

No Rhosts support in ssh 2

SSH protocol version 1 only allows negotiation of the symmetric encryption algorithm, all other things are hard corded(mac, compression etc)

SSH 2 supports certificates for public keys used

SSH 2 server can dictate the client to use multiple authentication methods in a single session to succeed. However ssh version 1 only supports one method per session

SSH version 2 allows the change of session key periodically.

[jumbojrp-blog]

Subversion on Command Line

Subversion is an open source version control system. Founded in 2000 by CollabNet, Inc. Subversion is developed as a project of the Apache Software Foundation (http://subversion.apache.org/)

To install Subversion (aka SVN), follow this link (http://subversion.apache.org/packages.html)

To work with SVN, one usually have 3 options

Plugin on IDE ( Ex- Subversive plugin on Eclipse IDE)

GUI tool ( Ex - Tortoise SVN on windows)

Command Line (CLI) - A great solution while working on server that doesn't have a GUI

In this post we will discuss about using SVN on Command Line.

Revision - SVN refers to different versions of repository as "Revisions" and each one has a number. Unlike in Git, where revision is a changeset, in SVN it refers to a snapshot. Changeset in SVN is the difference between two revisions.

Checkout - Create a working copy from the repository in the local directory

svn checkout http://svn.example.com/projects/project1 myproject

Update - Bring new changes from the repository into the working copy in the current directory

svn update

Status - To check the current status of working copy , showing files that are modified in the current directory or below it.

svn status

codes to explain the status:

Add - To add new file

svn add abc.txt

Revert - To remove local, uncommitted changes to a file

svn reverse pqr.txt

Commit - Sends local changes to the repository

svn commit -m "commiting my files"

All M & A status code files of current and below directory, will be committed. To commit specific files:

svn commit -m "committing specific file" abc.txt

Resolve - After "svn update", the command "svn status" shows the conflicted files with status code " C". These files need to be resolved before committing. The conflicted files look like this:

Resolve the conflict, remove ASCII art notations, when you are ready, run

svn resolve abc.txt

Then do, svn commit

Blame - Shows which user last changed which line in which version

svn blame abc.txt

Diff - Shows the difference between two revisions

svn diff svn diff http://svn.example.com/projects/stuff/trunk http://svn.example.com/projects/stuff/branches/branch1 svn diff -r 19:22

Log - Shows the story of changes that have been made.

svn log

Copy - To safely copy files and directories to a new location. This also copies the history of files/directories. The .svn folder gets updated with information.

svn copy fileA fileB

Remove 0r Delete - It removes the file from the directory. File gets status "D". The file will be deleted in next commit.

svn remove abc.txt svn delete abc.txt

Rename - Allows safe renaming of files. Old file gets status "D" and file with new name gets satus "A". The file gets committed in next commit.

svn move abc.txtx better_name.txt

Export - It gives you the files from your repository project, but without .svn folder.

svn export http://svn.example.com/projects/project1 myproject

Info - To give information about current working copy

svn info

Relocate - Allows different remote location to be associated with an existing working copy

svn relocate http://svn.example.com/projects http://example-svn.com/projects

Each project folder contains 3 subdirectories

Trunk - Main version of project

Branches - separate versions of the code, often used to work on a specific release or feature, allowing many commits over time and collaboration before the feature or branch is ready.

Tags - also copies, but these are not edited after they are created, Like label. To know which version of the code is currently live

Creating a new branch from the trunk of a project:

svn copy http://svn.example.com/projects/project1/trunk http://svn.example.com/projects/project1/branches/new-feature

Similarly tag can be created

To set a property to ignore files:

svn propset svn:ignore ‘*.pyc’ ./src/

This example adds an “ignore” rule to the src directory that stops files ending in .pyc from showing as new in svn status. This is very useful for generated content, cache files, or other items that don’t belong under source control. It’s also possible to supply multiple patterns, and to do so by supplying a file that holds these. The general format of the command is:

svn propset [property name] [value] [target]

to see which properties are set on a particular file or directory:

svn proplist index.php

To check the current values of any properties that are set, the svn propget command can be used:

svn propget svn:ignore src

If a property is set but should be removed, the svn propdel command will do so:

svn propdel svn:ignore greeting.php

Admin Features

to instantiate a new, empty Subversion repository

svnadmin create /path/to/new/repo

To safely duplicate a repository to a different location on the file system

svnadmin hotcopy /path/to/repo /path/to/copy/to

To export your repository

svnadmin dump /path/to/repo > repo.svndump

To import your repository

svnadmin load /path/to/repo

[jumbojrp-blog]

Learning Git

Git is a Distributed Version Controlling System (DVCS). it believes work-local, commit-often, publish-when-you-can methodology. It is primarily written in C.

To check the version :

git --version

To install Git, follow this url. it's easy. Configure your identity:

git config --global user.name “xyz” git config --global user.email “[email protected]”

These commands store your preferences in a file named .gitconfig inside your home directory (~ on UNIX and Mac, and %USERPROFILE% on Windows).

Github - github.com makes storing Git repositories and sharing code simple and easy.

To create a repository or repo :

github

Go to github.com and signup/signin

Click on "New Repository"

Give your repo name. No need to check "Initialize this repository with a README". Click on "Create repository"

Locally From a command prompt, change directories to an empty folder or an existing project directory that you want to put under version control. Then, initialize the directory as a Git repository by typing the following commands:

git init git commit -m "first commit" git remote add origin https://github.com/username/reponame.git git push -u origin master

If you want to fetch a Github repository to your local :

Go to repository page and click on "Clone or Download"

Copy the repo url

In your local, run this command:

git clone <repo-url>

You are ready to make changes now.

Overal Flow

Let's know a bit more about the flow ...

So you can see, there are FOUR areas - workspace (work area), index (stage area), local repository (committed area), remote repository (github repo)

Adding or Staging

When new or modified files are ready in your workspace for the next commit, they must first be staged with add command to move the files from workspace to stage or index area.

git add index.html =======> by filename git add javascript/ ======> by folder name git add *.js =============> wildcard pattern

More operations:

Moving

git mv abc.txt xyzDir

Removing

git rm removeFile.txt

Aborting If you want to abort all current, uncommitted changes and restore all files to the last committed state

git reset --hard ========> recursively discards all currently uncommitted (unstaged or uncommitted staged) changes

To abort not all, but a few items

git checkout modifiedfile.txt

To know the status of your local repository

git status =======> Very useful command

Committing

A commit command transactionally saves the pending additions to the local repository.

git commit -m "message for this commit"

To revert all committed changes

git checkout HEAD

Diff

A diff compares the patches between any two points in history or changes.

git diff =============> compares (uncommitted or last committed) and unstaged files git diff --staged =====> compares last committed and staged files git diff HEAD =========> compares last committed and modified files in workspace

Log

A full list of changes since the initialization of the repository, even when disconnected from all networks

git log git log -3 ==================> Last 3 commits git log --since=yesterday ===> all commits since yesterday

Blame

When one is trying to discover why and when a certain line was added

git blame <filename>

Branching

Git branch is very handy when developer wants to do a quick experiment that may be discarded or merged onto a well known branch depending on their success.

git branch <new branch name><from branch> git branch <new branch name>

Choosing a branch:

git checkout <branchname>

Alternatively, the above two steps can be performed in one command i.e.

git checkout -b <new branch name><from branch>

To list all local and remote branches:

git branch -a

Remote branches are prefixed by “remotes” and are shown in red.

To delete a branch

git checkout -d <branch-name>

Merging

Git allows you to merge one or more branches into the current branch.

git merge <branch one> git merge <branch one><branch two>

If any conflicts are encountered, with Git, a notification message is displayed and the files are internally marked with >>>>>>>>> and and git rebase

Confused between "git rebase" and "git merge"?

Both of these commands are designed to integrate changes from one branch into another branch—they just do it in very different ways. How?

Situation is like this, you are working on "feature" branch and in the mean time, some more commits happened in "master" branch and you need those changes for your work. Shown as below:

You have two options:

merge

rebase

Merge option

Merge master branch into the feature branch using below commands

git checkout feature git merge master

or,

git merge master feature

This creates a new "merge commit" in the feature branch, shown below

Merging is nice because it’s a non-destructive operation. The existing branches are not changed in any way. This avoids all of the potential pitfalls of rebasing (discussed below). Only problem is , If master is very active, this can pollute your feature branch’s history quite a bit.

Rebase option You can rebase the feature branch onto master branch using the following commands:

git checkout feature git rebase master

This moves the entire feature branch to begin on the tip of the master branch, effectively incorporating all of the new commits in master. But, instead of using a merge commit, rebasing re-writes the project history by creating brand new commits for each commit in the original branch, as shown below:

Though you get cleaner project history, you can’t see when upstream changes were incorporated into the feature. So, before you run git rebase, always ask yourself, “Is anyone else looking at this branch?” If the answer is yes, take your hands off the keyboard and start thinking about a non-destructive way to make your changes (e.g., the git revert command). Otherwise, you’re safe to re-write history as much as you like.

Commit hashes & Shorthand

Git marks each commit with a SHA–1 hash that is unique to the user committing the changes. A full SHA–1 hash is 40 hex characters, such as 64de179becc3ed324daab72f7238df1404723672. To efficiently navigate the history of hashes, several symbolic shorthand notations can be used as listed in the table below:

git diff HEAD~3

Advance, but Useful Commands

Stash (Save your work before doing another work)

You are in the mid of your work and your changes are not ready for commit yet. You want to move to last commit to fix a bug. The solution is :

Keep Calm and use git stash

Using stash on modified files places all uncommitted changes onto a stack

git stash

When you are ready to write the stashed changes back into the working copy of the files, simply pop them back off the stack.

git stash pop

Stashing will only set aside modified, tracked files. New, untracked files will remain untouched and in the working directory.

Tagging Git provides tagging to mark a specific commit in your timeline of changes, and serve as a useful identifier in history.

git tag <tag name>

To list all tags

git tag

Best Practice is:

"Branch out, Merge often and keep always in sync"

Working with remote repository

Unlike SVN's central repository model, Git follows "Collaboration model", which gives every developer their own copy of the repository. For example, the following diagram shows two remote connections from your repo into the central repo and another developer’s repo. Instead of referencing them by their full URLs, you can pass the origin and john shortcuts to other Git commands.

To get full address of your configured remote:

git remote -v

To add new remote :

git remote add <name><url>

To remove the connection to the remote repository

git remote rm <name>

To rename a remote connection from old name to new name

git remote rename <old-name><new-name>

Secrets of "origin"

When you clone a repository with git clone, it automatically creates a remote connection called origin pointing back to the cloned repository. This is useful for developers creating a local copy of a central repository, since it provides an easy way to pull upstream changes or publish local commits. This behavior is also why most Git-based projects call their central repository origin.

Confused between "origin", "master", "origin/master"? hmm..

origin : Central or remote repository name

master: A local branch

origin/master : A remote branch (which is a local copy of the branch named "master" on the remote named "origin")

Push

The push command performs a publishing action, and sends the Git commit history of one or more branches to an upstream Git repository.

git push <remote name><branch name>

Fetch

It imports commits from a remote repository into your local repo. The resulting commits are stored as remote branches instead of the normal local branches that we’ve been working with. This gives you a chance to review changes before integrating them into your copy of the project.

git fetch <remote> git fetch <remote><branch> ==========> Fetch a specified branch

Then you can use

git checkout master git merge orgin/master

The origin/master and master branches now point to the same commit, and you are synchronized with the upstream developments.

Pull (fetch + merge)

Fetch the specified remote’s copy of the current branch and immediately merge it into the local copy

git pull <remote>

How to fork a project or contribute to Open source project?

To contribute to other's project, we need follow two steps

Fork the repository (go to git hub repository and click on "fork"

After the changes done, make a Pull request

This is best described here and a MUST read (Don't miss it)

For GUIS

git gui

or

gitk --all

When Git flirts with SVN

To retrieve a local working copy of a Subversion repository that uses the traditional structure (trunk, tags, branches) and convert it into a Git repository,

git svn clone --stdlayout <svn-repo-url>

To commit to SVN repository

git svn dcommit

To make Git repository up-to-date

git svn rebase

When SVN flirts with Git

To perform a traditional SVN checkout against a repository on GitHub

svn checkout https://github.com/<username>/<project>

More sources:

http://rogerdudler.github.io/git-guide/

https://www.atlassian.com/git/tutorials/what-is-version-control