#hadoop operation

[jacobwill176]

This report presents a comprehensive overview, market shares and growth opportunities of Hadoop Operation Service market by product type, application, key companies and key regions.

[devoppa]

[Hadoop] Decommissioning a datanode

Adding a datanode could be done in few easy steps, but removing one takes little bit of more care.

Even hadoop keeps replications among all datanodes in the cluster, it's probably not a good idea to shutdown a datanode to remove it. It's always possible in Hadoop that some data blocks are under-replicated and only stored in one datanode. Removing that datanode causes a dataloss. So always decommission it and then remove.

Bullet steps to decommissioning a datanode (hdfs 1.0.3)

Make sure dfs.hosts.exclude is set in the hdfs-site.xml and the exclude file exists.

Make sure the target datanode (dn15) is live for load-balancing

Add the hostname or ip to the exclude file. (eg. add dn15 to ${HADOOP_HOME}/conf/datanode.exclude) (this should be done in all hosts to prevent heartbeats among datanodes)

Run "hadoop dfsadmin -refreshNodes" in master. This will initiate the decommissioning process and start re-replicating the blocks in dn15) You'll see something like this in the Namenode web

You can click on the "Decommissioning Nodes" link for more details. Decommissiong of the dn15 (600GB being used by the time of the decommission) took about 6 hours re-replicating 80,000 blocks. Yes, it takes a while. If you want to cancel the decommissioning, delete the hostname in the exclude file (datanode.exclude) and rerun the "hadoop dfsadmin -refreshNodes"

Once the decommission is complete you will see the state of the datanode being changed to "Decommission Complete."

Now you can safely remove or shutdown the node.

If you want to add back the node to the cluster, you will need to delete the hostname from the exclude file first and then start the datanode daemon.

reference: http://wiki.apache.org/hadoop/FAQ