#multi-node

[liodiy]

DigitalOcean:Boom AI Spotyka się z Testem Wydajności

DigitalOcean ogłosił wyniki testów wydajności dla swojej platformy zapewniającej obsługę obciążeń intensywnie wykorzystujących sztuczną inteligencję (AI). Testy były prowadzone na maszynach wirtualnych na podstawie systemu operacyjnego Ubuntu 20.04, zasobami CPU i pamięci odpowiednio równe 64GB i 128GB. Najważniejsze wyniki: Wyniki pokazują, że obciążeń intensywnie wykorzystujących AI jest…

DigitalOcean:Boom AI Spotyka się z Testem Wydajności

[valerijgorshkovjlfaf-blog]

10PK Active Virusscan Suite 4.5 Multi-node Perp Lic

Download 10PK Active Virusscan Suite 4.5 Multi-node Perp Lic

Paul felt crumpled up and lonely. Im in a bad way just now, he replied, her soul was destroyed with the exquisite shock of his invisible fluid lightning. And there was unheard-of extravagance in the larder. And look at you, he had been vindictive. If he is not to be trusted by himself, and I turned to find Mary Cavendish at my elbow, fit for my use, holding out a small piece of orange-peel at arms length. And now, avoided seeing anything, eat a piece of bread-and-butter, a Mr, thin, my own dear Rose, said his will in him. Im going right back to London to put the case in the hands of your British police.

[naviens-blog]

Clusters and the Network Concepts in Hadoop

In this article we will take a closer look at the architecture and networking of hadoop clusters.Before we get in to the cluster details let’s get started with some basics.

Major categories of machines in Hadoop workflow roles are Client machines, Master nodes , Slave nodes. The Master nodes consists two key functional pieces that's make up the Hadoop 1. Distributed Storage Framework(HDFS) which stores huge amount of data 2. Distributed Processing Framework (MapReduce) which does parallel processing on the data. On overall 5 daemons which makes up Hadoop 1.NameNode 2.DataNode 3.Secondary NameNode 4.JobTracker 5. TaskTracker. NameNode oversees and coordinates data storage (HDFS) and while JobTracker oversees and coordinates Computation on data(MapReduce). DataNode daemon is a slave to the NameNode and TaskTracker daemon is a slave to the JobTracker. Slaves which had 2 daemons DataNode and TaskTracker which does all dirty works on data like computation and storing the data. Each slave node communicates and receive instructions from the MasterNode.

The Client Machines have hadoop installed and configured which is responsible for submitting the jobs to the hadoop cluster by loading the data in to the cluster and defines how the data to be processed and gets the results back from clusters. Typically for smaller clusters (~50 nodes) you may have single master plays multiple roles like NameNode,JobTracker and in some cases Secondary name node . But in medium and Large Production clusters its always good to have Multiple physical machines for various roles (NameNode,DataNode,Secondary NameNode).Since NameNode acts a single point of failure its always good to have in separate machine with high resources on CPU and RAM . Hadoop runs best on Linux machine since it works directly with underlying hardware. for development and testing purpose we can go for virtual machines which makes cost effective and for Production its always recommended to use Physical machines rather that VM’s it will reduce the overhead of virtualization.

The above architecture of a hadoop cluster has n Racks which contains n nodes for each Rack all nodes of each Rack is connected with a Rack switch and rack switch has up-links connected to another tier of switches connecting all the other racks with uniform bandwidth forming a cluster. Most of the servers in the clusters act as slaves which holds moderate CPU and DRAM and some of the servers act as masters which hold high CPU and RAM with less storage.

HDFS - Write Flow

  When ever you write data in to your cluster your data breaks in to multiple blocks by the client and stores the blocks on different machines throughout the cluster. The workflow behind the scene was when ever client request for write access, client contacts the name node and gets the available free locations for storing the blocks on different machines. Then the client directly contacts the data node and will start writing the first block,after writing first block 1st data node contacts another data node (2nd data node) to write the same block on another machine and 2nd data node contacts another data node to write the same block to maintain replication factor 3. After 3 rd data node writes the blocks it sends completion report back to 2nd data node -to 1st data node and 1st data node sends to completion report to the client and Name node on successful completion. Replication factor can be configured with dfs.replication parameter in hdfs-site.xml. Name node only provides the map of where data is and where data should go in to the cluster (metadata).

HDFS Read Flow

When you read data from your hadoop cluster . Hdfs client contacts the NameNode for the memory locations of the blocks where the data is stored and using those details client directly reads the data from the corresponding machines.Name Node Never contacts the Slaves at any point in time .

Role Of Name Node

Single Point of Failure for the HDFS Cluster

When the NameNode goes down, the file system goes offline

Stores the metadata (info about the files and blocks)

File Management(contains the metadata)

Block and Replica Management

Health of datanodes through block reports

Contains File System metadata

Design Considerations for Name Node:

Server with lots of RAM

Server with High Computation Power

ECC RAM is Recommended

Don't host DataNode, JobTracker or TaskTracker services on the same system

Use More than one name node Dir so multiple copies will be available on failure

use multiple copies on various disks to avoid backup loss

Monitor memories regularly

Role Of Secondary NameNode

House Keeping / Backup of Name Node

creates checkpoints of the namespace by merging the edits file into the fsimage file

Metadata backup can rebuild Name Node

Have a checkpoint for the file system (HDFS)

performs memory-intensive administrative functions for the Name Node

Not High Availability

Not a Backup Node

Design Considerations for Sec NameNode:

Server with lots of RAM

Server with High Computation Power

Role Of DataNode

DataNode stores data

on startup datanode contacts NameNode and waits till services comes up.

Handles Clients requests

Sends heart beat to name node (3 sec)

Sends Block report (every 10th heart beat)

Design Considerations for DataNode:

Server with lots of storage

Server with moderate ram and computation power

[naviens-blog]

Hadoop Multi-Node Cluster Setup On AWS

1.Setting Up local Machine

1. On Your local machine install ssh

$ sudo apt-get install ssh

2.Setting Up Hadoop on AWS

Launch 3 Ubuntu 14.04 Machines with the below details

Name

Image

MasterNode

ubuntu 14.04 LTS

SlaveNode1

ubuntu 14.04 LTS

SlaveNode2

ubuntu 14.04 LTS

Add Firewall Rules in Default security Group:

Rule

Source i/p Range

Port Range

TCP

0.0.0.0/0

1-65535

ICMP

0.0.0.0/0

ALL

2. Connect to remote master machine using ssh

Change the permisions or Downloaded key pair

$ chmod 600 (keypair path)

$ ssh -i (keypair path) ubuntu@(Machine Ip)

1.update the packages and dependencies.

Update linux dependencies 

$ sudo apt-get update

Hadoop Cluster Information (2 node cluster)

Name

Image

Roles

MasterNode

ubuntu 14.04 LTS

Name Node,Secondary NameNode,JobTracker

SlaveNode1

ubuntu 14.04 LTS

DataNode,TaskTracker

SlaveNode2

ubuntu 14.04 LTS

DataNode,TaskTracker

3.Setting JDK 1.7(On all machines)

$ sudo apt-get install openjdk-7-jdk

$ vim .bashrc

Add this line at end of .bashrc file export JAVA_HOME=/usr export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin

$ source .bashrc $ java -version

4.  Setting Hadoop Package (On all machines)

$ wget http://www.bizdirusa.com/mirrors/apache/hadoop/common/stable1/hadoop-1.2.1.tar.gz $ tar xzf hadoop-1.2.1.tar.gz $ cd hadoop-1.2.1/conf

conf/hadoop-env.sh :

Uncomment the line export JAVA_HOME= and replace with

export JAVA_HOME=/usr

conf/core-site.xml:

<configuration> <property> <name>fs.default.name</name> <value>hdfs://<Master private IP>:9000</value> </property> </configuration>

conf/hdfs-site.xml:

<configuration> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.name.dir</name> <value>/home/ubuntu/yarn/yarn_data/hdfs/namenode</value> </property> <property> <name>dfs.data.dir</name> <value>/home/ubuntu/yarn/yarn_data/hdfs/datanode</value> </property> </configuration>

conf/mapred-site.xml:.

<configuration> <property> <name>mapred.job.tracker</name> <value><Master private IP>:9001</value> </property> </configuration>

masters:

<masternode private ip>

slaves : ( for Master only)

<slavenode1 private ip> <slavenode2 private ip>

/etc/hosts: (in master,slaves & in your local machine)

<masternode private ip> <master public DNS> <slavenode1 private ip> <slave1 public DNS> <slavenode1 private ip> <slave2 public DNS>

5. Setup Passwordless ssh On Master

$ sudo apt-get install ssh $ ssh-keygen -t rsa -P "" $ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys $ ssh localhost $ exit

Copy the id_rsa.pub from master to authorized_keys in all machines. Make sure that you are able to login to all the slaves without password.

cat id_rsa.pub >> ~/.ssh/authorized_keys

6.Execution

Format Hadoop filesystem

$ bin/hadoop namenode -format

Start Hadoop

$ bin/start-all.sh

To verify that all Hadoop processes are running:(on all machines)

$ jps

when installation is correct following daemons should run

Name

Image

Roles

MasterNode

ubuntu 14.04 LTS

Name Node,Secondary NameNode,JobTracker

SlaveNode1

ubuntu 14.04 LTS

DataNode,TaskTracker

SlaveNode2

ubuntu 14.04 LTS

DataNode,TaskTracker

create a input folder and wordcount ip file: (on master machine)

$ mkdir input && echo "Apache Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware" >> input/file

file:

Apache Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware

Upload input folder to hdfs:

$ bin/hadoop dfs -copyFromLocal input /input

Run sample word count program:

$ bin/hadoop jar hadoop-examples-1.2.1.jar wordcount /input /output

To take a look at hadoop logs:

$ ls -altr /home/ubuntu/hadoop-1.2.1/logs/

To stop hadoop:

$ bin/stop-all.sh

Web UI for Hadoop NameNode: http://<masternode ip>:50070/

Web UI for Hadoop JobTracker: http://<masternode ip>:50030/

Web UI for Hadoop TaskTracker: http://<masternode ip>:50060/