Online Bigdata and Hadoop courses

        To provide the best training in latest cutting-edge technologies across the globe and help learners carve their career. For more information visit our website. https://data-flair.training/    

[dataflairblog-blog]

https://data-flair.training/

[dataflairblog-blog]

Funny Images

[dataflairblog-blog]

Apache Spark Installation On Ubuntu- A Beginners Tutorial

This tutorial describes the first step while learning Apache Spark i.e. Apache Spark Installation On Ubuntu. This Apache Spark tutorial is a step by step guide for Installation of Spark, the configuration of pre-requisites and launches Spark shell to perform various operations. If you are completely new to Apache Spark, I would recommend you to read these introductory blogs- What is Spark, Spark ecosystem, Spark key abstraction RDD, Spark features, and limitations of Apache Spark.

2. Steps for Apache Spark Installation On Ubuntu

Follow the steps given below for Apache Spark Installation On Ubuntu-

 Deployment Platformi. Platform Requirements

Operating System: You can use Ubuntu 14.04 or later (other Linux flavors can also be used like CentOS, Redhat, etc.)

Spark: Apache Spark 1.6.1 or later

ii. Setup Platform

If you are using Windows / Mac OS you can create a virtual machine and install Ubuntu using VMWare Player, alternatively, you can create a virtual machine and install Ubuntu using Oracle Virtual Box.

2.2. Prerequisitesi. Install Java 7

a. Install Python Software Properties

$sudo apt-get install python-software-properties

b. Add Repository

$sudo add-apt-repository ppa:webupd8team/java

c. Update the source list

$sudo apt-get update

d. Install Java

$sudo apt-get install oracle-java7-installer

2.3. Install Apache Sparki. Download Spark

You can download Apache Spark from the below link. In the package type please select “Pre-built for Hadoop 2.6 and Later” http://spark.apache.org/downloads.html

Or, you can use direct download link: http://mirror.fibergrid.in/apache/spark/spark-1.6.1/spark-1.6.1-bin-hadoop2.6.tgz

ii. Untar Spark Setup

$tar xzf spark-1.6.1-bin-hadoop2.6.tgz

You can find all the scripts and configuration files in newly created directory “spark-1.6.1-bin-hadoop2.6”

iii. Setup Configuration

a. Edit .bashrc

Edit .bashrc file located in user’s home directory and add following parameters-

export JAVA_HOME=<path-to-the-root-of-your-Java-installation> (eg: /usr/lib/jvm/java-7-oracle/)

export SPARK_HOME=<path-to-the-root-of-your-spark-installation> (eg: /home/dataflair/spark-1.6.1-bin-hadoop2.6/)

2.4. Launch the Spark Shell

Go to Spark home directory (spark-1.6.1-bin-hadoop2.6) and run below command to start Spark Shell

$bin/spark-shell.sh

Spark shell is launched, now you can play with Spark

i. Spark UI

This is the GUI for Spark Application, in local mode spark shell runs as an application. The GUI provide details about stages, storage (cached RDDs), Environment Variables and executors

http://localhost:4040

2.5. Spark Commands / Operations

Once you installed Apache Spark, you can play with spark shell to perform the various operation like transformation and action, the creation of RDDs. Follow this guide for Shell Commands to working with Spark.

For more info https://data-flair.training/blogs/apache-spark-installation-on-ubuntu/

[dataflairblog-blog]

Hadoop Interview Questions and Answers Part- 1

Hadoop Interview Questions and Answers

In this section of Hadoop Interview Questions, we will cover the Basic Hadoop Interview Questions and Answers for freshers and experienced. After this general Hadoop Interview Questions sections we will have different sections on the Hadoop Interview Questions on HDFS and MapReduce.

1) What is Apache Hadoop?

Hadoop emerged as a solution to the “Big Data” problems. It is a part of the Apache project sponsored by the Apache Software Foundation (ASF). It is an open source software framework for distributed storage and distributed processing of large datasets. Open source means it is freely available and even we can change its source code as per our requirements. Apache Hadoop makes it possible to run applications on the system with thousands of commodity hardware nodes. It’s distributed file system has the provision of rapid data transfer rates among nodes. It also allows the system to continue operating in case of node failure. Apache Hadoop provides:

Storage layer – HDFS

Batch processing engine – MapReduce

Resource Management Layer – YARN

2) Why do we need Hadoop?

The picture of Hadoop came into existence to deal with Big Data challenges. The challenges with Big Data are-

Storage – Since data is very large, so storing such huge amount of data is very difficult.

Security – Since the data is huge in size, keeping it secure is another challenge.

Analytics – In Big Data, most of the time we are unaware of the kind of data we are dealing with. So analyzing that data is even more difficult.

Data Quality – In the case of Big Data, data is very messy, inconsistent and incomplete.

Discovery – Using a powerful algorithm to find patterns and insights are very difficult.

Hadoop is an open-source software framework that supports the storage and processing of large datasets. Apache Hadoop is the best solution for storing and processing Big data because:

Apache Hadoop stores huge files as they are (raw) without specifying any schema.

High scalability – We can add any number of nodes, hence enhancing performance dramatically.

Reliable – It stores data reliably on the cluster despite machine failure.

High availability – In Hadoop data is highly available despite hardware failure. If a machine or hardware crashes, then we can access data from another path.

Economic – Hadoop runs on a cluster of commodity hardware which is not very expensive

Here is the detailed explanation of features of Hadoop

3) What are the core components of Hadoop?

Hadoop is an open-source software framework for distributed storage and processing of large datasets. Apache Hadoop core components are HDFS, MapReduce, and YARN.

HDFS- Hadoop Distributed File System (HDFS) is the primary storage system of Hadoop. HDFS store very large files running on a cluster of commodity hardware. It works on the principle of storage of less number of large files rather than the huge number of small files. HDFS stores data reliably even in the case of hardware failure. It provides high throughput access to an application by accessing in parallel.

MapReduce- MapReduce is the data processing layer of Hadoop. It writes an application that processes large structured and unstructured data stored in HDFS. MapReduce processes a huge amount of data in parallel. It does this by dividing the job (submitted job) into a set of independent tasks (sub-job). In Hadoop, MapReduce works by breaking the processing into phases: Map and Reduce. The Map is the first phase of processing, where we specify all the complex logic code. Reduce is the second phase of processing. Here we specify light-weight processing like aggregation/summation.

YARN- YARN is the processing framework in Hadoop. It provides Resource management and allows multiple data processing engines. For example real-time streaming, data science, and batch processing.

Read: Hadoop Ecosystem Components in detail.

4) What are the Features of Hadoop?

The various Features of Hadoop are:

Open Source – Apache Hadoop is an open source software framework. Open source means it is freely available and even we can change its source code as per our requirements.

Distributed processing – As HDFS stores data in a distributed manner across the cluster. MapReduce process the data in parallel on the cluster of nodes.

Fault Tolerance – Apache Hadoop is highly Fault-Tolerant. By default, each block creates 3 replicas across the cluster and we can change it as per needment. So if any node goes down, we can recover data on that node from the other node. Framework recovers failures of nodes or tasks automatically.

Reliability – It stores data reliably on the cluster despite machine failure.

High Availability – Data is highly available and accessible despite hardware failure. In Hadoop, when a machine or hardware crashes, then we can access data from another path.

Scalability – Hadoop is highly scalable, as one can add the new hardware to the nodes.

Economic- Hadoop runs on a cluster of commodity hardware which is not very expensive. We do not need any specialized machine for it.

Easy to use – No need of client to deal with distributed computing, the framework take care of all the things. So it is easy to use.

Read Hadoop Features in detail

5) Compare Hadoop and RDBMS?

Apache Hadoop is the future of the database because it stores and processes a large amount of data. Which will not be possible with the traditional database. There is some difference between Hadoop and RDBMS which are as follows:

Architecture – Traditional RDBMS have ACID properties. Whereas Hadoop is distributed computing framework having two main components: Distributed file system (HDFS) and MapReduce.

Data acceptance – RDBMS accepts only structured data. While Hadoop can accept both structured as well as unstructured data. It is a great feature of Hadoop, as we can store everything in our database and there will be no data loss.

Scalability – RDBMS is a traditional database which provides vertical scalability. So if the data increases for storing then we have to increase particular system configuration. While Hadoop provides horizontal scalability. So we just have to add one or more node to the cluster if there is any requirement for an increase in data.

OLTP (Real-time data processing) and OLAP – Traditional RDMS support OLTP (Real-time data processing). OLTP is not supported in Apache Hadoop. Apache Hadoop supports large-scale Batch Processing workloads (OLAP).

Cost – Licensed software, therefore we have to pay for the software. Whereas Hadoop is open source framework, so we don’t need to pay for software.

For more interview Question follow this link  Hadoop Interview Questions for Freshers

[dataflairblog-blog]

Python Tutorial

Python Tutorial: Objective

In this Python tutorial, we will learn about the introduction to Python programming, Python features, an overview of Python architecture, and Python applications in real industry. We will also learn about available python frameworks like Django, Flask, Pyramid etc.

The transition from consuming games and web apps to being the one behind them is exciting. Imagine if, with one technology, you could create all that? The world would be at your fingertips. Well, what if we told you that it’s possible? 1991 changed a lot of things for the programming industry and many others. We’re talking about Python programming, a friendly general-purpose programming language. And in a world of machine learning, Python is here to stay. So let us start with Python tutorial.

What is Python?

Python is an object-oriented language, which means that it can model real-world objects. It is also dynamically-typed because it carries out type-checking at runtime. It does so to make sure that the type of a construct matches what we expect in that context. The distinctive feature about Python is that it is an interpreted language. The Python IDLE (Integrated Development Environment) executes instructions a line at a time.

Etymology states that Guido van Rossum named it after the comedy group Monty Python. That is why the metasyntactic variables used here are ‘spam’ and ‘eggs’ instead of ‘foo’ and ‘bar’. Unlike as expected, it does not refer to the reptile species. A lot of implementations today run version 2.x, but the future belongs to Python 3.x. It is also called ‘Python 3000’ or ‘Py3K’. CPython, written in C, is the most common implementation of Python. It compiles a Python program into intermediate bytecode.

Apart from the constructs that Python provides, you can use the PyPI (Python Package Index). It is a repository of third-party modules, you can install it using a program called pip. Run the following command in Command Prompt:

pip install library_name

Python or R: To learn the difference between Python and R, please follow Python vs R. For now let is move ahead with the current python tutorial.

Python History

Python programming language was conceived in the late 1980s and was named for the BBC TV show Monty Python’s Flying Circus. Guido van Rossum started python implementation at CWI in the Netherlands in December 1989. This was a successor to the ABC (programming language) which was capable of exception handling and interfacing with the Amoeba operating system.

On October 16, 2000, python 2.0 release was there and it had many major new features, that includes cycle-detecting garbage collector for memory management and support for Unicode.

Next version of Python 3.0 was released on December 3, 2008.

Architecture

This is the architecture of a CPython Compiler/Interpreter.

a. Parser

It uses the source code to generate an abstract syntax tree.

b. Compiler

It turns the abstract syntax tree into Python bytecode.

c. Interpreter

It executes the code line by line.

Components of Python

Functions

A function is a collection of statements named into one. You can use it when you want to execute all those statements at a time. You can call it wherever you want in a program. A function may return a value.

Classes

As we discussed, Python is an object-oriented language. It supports classes and objects. A class is an abstract data type. In other words, it is a blueprint for an object of a certain kind. It holds no values.

Modules

A Python module is a collection of related classes and functions. We have modules for mathematical calculations, string manipulations, web programming, and many more.

Packages

A package is a collection of related modules. You can either import a package or create your own.

Now let us learn about the features of Python in this python tutorial. Based on these features you will be able to choose a programming language for your new project.

For more learn  Features of Python

[dataflairblog-blog]

MapReduce is the processing layer of Hadoop. MapReduce programming model is designed for processing large volumes of data in parallel by dividing the work into a set of independent tasks. You need to put business logic in the way MapReduce works and rest things will be taken care by the framework. Work (complete job) which is submitted by the user to master is divided into small works (tasks) and assigned to slaves.

More learn to check this

https://data-flair.training/blogs/hadoop-mapreduce-tutorial/

[dataflairblog-blog]

DataFlair provides certified training courses for Big Data and Hadoop, Apache Spark and Scala, Flink, Data Science, Kafka, Storm, Hadoop Admin, Hadoop Architect.For more information visit our website. https://data-flair.training/

[dataflairblog-blog]

[dataflairblog-blog]

[dataflairblog-blog]

Apache Spark vs. Hadoop MapReduce

Objective

Apache Spark is an open-source, lightning fast big data framework which is designed to enhance the computational speed. Hadoop MapReduce, read and write from the disk, as a result, it slows down the computation. While Spark can run on top of Hadoop and provides a better computational speed solution. This tutorial gives a thorough comparison between Apache Spark vs. Hadoop MapReduce.

In this guide, we will cover what is the difference between Spark and Hadoop MapReduce, how Spark is 100x faster than MapReduce. This comprehensive guide will provide feature wise comparison between Apache Spark and Hadoop MapReduce.

Comparison between Apache Spark vs. Hadoop MapReduce

           Introduction

Apache Spark – It is an open source big data framework. It provides faster and more general purpose data processing engine. Spark is basically designed for fast computation. It also covers a wide range of workloads for example batch, interactive, iterative and streaming.

Hadoop MapReduce – It is also an open source framework for writing applications. It also processes structured and unstructured data that are stored in HDFS. Hadoop MapReduce is designed in a way to process a large volume of data on a cluster of commodity hardware. MapReduce can process data in batch mode.

          Speed

Apache Spark – Spark is lightning fast cluster computing tool. Apache Spark runs applications up to 100x faster in memory and 10x faster on disk than Hadoop. Because of reducing the number of read/write cycle to disk and storing intermediate data in-memory Spark makes it possible.

Hadoop MapReduce – MapReduce reads and writes from disk, as a result, it slows down the processing speed.

          Difficulty

Apache Spark – Spark is easy to program as it has tons of high-level operators with RDD – Resilient Distributed Dataset.

Hadoop MapReduce – In MapReduce, developers need to hand code each and every operation which makes it very difficult to work.

           Easy to Manage

Apache Spark – Spark is capable of performing batch, interactive and Machine Learning and Streaming all in the same cluster. As a result makes it a complete data analytics engine. Thus, no need to manage different component for each need. Installing Spark on a cluster will be enough to handle all the requirements.

Hadoop MapReduce – As MapReduce only provides the batch engine. Hence, we are dependent on different engines. For example- Storm, Giraph, Impala, etc. for other requirements. So, it is very difficult to manage many components.

          Real-time analysis

Apache Spark – It can process real-time data i.e. data coming from the real-time event streams at the rate of millions of events per second, e.g. Twitter data for instance or Facebook sharing/posting. Spark’s strength is the ability to process live streams efficiently.

Hadoop MapReduce – MapReduce fails when it comes to real-time data processing as it was designed to perform batch processing on voluminous amounts of data.

          latency

Apache Spark – Spark provides low-latency computing.

Hadoop MapReduce – MapReduce is a high latency computing framework.

          Interactive mode

Apache Spark – Spark can process data interactively.

Hadoop MapReduce – MapReduce doesn’t have an interactive mode.

          Streaming

Apache Spark – Spark can process real-time data through Spark Streaming.

Hadoop MapReduce – With MapReduce, you can only process data in batch mode.

          Ease of use

Apache Spark – Spark is easier to use. Since its abstraction (RDD) enables a user to process data using high-level operators. It also provides rich APIs in Java, Scala, Python, and R.

Hadoop MapReduce – MapReduce is complex. As a result, we need to handle low-level APIs to process the data, which requires lots of hand coding.

          Recovery

Apache Spark – RDDs allows recovery of partitions on failed nodes by re-computation of the DAG while also supporting a more similar recovery style to Hadoop by way of checkpointing, to reduce the dependencies of an RDD.

Hadoop MapReduce – MapReduce is naturally resilient to system faults or failures. So, it is a highly fault-tolerant system.

[dataflairblog-blog]

Advantages and Disadvantages of Python Programming Language

 Advantages and Disadvantages of Python Programming Language

When we want to choose a language for a project, we want to be thorough with what we can do with it. We want to be aware of how it can help us be efficient at what we want to do, but we also want to be careful of the problems that can arise. So, we believe it is worthwhile to take out some time and find out more. In this advantages and disadvantages of python programming language tutorial, we will learn the advantages and disadvantages of python programming language that will help you in knowing benefits of learning Python programming.

Advantages and Disadvantages of Python Programming Language

Advantages of Python Programming Language

Let’s see why Python is cool.

Extensive Libraries

Like we mentioned in our article on Python features, it downloads with an extensive library. These contain code for various purposes like regular expressions, documentation-generation, unit-testing, web browsers, threading, databases, CGI, email, image manipulation, and more. So, we don’t have to write the complete code for that manually.

 Extensible

As we have seen earlier, Python can be extended to other languages. You can write some of your code in languages like C++ or C. This comes in handy, especially in projects.

Embeddable

Complimentary to extensibility, Python is embeddable as well. You can put your Python code in your source code of a different language, like C++. This lets us add scripting capabilities to our code in the other language.

 Improved Productivity

The language’s simplicity and extensive libraries render programmers more productive than languages like Java and C++ do. Also, the fact that you need to write less lets more get done.

IOT Opportunities

Disadvantages of Python Programming Language

So far, we’ve seen why Python is a great choice for your project. But if you must choose it, you should be aware of its consequences as well. Let’s now see the downsides of choosing Python over another language.

 Speed Limitations

We have seen that Python code is executed line by line. But since Python is interpreted, it often results in slow execution. This, however, isn’t a problem unless speed is a focal point for the project. In other words, unless high speed is a requirement, the benefits offered by Python are enough to distract us from its speed limitations.

Weak in Mobile Computing and Browsers

While it serves as an excellent server-side language, Python is much rarely seen on the client-side. Besides that, it is rarely ever used to implement smartphone-based applications. One such application is called Carbonnelle.

The reason it is not so famous despite the existence of Brython is that it isn’t that secure.

Design Restrictions

As you know, Python is dynamically-typed. This means that you don’t need to declare the type of variable while writing the code. It uses duck-typing. But wait, what’s that? Well, it just means that if it looks like a duck, it must be a duck. While this is easy on the programmers during coding, it can raise run-time errors.

In Any query regarding the advantages and disadvantages of Python programming language tutorial feel free to drop a comment.

Underdeveloped Database Access Layers

Compared to more widely used technologies like JDBC (Java DataBase Connectivity) and ODBC (Open DataBase Connectivity), Python’s database access layers are a bit underdeveloped. Consequently, it is less often applied in huge enterprises.

e. Simple

[dataflairblog-blog]

Types of jobs In python and salary

[dataflairblog-blog]

Big Data History, Technologies and Use cases

Overview

This tutorial provides you a detailed introduction to big data and big data history. We will also discuss the big data technologies like – Hadoop, Apache Spark, and Flink. Various real-life use cases of big data are also discussed in this tutorial.

 Big Data – Introduction

Day by day the big world of internet is creating 2.5 quintillion bytes of data on regular basis according to the statistics the percentage of data that has been generated from last two years is 90%. This data comes from many industries like climate information collects by the sensor, different stuff from social media sites, digital images and videos, different records of the purchase transaction. This data is big data.

Big Data History

This section of tutorial gives you a clear picture of big data history-

Research & Development in these Big Data native businesses are very close, and very close to the research and open source community.

Each paper on the cost-efficient innovative information processing techniques has been accompanied by open source adoption within an ever-growing ecosystem called Hadoop.

Two major milestones in the development of Hadoop also added confidence into the Power of open source and Big Data Technologies. Only two years after its first release, in 2008, Hadoop won the terabyte sort benchmark in big data history. This is the first time that either a Java or an open source program has won. In 2010 Facebook claimed that they had the largest Hadoop cluster in the world with 21 PB of storage for their social messaging platform.

Facts and Figures

91% of leaders belong to marketing believe successful brands use customer data to drive business decisions.

The overall percentage of the world’s total data has been created just within the past two years is 90%.

87% companies agree capturing and sharing the right data is important to effectively measure ROI in their own company.

500 million calls record daily analyzed by IBM to predict the customer’s churns.

350 billion annual meter readings converted by IBM through Big Data to better predict power consumption.

On Facebook, 30 billion pieces of content are sharing by users in each month.

Fascinated by Big Data? Enroll Now!

 Big Data Technologies

While the topic of Big Data is broad and encompasses many trends and new technology developments, the top emerging technologies are given below that are helping users cope with and handle Big Data in a cost-effective manner.

For details about  Big Data Technologies check this website

https://data-flair.training/blogs/big-data-history-use-cases/

[dataflairblog-blog]

For more information follow link

https://data-flair.training/blogs/what-is-big-data/

[dataflairblog-blog]

Apache Hadoop Article for Beginners

Objective

Hadoop article is a comprehensive guide on Big Data Hadoop that covers what is Hadoop, what is the need of Apache Hadoop, why Apache Hadoop is most popular, How Apache Hadoop works?

Apache Hadoop is an open source, Scalable, and Fault-tolerant framework written in Java. It efficiently processes large volumes of data on a cluster of commodity hardware. Hadoop is not only a storage system but is a platform for large data storage as well as processing. This Big Data Hadoop tutorial provides a thorough Hadoop introduction.

We will also learn in this Hadoop tutorial about Hadoop architecture, Hadoop daemons, different flavors of Hadoop. At last, we will cover the introduction of Hadoop components like HDFS, MapReduce, Yarn, etc.

​

What is Hadoop?

Hadoop is an open-source tool from the ASF – Apache Software Foundation. Open source project means it is freely available and we can even change its source code as per the requirements. If certain functionality does not fulfill your need then you can change it according to your need. Most of Hadoop code is written by Yahoo, IBM, Facebook, Cloudera.

It provides an efficient framework for running jobs on multiple nodes of clusters. Cluster means a group of systems connected via LAN. Apache Hadoop provides parallel processing of data as it works on multiple machines simultaneously.

By getting inspiration from Google, which has written a paper about the technologies. It is using technologies like Map-Reduce programming model as well as its file system (GFS). As Hadoop was originally written for the Nutch search engine project. When Doug Cutting and his team were working on it, very soon Hadoop became a top-level project due to its huge popularity.

Apache Hadoop is an open source framework written in Java. The basic Hadoop programming language is Java, but this does not mean you can code only in Java. You can code in C, C++, Perl, Python, ruby etc. You can code the Hadoop framework in any language but it will be more good to code in java as you will have lower level control of the code.

Big Data and Hadoop efficiently process large volumes of data on a cluster of commodity hardware. Hadoop is for processing huge volume of data. Commodity hardware is the low-end hardware, they are cheap devices which are very economical. Hence, Hadoop is very economic.

Hadoop can be set up on a single machine (pseudo-distributed mode, but it shows its real power with a cluster of machines. We can scale it to thousand nodes on the fly ie, without any downtime. Therefore, we need not make any system down to add more systems in the cluster. Follow this guide to learn Hadoop installation on a multi-node cluster.

Hadoop consists of three key parts –

Hadoop Distributed File System (HDFS) – It is the storage layer of Hadoop.

Map-Reduce – It is the data processing layer of Hadoop.

YARN – It is the resource management layer of Hadoop.

In this Hadoop Article for beginners we will all these three in detail, but first let's discuss the significance of Hadoop.

Why Hadoop?

Let us now understand in this Hadoop tutorial that why Big Data Hadoop is very popular, why Hadoop capture more than 90% of big data market.

Apache Hadoop is not only a storage system but is a platform for data storage as well as processing. It is scalable (as we can add more nodes on the fly), Fault-tolerant (Even if nodes go down, data processed by another node).

Following characteristics of Hadoop make it a unique platform:

Flexibility to store and mine any type of data whether it is structured, semi-structured or unstructured. It is not bounded by a single schema.

Excels at processing data of complex nature. Its scale-out architecture divides workloads across many nodes. Another added advantage is that its flexible file-system eliminates ETL bottlenecks.

Scales economically, as discussed it can deploy on commodity hardware. Apart from this its open-source nature guards against vendor lock.

Learn Hadoop features in detail.

What is Hadoop Architecture?

After understanding what is Apache Hadoop, let us now understand the Big Data Hadoop Architecture in detail in this Hadoop tutorial.

​

Hadoop works in master-slave fashion. There is a master node and there are n numbers of slave nodes where n can be 1000s. Master manages, maintains and monitors the slaves while slaves are the actual worker nodes. In Hadoop architecture, the Master should deploy on good configuration hardware, not just commodity hardware. As it is the centerpiece of Hadoop cluster.

Master stores the metadata (data about data) while slaves are the nodes which store the data. Distributedly data stores in the cluster. The client connects to master node to perform any task. 

Hadoop Components

There are three most important Apache Hadoop Components. In this Hadoop article, you will learn what is HDFS, what is Hadoop MapReduce and what is Yarn Hadoop. 

Hadoop HDFS or Hadoop Distributed File System

follow this link

[dataflairblog-blog]

To provide quality education at an affordable price to help everyone develop their career in latest technologies. We aim to reach the mass through our unique pedagogy model for Self-paced learning and Instructor-led learning that includes personalized guidance, lifetime course access, 24×7 support, live project, resume and interview preparation and ready to work level learning. We aim to provide real-time technical experience to learners through our expert instructors. We will strive to continuously upgrade our offerings ensuring that we can cover all technologies and courses are up-to-date with latest industry needs. For more information visit our website. https://data-flair.training/

[dataflairblog-blog]

Apache Spark Tutorial for Beginners

What is Spark? Why there is a serious buzz going on about this technology? I hope this Spark introduction tutorial will help to answer some of these questions. Apache Spark is an open-source cluster computing system that provides high-level API in Java, Scala, Python and R. It can access data from HDFS, Cassandra, HBase, Hive, Tachyon, and any Hadoop data source. And run in Standalone, YARN and Mesos cluster manager.

What is Spark tutorial will cover Spark ecosystem components, Spark video tutorial, Spark abstraction – RDD, transformation, and action in Spark RDD? The objective of this introductory guide is to provide Spark Overview in detail, its history, Spark architecture, deployment model and RDD in Spark.

What is Spark?

Apache Spark is a general-purpose & lightning fast cluster computing system. It provides high-level API. For example, Java, Scala, Python and R. apache Spark is a tool for Running Spark Applications. Spark is 100 times faster than Bigdata Hadoop and 10 times faster than accessing data from disk.

Spark is written in Scala but provides rich APIs in Scala, Java, Python, and R.

It can be integrated with Hadoop and can process existing Hadoop HDFS data. Follow this guide to learn How Spark is compatible with Hadoop?

It is saying that the images are the worth of thousand words. To keep this in mind we have also provided Spark video tutorial for more understanding of Apache Spark.

History Of Apache Spark

Apache Spark was introduced in 2009 in the UC Berkeley R&D Lab, later it becomes AMPLab. It was open sourced in 2010 under BSD license. In 2013 spark was donated to Apache Software Foundation where it became top-level Apache project in 2014.

Why Spark?

After studying Apache Spark introduction lets discuss, why Spark come into existence?

Hadoop MapReduce can only perform batch processing.

Apache Storm / S4 can only perform stream processing.

Apache Impala / Apache Tez can     only perform interactive processing

Neo4j / Apache Giraph can only     perform graph processing

Hence in the industry, there is a big demand for a powerful engine that can process the data in real-time (streaming) as well as in batch mode. There is a need for an engine that can respond in sub-second and perform in-memory processing.  

Apache Spark Definition says it is a powerful open-source engine that provides real-time stream processing, interactive processing, graph processing, in-memory processing as well as batch processing with very fast speed, ease of use and standard interface. This creates the difference between Hadoop vs Spark and also makes a huge comparison between Spark vs Storm.

In this What is Spark tutorial, we discussed the definition of spark, history of spark and importance of spark. For more information about 

Apache Spark Components