SQL-on-Hadoop: The dos and the don’ts
While considering using SQL to query data from Hadoop, the first question which comes to mind is “Which is the tool to be used?” There are a lot of tools out there but they don’t truly provide with the promised effectiveness and efficiency. While some turn to MapReduce, they are unable to achieve the desired results. On the other hand, MR framework works well with data on a massive scale and caters to fault tolerance effectively. MapReduce was dedicatedly built for processing Big Data on large scale which we mostly refer to as batch processing.
As enterprises have started using Hadoop as a Big Data repository for the inflow of their data (originating from various sources varying from the operational systems, smart sensors, mobile devices such as smartphones and tablets, along with various applications over the Internet), SQL becomes an optimal choice. A fundamental reason being, most enterprise management and analytical tools rely heavily on SQL. Also that the talent pool for SQL is much bigger than that of data scientists and MR developers.
Hadoop is the best tool in the market for Big Data processing, there are many tools and frameworks available in the market which make use of SQL to provide the functionalities. That said, some of them even make SQL-on-Hadoop better than native SQL tools. As a means of query execution, SQL processing benefits from years of research, user experiences and optimizations.
MapReduce might be appropriate for ad-hoc analysis but that is as far as it can go with the current stage of technology. It serves the requirements well if they adhere to general aspects of the data, but to perform complex queries and analytics SQL-on-Hadoop is a necessity rather than an accessory. As discussed there are a lot of tools in the market which provide us with this capability, but to get the one that fits your needs perfectly, you need to perform a thorough research.
Some of the major tools which provide us with SQL via third party development on Hadoop are:
- Apache Hive
- Cloudera Impala
- Presto by Facebook
- Big SQL 3.0 by IBM
All of these can provide with the required functionality, but where one gains advantage over the other in terms of functionality, the others might take advantage in terms of pricing. Usually these tools are provided for free, with minimal usage fee but you might not be in a position to pay that fee either. So make sure that you perform the required research well before hand, and avoid dedicated hiring for Map Reduce. Hiring from the SQL resource pool will ensure that you get the work done effectively and efficiently. The SQL professionals have economically priced services and can query data in ways others simply cannot. It’s a win-win situation for all, you get SQL on Hadoop and they get to play around with Big Data.