Spark Recipes

[spark-recipes]

Recipe 2: Install/Start Spark/Scala REPL

Scala has an interactive shell, or prompt, or interpreter, or, more formally, REPL -- Read-Evaluate-Print-Loop, a venerable term descending from Lisp, the functional ancestor of Scala.

Both standard Scala and Spark distributions come with diver programs wrapping the REPL invocation. For Scala it is called scala, and for Spark it is spark-shell.

You can download both Scala and Spark as tarballs and have a REPL ready to go. We'll focus on the Spark REPL as it is in fact a Scala REPL with the Spark libraries made available and SparkContext defined.

Instead of getting binary downloads of JVM jars, it is much more general and wise to use a build tool like Maven or SBT to keep all of the jars in the same local repository. Maven is a general-purpose build tool for the JVM based on XML configuration files, and SBT started as a Simple Build Tool to become a powerful if certainly not so simple standard for Scala projects. It has a DSL (domain-specific language) to define builds which can contain both Scala and Java dependencies.

Let’s get SBT first. It has its own home at scala-sbt.org, and there are several options listed there. In essence SBT is just a jar and a driver shell script. If you have brew you can just do brew install sbt, or get the jar and a shell script from the Manual Installation link. SBT has its own versions, which use their own version of Scala, so if you ever need excruciatingly fine-grained control over versions of Java, Scala, and SBT from the very first SBT invocation, you can get Paul Phillips’ excellent sbt-extras driver version instead of the vanilla one. In that case make sure it’s on your shell PATH first.

Now that we got the shiny new sbt available on the command line, let’s write a build.sbt file. This file is a build definition for a Scala project, and at a minimum, it defines several names and optional dependencies. Here’s what our first build.sbt looks like:

name := "first-spark" organisation := "spark.recipes" version := "0.1" scalaVersion := "2.10.4" libraryDependencies += "org.apache.spark" %% "spark-core" % "1.0.0" initialCommands in console := """val sc = new org.apache.spark.SparkContext("local[4]", "demo")"""

Now from the project directory say sbt, and there at the prompt say console, and you are at the Spark REPL prompt!

scala> sc res5: org.apache.spark.SparkContext = org.apache.spark.SparkContext@798ddf93

[spark-recipes]

Recipe 1: Use Scala

The first recipe is: use Scala. The reason to use Scala is the same reason why Spark is written in Scala: it is the modern language built for parallel and distributed systems on the JVM.

Scala is the only language really suitable for Big Data as it will evolve in the next few years. This is a remarkable claim which may sound outrageous. While at it, let's add some far-out predictions. Hadoop, or rather its Java-based Map/Reduce workhorse herd, as we know it, is galloping away into the sunset; Spark is the core of the new Hadoop. Depending on the strength of the Hadoop brand, it may persist, but built around Spark.

Scala is the only widespread programming language which has all of the following features:

runs on the JVM

mature and used in the core of global public companies

strongly statically typed with type inference (brevity)

functional

object-oriented

taking full advantage of multi-core architectures

taking full advantage of commodity clusters

taking full advantage of datacenter computers such as Apache Mesos

growing big developer communities

providing rich and uniform object-functional collections API

Hadoop, HBase, and most of the Big Data as we know it runs on the JVM, because JVM is the most mature VM easy for deployments, with man-years of garbage-collection, numerics, text processing, and other kinds of optimization which make it perform on par with C/C++ -- a claim which was hard to believe ten years ago. While other VM-based languages exist, they do not offer computational efficiency, devops tooling, and community support of JVM. To JVM, your Scala code is just a library. You can use all of the existing infrastructure from day one. The only thing remaining is to learn Scala.

In this book, we propose a radical view of Spark: Spark is distributed Scala. Apache Spark works extremely hard to conform to the Scala collections API. We can view it as a form of Scala distributed collections. This view will allow us to teach Spark and Scala together, to inform the intuition of the programmer so that she knows what methods to use in Spark from just using Scala before, and apply the same knowledge across other distributed Scala frameworks such as Scoobi or Scalding.

Spark offers two other APIs, wrapping its distributed Scala heart: Java and Python. As a data scientist you may use Python and be very fond of it. As a Java hand you may be fine with Java. You may be quite satisfied with either Java or Python and try to stick with them while learning Spark in one of those APIs. Let's try to learn Scala and Spark together instead.

Yes, Spark guys work hard to make it easy, enabling adoption. But you will want to learn the true Scala way later anyway. Take the plunge now and learn the simple subset of Scala which explains and enables all of Spark. Call any and all of your Java from it.

The case against Python is based on an old polarizing argument between dynamically and statically typed languages, but it is actually valid and relevant in Big Data space and thus warrants early attention. The original Map/Reduce framework of Hadoop fails to enforce type safety across Map and Reduce jobs, reducing your Java program to a kind of distributed scripting language, in the sense that an input of the next step is not checked for compatibility with the output of the previous step at compile time. Huge resources, dozens of hours of thousands of machines, may be wasted only to find that a map and a reduce step are making wrong assumptions about the data they hand off from one to the next. Scoobi and Scalding solve this problem by making each step a line in a single program and checking the input and output types across them. (We call a short segment of code a line here, such as a lambda function.)

In Python, each line is a map or a reduce step looking very much like in Scala, but Python does not check their compatibility. It is as easy to make a typo which will only trigger a fault at runtime in a Python Spark job as it is in a regular Python script, except that it will fail on a cluster. The same amount of Scala code will eliminate dramatically more most common trivial bugs than a Python program.

We love Java and Python, but in the context of Spark, we propose you use the old adage: if you love something, let it go. Learn Scala and Spark together. Let's begin the journey to Spark with Scala! Well begun, half done.

[spark-recipes]

Spark Recipes: the first Apache Spark Cookbook by Alexy Khrabrov

This is the site of Spark Recipes, the first Apache Spark cookbook, being written by Alexy Khrabrov. The book is developed as a series of common Spark tasks and tutorials, which will be published here piecemeally, and later collated into a book.

The recipes are designed to be especially useful in Spark and Scala Basics, a By the Bay Training course by Alexy Khrabrov, the founder and organizer of SF Scala, the biggest Scala meetup in the world, founding co-organizer of Spark Users, founder and co-organizer of Funconf and Scala by the Bay.