Prassee's Blog @prassee-blog - Tumblr Blog

My Emacs config for Ensime

My Emacs Config

My config for emacs and ensime usage. Emacs looks for a file called init.el under ~/.emacs.d to load all the required plugins,settings and config. In this post Im detailing my emacs configuration. The varios sections are explain one by one so as a reader you can just copy the contents from top to bottom and paste it in your init.el file before you start using Ensime in Emacs.

The entire config can be found here

Disclaimer : The steps explained here are done in my Macbook. The same should work for linux machines as well excpet for few configs

Defining Package Management and archives

Like any other text editor emacs also have a lot of plugins solving various purposes. Package are hosted in a repository called melpa. The first thing is to configure those repository and install use-package whcih is like the seed package to download others.

;; the package manager ;; =================== (require 'package) (setq package-archives '(("gnu" . "http://elpa.gnu.org/packages/") ("org" . "http://orgmode.org/elpa/") ("melpa" . "http://melpa.org/packages/") ("melpa-stable" . "http://stable.melpa.org/packages/")) package-archive-priorities '(("melpa-stable" . 1))) (package-initialize) (when (not package-archive-contents) (package-refresh-contents) (package-install 'use-package)) (require 'use-package)

Configuring packages

the below section details about the various plugins which I've configured. This section is heavily inpired from here. Thanks to Fommil the maintainer of Ensime project / comminity.

ensime

the fist package we are going to install is ensime which pulls out the ensime server and other goodies like code autocompletion, formatting and sbt compatibility etc. The complete features and functionalities of ensime is listed here

(use-package ensime :ensure t :pin melpa-stable)

Projectile

simple project management as seen in Eclispe,IDEA but without lot of config files .idea or .project etc its similar to .gitconfig.

(use-package projectile :demand :init (setq projectile-use-git-grep t) :config (projectile-global-mode t) :bind (("s-f" . projectile-find-file) ("s-F" . projectile-grep)))

Undo Tree

simple visualization of changes happed to the buffer undo-tree

(use-package undo-tree :diminish undo-tree-mode :config (global-undo-tree-mode) :bind ("C-S-t" . undo-tree-visualize))

flx-ido

emacs minor mode auto complete

(use-package flx-ido :demand :init (setq ido-enable-flex-matching t ido-show-dot-for-dired nil ido-enable-dot-prefix t) :config (ido-mode 1) (ido-everywhere 1) (flx-ido-mode 1))

misc

(use-package highlight-symbol :diminish highlight-symbol-mode :commands highlight-symbol :bind ("C-S-h" . highlight-symbol) ) (use-package goto-chg :commands goto-last-change :bind (("C-." . goto-last-change) ("C-," . goto-last-change-reverse))) (use-package popup-imenu :commands popup-imenu :bind ("M-i" . popup-imenu))

magit

the awesome git client for emacs

(use-package magit :commands magit-status magit-blame :init (setq magit-revert-buffers nil)) (use-package git-gutter)

**helm & helm projectile **

its documentation provides much more info and its a worthy read :)

;; the great helm mode (use-package helm :diminish helm-mode :init (progn (require 'helm-config) (setq helm-candidate-number-limit 100) ;; From https://gist.github.com/antifuchs/9238468 (setq helm-idle-delay 0.0 ; update fast sources immediately (doesn't). helm-input-idle-delay 0.01 ; this actually updates things ; reeeelatively quickly. helm-yas-display-key-on-candidate t helm-quick-update t helm-M-x-requires-pattern nil helm-ff-skip-boring-files t helm-autoresize-mode t) (helm-mode)) :bind (("C-c h" . helm-mini) ("C-h a" . helm-apropos) ("C-x C-b" . helm-buffers-list) ("C-x b" . helm-buffers-list) ("M-y" . helm-show-kill-ring) ("M-x" . helm-M-x) ("C-x c o" . helm-occur) ("C-x c s" . helm-swoop) ("C-x c y" . helm-yas-complete) ("C-x c Y" . helm-yas-create-snippet-on-region) ("C-x c b" . my/helm-do-grep-book-notes) ("C-x c SPC" . helm-all-mark-rings))) (use-package helm-projectile :init (progn (require 'helm-projectile) (helm-projectile-on)) :bind (("C-x t" . helm-projectile)))

move or swap lines

move lines or selection up and down

;; move lines with ease (use-package drag-stuff :init (progn (drag-stuff-mode t) (drag-stuff-define-keys) (drag-stuff-global-mode 1)))

smooth scrolling

much refined scrolling in emacs

(use-package smooth-scrolling :init (progn (require 'smooth-scrolling) (smooth-scrolling-mode 1)))

persistent scratch

(use-package persistent-scratch :init (progn (persistent-scratch-setup-default)))

only for Mac users

emacs confuses between the command , option and control keys in Mac keyboard, hence we can remap them using the following config. We also enable copy paste by cmd + c and cmd + v like any other mac app.

;; changing the command to control mac specific (setq mac-command-modifier 'control) (setq mac-option-modifier 'meta) ;; mac specific (cua-mode t)

Configuring the Plugins

now that we have installed the plugins though most of them does not really need special configurations but few of them needs. The comments in-line explain their purpose.

;; show line numbers globally (global-linum-mode 1) ;; let the buffer fringe on the left show git status (global-git-gutter-mode +1) ;; highlight matching parenthesis (show-paren-mode 1) (column-number-mode 1) ;; don't show tool bar : use more screen space (tool-bar-mode -1) ;; modes (electric-indent-mode 0) ;; global keybindings (global-unset-key (kbd "C-z")) ;; keep emacs maximised while startup (add-to-list 'default-frame-alist '(fullscreen . maximized)) ;; use default font across buffer, replace with your fav one (set-frame-font "PragmataPro 14")

Shortcuts

line any IDE emacs has many many shortcuts for various actions. Besides the stock ones its also let us to define our own and here are the ones which I configured , feel free to configure your own :)

;; setting shortcut to find file (fset `yes-or-no-p `y-or-n-p) (global-set-key (kbd "C-x g") 'magit-status) (global-set-key (kbd "C-c d") 'kill-whole-line) (global-set-key (kbd "C-x e") 'ensime) (global-set-key (kbd "C-x o") 'ensime-refactor-organize-imports) (global-set-key (kbd "C-x a") 'org-agenda) (global-set-key (kbd "<c-up>") 'shrink-window) (global-set-key (kbd "<c-down>") 'enlarge-window) (global-set-key (kbd "<c-left>") 'shrink-window-horizontally) (global-set-key (kbd "<c-right>") 'enlarge-window-horizontally) (global-set-key (kbd "C-c <left>") 'windmove-left) (global-set-key (kbd "C-c <right>") 'windmove-right) (global-set-key (kbd "C-c <up>") 'windmove-up) (global-set-key (kbd "C-c <down>") 'windmove-down) (global-set-key (kbd "M-x") 'helm-M-x) (global-set-key (kbd "<c-s-up>") 'buf-move-up) (global-set-key (kbd "<c-s-down>") 'buf-move-down) (global-set-key (kbd "<c-s-left>") 'buf-move-left) (global-set-key (kbd "<c-s-right>") 'buf-move-right)

#Emacs #Scala #scala-basics #scala-tutorial #scala step by step #ensime #magit #projectile

Context Bounds in Scala

In Scala you can impose a constraint on a type parameter (T) saying that, T has to have a binding with another type. I hope this sounds too abstract. Lets write an example,

Consider there are 2 types

case class Address(doorNo: Int, streetName: String, area: String, city: String, pincode: Int, country: String) case class Name(sal: String, fn: String, ln: String)

we are to build a formatted String out of these two types meaning, the common agenda is to build a formatted string . Lets define a type for that too so that we can pass any type.

trait LabelMaker[T] { def printLabel(t: T): String }

Now lets implement the trait to implement formatted string for Address and Name case classes.

object LabelMaker { implicit object IndianAddressLabelMaker extends LabelMaker[Address] { def printLabel(t: Address): String = s"""%d, %s, |%s, |%s - %d, |%s """.stripMargin.format(t.doorNo, t.streetName, t.area, t.city, t.pincode, t.country) } implicit object NameLabelWriter extends LabelMaker[Name] { override def printLabel(t: Name): String = """%s.%s %s""".format(t.sal, t.fn, t.ln) } }

(Note : type enthusiast would have figured out that Iam speaking about type classes !!!)

Now Lets use LabelMaker to format our types a naive implementation would like below

def printLabel[T](t:T)(lm:LabelMaker[T]) = lm.printLabel(t) val x: (LabelMaker[Address]) => String = printLabel(address)

it works, but x results in another function where we have to pass an implementation of LabelMaker.

Lets improve this such that we can infer the LabelMaker from surrounding context using implicits.

val address = Address(4, "pkwms", "sw", "c", 600015, "I") def printLabel[T](t:T)(implicit lm:LabelMaker[T]) = lm.printLabel(t) val x: String = printLabel(address)

now that we are not passing any LabelMaker implementation.

Here comes the deal breaker, now lets say that we have another type called DateOfBirth and we want to create a formattedString out of it.

case class DateOfBirth(day: String, dateOfMonth: Int, month: String, year: Int)

so If we use it saying

val dob = DateOfBirth("Fri", 15, "Jun", 1984) printLabel(dob)

this won't compile as we don yet have the implementation for DateOfBirth. (Note: none of the IDE's today could catch this error inline as you code, it get caught when you compile)

Lets change the printLabel function,

def printLabel[T: LabelMaker](t: T) = implicitly[LabelMaker[T]].printLabel(t)

As you can see we removed the implicit parameter and used something called implicitly , this is where context bounds could be used , If you see in the PrintLabel object we have implicit object which Implement LabelMaker such that we are saying that Address and Name types has a bound to an Implementation of LabelMaker. This is being represented in the function such that it expects that the generic type parameter T has a bound to LabelMaker as defined in [T:LabelMaker]. The compiler will throw an error if it could not find the binding to LabelMaker for T.

In the right side of the function we use implicitly keyword to access the printLabel funciton from the implicit parameter.

so when we use this nothing really changes ....

printLabel(address)

Consider having all the traits , case classes and companion traits in the same package to avoid confusion.

I hope this post had helped you to get started with context bounds..

#Scala #typeclass

My Simple Leader Election Alogrithm

Why Leader Election

Lets say that we have a cluster of nodes with out SPOF. There should be a node in the cluster ready to receive the value from client and save it (disk or memory). How do we find that node,hence we need to select a node from the cluster as Master and that in turn replicates to other nodes. Instead of us making the decision of hand-pick the node, this alogrithm is going to pick node as master, this process is called Leader Election and is a common use case in Distributed Systems.

I had implemented a simple Leader Election algorithm that does this using Akka. This post discusses the implementation.

Algorithm

Each node is considered as a State machine in this algorithm where in it could be in any of the following state

Idle - node start with this state.

Candidate - when node becomes ready for election.

Leader - the master of the cluster.

Idle to Candidate

on start of each node it should be aware ofthe other nodes

a timer executes and it checks if master is elected

if elected it updates its reg to the master

Else it starts the election processs where in it initaites "Election" msg which of type

// Election(ActorRef , Long)

is sent to other nodes here by it (and other nodes) un-become "Idle" i.e it goes back to the Idle state.

Candidate to Leader

While its candidate it also receives election message from other nodes and it stores all such messages in a cache.

// List(Election)

At this state another scheduler kicks in to elect the leader such that all nodes finds the actor ref for corresponding oldest timestamp basically a simple search in the cache and is sent a Leader Elected message.

The node which gets the Leader elected message is the new Leader thus it becomes "Leader" now the leader send other nodes Leader message.

Other nodes remains as candidate and update their master reference to the sender of the "Leader" message.

When Master node goes down

Now that master is elected and oher nodes acting in Idle state. Its also possible hat Master node might go down in such situations and other nodes should be aware of the state master in equal intervals of time.

Each node sends a Heart-Beat message to the master for every 2 secs and receive a Ack msg from the master within 2 secs.

If the Ack is received the node considers that master is alive if not the node gets a time out or something it starts a election thus becomes candidate.

More steps

So far I have implmented only the alorithm and its not complete with a implementation.

Code

Below is the complete implmentation of the algorithm in Akka. This could be run in a cluster of nodes using Akka cluster.

class Node extends Actor { val cluster = Cluster(context.system) val peersBuffer = ListBuffer[ActorRef]() val nodeData = new NodeData() var masterElected = false var electedMaster: ActorRef = null var lastTimeStamp = 0l // subscribe to cluster changes, MemberUp // re-subscribe when restart override def preStart(): Unit = { cluster.subscribe(self, classOf[MemberUp]) cluster.subscribe(self, classOf[UnreachableMember]) } override def postStop(): Unit = cluster.unsubscribe(self) def receive = idle import DurationImplicits._ import scala.concurrent.ExecutionContext.Implicits._ context.system.scheduler.schedule(2.toSecs, 2.toSecs, self, IsMasterElected) context.system.scheduler.schedule(4.toSecs, 10.toSecs, self, ElectionOver) context.system.scheduler.schedule(4.toSecs, 2.toSecs, self, CheckMasterHB) /** * this stage node collects its peers before it can conduct election * when the node */ def idle: Receive = { case MemberUp(m) => register(m) case UnreachableMember(x) => case IndexerNodeUp => if (masterElected) { sender() ! PreElectedMaster(electedMaster) } peersBuffer.+=(sender()) println(s"A Indexer node is brought up now the peers are $peersBuffer") case a: PreElectedMaster => masterElected = true electedMaster = a.actorRef case IsMasterElected => println("schduler kicked in to elect master") if (!masterElected) { println("Cluster's master does not exist , we may need election") lastTimeStamp = System.currentTimeMillis() println(s"Sending Election message to peers $lastTimeStamp -> $peersBuffer") println("Iam a candidate now contesting on election") context.become(candidate) peersBuffer // .filter(ar => ar.path != self.path) .foreach(ar => ar ! Election(lastTimeStamp)) } else context.become(candidate) case DataRequest => println("back to Idle state") case CheckMasterHB => try { Await.result(electedMaster ? HBMaster, 2.toSecs) } catch { case e: Exception => // this means master has not responsed withtin 2 secs hence master failure println("master failed time for the node to become candidate and contest election") } finally {} } def candidate: Receive = { case IndexerNodeUp => if (masterElected) { sender() ! PreElectedMaster(electedMaster) } peersBuffer.+=(sender()) println(s"A Indexer node is brought up now the peers are $peersBuffer") case election: Election => println("Received vote from peer") nodeData.addVote(sender(), election.ts) case ElectionOver => println(s"schduler kicked in to announce election over master $masterElected") if (!masterElected) nodeData.findOldest ! LeaderElected case LeaderElected => println("Oh my god ... Iam elected as leader ") masterElected = true println("Sending the new leader") peersBuffer.foreach(ar => ar ! NewLeader) case NewLeader => println("Welcome new leader") masterElected = true electedMaster = sender() if (sender().path == self.path) { println("Iam elected Let me sworn in as leader ") context.become(leader) self ! "First Msg" } nodeData.invalidateVotes() context.unbecome() // self ! DataRequest } def leader: Receive = { case s: String => println(s"After elected as a leader $s") case HBMaster => sender() ! MasterHBAck case IndexerNodeUp => if (masterElected) { sender() ! PreElectedMaster(electedMaster) } peersBuffer.+=(sender()) println(s"A Indexer node is brought up now the peers are $peersBuffer") } def register(member: Member): Unit = if (member.hasRole("indexer")) context.actorSelection(RootActorPath(member.address) / "user" / "indexBackend") ! IndexerNodeUp } /** * Class to hold election data for a node */ class NodeData { private val votes = scala.collection.mutable.Map[Long, ActorRef]() def addVote(actorRef: ActorRef, ts: Long) = votes.+=(ts -> actorRef) def findOldest = { val oldest = votes.keySet.toSeq.sortWith((a, b) => a < b).head votes(oldest) } def invalidateVotes() = votes.clear() }

The messages handled by the Actor are

import akka.actor.ActorRef case object IndexerNodeUp case object IsMasterElected case class Election(ts: Long) case object ElectionOver case object LeaderElected case object NewLeader case class PreElectedMaster(actorRef: ActorRef) case object DataRequest case object CheckMasterHB case object HBMaster case object MasterHBAck

The entire code is hosted in github . Suggestions & PRs are welcome !!!!!

More to come

Now that we have the algorithm and a skelteton implmentation. I will further extend this post by explaining the Akka code and there by write a small cluster to build a distributed index.

#scala #distributed-systems #akka #akka-cluster

Mining Unstructured Data with Scala and Akka - Part 1

In this post I showcase how to process un-structured data using scala and akka. I am trying best to write this as a series of posts.

approach

Chunk Given a unstructured file , chunk them in to finit no of bytes rouding to nearest line break.

Preprocess We need a descriptor file which has all regex or string pattern which we are interested to parse in the unstructued file.

We also need to distribute each chunk of file (from the prevous step) to an Actor and pass the descriptor along and look for the pattern in the chunk if a pattern is found mark the start bytes, write to db. By doing this we could figure out what are the byte boundaries for each of regex. so we can effectively distribute further to parse only the bytes in the parse stage.

Parse

In the preprocess stage we mark the byte boundaries for a file. now we are to parse only those bytes , here we use the descriptor which carrries the dataformat for each regex / pattern.

our input

un-strutured log file

here is a portion of a log file which we need to parse

Tue Oct 13 09:37:32 IST 2015 Removing old temporary files: Cleaning out old system announcements: Removing stale files from /var/rwho: Removing scratch fax files Disk status: Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk1 112Gi 75Gi 36Gi 68% 19807270 9515992 68% / Network interface status: Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll lo0 16384 <link> 5648 0 5648 0 0 lo0 16384 localhost ::1 5648 - 5648 - - lo0 16384 127 localhost 5648 - 5648 - - lo0 16384 fe80::1%lo0 fe80:1::1 5648 - 5648 - - gif0* 1280 <link> 0 0 0 0 0 stf0* 1280 <link> 0 0 0 0 0 en0 1500 <link> a0:99:9b:17:0f:3d 1486495 0 1095956 0 0 en0 1500 prassees-ma fe80:4::a299:9bff 1486495 - 1095956 - - en0 1500 192.168.0 192.168.0.100 1486495 - 1095956 - - p2p0 2304 <link> 02:99:9b:17:0f:3d 0 0 0 0 0 awdl0 1484 <link> 8e:9d:61:a4:84:18 0 0 157 0 0 awdl0 1484 prassees-ma fe80:6::8c9d:61ff 0 - 157 - - en1 1500 <link> 4a:00:02:10:9e:80 0 0 0 0 0 en2 1500 <link> 4a:00:02:10:9e:81 0 0 0 0 0 bridg 1500 <link> a2:99:9b:71:cd:00 0 0 1 0 0 Local system status: 9:37 up 2 days, 16:11, 2 users, load averages: 2.09 1.62 1.42

Lets look at un-structued file we are going to parse the lines enclosed with Disk status, network interface status and Local sysem status. Since this is a portion of a log file meaning there could be numerrous repetition in a single file or distrubuted in as many file.

Segment Descriptor

We can represent the three keywords as segements since we only look for those lines. Lets represent this in a JSON file we call this as Segment Descriptor file.

{ "segments": [ { "name": "Disk Status Data", "startRegex": "Disk status:", "endRegex": "", "format": "CSV" }, { "name": "Network Interface Data", "startRegex": "Network interface status:", "endRegex": "", "format": "CSV" }, { "name": "Local System Status", "startRegex": "Local system status:", "endRegex": "", "format": "text" } ] }

Now that we have the input, we are going to use Scala and Akka to give a shpae to our apprach. In the next blog post I explain the implmentation of the approach.

#Scala #akka #big data #data #data mining

Configure Ensime + Emacs for your own good

No wonder Emacs is a powerful text editor of all time it been used by many coders / geeks for the last 30+ years. I am writing Scala code and started using Emacs with ENSIME plugin.

Though abbreviated as ENSIME - Stands for Enhanced Scala Interaction Mode for Emacs it does support Sublime,Vim and Atom. Im going to show how I configured ENSIME and have fun.

Configuring Ensime globally

For Scala , you must be using SBT (0.13.x and above) as your build tool. Put the following lines under ~/.sbt/0.13/plugins/plugins.sbt

addSbtPlugin("org.ensime" % "ensime-sbt" % "0.1.7")

This config would help to generate .ensime file for all our Scala projects. The .ensime file is used by the ensime server process created from with emacs editor to provide auto-completion and other goodies.

Configuring .emacs file

The next step is to install Emacs download for ur desired platform after installation create a file called .emacs under ur home director y and paste the below code

;; if you're new to the MELPA package manager, just include ;; this entire snippet in your `~/.emacs` file and follow ;; the instructions in the comments. (require 'package) (add-to-list 'package-archives '("melpa" . "http://melpa.milkbox.net/packages/") t) (package-initialize) (when (not package-archive-contents) (package-refresh-contents)) ;; Restart emacs and do `M-x package-install [RETURN] ensime [RETURN]` ;; To keep up-to-date, do `M-x list-packages [RETURN] U x [RETURN]` ;; If necessary, make sure "sbt" and "scala" are in the PATH environment ;;(setenv "PATH" (concat "/path/to/sbt/bin:" (getenv "PATH"))) ;; (setenv "PATH" (concat "/path/to/scala/bin:" (getenv "PATH"))) ;; ;; On Macs, it might be a safer bet to use exec-path instead of PATH, for instance: (setq exec-path (append exec-path '("/usr/local/bin"))) (require 'ensime) (add-hook 'scala-mode-hook 'ensime-scala-mode-hook)

lot of Emac'ish things happening on the above code in nutshell we are configuring Ensime plugin. Save the .emacs file and restart emacs.

Now you should have received an error while emacs bootstrap saying

(require ensime) is not found or something (require neotree) is not found or something

This means that we have configured ensime but not yet installed the plugin. Lets install the plugin.

Install Ensime plugin

While you're in Emacs say Alt + x and say package-install and press enter. Emacs would prompt for Install Package: enter ensime (and followed by neotree)in the prompt and ensile would be installed in your machine.

Configuring Ensime for Scala Project

Lets configure ensime for a Scala project this essentially means to create a file called .ensime in your project root so that it is used by the Emacs editor to give you all the IDE goddies.

under sbt prompt of your project say,

gen-ensime

this command configures ensime for your project.

Starting Ensime daemon from Emacs

Now that we are almost configured emacs and created .ensime file for our project. Open emacs and press F8 it presents a file navigation window on the left side navigate to ur project folder and press ctrl + c and ctrl + c (yes two times) to change the root of the project (though this is not necessary about ensime its useful).

Press Alt + x and say ensime this would start ensime daemon for your proejct running in background this is per project and you have to repeat when u move to another project. Ensime supports muti-module SBT projects as well :). Once ensime is sucessfully started

you may get a message in the bottom message bar saying

Connecting to Swank on port 57754.. ENSIME ready. Let the hacking commence!

a sample working ensime screenshot

Thats it enjoy hacking Scala with Ensime

#emacs #scala

Writing a Single Page Application with Widok

Widok is reactive UI framework based out of ScalaJS This can be compared to ReactJS and the likes (in fact there is a ScalaJS port for ReactJS).

Lets see how to write a Single Page Application Layout with Widok. To get started on how to setup a Widok project follow their official docs.

Widok provides classes to write an application

object Main extends PageApplication { def view() = Inline( HTML.Heading.Level1("Welcome to Widok!"), HTML.Paragraph("This is your first application.") ) def ready() { log("Page loaded.") } }

here the view gets called when the page loads and it generates a view in the HTML. The above code is a good candidate for a static page, but we are interested in creating an layout based application which has a static header and the content changes dynamically. The idea is to break the UI in to individual page , connect pages with a router and an Application.

Application -- the starting point of the SPA a class that extends RoutingApplication.

View -- class that extends Page class.

Routers -- to change the UI : a set of Route each point to a Page instance.

Writing the Application

object Routes { val root = Route("/", page.Root) // dynamic route val test = Route("/test/:param", page.Test) val notFound = Route("/404", page.NotFound) val routes = Set(root, test, notFound) } // Set of routes and the Fallback route object MainPage extends RoutingApplication(Routes.routes, Routes.notFound)

The above is mostly self-explanatory, this is the starting point of the application unlike the previous example it extends RoutingApplication instead of PageApplication. The Route instances point to instances of Page class.

Breaking the view into Page

A view in Widok is created by extending the Page class. Since in a SPA multiple view have to share the same layout we are creating a base layout page so other view can extend them.

The LayoutPage extends the Page and mixes in with Navbar. The view function creates the base view which has the header and body inside the container.

The individual view has to extends LayoutPage and should provide implementation of body method.

layout

trait LayoutPage extends Page with Navbar { def body(): View def view() = { Inline(header(), Container(body())) } } trait Navbar { def header() = { NavigationBar( Container( NavigationBar.Header( NavigationBar.Toggle() ) ) ) } }

Individual View

case class Root() extends LayoutPage { def body() = { div(HTML.Anchor("Link to second page").url(Routes.test("param", "first page"))) } def ready(route: InstantiatedRoute) {} }

This is the root view i.e the landing page of our application, here we are linking to the next view test we also pass a parameter called param.

case class Test() extends LayoutPage { val query = Channel[String]() def body() = { Inline("received ", query) } def ready(route: InstantiatedRoute) = { query := route.args("param") } }

In Test page we simple print the value of the query string passed from the root view.

#scala #widok #html5 #css3 #js

Writing micro-services with Colossus

Micro-services has been the hot topic these days. There are handful of options that exist to write micro-services like Finagle,Spray etc.

I tried both Finagle and Spray but found that Finagle is more easy to write services than Spray. Though Spray is backed by the Akka actor model so Services could be written by extending Actor. Finagle on the other hand Finagle uses Futures.

For me I was looking for kind of framework combining the best of these worlds i.e a framework backed by actors and uses Futures like semantics. Thats how I found Tumblr's Colossus, in this post I am giving an intro on how to write a micro-service using Colossus.

Colossus is a micro-service framework which is purely built on Typesafe stack using the actor model.

To get started, in your build.sbt add the following

"com.tumblr" %% "colossus" % "0.6.1"

Lets build a micro-service using HTTP protocol

object HttpMicroService extends App { implicit val io_system = IOSystem() val responseTypes = List(("Content-Type", "application/json")) Service.serve[Http]("KVStore", 10200) { context => val kvs = KVStoreBackend context.handle { connection => connection.become { case request@Get on Root => Callback.successful(request.ok("{\"sim\" : \"data\"}", responseTypes)) case request@Get on Root / "api" / "create" / store => Callback.successful(request.ok("{\"status\" : \"" + kvs.createStore(store) + "\"}", responseTypes)) case request@Get on Root / "api" / store / key / value => Callback.successful(request.ok("{\"status\" : \"" + "saved" + "\"}", responseTypes)) case request@Get on Root / "api" / store / key => Callback.successful(request.ok("{\"result\" : \"" + kvs.readValue(store, key) + "\"}", responseTypes)) } } } }

the line which says

Service.serve[Http]("KVStore", 10200)

is where we define our service's protocol ,name and port where the service is supposed to run. Colossus relies on something called event loop so each request is executed within the event loop. However for writing a service we just need not worry about that but just create a implicit variable

implicit val io_system = IOSystem()

Like all other IO frameworks, the data is handled via a handler we create handler from the context there by obtain a connection which exposes a actor. We implement the behaviour of the service within this actor by using the become method.

We match for the URL pattern "/" by

case request@Get on Root => Callback.successful(request.ok("{\"sim\" : \"data\"}",responseTypes))

In the example I have KVStoreBackend which get initiated on the handler bootstrap. Lets implement creating a store by calling the url "/api/create/"

case request@Get on Root / "api" / "create" / store => Callback.successful(request.ok("{\"status\" : \"" + kvs.createStore(store) + "\"}"

In this post, I just scratched the surface by writing a service for a simple KeyValue store. In colossus a service could be a client for another service .

The code in this post is shared in Github.

#Scala #micro-service #finagle #spray #colossus

My experience with Scala IDE 4.0 so far ....

Eclipse was my IDE of choice since I started coding in Scala. But things have changed now either of the IDE's have matured a lot (and equally).

Ubuntu linux has been my OS of choice , Iam right now using Ubuntu Gnome 14.04 it runs on GTK 3.12, the latest Scala IDE is based out of Eclipse Luna which have some issues i.e the select workspace prompt on eclipse boot appears but when you switch to any other app while leaving this prompt open you cannot go back with (alt + tab).

As you can see from the below screenshot that eclipse prompt is not dislayed in the desktop window overlay.

However this task keeps running on the background.

Eventhough I can be patient enough to wait for the eclipse to start completely after selecting the workspace I have a issue the below "force quit" prompt appears for several time. This is wher I got really disappointed.

(Note : this issue prevails in Kepler and Luna)

I have this issue for a very long time its the same reason I avoided Luna and used Kepler but Iam also concerned with Kepler support will be droped in the near future.

Solution

Easy one:- I picked Intellij IDEA Community Edition instead of Eclipse :)

Hard one:- I could also pick ENSIME but Iam not an Emacs expert :)

#Scala #intellij idea #eclipse #scala-ide

DAG (Directed Acyclic Graph) for data processing - Part 1

A DAG is a form form of a graph which does not have start - end.

How a DAG look like ?

The graph starts with node0 and connects node 1.1 and node1.2 which in turn have connected nodes , you could also observe that edges connect the nodes never loop back to any of the previous node so they are "acyclic" also the edges direct the connection between the nodes hence "directed" , so the name Directed Acyclic Graph.

The idea of building a Data processing with DAG DAG are good for data processing the idea is to break data processing logic in to a chain of tasks and chain them together (output of a previous node is the input of the current node) as a DAG.

Thus a program to could be built to process the DAG alongside take a input which does the data processing while traversing through the graph.

But DAG's are not linear As mentioned in the diagram a node is connected to more than one node meaning that DAG are not linear the reason is while data processing huge data sets we might fork task that run in parallel. This is one of the reason that DAG's are suitable for data processing.

#dag #graph #data structures #algorithm #java #Scala

Scala + DWR + AngularJS = fun webapps

This blog post I tell about how to write a Scala webapp with DWR remoting. DWR is a Java remoting API which generate JS proxies for your Java classes. The result is you can call Java methods from your JS.

The example we have taken here would use Scala as a language to write DWR remoting layer and a UI with AngularJS. The code for this project is hosted in my github here feel free to fork and clone.

Iam not going deep in to every detail. But the crux of the post is to show how to add required dependencies and get going with your development....

We take a simple use case of adding job , we provide two screens one for creating and another for displaying some jobs.

I use sbt for building my webapp here the content of build.sbt

name := "kanban" scalaVersion := "2.10.0" seq(webSettings :_*) libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.0.M5b" % "test" libraryDependencies += "org.directwebremoting" % "dwr" % "2.0.3" libraryDependencies += "commons-logging" % "commons-logging" % "1.1" libraryDependencies += "org.eclipse.jetty" % "jetty-webapp" % "8.0.1.v20110908" % "container"

Along with this we should have a plugins.sbt which has the dependency for setting up web-container specific commands i.e start,stop,reload etc. This file should go within your project folder.

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.2.0") addSbtPlugin("com.earldouglas" % "xsbt-web-plugin" % "0.2.12")

Iam using eclipse as my IDE , you can also include your IDE specific plugin information as well. Now that we are done with the environment ready.

In your terminal

> sbt

this should load the project with dependencies so when u lightup your browser with

http://localhost:8080 you can see the app running.

Lets see how I did remoting with DWR in Scala. Its not a big deal since all DWR is a API which is completely isolated from application code, writing a remoting layer in DWR is a no-brainer. So here is the code

object UserInterfaceRemote { private val jobs: ArrayBuffer[Job] = ArrayBuffer[Job]() def createJob(name: String, desc: String): Unit = { jobs.+=:(Job(name, desc)) } def listJobs(): String = { val resp = new StringBuffer resp.append("{\"jobs\":[") jobs.foreach(job => { resp.append("{\"name\":\"" + job.name + "\"},") }) resp.append("{\"name\":\"John\"}") resp.append("]}") resp.toString() } } // Model.scala case class Job(name: String, desc: String)

the code above holds a Array of Job class and methods to add and list Job instances. If you have some familiarity with scala case classes are good fit for this case to transfer the data to-and-fro UI and remoting. So the summary is we have just written our remoting class in Scala and is ready to get invoked from client side.

Our client is an HTML mixed in correct combination of AngularJS topping. Here is the code

angular.module('kanban', []). config(function($routeProvider) { $routeProvider. when('/', {controller: IndexCtrlr, templateUrl: 'html/jobs.html'}). when('/create', {controller: JobCtrlr, templateUrl: 'html/create.html'}). otherwise({redirectTo: '/'}); }); function JobCtrlr($scope,$location) { $scope.createJob = function() { UIRemote.createJob($scope.job.name,$scope.job.desc); $location.path('/'); }; } function IndexCtrlr($scope,$location) { $scope.jobs = []; UIRemote.listJobs({ callback : function(str) { var obj = JSON.parse(str); for(job in obj.jobs) { $scope.jobs.push(obj.jobs[job]); } } }); }

#Scala #dwr #angularjs

Finite State Machines in Akka

Whats FSM

Akka supports creating Finite State Machines (FSM) out of the box. Essentially FSM encapsulte state so based on the data and to the event it listnes to , it toggles from one state to another state.

The typical example to illustrate FSM Traffic Signal yes !! the traffic signal changes itself/manual intervention based on the flow of traffic it switches the traffic flow on the either side of the road.

I'm going to explain FSM in Akka with the same example explained above.

"FSM" Actor Subtype

To implment an FSM we are to create an Actor by extending the trait called FSM this comes out-of-the box in Akka.

The FSM trait is of the structure

trait FSM[State,Data]

hence if we implement this we comeup with an implementation saying

class SignalChangeFSMActor extends Actor with FSM[FSMSignal,SignalColorData]

Implementing State and Data

FSM needs types of State and Data to switch between the FSM state and use data when it responds to events. Lets write those to reflect the traffic signal scenario.

// state - sealed trait FSMSignal case object RedSignal extends FSMSignal case object YellowSignal extends FSMSignal case object GreenSignal extends FSMSignal // data case class SignalColorData(str: String) // event case object ChangeSignal case object RetainSignal

Now we are to write the FSM itself.

/** * Actor */ class SignalChangeFSMActor extends Actor with FSM[FSMSignal, SignalColorData] { // initial state of FSM startWith(RedSignal, SignalColorData("data")) when(RedSignal) { case Event(ChangeSignal, _) => goto(YellowSignal) case Event(RetainSignal, _) => stay } when(YellowSignal) { case Event(ChangeSignal, _) => goto(GreenSignal) case Event(RetainSignal, _) => stay } when(GreenSignal) { case Event(ChangeSignal, _) => goto(RedSignal) case Event(RetainSignal, _) => stay } onTransition { case RedSignal -> YellowSignal => println("Changing from red to yellow signal - get ready to go") case YellowSignal -> GreenSignal => println("Changing from yellow to green signal - wroooom") case GreenSignal -> RedSignal => println("Changing from green to red signal - stop !!! :(") } initialize }

Thats it !!!! we have created our first FSM in Akka !!!!

This project is shared in Github FSMExperiment

#Scala #akka #fsm #sbt

Publish your own API's to Github and use it other project

Lets host a scala project in a github account and use it in another project for this post I'm gonna take a project CoreNLP and publish it in my github driven maven project moma

first lets clone both of these repo in local

CoreNLP-Scala

git clone <https://github.com/gangeli/CoreNLP-Scala>

moma

git clone [https://github.com/prassee/moma](https://github.com/prassee/moma/raw/master/snapshots)

from your CoreNLP-Scala project's sbt prompt just say publishLocal

now that the project articfacts are being published in local ivy cache under my home folder.

~/.ivy/local/corenlpscala/corenlpscala_2.10/1.0-SNAPSHOT.

cd in to the moma folder where we have cloned our maven repo from github and copy the corenlpscala folder from ~/.ivy/local/ and paste it to curr dir, so our current dir looks like this.

As you could see the current dir does contain the copied "corenlpscala" dir. Now lets commit and push it so that it appears in my remote repo.

here the project details project Id is corenlpscala artifact Id is corenlpscala_2.10 version is 1.0-SNAPSHOT

Now its time to use the publishsed artifact in a project in your build.sbt (of any project)

// other project specific settings scalaVersion := "2.10.2" libraryDependencies ++= Seq("corenlpscala" % "corenlpscala_2.10" % "1.0-SNAPSHOT") resolvers ++= Seq( "moma" at "https://github.com/prassee/moma/raw/master/snapshots" )

#git #github #sbt #Scala #maven

Absolute Intro to LXC

Its been very long since I wrote a blog post so to keep me motivated Iam (re)starting with a lighter one. My last post I spoke about advantages of using linux, this post Iam trying to share how to do kernel level virtualization with with LXC - a container virtualization.

(before you read through I assume you fairly have some rough idea about virutalization else please come back and gather some inforamtion about it )

Why LXC?

LXC is a very light weight and easy to maintain virtualization technique , lets say that you want to run Hadoop , Spark cluster we need a cluster of machines, given a desktop or laptop with single NIC card how is it ever possible to build a cluster? yes you can !!! thats what exactly LXC is for us you can build linux virtual machines in an existing linux OS.

Installing LXC

LXC is a kernel level feature and the processor should be VT enabled (most of the 64bit processor are) so make sure you are running a 64 bit OS this post Iam using ubuntu, so to install LXC

sudo apt-get install lxc

thats it you are good to create virtual linux machines from within you own linux OS.

How to use LXC?

Lets see how to create a new LXC

sudo lxc-create -n jupiter -t ubuntu

this command creates a virtualized linux machine with name jupiter (-n option) of type ubuntu (-t option).

To list all LXC created

sudo lxc-ls -f NAME STATE IPV4 IPV6 AUTOSTART ---------------------------------------- jupiter STOPPED - - NO

To start a LXC

once a lxc is created though its ready for use it has to be started in the background so you can login to that LXC .

sudo lxc-start -n jupiter -d

this starts the jupiter the -d option is to daemonize the lxc so it runs in background.

To login to LXC

Now that we have started the lxc now lets login to the lxc

sudo lxc-console -n jupiter

its advised to execute this command on a new tab in the terminal because it gets you to new prompt

to login to the LXC use ubuntu as the user id and password.

#lxc #linux #Docker #cloud #virtualization #ubuntu

Why in the age of cloud computing Linux is an ideal choice for developers

First of all this entry is just my own views Iam a fan of both Linux and Mac OS. I use Macbook Pro and in work I use linux.

When I started using linux with out completely understanding its underlying files system and it edge over windows. I got myself an MacBook 5 years back and used it as how I used Linux.

Slowly as I started to work with distributed computing like Hadoop,Spark,Storm and wanna implement my own ditributed computing structues I needed more than one system.

One of my senior colleagues introduced me about LXC (Linux Containers) it was a ground-breaking as I could able to virtualize / emulate as many as linux machines on my Linux OS this was my first understanding about virtualization and I started understanding about cloud.

By leverating LXC I could build my own personal micro cloud where I deploy my distributed systems. At times when I want to experiment with some of the or build my own I could not do that in my Macbook Pro. Because hese goodies are not allowed in Mac OS.

Though there are choices like

VirtualBox AWS or IAAAS services Docker

Iam not convinced the reason are Virtualbox it would take more CPU and memory AWS or IAAAS Services I should get connected to the internet always right now Iam choosing this Docker they don't support OSX

Mac OS doesnot comes with any virtualization coz of its hardware limitations, though its base BSD kernel does have Jail, a container technology. It could not be installed on Mac hence Iam left with no choice. Infact I had put hold on continuing with some of my self initiated opensource projects (shameless self blame !!!)

All these made me think about installing linux on my Mac though I would lose a good OS for desktop my data and computation requirements has not been fullfilled hence I think it would be wise to move over to Linux. May be I would write an followup post if I suceed migrating to linux.

#cloud #aws #docker #linux #windows #distributed-systems #java #scala

Akka Actor as a Resource for Storm Spout

This blog post would explain how to use Akka actor as an input for Spout in any given Storm topology. Since Spouts are designed to tap out data from any given data source I thought of Akka Actor could also be an eligible candidate as supplier of data for a Spout in a Storm topology.

Data Staging The whole idea is depicted in the above diagram. FileReader reads a file line by line and post that toa TCP port. For each line posted the Actor holds it to the local queue.

Data Processing When the Storm cluster gets started the Spout in could read from this Actor based queue (one at a time) and process it with various combinations of Storm Bolt/s.

Why a Actor Based Queue for Storm? When there is a need to feed a big File as input to Storm's topology , Spout it wouldn't be a suitable abstraction, since a Storm toplogy essentially runs in multiple threads across multiple nodes (in a cluster environment) file reading behaviour if embedded in Spout we would read the same file multiple times which not we want and hence the we delegate that to a Actor which holds a queue of lines read by the FileReader.

Code for Akka Based Queue

case class ParsedLineData(str: String) class TCPServerActor(port: Int) extends Actor { val queue = new Queue[String] override def preStart { IOManager(context.system) listen new InetSocketAddress(port) } def receive = { case ParsedLineData(str) => queue.enqueue(str) case IO.NewClient(server) => server.accept() case IO.Read(rHandle, bytes) => rHandle.asSocket.write(ByteString(queue.dequeue() + "\n".getBytes()).compact) } }

Code that kickstarts Queue Actor

object TCPServerActorApp extends App { def startTCP() { println("exposing new TCP port for message queue @ 6789") val server = ActorSystem().actorOf(Props(new TCPServerActor(6789))) val fileSrc = scala.io.Source.fromFile(this.getClass().getResource("/gitsActorRead.csv").getPath()) fileSrc.getLines().foreach(line => { server.!(ParsedLineData(line)) }) } startTCP() }

Message Reading Spout

class LineBasedProcessingSpout() extends StormSpout(outputFields = List("Line")) { def nextTuple { val next = nextTupleImpl emit(next) } def nextTupleImpl = { val socket = new Socket("localhost", 6789) val br = new BufferedReader(new InputStreamReader(socket.getInputStream())) val dos = new DataOutputStream(socket.getOutputStream()) dos.writeChars("\n") val x = br.readLine() dos.close() br.close() x } }

The whole code base has been published in GitHub

#storm #akka #Scala #Java IO #java

Websocket with Scala.js

I was playing with Scala.js, a Scala to JavaScript compiler. Scala.js essentially helps us to write JS for HTML in pure Scala. So you can full utilize Scala to code for the DOM checkout the project here.

If you have a scala development environment then you are already ready to rock n roll with Scala.js all I will do is jump to the crux of the code.

This post briefs about how to write web-socket client in Scala.js we develop the classical chat application

Before diving in I have implemented a websocket server for myself and assuming that its running on port 9898. Scala.js is a client side hence we our code is essentially a client of the websocket server. Lets jump in to the code.

Scala.js implements the latest JS standard hence API for websocket comes out of the box. A classical JS version of websocket looks like this.

I created a class called ChatExample. The line which says

val ws = new dom.WebSocket("ws://127.0.0.1:9898")

creates a instance of client websocket in Scala.js and it points to the so-called websocket server running in localhost. The instance of this class we implement its lifecycle callbacks as closure in the lines

ws.onmessage = (x: MessageEvent) => populate(x.data.toString) ws.onopen = (x: Event) => {} ws.onerror = (x: ErrorEvent) => Console.println("some error has occured " + x.message) ws.onclose = (x: CloseEvent) => {}

import scala.scalajs.js.String import org.scalajs.dom import org.scalajs.dom._ /** * */ object ChatExample { val d = dom.document val chatField = d.getElementById("playground") val errorField = d.getElementById("error") val infoField = d.getElementById("info") // websocket val ws = new dom.WebSocket("ws://127.0.0.1:9898") def populate(data: String) { console.log("data " + data) val out = data.split("#") val msg = out(1) console.log("msg " + msg) val prefix = out(0).toString prefix match { case "chat " => { val p = d.createElement("p") p.className = "small" p.innerHTML = s"${msg}" chatField.innerHTML = "" chatField.appendChild(p) } case "error " => { errorField.innerHTML = "" errorField.innerHTML = "" + msg + "" } case "info " => { infoField.innerHTML = "" val alert = d.createElement("div") alert.className = "alert alert-success" alert.innerHTML = msg infoField.appendChild(alert) } } } def initWS() { ws.onmessage = (x: MessageEvent) => populate(x.data.toString) ws.onopen = (x: Event) => {} ws.onerror = (x: ErrorEvent) => Console.println("some error has occured " + x.message) ws.onclose = (x: CloseEvent) => {} } }

The method populate does most of the decorating work. It is assumed that the websocket server responds like

"chat # 'chat mes' " "info # 'notification for any event' " "error # 'incase of connection errors etc' "

Hence the method essentially splits the incoming response and updates the DOM.

The code is shared in github. Feel free to fork it ....

#scala #scala.js #html #html5 #css3 #websockets #sbt

Enable "Tamil Typing" on your Mac

I was wondering for a long time how people do Tamil posts facebook, twitter etc. Yesterday I got a chance to meet one of my FOSS friends we was discussing about writing articles in Tamil, very lately I was informed by him that there is a way to type in Tamil in Mac (as Iam a Mac OSX user).

Here we go how to do that ,

Go to System Preferences -> Languages & Text

Go to Input Sources tab, search for Tamil in the text field to locate & enable Tamil as input source also you can check the "show input menu in menu bar".

Essentially this enables you 3 types of keyboard layouts English US , Tamil 99 and Anjal. In this the latter two are Tamil keyboard layout. Anjal is a phonetic based one meaning when typed "jaavaa" it displays "ஜாவா".

At any point of time you can switch between English and Tamil keyboard layouts by META + SPACE or ALT + META+ SPACE keyboard shortcuts. Once you are done you can see US flat added in the menu bar just click on it

#mac #tamil #tamil keyboard in mac

Trending Blogs

Recently Viewed Blogs

Prassee's Blog