Arche(type)

[harph]

Managing Scale and Complexity. An Event Driven Approach

Coupled services, tangled endpoints and pooling mechanisms are usually the result of an organically growth architecture. Unfortunately, as the scale increases, the complexity and constraints do as well.

The purpose of this talk is to expose how quickly complexity grows, and how a unified event driven architecture approach, enables the support of more complex infrastructures that require to be flexible and scalable. Additionally, it covers best practices, pros and cons; and how I have used this approach at Netflix to drive a platform that leverages more than 300 million events a day.

This talks aims to target mid to advanced level engineers who have experienced problem scaling services or dealing with complex architectures. The expectation is that after this talk, everyone walks away understanding the power of events, and how they can be leveraged to design architectures for large scale problems.

[harph]

Path to Principal Interview

I got the pleasure of being invited to be part of one of the Path to Principal Podcast episodes. Click here to listen to my interview.

[harph]

When Booleans Are Not Enough... State Machines?

Usually, we tend to represent objects status with boolean attributes. At first, this seems right and simple enough, but as the code base evolves and gets bigger, status become complex; so do the transitions between them. Transition between status can be seen as behaviors or actions. Therefore, we find ourselves implementing rules to enforce behaviors and validating transitions; or even worse, not enforcing or validating anything at all. Considering using a state machine to represent the status of an object may be ideal. As each state on the machine can represent an object status, the transitions between states can represent well defined actions or behaviors that can be performed between status.

This talk focuses on how to identify when booleans are not the right type to represent status, and how using a state machine may lead you to a better and cleaner design that can enforce conditions between status by simply relying on the definition of a state machine.

Here you can find my slides for this presentation.

[harph]

Quick And Easy MongoDB Replica Set Tutorial

This post summarizes what I’ve learned about setting up a MongoDB Replica Set. For this tutorial I’m using Vagrant to create the instances but you can use the information explained here and adapt it for actual machines.

VMs Setup

Since we are going to set up a replica set with 3 members, the first thing we have to do is create the VMs, one for each member.

# Create a directory for each VM. mkdir -p mongo01 mkdir -p mongo02 mkdir -p mongo03 # Create the Vagrant files. vagrant init hashicorp/precise64 --output mongo01/Vagrantfile vagrant init hashicorp/precise64 --output mongo02/Vagrantfile vagrant init hashicorp/precise64 --output mongo03/Vagrantfile

After executing the code above we end up with 3 Vagrant environments, one in each directory. Next, edit each Vagrantfile. Set the value of config.vm.hostname and config.vm.network :private_network, ip.

# The IPs and the names are up to you. # For mongo01 config.vm.hostname = "mongo01" config.vm.network :private_network, ip: "192.168.33.91" # For mongo02 config.vm.hostname = "mongo02" config.vm.network :private_network, ip: "192.168.33.92" # For mongo03 config.vm.hostname = "mongo03" config.vm.network :private_network, ip: "192.168.33.93"

Now go ahead and execute vagrant up inside each directory to create the VM. After this, execute vagrant ssh to access the VM.

Next, we need to make sure that each VM is accessible via DNS or hostname. In this case we are going to use the /etc/hosts file to set this up. Open the file on each VM and add the following lines:

# The IPs listed here must match the ones defined in the Vagrantfiles. 192.169.33.91 mongo01 192.169.33.92 mongo02 192.169.33.93 mongo03

Now you should be able to ping each VMs from one to another by using the hostnames.

MongoDB Replication Setup

Let’s install MongoDB on each VM.

sudo apt-get update sudo apt-get install mongodb

Next, edit /etc/mongodb.conf. Remove the line bind_ip = 127.0.0.1 and add the following line:

replSet = repl01 # repl01 is the name of the replica set. You can change this name for something else.

Important: The name you choose for the replica name has to be the same in all the VMs, if not, you are going to get the following error when you try to add a new replica:

{ "assertion" : "set name does not match the set name host repl01:27017 expects", "assertionCode" : 13145, "errmsg" : "db assertion failure", "ok" : 0 }

Note: Removing the bind_ip = 127.0.0.1 line allows incoming connections from all sources. Do not use this in production. Instead of removing this line, specify the IPs that are allowed to access this server.

Next, you have to restart the mongodb service to make the changes effective. For this run the following command on each VM:

sudo service mongodb restart

After this you are almost done. Before continuing, check the connectivity between instances by making sure that each MongoDBinstance is reachable by the others. For this execute:

# In mongo01 mongo --host mongo02 mongo --host mongo03 # In mongo02 mongo --host mongo01 mongo --host mongo03 # In mongo03 mongo --host mongo01 mongo --host mongo02

You should be able to establish connection between servers.

Finally, execute the following command in the mongo shell in mongo01:

# Initiate replica set use admin rs.initiate() # Verify the configuration rs.conf() # Add mongo02 as a member or the replica set rs.add("mongo02:27017") # Add mongo03 as a member or the replica set rs.add("mongo03:27017")

You can check the status of the replica set by executing:

rs.status()

The output of this should look like:

{ "set" : "repl01", "date" : ISODate("2014-10-28T23:12:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "mongo01:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "optime" : { "t" : 1414522392000, "i" : 1 }, "optimeDate" : ISODate("2014-10-28T18:53:12Z"), "self" : true }, { "_id" : 1, "name" : "mongo02:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15754, "optime" : { "t" : 1414522392000, "i" : 1 }, "optimeDate" : ISODate("2014-10-28T18:53:12Z"), "lastHeartbeat" : ISODate("2014-10-28T23:12:29Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "mongo03:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 7953, "optime" : { "t" : 1414522392000, "i" : 1 }, "optimeDate" : ISODate("2014-10-28T18:53:12Z"), "lastHeartbeat" : ISODate("2014-10-28T23:12:27Z"), "pingMs" : 0 } ], "ok" : 1 }

The replica set is already setup. You should be able to create a database and a colletion in mongo01. You can verify that it gets replicated by opening the mongo shell in mongo02 or mongo03 and executing:

show dbs

You can only insert, modify or delete data in the primary member. If the primary member goes down, one of the secondary members will become primary.

Note: Mongo replica set requires an uneven number of members greater or equal than 3. If you have an even number of members then you must add an arbiter. The arbiter will be responsible for breaking the ties in the election of the primary member. You can find more information about this here.

I hope you find this useful.

Cheers.

[harph]

Setting NSMutableURLRequest Cookies

I've been looking for a easy a simple explanation/tutorial of how to add cookies to a NSMutableURLRequest. After experimenting and reading a few forums and blogs, I finally understand how to do it and therefore, I decided to sumarized this in a way I would like someone to explain it to me.

There are two ways you can set cookies. The first one makes use of the setValue function of the NSMutableURLRequest instance. It basically requires you to create a string with the following format COOKIE_NAME_1=COOKIE_VALUE_1 and set it as a header. If you want to add multiple cookies, use a semicolon (;) to separate them. This is what your code would look like:

var request = NSMutableURLRequest(URL: NSURL(string: "http://example.com")) request.setValue( "cookie_name_1=cookie_value1; cookie_name2=cookie_value_2", forHTTPHeaderField: "Cookie")

As you can see is very simple.

The second way seems a bit less simple but it does not involve creating a string with a specific format. In this case, each cookie is defined by a NSHTTPCookie instance. Here is a snippet of what the code would look like:

var request = NSMutableURLRequest(URL: NSURL(string: "http://example.com")) // Cookie1 properties var cookieProperties1 = [ NSHTTPCookieOriginURL: "http://example.com", NSHTTPCookiePath: "/", NSHTTPCookieName: "cookie_name_1", NSHTTPCookieValue: "cookie_value1"] // Create an instance of NSHTTPCookie for cookie1 var cookie1 = NSHTTPCookie.cookieWithProperties(cookieProperties1) // Cookie2 properties var cookieProperties2 = [ NSHTTPCookieOriginURL: "http://example.com", NSHTTPCookiePath: "/", NSHTTPCookieName: "cookie_name_2", NSHTTPCookieValue: "cookie_value2"] // Create an instance of NSHTTPCookie for cookie2 var cookie2 = NSHTTPCookie.cookieWithProperties(cookieProperties2) // Create an array of cookies var cookies[NSHTTPCookie] = [cookie1, cookie2] // Create cookies as headers var cookieHeaders = NSHTTPCookie.requestHeaderFieldsWithCookies(httpCookies) // Set cookies as part of the request header request.allHTTPHeaderFields = cookieHeaders

The first thing you do is to create a dictionary that contains the cookie properties. Once you have that, you create an instance of NSHTTPCookie by calling the cookieWithProperties function and passing the properties dictionary. Repeat this process for each cookie you want to set and append it to an array. Once you are done, use the requestHeaderFieldsWithCookies function to create the cookie headers. Last, assign the cookie headers to the request by using the allHTTPHeaderFields attribute.

NOTE: Assigning the cookies using allHTTPHeaderFields function, will override the previous headers you may have set using the addValue function. If you want to set headers to the request, do it after you set cookies if you are using the second way.

I hope you find this useful. Cheers.

[harph]

NSHTTPCookie.cookieWithProperties Return nil

If you are trying to create a NSHTTPCookie using the cookieWithProperties function and getting back nil instead of a NSHTTPCookie instance, the dictionary that you are passing to the function must be wrong or incomplete.

// Wrong properties dictionary var properties = [NSHTTPCookieName: "YOUR-COOKIE-NAME", NSHTTPCookieValue: "Cookie Value"] // nil cookie var cookie = NSHTTPCookie.cookieWithProperties(wrongProperties)

After executing this code, the cookie will be nil. This occurs because the properties dictionary does not contain the right keys and values. The documentation of the function cookieWithProperties says: "To successfully create a cookie, you must provide values for (at least) the NSHTTPCookiePath, NSHTTPCookieName, and NSHTTPCookieValue keys, and either the NSHTTPCookieOriginURL key or the NSHTTPCookieDomain key.".

Following exactly what I just quoted, our code will look like this:

var properties = [ NSHTTPCookieOriginURL: "http://example.com", NSHTTPCookiePath: "/", NSHTTPCookieName: "foo", NSHTTPCookieValue: "bar" ] var cookie = NSHTTPCookie.cookieWithProperties(properties)

The execution of the code above will end up with the cookie being an instance of NSHTTPCookie.

Cheers.

[harph]

Swift REPL

Read-Eval-Print-Loop - REPL. is an interactive environment for computer programming.

Being used to languages like Python, REPL is a powerful tool that you use on a daily basis for experimental purposes, exploring functions and algorithms, debugging and even for learning a new programming language.

Using Swift REPL

First you need to have Xcode6 installed. Then, execute the following command in the terminal:

sudo xcode-select -s /Applications/Xcode6-Beta.app/Contents/Developer/

Once you have executed the command above, you are ready to start. Go ahead and execute:

xcrun swift

This is going to start a Swift interactive console. Next, type:

1> 1 + 1 $R0: Int = 2 // This is the result of the operation

You can declare variables, functions, classes, etc. You may want to import Foundation depending on what you want/need to use. i.e.:

1> var url = NSURL(string: "http://blog.hjoseph.com") :1:11: error: use of unresolved identifier 'NSURL' var url = NSURL(string: "http://blog.hjoseph.com")

As you can see you get back an error telling you that it doesn't recognize NSURL. If you import Foundation you don't get this error anymore.

1> import Foundation 2> var url = NSURL(string: "http://blog.hjoseph.com") url: NSURL = "http://blog.hjoseph.com"

You can also run your Swift scripts by executing the following command:

xcrun swift -i YOUR_FILE.swift

You can even make the file executable by adding the following line at the begining of the file:

#!/usr/bin/env xcrun swift -i

Then, change the file mode to executable.

chmod +x YOUR_FILE.swift

Now, you can run your file by executing:

./YOUR_FILE.swift

Cheers!

[harph]

Xcode6 - Swift: Create an Empty Project With **NO** Storyboard

I tend to be the kind of developer that likes to start from scratch and understand from the bottom up everything I'm doing and how I can hack it. This is the reason why I decided to start an empty project with no storyboard or anything pre-defined. Using a storyboard may be easier at the beginng but probably, in my opinion, this level of abstraction just makes things a bit too messy when you have a big project with multiple developers involved. I also feel that writing and having the whole application flow in my code is that right way to do stuff and definitely a better way to debug when things are not working the way you expected.

Creating an Empty Project

Assuming that you are using Xcode6 (I'm using the Beta2).

File > New > Project

Once the Project dialog Poped-up you should see something like this:

Now, go ahead and select from the left menu, Application under the iOS option. Next, select Empty Application and hit Next. Then we have to name the product, in this case let's call it HelloWorld. Also, in the Language dropdown select Swift.

Finally, hit Next and select the location where you want to store the project.

You should have a project that looks like this:

Creating a View

Now, let's create a view.

File > New > File

Next, from the left menu on the template dialog select User Interface under iOS. Then select View and hit Next.

After this select the Device Family (iPhone in my case) and hit Next.

Finally, name the file RootView and select its location. Make sure to place the file inside the HelloWorld folder in your project. Hit Create and you'll see a new file called RootView.xib.

Now, let's add a label to the view. First, select the RootView from your project tree. You should see the Interface Builder XIB Document open.

Next, drag a label from the right bottom side and drop it in the midle of the view. Then double click the label to change to "edit mode" and type "Hello". You should have something like this:

Creating a View Controller

Now let's create the view controller that is going to be in charge of the actual view.

File > New > File

Next from the left menu on the template dialog select Source under iOS.

Then select Swift File and hit Create.

Now, let's write the following code on this file:

import UIKit class RootViewController: UIViewController { }

As you can see, we are defining a class called RootViewController that inherits from UIViewController. Our class is empty. Here, you can add functions and attributes that can be connected to elements on the view.

Connecting The View With The View Controller

Now that we have the view and the controller defined, we need to create the connection between them so the view identifies its controller. For this we have to follow these steps:

Click on the RootView.xib file.

Once the file is open, click on the File's owner icon (the yellow 3d-box).

Select the thrid tab on the inspector menu.

Set the Custom Class - Class to RootViewController and the Custom Class - Module to HelloWorld.

Control-click (right-click) the File's Owner icon and drag it to the View icon. This will define the view as an outlet of the File's Owner.

Defining a Navigation Controller

Lastly, we need to do some work in the AppDelegate.swift file. First, let's add a UINavigationController attribute to the AppDelegate class. Add the following code after the var window: UIWindow? declaration.

var navigatorController: UINavigationController?

Next, right before the return true instruction of the func application add the following code:

var rootViewController = RootViewController(nibName: "RootView", bundle: nil) self.navigatorController = UINavigationController(rootViewController: rootViewController) self.window!.rootViewController = self.navigatorController

The code above creates an instance of RootViewController specifying the RootView.xib as its nib. Next, it initializes the navigationController setting the instance of RoowViewController as the rootViewController. Finally, it sets the navigatorController as the rootViewController of the AppDelegate window.

We are done. Now you should be able to hit run and see the app in the iOS Simulator.

Stay tuned for more!

[harph]

Resize a VirtualBox VM Virtual Disk

Firstable stop the VM. Then execute:

cd <PATH_TO_VIRTUALBOX_VMs>/<YOUR_VM> VBoxManage clonehd --format VDI <YOUR_VM_DISK>.vmdk <NEW_VM_DISK_NAME>.vdi VBoxManage modifyhd <NEW_VM_DISK_NAME>.vdi --resize 60000

This can only be used to expand (not shrink) the capacity.

Once you have done this, open VirtualBox, right-click on your VM and select Settings. Once the Settings window is open go to the Storage section, click on "Add new IDE controller" and select the new VM_Disk file. Then select the previous disk and remove it from the attachment. Click ok and restart the VM.

[harph]

Extracting Text Document Terms

I've been working, experimenting and playing for a while with different ways to extract information from text documents. One of the most important task is determining the terms (keywords) that define a text document. This task is the based of full text search, indexing and many other applications. 

Determining the terms from a text can be very simple, but it can also get really complicated. All this depends on the final application. These are the basic steps to get the keywords:

Remove all the punctuation signs.

Strip all the simple/stop words. You can find lists of these online, but you can also create your own depending on the final results and the context of the text.

The remaining words are going to be consider terms. Split the text by spaces and  count how many times each term appears and divide that number by the total number of terms. This would be the a frequency factor that you can use to rank the term in that document.

It's important to mention that not always the term with higher frequency factor is the one that best defines the document. Sometimes it's better to use the inverse value or the IDF (Inverse Document frequency value). You can read more about this here.

Cheers.

[harph]

Updating Nginx to The Latest Stable Version In Ubuntu

Open /etc/apt/sources.list and add the following lines:

deb http://nginx.org/packages/ubuntu/ precise nginx deb-src http://nginx.org/packages/ubuntu/ precise nginx

If you have a Nginx version already installed run the following commands to avoid conflicts with the new versions:

sudo apt-get remove nginx-full sudo apt-get remove nginx-common

Then run:

wget http://nginx.org/packages/keys/nginx_signing.key cat nginx_signing.key | sudo apt-key add - sudo apt-get update sudo apt-get install nginx

Now to check the version installed run:

nginx -v

You should get the latest stable version.

Cheers.

[harph]

Finally a working version of my current project. #robotic #arm #arduino

[harph]

Transifex Exception: 400: Error compiling the translation file

transifex.exceptions.TransifexAPIException: 400: Error compiling the translation file

If you are trying to download a translation using Transifex graphic interface or pulling them down using manage.py tx pull_translations and you are getting an error like:

transifex.exceptions.TransifexAPIException: 400: Error compiling the translation file: ‘ascii’ codec can’t encode character u’\u2019’ in position 9017: ordinal not in range(128)

You should set the variable LC_ALL to your ~/.bashrc or ~/.profile.

export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8 export LANGUAGE=en_US.UTF-8

Instead of setting that to your shell you can also set it on your Gunicorn settings.

environment=LC_ALL='en_US.UTF-8'

Cheers.

[harph]

Set a class function with parameters as an Attribute

Sometimes we have to set/inject/monkey patch attributes or functions to objects. This is fairly simple when we have the instance or the class and we don't need to pass any parameters. In that case we would do:

class Foo(object): def bar(self): print 'bar' def foo(self): print 'foo' self.bar() # You can set the attribute by doing: setattr(Foo, 'foo', foo) # or Foo.foo = foo

Then once you create and instance of Foo you can call the foo.

foo_instance = Foo() foo_instance.foo() # This will print 'foo' and 'bar'.

All this is pretty simple and you can also do the same with an instance, but how about setting a function that receives parameters? In that case you need to use a wrapper.

from functools import wraps def partial(func, *parameters, **kparms): @wraps(func) def wrapper(*args, **kwargs): kw.update(kparms) return func(*(args + parameters), **kwargs) return wrapper

This would allow you to do things like:

def dynamic_foo(self, value1, *args, **kwargs): print value1 print args print kwargs self.bar() setattr(Foo, 'foo1', partial(dynamic_foo, 'this is foo 1')) setattr(Foo, 'foo2', partial(dynamic_foo, 'this is foo 2', 'value 1', 'value 2')) setattr(Foo, 'bar_x', partial(dynamic_foo, 'this is bar x', val1='value 1', val2='value 2'))

As you can see this is setting dynamic functions to Foo with different parameter values. This allow you to do:

foo_instance = Foo() foo_instance.foo1() # This will print: # 'this is foo 1' # () # {} # 'bar' foo_instance.foo2() # This will print: # 'this is foo 2' # ('value 1', 'value 2') # {} # 'bar' foo_instance.bar_x() # This will print: # 'this is bar x' # () # {'val1': 'value 1', 'val2': 'value 2'} # 'bar'

Cheers.

[harph]

PySerial communication with Arduino

PySerial is a great Python Library used to access the serial port. I've been playing with this to interface with Arduino.

This is a basic example:

from serial import Serial serial_port = Serial('/dev/tty.usbmodemfd') foo = serial_port.read() serial_port.close()

As you can see, it's pretty simple. The problem appears when you want to write to the serial port. Let's say that we want to send an integer. Due to the fact that PySerial sends the data as ASCII code, the Arduino receives the ASCII representation of the number you send. To get around this you have to pack the integer as a binary.

import struct foo = struct.pack('B', 1) # here you are packing the number 1 as a binary. serial_port.write(foo)

Cheers.

[harph]

Django Upload File Permission

If you have ever had permission problems serving/rendering/displaying files uploaded with Django, maybe this can be a solution for you.

Open you settings.py file and add the following line:

FILE_UPLOAD_PERMISSIONS = 0644

So, basically you're setting that permission to all the files that are going to be uploaded through Django.

Hope this works for you.

[harph]

Django get_object_or_404 instead of except DoesNotExist

We always define URLs in which we specify parameters like and ID, slug, name, code or whatever to link and specific object in our web app. But what happens when the value doesn't link anything?. Let's see an example:

Let's suppose we have this URL:

url(r'^post/(?P\d+)/$', name = 'get_post', 'blog.views.get_post')

This means that we can go directly to a post using a URL like http://example.com/post/1. Executing that URL we are requesting the information/page associated with the post with id = 1. Now, the problem occurs when we don't have a post in our database that fulfills that condition, because usually we get the post using a function like this:

def get_post(request, id): post = Post.objects.get(id = id) ...

Django executes that query and assigns the result to the post variable, but when Django doesn't find the post that fulfills the condition, it throughs a DoesNotExitException which at the end if we don't handle it, generates a 500 error page (Internal server error). In my opinion, we shouldn't take the user to that page for a reason like this. Instead of a 500 error we should display a 404 error page (Not found error). So in the end our code looks like this:

def get_post(request, id): try: post = Post.objects.get(id = id) ... except Post.DoesNotExist, e: raise Http404

This works pretty well but, it implies that you have to add try and except to every query of this kind. Instead of doing this, why don't we use the Django shortcut get_object_or_404? This is how it works:

from django.shortcuts import get_object_or_404 ... def get_post(request, id): post = get_object_or_404(Post, id = id) ...

This is pretty obvious. The function is going to try to get the object based on a condition and if it can't find it, it is going to take the user to a 404 page. You can use this in combination with the objects functions, i.e.:

post = get_object_or_404(Post.objects.select_related(depth=2), id = id)

Enjoy.