Wrongbat

My notes on what I do on my PC and stuff

[wrongbat]

[wrongbat]

Making water Kefir

A friend of mine started talking to me about Kefir, and how healthy it is. Apparently it's supposed to be the ultimate pro-biotic. That is, it stimulates the growth of the right kind of bacteria in your gut. I thought it'd be fun to try to make some, as I like making fermented foods, but I didn't know where to buy it in Tokyo (where I live). After googling around a bit, I found this Facebook group and asked where to buy it. A very kind woman offered to send me some grains just minutes after I posted in the group. So already the day after first hearing about Kefir, I had some in my postbox.

There're several kinds of Kefir. The bacteria live in little chunks called "grains". You can add the grains sugar water. The bacteria like to eat the sugar which they convert into other things, such as lactic acid and carbon dioxide. According to this site they even produce alcohol, yay! As they eat the sugar they also grow larger, so you will end up with more grains than when you started the fermentation process.

I started with with dissolving two spoons of brown sugar into 500mL water, and then I added two tablespoons of Kefir grains to that. I also added some sterilized egg shells as I was told they grains like calcium.

After six days of fermenting this stuff at room temperature bubbles had formed in the liquid, and I filtered the it through a colander to separate the grains. This is the finished product that I’m currently drinking.

Two two spoons of grains were now four spoons of grains, so I’m currently making 1L of drink. I also tried adding a bit more sugar this time to get more carbon dioxide, as I like fizzy drinks. Below is the new brew to the left, and the fizzy fermented drink to the right.

If you live in Japan and want to try out making your own water Kefir

[wrongbat]

zeromq Context destroy method

The pyzqm tutorial is very good for learning the basics of how to use zmq. I've learned a lot from reading it. Now I'm reading the PyZMQ API documentation because I feel that investing some time in learning the finer apsects of ZMQ is worth the effort. I'm discovering some handy tricks that I thought I'd share on my blog.

You know how you use a Context instance to create sockets? Well, the Context class has a very nice method for closing all sockets that were created by that context. Here's how you do it.

Here's a program that creates two servers and two clients that sends a message every second. After three seconds a timer kicks in and closes all

import zmq from time import sleep from multiprocessing import Process from threading import Timer, Thread context = zmq.Context() def server(name, port): server = context.socket(zmq.ROUTER) server.bind('tcp://*:{}'.format(port)) while True: print('{}: {}'.format(name, server.recv_multipart())) def client(name, port): client = context.socket(zmq.DEALER) client.connect('tcp://localhost:{}'.format(port)) while True: client.send_multipart(["", "From {} with love".format(name)]) sleep(1) def destroy(): context.destroy() if __name__ == "__main__": Process(target = server, args = ('server1', 5563)).start() Process(target = server, args = ('server2', 5564)).start() Process(target = client, args = ('client1', 5563)).start() Process(target = client, args = ('client2', 5564)).start() Timer(3, destroy).start()

However, this program won't work as intended. The clients will keep sending and the servers will keep printing what they receie. Why is this? Because we created the servers and clients as processes! When you create a process a copy is made of all the objects in scope. So the context object that the timer closed was never used to create any sockets! Aha, basic lesson in processes. Actually I made exactly that mistake :).

So let's create the servers and clients as threads instead:

if __name__ == "__main__": Thread(target = server, args = ('server1', 5563)).start() Thread(target = server, args = ('server2', 5564)).start() Thread(target = client, args = ('client1', 5563)).start() Thread(target = client, args = ('client2', 5564)).start() Timer(3, destroy).start()

This time the message flow stops, but not in a very nice way. The server threads throw an ContextTerminated exception, the clients throw ZMQError: Socket operation on non-socket exception, and then the program hangs. It's actually written in the docs:

destroy involves calling zmq_close(), which is NOT threadsafe. If there are active sockets in other threads, this must not be called.

So finally my conclusion is that context.destroy() is mainly useful when you have a single thread using many sockets. There are plenty of use cases for that. Here's a simple example where a single thread has two threads talking to each other (pretty useless, but just for the purpose of demonstation :)

import zmq from time import sleep from multiprocessing import Process from threading import Timer, Thread from itertools import count context = zmq.Context() def main(): server = context.socket(zmq.ROUTER) server.bind('tcp://*:5563{}') client = context.socket(zmq.DEALER) client.connect('tcp://localhost:5563') for i in count(): if i == 3: context.destroy() if not client.closed: client.send_multipart(["", "From client with love"]) sleep(1) print(server.recv_multipart()) else: break if __name__ == "__main__": main()

[wrongbat]

Why did I start learning zeromq?

I started out learning zeromq because I wanted to learn how to send messages between many programs running of different computers. I had never done anything like this before. I was aware that there was something called sockets (I’d heard of TCP sockets and so on), but learning how to use them properly seemed like a huge effort to me, so I never really got started. I kept thinking something like “sockets are for network engineers, which I’m not”. 

When I was at university I was interested in multi-agent simulation. Well, the idea of simulating complex systems using many simple agents cinhabiting an environment and communicating with each other seemed interesting anyway. For one reason or another I decided to try to simulate a financial market and figure out why flash crashed were happening. I think it was because I’d read “The Black Swan” by Nicholas Taleb. His writing style felt inspiring to me at the time. So anyway, I started writing a simulation engine in Java in spite of being mainly a Python programmer. Something with strong opinions about the speed of Python vs. within the lab where I was studying. Anyway, I wrote the whole thing as a giant Java program to run in a single thread, and it became a monster, but one that worked for the purposes of doing my simple research. (check it out here). I graduated, and I thought “that’s pretty much it for me and multi-agent systems”.

I went and got a job in an IT company where I started learning about web-programming and having a fun time doing it. I didn’t think much about multi-agent systems until almost a year later. I started to think that I could maybe combine what now knew about web stuff and what I knew about multi-agent stuff, mix it and make something out of it. My first idea was to create a trading platform where people could play little trading games against each other or program algorithms to trade for them in the same games. I’m still working on that, and it’s a much larger project than I first thought (usually this is what happens for me). But even though I’m just getting started I’ve already learned so much! 

My first attempt was to make a web interface that connected the user to a program running an auction. The web app was based on flask and used web sockets to connect the user to the auction. The proof-of-concept prototype was more or less working (for a single user trading with himself...), but I had no idea on how to expand it into a larger system with many auctions running at the same time. And then I started writing integration tests to make sure that what I had already made would continue to work in the future. And honestly all that extra effort was very demotivating to me, and I gradually stopped working on the app. 

Then again a few months later, one of my friends told me about this thing called zeromq. I looked at the front page of zeromq and didn’t really understand much of what was written there. Something about sockets on steroids and so on. To someone like me for who the concept of a socket was still pretty vague and scary it didn’t really make much sense. But I liked the friendly way that the documentation was written in, so I decided to give it a shot an bought the zeromq book. I have to say it’s one of the only books on programming that I’ve actually enjoyed reading. Not that other programming books are not good, but they’re usually pretty dry and devoid of humor. This is not the case with the zeromq book, and you’ll often find yourself smiling when you’re reading it. Apart form the joy of reading a well written book, I also started to understand how incredibly useful a tool zeromq is. 

You know that feeling when you learn something, and that new knowledge makes you feel powerful? That’s what I felt like when I was reading about zeromq. Basically zeromq is the first tools that has enabled me to feel some confidence about writing distributed systems. Of course I’m still facing many big issues, and writing a market app is still a lot of work, but thanks to zeromq I now have an idea of how to make it. 

[wrongbat]

Agent class using PyZqm and tornado ioloop

I created a simple class that will be the parent class for all the various types of agents in my multi-agent simulation. The class inherits from Process, so it needs to define a run() method. This run method will be called when the process is started. 

Agent has an abstract method, setup(), the implementation of which is done in the sub-class. In setup() the sockets are created, and the agent sends initial requests/init messages.

I’ll figure out how to deal with heart-beating stuff and reliability later. For now this class will do for prototyping applications.

Here’s the code:

import zmq from zmq.eventloop import ioloop, zmqstream from multiprocessing import Process import abc from datetime import datetime class Agent(Process): __metaclass__ = abc.ABCMeta def __init__(self, name): Process.__init__(self) self.context = zmq.Context() self.loop = ioloop.IOLoop.instance() def run(self): self.say('Setting up agent...') self.setup() self.say('Starting ioloop...') self.loop.start() @abc.abstractmethod def setup(self): return def say(self, msg): print('{} - {}: {}'.format(datetime.now().strftime('%H:%M:%S'), self.name, msg)) class Server(Agent): def handle_socket(self, msg): address, m = msg[0], msg[2] self.say(m) self.socket.send_multipart([address, '', 'REQUEST OK']) def setup(self): self.socket = self.context.socket(zmq.ROUTER) self.socket.bind('tcp://*:6000') stream_pull = zmqstream.ZMQStream(self.socket) stream_pull.on_recv(self.handle_socket) class Client(Agent): def handle_socket(self, msg): self.say(msg) self.socket.send_multipart(["", 'NEW REQUEST']) def setup(self): self.socket = self.context.socket(zmq.DEALER) self.socket.connect('tcp://localhost:6000') stream_pull = zmqstream.ZMQStream(self.socket) stream_pull.on_recv(self.handle_socket) self.socket.send_multipart(["", 'NEW REQUEST']) if __name__ == '__main__': Server(name = 'server').start() Client(name = 'client').start()

[wrongbat]

request/reply (DEALER/ROUTER) with zeromq IOLoop

In my previous post I tried out the tornado-based IOLoop with PUSH/PUSH zmq sockets. Here’s an example for for to use the asynchronous DEALER/ROUTER sockets: 

import zmq from zmq.eventloop import ioloop, zmqstream from time import sleep from multiprocessing import Process ioloop.install() context = zmq.Context() def server(): def handle_socket(msg): address, m = msg[0], msg[2] print('In server: {}'.format(m)) socket.send_multipart([address, '', 'REQUEST OK']) socket = context.socket(zmq.ROUTER) socket.bind('tcp://*:6000') stream_pull = zmqstream.ZMQStream(socket) stream_pull.on_recv(handle_socket) ioloop.IOLoop.instance().start() def client(): def handle_socket(msg): print('In client: {}'.format(msg)) socket.send_multipart(["", 'NEW REQUEST']) socket = context.socket(zmq.DEALER) socket.connect('tcp://localhost:6000') stream_pull = zmqstream.ZMQStream(socket) stream_pull.on_recv(handle_socket) socket.send_multipart(["", 'NEW REQUEST']) ioloop.IOLoop.instance().start() if __name__ == '__main__': Process(target = server).start() Process(target = client).start()

The main point is here that both client and server make an instance of the ioloop, so everything is event driven. Hence, in order to get messages flowing, the client has to send out a message before starting the ioloop. I'd like to try to code a PING/PONG heartbeating for reliability using this ioloop class.. (http://zeromq.org/deleted:topics:heartbeating)

[wrongbat]

IO loop in zeromq

I've been writing code for implementing a reactor loop with handler functions for sockets. The idea was that it should be easy to create agents with an arbitrary number of sockets, and which then have a central loop which polls the sockets and call the appropriate handlers. Of course this has already been done, and since my own efforts were not going so well I started trying out the tornado IOLoop. It's pretty awesome. Check out the example below.

import zmq from zmq.eventloop import ioloop, zmqstream from time import sleep from multiprocessing import Process ioloop.install() context = zmq.Context() def server(): socket = context.socket(zmq.PUSH) socket.bind('tcp://*:6000') while True: socket.send('Yo') sleep(1) def worker(): def handle_socket(msg): print('In worker 1: {}'.format(msg)) socket = context.socket(zmq.PULL) socket.connect('tcp://localhost:6000') stream_pull = zmqstream.ZMQStream(socket) stream_pull.on_recv(handle_socket) ioloop.IOLoop.instance().start() def worker2(): def handle_socket(msg): print('In worker 2: {}'.format(msg)) socket = context.socket(zmq.PULL) socket.connect('tcp://localhost:6000') stream_pull = zmqstream.ZMQStream(socket) stream_pull.on_recv(handle_socket) ioloop.IOLoop.instance().start() if __name__ == '__main__': Process(target = server).start() Process(target = worker).start() Process(target = worker2).start()

The example is based on code in the pyzmq documentation: http://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/multisocket/tornadoeventloop.html

[wrongbat]

Python multi key dictionary

Normal Python dictionaries have a simple one-to-one mapping, where a single key maps to a single value. However, sometimes it’s useful to be able to access a value via several different keys. If you want to do that, you can use multi_key_dict. Just do

$ pip install multi_key_dict

Here’s an example of how this data structure works:

So basically you can specify multiple keys and get the value using either of them. Note that this is not the same as using a tuple in a normal dictionary, since in that case you will have to use the entire tuple to get the value. 

If you try to add a value with a key which overlaps with an already existing key, an exception is thrown, which means is significant since it means you can be sure that each key maps to a single value. 

One place where you need to be careful when using this data structure is that values are overwritten when placing a value referenced by a key which is part of a key already in the dictionary:

This is a bit confusing since 1 will suddenly map to ‘b’ instead of ‘a’, even though this mapping was never explicitly entered in the dictionary. Fortunately the class contains a method which you can use to know when this happens.

[wrongbat]

Connecting a zmq ROUTER socket to multiple peers

Using zeromq it’s very easy to connect the asynchronous DEALER socket to more than oner peer. When you send a message on the DEALER socket, zmq will choose the least recently used peer to send it to. I’ve made a little example below to illustrate.

import zmq from multiprocessing import Process from time import sleep addr1 = 'ipc://broker1' addr2 = 'ipc://broker2' context = zmq.Context() def worker(): socket = context.socket(zmq.DEALER) socket.connect(addr1) socket.connect(addr2) while True: socket.send_multipart(["", "Hello"]) print(socket.recv_multipart()) sleep(1) def broker_1(): socket = context.socket(zmq.ROUTER) socket.bind(addr1) while True: addr, e, msg = socket.recv_multipart() print("In broker 1: %s"%addr) socket.send_multipart([addr, "", "hello from broker 1"]) def broker_2(): socket = context.socket(zmq.ROUTER) socket.bind(addr2) while True: addr, e, msg = socket.recv_multipart() print("In broker 2: %s"%addr) socket.send_multipart([addr, "", "hello from broker 2"]) if __name__ == '__main__': Process(target = worker).start() Process(target = broker_1).start() Process(target = broker_2).start()

[wrongbat]

Simple arithmetics with opamps

I’ve read that you can use opamps to do arithmetic operations like adding, subtracting, etc, but I was like “wtf, how?” until I just stumbled into it by accident by tinkering with variations of the circuit below:

The opamp doesn’t really do anything here except inverting the signal. This is because of two sets of facts. 

First, Vin is a sine wave with a 5VDC offset and V1 is 5VDC. That means that when Vin = 5V, no current will flow through R1 and R2, which means that there’s no voltage drop over neither resistor. Then because the signal is connected to the inverting input of the amplifier, Vout = -Vin.

Second, since R1 = R2 and since the current that flow between two two resistors is the same, the voltage drops are also the same. So when Vin is at its peak value of 6V, this pulls a current of (6V - 5V)/10kΩ = 100µA through R1. This same current flows through R2 and creates a voltage drop 1V, making Vout = 5V - 1V = 4V. Similarly when Vin = 4V, Vout = 6V. 

Multiplication and division

It’s easy to multiply or divide Vin. Simply change the ratio R1/R2. When R2 < R1, more current will flow through R2 than R1, and thus the voltage drop over R2 will be smaller than over R1, and you will have scaled down (divided) the signal.

When R2 > R1 the signal is amplified, that is, multiplied. 

Addition and subtraction

Now let’s see what happens when we change the level of the voltage supply going into the positive input on the opamp. 

When Vin is 6V, a current of 100µA will flow, causing a voltage drop of 1V over R2. Then Vout = 7V + 100µA * 10kΩ = 8V. Hmm. When Vin = 4V a current of 300µA will flow through in the reverse direction through R2, causing a voltage *gain* of 3V, making Vout = 10V. When Vin is 5V Vout is 9V. 

So we’ve managed to add 4V to Vin! That’s pretty awesome. You can subtract by setting V+ less than 5V (the mean of Vin). 

[wrongbat]

Simple way to use Python decorators

Probably the simplest way to use a decorator in Python is to do something before the decorated object (classes and functions) is defined. In the example code below the class A is never instantiated, but the code in the decorator is still executed. Also it's important to keep in mind that the code in decorator is executed exactly once each time it's used to decorate an object. If you want to make a wrapper function that does something before and/or after the

def decorator(obj): print "In the decorator now, where we can do something before obj is defined" return obj @decorator class A: pass

which generates the output

In the decorator now, where we can do something before obj is defined

This is useful if you want to register an object in a global data structure, like this:

animals = [] def register_animal(obj): print 'Registering animal: ', obj animals.append(obj) return obj @register_animal class Cat: pass @register_animal class Dog: pass class Teapot: pass print 'Registered animals: ', animals

which results in the output

Registering animal: __main__.Cat Registering animal: __main__.Dog Registered animals: [<class __main__.Cat at 0x10e538f58 >, <class __main__.Dog at 0x10e565120>]

So what’s the deal with the at-mark? It’s a neat syntax. What happens is this. When the interpreter reaches the @, it calls decorator(func) where func is the function f. It then replaces the definition of f with what’s returned by the decorator. In this case the original function. We can check this:

def decorator(func): print "In decorator: ", func return func @decorator def f(): print 'In f()' if __name__ == '__main__': print f

Output:

In decorator: <function f at 0x104432de8> <function f at 0x104432de8>

You can see that the address of func and f are the same. So really, that’s the simplest way to use decorators. In later blog posts I will show a few other examples of more advanced ways of using decorators. Mainly how to pass arguments to the decorator, how to create decorators that wraps the decorated function, and how to use classes as decorators (my personal favorite :)

[wrongbat]

A few LTSpice resources

This is mostly for my own reference. 

LTSpice keyboard shortcuts for OSX can be found here:

http://cds.linear.com/docs/en/software-and-simulation/LTspiceShortcutsForMacOSX.pdf

And this page has some good tutorials for learning how to use LTSpice. I’ve basically been following his tutorials until now: http://www.simonbramble.co.uk/lt_spice/ltspice_lt_spice.htm

[wrongbat]

First time simulating opamp in LTSpice

I’m trying to learn how to use opamps, so I’m  starting with one of the simplest circuits, using a single-supply amplifier because I then only have to use one voltage source to power the amplifier. The circuit below uses negative feedback to amplify Vin. I wanted the input voltage to vary during simulation, so I changed it to a piece-wise linear function. 

To do this go to advanced properties of the voltage source (on mac you simply right click on the voltage source and press ‘Advanced’). Then you choose the style PWL (piece-wise linear) and fill in the points as follows:

This makes the voltage source jump from 0V to 1V at 0.001s, stay 1V for another millisecond and then jump back to 0V. The figure below shows the simulation output, and it shows that the amplifier has a gain of 3.

[wrongbat]

one-to-one map in Python

There’s a module called bidict which implements a one to one mapping. In a normal dict keys are unique whereas values can be duplicate. In a bidict both keys and values are unique. A key points to a single values, which points back to the same key. So just as putting a key-value pair in a dictionary where the key already exists will cause the old value to be replaced with the new value, so will placing a key-value pair in a bidict where the value already exists cause the old value to be replaced by the new value:

In [78]: b = bidict({1:2}) In [79]: b Out[79]: bidict({1: 2}) In [80]: b.put(3, 2) In [81]: b Out[81]: bidict({3: 2})

As for speed, bidict somewhat slower than a standard dictionary, but still pretty fast:

In [53]: d = {1:2} In [54]: b = bidict({1:2}) In [55]: %timeit d.get(1) 10000000 loops, best of 3: 134 ns per loop In [56]: %timeit b.get(1) 1000000 loops, best of 3: 363 ns per loop In [57]: %timeit b.inv.get(2) 1000000 loops, best of 3: 620 ns per loop

[wrongbat]

Writing a wrapper function WITHOUT using decorators

When I first started learning about Python decorators I found them pretty confusing an unintuitive, and it took me a while before I started using them. What I often want to do is modify a function with some extra functionality without having to add the extra code in my original function. Decorators are excellent for doing that, but if you find them confusing (which I did) then you can write a wrapper like this:

def wrap(agent): assert hasattr(agent, 'loop') lfunc = agent.loop def before_loop(): print('Before loop') def after_loop(): print('Doing stuff after loop') def wrapped(*args, **kwargs): before_loop() try: lfunc(*args, **kwargs) except Exception, e: print(e) after_loop() agent.loop = wrapped class TestAgent: def loop(self, n): for i in range(n): print('In loop') print(x) if __name__ == '__main__': a = TestAgent() wrap(a) a.loop(5)

if you run the script you will see that the before_loop and after_loop are also executed:

Before loop In loop In loop In loop In loop In loop Doing stuff after loop

The try/except block in wrapper() makes sure that after_loop() is called even if loop() throws an exception.

You should not forget to use a copy of the loop function (third line in the example above) as it will cause a RuntimeError due to infinite recursion. So this will crash:

def wrap_broken(agent): assert hasattr(agent, 'loop') def before_loop(): print('Before loop') def after_loop(): print('Doing stuff after loop') def wrapped(*args, **kwargs): before_loop() agent.loop(*args, **kwargs) after_loop() agent.loop = wrapped class TestAgent: def loop(self, n): for i in range(n): print('In loop') if __name__ == '__main__': a = TestAgent() wrap_broken(a) a.loop(5)

[wrongbat]

Python atexit module in multi-process program

I’m trying to figure out an easy way of knowing when . The python module atexit let’s you specify a callback which is executed after the program finishes. To be precise the documentation states that

"The atexit module defines a single function to register cleanup functions. Functions thus registered are automatically executed upon normal interpreter termination.”

So here’s a little example that I wrote to exactly understand what that means.

from __future__ import print_function from multiprocessing import Process, current_process import atexit def exit_handler(): print('In exit handler') def work(): print('(in {}) done: {}'.format(current_process().name, sum(range(10000000)))) if __name__ == '__main__': print('Before starting') Process(target = work, name = 'worker-1').start() Process(target = work, name = 'worker-2').start() atexit.register(exit_handler) print('Back in main')

Running the program will generate the following output:

Back in main In exit handler (in worker-1) done: 49999995000000 (in worker-2) done: 49999995000000

So you see that the exit handler is called when the interpreter exits the main script, but not after each of the processes are terminated.

[wrongbat]

Kill script without having to manually find the PID

For instance, find process running python script named market_v4.py and kill it:

pgrep python | xargs ps | grep market_v4.py | sed 's/s.*//' | xargs kill

The command works as follows.

pgrep python

shows the PID of all python processes

xargs ps

shows the pid and COMMAND used to start the process

grep market_v4.py

matches the line where the command is python market_v4.py

sed ‘s/s.*//’

subsitutes everything except the PID with nothing (leaving only the PID). And finally:

xargs kill

kills the process :).

There’s probably a shorter way to do it, but I think this is pretty straightforward.