toThinkOrNotToThink @tothinkornottothink-blog - Tumblr Blog

React Native Images in Android

I've been lately working on a personal React Native project. This app renders a lot of pictures in a ListView within the home page and you can tap on any picture and get a full resolution image displayed that can be pinched-and-zoomed. This seemed like a good fit for RN in my head, although after a lot of suffering I'm no longer sure of this though. Actually today I decided to write about the state of image loading and rendering in RN Android as a way of organizing my mind.

First my application is suffering a lot of memory issues, that crash the application in an OutOfMemoryError exception. I found an open issue about this exact topic, issue #7972. The issue had been idle for a while, so I decided to post my findings there, nothing incredible.

Facebook guys decided to integrate their Android open source library for managing images, named fresco, to handle images in Android with RN. After a lot of digging in React Native issues, you can get the feeling that fresco/RN integration has issues, maybe memory leaks issues. So if you are considering creating your next Android mobile photo album application with React Native, maybe it's not yet quite the time.

#react native

React Native navigation evolution and thoughts on it

When I started coding in React Native, Android support was fresh off the bench and navigation at that time was simple but limited. There were two choices: Navigator and NavigatorIOS. The choice for new comers was clear, NavigatorIOS was limited to iOS as the name implies, it had several bugs and there was an official statement in the docs recommending to use Navigator at that time. However, after some time using Navigator, you could really feel that it was the black sheep of the React Native family. First, its API was imperative, it was quite basic handling nested routes, it was limited in many ways, you had to pass navigator as a prop everywhere, etc.

React Native core team, started soon working on a different approach, that they called NavigatorExperimental. They haven't fully documented it and they warned everyone that the API could change drastically with every new release of RN. Today, you can see how it is being discussed when NavigatorExperimental will turn into maybe simply Navigation and join the stable API lines, deprecating the old fashioned but popular in many projects Navigator.

#react-native #react

Raspberry Pi + SimpleCV + OpenCV + Raspicam CSI camera

In this Codemotion Madrid 2013, I'm giving a talk about Computer Vision. It's going to be a beginner's talk, an introduction to high level concepts of artificial vision using SimpleCV, OpenCV and Python. For making this talk more interesting I'm going to develop a software prototype, whose internals I will explain at the end of the talk, putting in play some of the concepts introduced during the talk, so people see a real application of things learned.

In the abstract of the talk, I stated I was going to run this prototype in a Raspberry Pi. I have some experience with the Raspi using USB cameras and compact cameras handled with libgphoto2, but many of these hog Pi's CPU, their framerate is too slow, the camera is too expensive or too big.

We've all heard of Raspberry CSI camera board or Raspicam. It's specs are great and one of its great strengths is that is uses ~2% CPU. That's crucial as I'm going to need every CPU cycle available for analyzing frames looking for patterns. The problem is that the camera is not USB, so I knew it wasn't going to be an easy fit.

I couldn't find a single tutorial on how to make Raspicam work with SimpleCV, only questions about it online. Actually, I only found one for OpenCV, a seven part tutorial by Pierre Raufast, the guy behind Magic Mirror project. The idea of this tutorial is to modify source code of raspivid and raspistill to use buffer memory of the camera to feed OpenCV image objects. This was a great step in making it work, but not an easy one, everyone can follow.

#raspberry #simplecv #opencv #raspicam #python

Django Forms I: Custom fields and widgets in detail

On Thursday September 8th I copresented with Daniel Greenfeld my first talk "Advanced Django Form Usage" at DjangoCon.us 2011. It was my first talk for many reasons. I had never spoken in front of such a large audience before, less in English and this was also my first Python talk. If you haven't seen the video of the talk and the slides, you probably want to, because this post will try to extend, clarify and recap it somehow. Also this will be my official erratum for things that were not exactly true or right that appeared in the slides

Some people really loved the talk and came after it to comment it with me and ask some questions. By that time, I was already thinking that the topic did deserve its own series of posts. Gabriel Hurley, Django committer and documentation specialist, stated:

Apparently some core developers were asking to use the code examples and a transcript from the talk as a starting point for this task. Well, I wasn't sure if the transcript would be good enough, many things were left to say, but I agreed with Gabriel that it's a good starting point to base docs on, which is better than starting from scratch. I still feel this necessity of writing these series of posts, which will hopefully be a better starting point than the transcript. I've been extremely busy since I gave the talk, but now I happen to have some spare time, so let's start.

#django #forms #django-uni-form #django-floppyforms #python #fields #widgets #MultiWidget #MultiWidgetLayout #JSONField

Django-uni-form 0.9.0 is out [security fix]

Version 0.9.0 is out and you should update because of security reasons. An XSS bug has been fixed, thanks to Charlie Denton for reporting it. If you are using django-uni-form and a form field that renders the input of a field as part of the error message without sanitizing it, such as ChoiceField, you are vulnerable to it. This is because errors are rendered using |safe filter.

This has been addressed as fast as possible. Fixes for previous versions might come in the soon future. PyPi package has been update so you can do:

pip install --upgrade django-uni-form

This is the commit that introduced the security bug, as you can see it affects version 0.7.0 and forward versions.

I want to thank every contributor and user that has made this version rock even more than previous ones.

Bug fixes

It's got support for show_hidden_initial, Fieldset's legends internationalization, error_css_class and required_css_class. MultiField layout object has fixed markup and has been optimized.

Performance boost

The project has seen several performance tune ups, related to templates handling, that make it run blazing fast. You can run now django-uni-form without template caching only one second slower than the cached version.

Customizable templates

All layout objects have now an attribute named template that points to a template file. This template is in charge of how the rendering of the layout object. Using you custom templates for layout objects is not a hassle anymore.

Improved docs

The docs are on their way to be easier to grok for new comers. New sections have been added that explain some of the new 0.9.0 features and some of the most hidden features of django-uni-form. It's got now more examples and use cases, that will hopefully resolve some of the most seen questions. For better readability we are now using kenneth Reitz kr sphinx themes, based on Flask themes, for read the docs.

Layout inheritance

If you have a layout shared by many forms, you will be able to do some neat layout inheritance with it. That will help you being more DRY.

Better organization

helpers.py file has been divided into three different files: helper.py, layout.py and utils.py. You can still import everything using helpers, but a deprecation warning has been added, so that you can start moving to the new importing structure. helper holds FormHelper class, layout holds the Layout class and layout objects, while utils holds render_field, an internal function used for rendering your django-uni-form layouts.

More tests and better test coverage

This is the only way we will be able to mantain the quality of this project and keep up the pace of faster release cycles.

#django #python #django-uni-form #0.9.0 #uni-form #forms #layout #DRY #layouts

New kung-fu in django-uni-form 0.8.0

This won't be a new official release announcement as Daniel already did the honors. I'm now the co-lead developer of django-uni-form and in this post I will try to go over some examples that teach you new kung-fu techniques. But before doing so, let me emphasize how cool this project is and how many new nice features are packed in the new version 0.8.0. If you like DRY principles and don't think your Django forms are DRY enough using built-in filters, you should check it! if you haven't yet.

To explain some of the new features, let's use a real form and a real example, so it's easier to follow how django-uni-form works and what has been enhanced.

The following form will be used from now onwards to exemplify the new features in django-uni-form. I'm using different types of form fields to demonstrate how django-uni-form can work with any built-in widget or form field or any other custom field that you have defined. That is, django-uni-form plugs well with other form apps.

from django import forms class ArticleFeedbackForm(forms.Form): like_example = forms.TypedChoiceField( label = _("¿Do you like this article?"), choices = ((1, "Yes"), (0, "No")), coerce = lambda x: bool(int(x)), widget = forms.RadioSelect, initial = '1', required = True, ) reason = forms.CharField( label = "Why did you like it?", max_length = 80, required = True, ) rating = forms.IntegerField( label = "Rating (from 0 to 10)", min_value = 0, max_value = 10, required = False, ) notes = forms.CharField( label = "Additional notes or feedback", max_length = 200, widget = forms.Textarea, required = False, )

To load django-uni-form template tags in our template we will have to add:

{% load uni_form_tags %}

Django-uni-form still has |as_uni_form filter, which is used exactly the same way. The only difference is that it now supports rendering formsets. This is how you would use it, supposing we have the form in the template context as example_feedback_form:

<form ...> {{ example_feedback_form|as_uni_form }} </form>

In my opinion, the real power in django-uni-form comes from its {% uni_form %} tag and being able to set layouts. But let's go step by step to avoid confusion.

Django-uni-form implements a class called FormHelper which is attached to the form. This class defines the form rendering behavior.

Let's add to our form an instance-level form helper variable:

from django import forms from uni_form.helpers import FormHelper class ExampleFeedbackForm(forms.Form): like_example = forms.TypedChoiceField( label = _("¿Do you like this example?"), choices = ((1, "Yes"), (0, "No")), coerce = lambda x: bool(int(x)), widget = forms.RadioSelect, initial = '1', required = True, ) reason = forms.CharField( label = "Why did you like it?", max_length = 80, required = True, ) rating = forms.IntegerField( label = "Rating (from 0 to 10)", min_value = 0, max_value = 10, required = False, ) notes = forms.CharField( label = "Additional notes or feedback", max_length = 200, widget = forms.Textarea, required = False, ) def __init__(self, *args, **kwargs): self.helper = FormHelper() self.helper.form_id = 'example_feedback_form' return super(ExampleFeedbackForm, self).__init__(*args, **kwargs)

Now that we have a helper, we can use helper methods to set how the form should render. In this case I've just set the form DOM id to 'example_feedback_form'.

New FormHelper attributes

Version 0.8.0 introduces new attributes that can be set in a helper:

form_error_title: If you are rendering a form using {% uni_form %} tag and it has non_field_errors to display, they are rendered in a div. You can set the title of the div with this attribute. Example: “Form Errors”.

formset_error_title: If you are rendering a formset using {% uni_form %} tag and it has non_form_errors to display, they are rendered in a div. You can set the title of the div with this attribute. Example: “Formset Errors”.

form_style: If you are using uni-form CSS, it has two different form styles built-in. You can choose which one to use, setting this variable to "default" or "inline".

Creating a Layout

Django-uni-form defines another powerful class called Layout. You can create your Layout to define how the form fields should be rendered. With "how" I mean things like the order of the fields, wrap them in divs or other html structures with ids or classes set to whatever you want, add things in between, etc. And all that without writing a custom template, rather fully reusable without writing it twice.

A Layout is constructed by what I like to call layout objects, which can be thought of as form components. You assemble your layout using those. For the time being, your choices are: Button, ButtonHolder, Column, Fieldset, HTML, Hidden, MultiField, Reset, Row and Submit.

All these components are explained in helper API docs. What you need to know now about them is that every component renders a different template. Let's write a couple of different layouts for our form, continuing with our form class example (note that I'm not showing the full form again):

from uni_form.helpers import Fieldset, Submit (...) def __init__(self, *args, **kwargs): self.helper = FormHelper() self.helper.form_id = 'example_feedback_form' layout = Layout( Fieldset('The first arg is the legend of the fieldset', 'like_example', 'reason', 'rating', 'notes', ) ) self.helper.add_layout(layout) self.helper.add_input(Submit('register', 'Register')) return super(ExampleFeedbackForm, self).__init__(*args, **kwargs)

Now if we want to render our form example_feedback_form using what we've set in the helper and our layout, we would do in the template:

{% uni_form example_feedback_form example_feedback_form.helper %}

This is exactly the form that we would get rendered using the previous tag:

<form action="." class="uniForm" method="post" id="example_feedback_form"> <fieldset> <legend>The first arg is the legend of the fieldset</legend> <div id="div_id_like_example" class="ctrlHolder "> <label for="id_like_example" class="requiredField"> ¿Do you like this article?<span class="asteriskField">*</span> </label> <ul> <li><label for="id_like_example_0"><input checked="checked" name="like_example" value="1" id="id_like_example_0" type="radio" class="radioselect" /> Yes</label></li> <li><label for="id_like_example_1"><input value="0" type="radio" class="radioselect" name="like_example" id="id_like_example_1" /> No</label></li> </ul> </div> <div id="div_id_reason" class="ctrlHolder "> <label for="id_reason" class="requiredField"> Why did you like it?<span class="asteriskField">*</span> </label> <input id="id_reason" class="textinput textInput" type="text" name="reason" maxlength="80" /> </div> <div id="div_id_rating" class="ctrlHolder "> <label for="id_rating" > Rating (from 0 to 10) </label> <input id="id_rating" type="text" name="rating" class="textinput textInput" /> </div> <div id="div_id_notes" class="ctrlHolder "> <label for="id_notes" > Additional notes or feedback </label> <textarea id="id_notes" class="textarea" rows="10" cols="40" name="notes"></textarea> </div> </fieldset> <div class="buttonHolder"> <input type="submit" name="register" value="Register" class="submit submitButton" id="submit-id-register"/> </div> </form>

Note that every field is wrapped by default in a div with an id. This makes it very easy to style any field using CSS.

Buttons are now Layout objects

Now, I will show another layout example that would go in the same place shown before:

from uni_form.helpers import Fieldset, Submit, Row, ButtonHolder, HTML (...) Layout( Row( HTML("""<p>Did you know that some layout objects can hold others within?</p>""") 'like_example', 'reason', ), MultiField("Additional information", 'rating', 'notes', ), ButtonHolder( HTML(""" {% if message %} <span class="notification">{{ message }}</span> {% endif %} """), Submit('save', 'Save'), ) )

You can wrap fields in divs, like Row. MultiField is a layout object that comes from a uni-form adaptation: instead of rendering errors surrounding inputs in boxes, it renders them in a list and boxes all inputs affected. Sometimes the best way to familiarize yourself with these layout objects is getting your hands dirty writing and rendering some layouts to see what they do. They are harder to explain than to understand once you see them working.

Note that HTML is a layout object that can be used to input arbitrary HTML code in the layout.

In this version of django-uni-form you can have buttons (Button, Hidden, Reset and Submit) as layout objects. The recommended way of doing so is having them wrapped in a ButtonHolder which renders to a <div class="buttonHolder">, that is, a div that uni-form CSS magically knows how to position.

Django-uni-form is more context-aware

In the last example, you've seen how HTML can now be part of a Django template while being fully context-aware, which means that it's got access to the full context of the template in which it's being rendered. This gives you some really nice extra power that I find very useful.

However, not only HTML is context-aware. Fieldset legends behave the same way:

Fieldset("Please fill in your data {{ user.username }}", 'like_example', [...] )

Setting DOM ids and classes in Layout objects

Now you can set DOM ids and classes in every layout object, using css_id and css_class. Let's see a couple of examples:

Submit('Save', 'Guardar', css_class='button white') Fieldset("legend", 'field1', 'field2', css_id="iLoveThisFieldset", css_class="fieldsets_that_rock" )

Also, the ids and classes are left untouched; they are not slugified anymore so you can use camelCase or whatever you want.

Formsets support

Now you can render formsets using django-uni-form. Let's see how:

from django.forms.models import formset_factory ExampleFeedbackFormSet = formset_factory(ExampleFeedbackForm, extra = 3) formset = ExampleFeedbackFormSet()

You can use {% uni_form %} tag the same way as with a form:

{% uni_form formset formset.form.helper %}

Note that you can still use a helper (in this case we are using the helper of the form used for building the formset). The main difference here is that helper attributes are applied to the form structure, while the layout is applied to the formset's forms. Rendering formsets injects some extra context in the layout rendering so that you can do things like:

HTML("{% if forloop.first %}Message displayed only in the first form of a formset forms list{% endif %}", Fielset("Item {{ forloop.counter }}", 'field-1', [...])

Django-uni-form fails silently

When django-uni-form encounters errors it failed in a very intrusive way. Now, by default, it will log errors and continue working if possible. A settings variable called UNIFORM_FAIL_SILENTLY has been added so that you can control this behavior. If you want django-uni-form to raise errors instead of logging, telling you what's going on when you are developing in debug mode, you can set it to:

UNIFORM_FAIL_SILENTLY = not DEBUG

Help texts are wrapped in divs

This means that in your form fields you can set your help_text to things like:

help_text = "<ul><li>First help note</li><li>Second help note<li></ul>"

Layouts and Layout objects are now dynamic

The first steps have been taken to make adapting a Layout or a Layout object on the go possible, so that django-uni-form plays nice with dynamic form generation. Version 0.8.1 will hopefully come with an API that makes handling this a breeze.

Conclusions

Django-uni-form is in active development. We are aiming to build the DRY-est form building solution for Django. It comes with better docs in RTD, better error messages, a new full testing suite with high coverage and new code carefully cleaned and optimized for performance.

We are already working on version 0.8.1 which will be released in less than a month from now, packed with new killer features.

Special thanks go to Javier Maestro for reviewing my writing in this article.

#django #django-uni-form #uni-form #python #0.8.0 #DRY #forms #layout

Be careful how you use static variables in forms

Working on django-uni-form I came across with a weird situation and it took me some time to figure out what was going on. I wouldn't say it was a bug, it was more a misuse or a documentation problem. After that, I realized other coders make the same mistake that was affecting us, so I thought about writing an article for warning Python/Django programmers.

If you are building a Django application in which you add attributes to a form, that are not form fields, like in this example:

class ExampleForm(forms.Form): description = forms.CharField( label = u"Short description", max_length = 40, ) type_of_form = 'dangerous'

You should know that every attribute that is not a form field, such as type_of_form, is static or a class-wide variable. Let's see what this means and please stay with me till the end, don't switch channel. If you handle a form instance in one of your views and change type_of_form value:

form = ExampleForm(request.POST) form.type_of_form = "secure"

Then type_of_form turned into an instance-level variable. Which means only that specific form instance has a type_of_form set to 'secure'. You can still access the class wide variable, using the class name:

>>> ExampleForm.type_of_form 'dangerous'

Things get complicated if we turn type_of_form into a list. For example:

class ExampleForm(forms.Form): description = forms.CharField( label = u"Short description", max_length = 40, ) type_of_form = []

If we work with the list without instantiating a new one, the list behaves like a Singleton, there is only one instance of that object. Now imagine someone posts the form once and we do:

form = ExampleForm(request.POST) >>> id(form.type_of_form) 140281019424640 form.type_of_form.append(1)

type_of_form is now the list [1] for this and every future object we instantiate. So if a second person posts to the same view:

form = ExampleForm(request.POST) >>> id(form.type_of_form) 140281019424640 >>> form.type_of_form [1]

If you see type_of_form is not an empty list, it's not instantiated again, we share the same object than before in memory, it persists. If we manually create a new list:

form.type_of_form = []

In that case we have created a class-level variable, but the class wide variable still exists and hasn't changed:

>>> ExampleForm.type_of_form [1]

Remember that next time we instantiate a form, unless we manually change the variable for a new object, type_of_form will be [1], It persists

form = ExampleForm(request.POST) >>> id(form.type_of_form) 140281019424640 >>> form.type_of_form [1]

Some of you, might be thinking, but everything in Python is an object, true. The only difference here is that if type_of_form is a string, as strings are immutable in Python, every time we manipulate it we create a new string, a new instance-level variable. But with a list or any other object, it's very easy to get tempted to manipulate the class-wide variable, without instantiating a new one, if done without knowing what we do, it will provoke side effects.

Of course, this not a Django bug, this is how Python classes like Django forms work. The problem is many coders use Django forms this way without being aware of what they do. Therefore when doing this in your open source application, try to be clear in your docs and use static variables where due.

Real example

This is the way Django-uni-form advised to add helpers to a form:

class ExampleForm(forms.Form): [...] helper = FormHelper() helper.form_id = 'publishTransportSecondStep'

That made it easy for someone to manipulate the helper the wrong way, doing:

def view(request): # Create the form form = MyForm() helper.add_input(Reset('reset','reset button'))

See the problem? Now we have added to helper, which behaves like a Singleton, a new button. That button will be shown in every form of MyForm type we create. Every time we go into the view, a new button will be added, so the form will end up having tons of buttons. Thus we changed the advised way of adding helpers to forms.

This way, if we manipulate the helper in a form, we do it for that specific form helper. But sometimes people don't manipulate helpers in views, or if they do, they do it the right way, thus, some people might want to have helpers as static variables, for improving performance and memory footprint.

Solution

Conclusion is that you should use static variables carefully and in the situations they should be used, always understanding what you do. If you don't want a variable to be static, wrap it in the constructor and turn it into an instance-level variable:

class ExampleForm(forms.Form): description = forms.CharField( label = u"Short description", max_length = 40, ) def __init__(self, *args, **kwargs): self.type_of_form = [] super(ExampleForm, self).__init__(*args, **kwargs)

Note that turning type_of_form into a property will not solve the problem:

class ExampleForm(forms.Form): description = forms.CharField( label = u"Short description", max_length = 40, ) @property def type_of_form(self): return []

If you do something like this:

form.helper.append(1) form.helper.append(2) >>> form.helper []

In this example, every time you call form.helper you get a new list, a new object in memory, so changes don't persist. So wrapping a variable in a property should not be used as an alternative to instance-level variables.

Thanks to Jonathan Stoppani for his corrections and help.

#django #forms #python #static

Using jQuery templating ICanHaz.js with Django

When working on a Django project with an important part of the user experience based on AJAX, you better use a frontend templating language. The same way Django has a templating language that we use to populate values easing your coder's life, jQuery has client side alternatives. One of these plugins and my personal favorite is ICanHaz.js created by @HenrikJoreteg. It's only 4.4 Kb.

ICanHaz.js syntax will be very familiar to Django devs, because a variable is defined using double curly brackets {{ variable }}. But I will not enter into details, the project has very easy to follow setup instructions and code examples.

When people start using ICanHaz.js in their Django templates, they have a very hard first time trying to get it to work. The problem is simple, Django and ICanHaz.js share the same variable definition, so Django interprets the script templates, breaking them.

When I found this out, I looked over the Internet until I found a fix by Eric Florenzano (@ericflo). He coded a templatetag called verbatim, that avoids that Django interprets all content within the block tag. He published it as a gist in Github. You simply have to put this within one of your app's templatetags directory. The way you use it in a template:

{% verbatim %} <script id="user" type="text/html"> <li> <p class="name">Hello I'm {{ name }}</p> <p><a href="http://twitter.com/{{ twitter }}">@{{ twitter }}</a></p> </li> </script> {% endverbatim %}

simple, right? Well, I recently work on an enhancement for this gist. Eric Florenzano's version of the verbatim tag escapes everything, which means that you can't use tags like {% url %} {% csrf_token %} or others. These are tags and not variables, so I didn't see any reason for not having them around. Therefore I forked Ericflo's gist and added tags support. Now you can do:

{% verbatim %} <script id="user" type="text/html"> <a href="{% url addItem %}">Add Item</a> </script> {% endverbatim %}

Working on this wasn't as easy as other tags I have worked on. I had to go inside the guts of django.template.defaulttags and django.template.base (former __init__.py in Django 1.2) to understand how the Lexer, Token, Parser, Filter/Tag, Node, NodeList, Template structure in Django works. I couldn't just simply parse the Tokens and build a NodeList, I had to write my own template parser and I based my code on Parser.parse. Pretty much I extracted the bits I needed from there.

#django #jquery #template #icanhaz.js

Django PositiveNormalizedDecimalField

Some weeks ago I was working on a Django project that was using some FloatFields where they should not be used, the database backend was Postgres 8.4. If you are not aware of it, FloatFields are turned into double precission types, that means 64-bit floating-point numbers. Those fields were storing values that didn't need such precission. For example I don't think you need them to store items' weights, do you?

I migrated those fields to DecimalFields using an average (max_digits=7, decimal_places = 2), which means a total of 7 digits, with 2 decimals. Of course I had to migrate the database, doing some manual alters, as Django-south was not being used. By default DecimalFields render the number with all its decimal digits, which was not what I was looking for. I needed a decimal field that:

By default render decimal values normalized using non-scientific notation. See Decimal("10.400").normalize()

Only allowed positive values

I could have used floatformat built-in filter, but that would have supposed to change hundreds of different templates. Even though grep and sed are a big allies, the process would be quite error prone. I simply needed a centralized solution. Obviously It was time to create a custom model field, I decided to call it PositiveNormalizedDecimalField. I started reading Writing custom model fields, which gave me a good idea of how things work.

I decided to start subclassing DecimalField and rendering values normalized. I started overwritting to_python method. From the docs I understood to_python was in charge of turning the value stored in the database into a python object. But things were not working. This is the code I had at that time

class PositiveNormalizedDecimalField(models.DecimalField): def to_python(self, value): if value is None: return value try: # we want to return a normalized non-scientific notation decimal return decimal.Decimal('{0:f}'.format(decimal.Decimal(str(value)).normalize()))

However, to_python wasn't working as django docs prayed. So I thought that maybe Django had a bug, honestly I was pretty much sure I was doing something wrong. So I got into Django's code and put some breakpoints using pdb, don't do this in your production server. Then I found out that to_python was not even being called, damn! So I filled a ticket for documentation improvement quite angry after loosing some hours on this.

Thanks to Gabriel Hurley I found out that I needed to use the metaclass SubFieldBase to have to_python being called. That finally made the trick and I got it working. Here is the final code:

class PositiveNormalizedDecimalField(models.DecimalField): __metaclass__ = models.SubfieldBase default_error_messages = { 'positive': _('Introduce a positive number.'), } def to_python(self, value): if value is None: return value try: # we want to return a normalized non-scientific notation decimal number = decimal.Decimal('{0:f}'.format(decimal.Decimal(str(value)).normalize())) # This ensures no negative number can get to the model (model validation) if number < decimal.Decimal(0): raise exceptions.ValidationError(self.error_messages['positive']) return number except decimal.InvalidOperation: raise exceptions.ValidationError(self.error_messages['invalid'])

After talking to some developers in the django IRC channel I realized adding a field like this into the core was not going to make it. So instead I'm publishing it here and I might be releasing a Django-app in the future with some more model fields.

Some Built-in fields such as PositiveIntegerField have a database CHECK constraint to allow only positive numbers, that constraint is located at django/db/backends/*.py This could be another option to check positive values, however modyfing Django's code without using official accepted patches is not a recommended option.

This is also a good example in any case you need to redefine default rendering behavior of a model field. Bare in mind, that Django use of to_python is a little bit inconsistent. So when doing aggregate queries to get minimums or maximums, you will need to filter output manually.

#decimalfield #django #field #model #positivenormalizeddecimalfield #to_python

Last hints on Erlang

Soon I will start writing some posts on Django, my current passion, in which I work fulltime at the moment. But before, I really want you to understand how Erlang real applications are built. If you look at Wikipedia's Erlang article section: Project's using Erlang, is simple to understand Erlang is used for what is was built, concurrency. These projects have a common denominator. Most need to be real-time, they need to be efficient and distributed. Erlang excels at this.

How do I build my GUI?

When people start learning Erlang, they feel the necessity of building some sort of GUI (Graphical User Interface). Since Erlang's R13B release, Erlang has a GUI built-in library called wxErlang, which can be a good option building a desktop application that needs multicore usage or controls a distributed cluster.

However, most of the times Erlang applications are built-in using a server-client architecture. We code a web server in Erlang that expects requests coming to be in a specific protocol. For example ejabberd expects requests coming in XMPP protocol (Extensible Messaging and Presence Protocol), which is based on XML. Other Erlang applications are RESTful or use SOAP. So many times, we have a frontend application, that draws the GUI, that communicates with the Erlang application using RPC or raw sockets. So you can build GUIs using java libraries, C#, Python, etc. or you can create web services.

Why so I do this effort?

Of course not everyone needs their applications to be distributed or concurrent. I would say everyone would love to have fault tolerant applications. So if you got here, after reading all these articles and you still wonder why should I write a server, then a frontend GUI or web service and so on. That's because you didn't pay attention. If you need what Erlang offers, you should pick Erlang.

There are two Erlang's features that really impressed me when I started:

One was the ability to hot load code without stopping the application.

The other was being able to restart little portions of an application, if something failed you didn't need to restart everything, just that part. And if it continued failing you could have the rest working until you came up with a hotfix.

Open Source Erlang projects I can look into

The bulk of Erlang projects and Erlang hackers have moved to Github and I can understand why, but my Github love can wait for another post. These are 5 Erlang projects that you should know:

Erlang/OTP itself

Nitrogen: A web framework in Erlang. There are some demos to see it working

CouchDB the DB that is getting very known

RabbitMQ is a great messaging system

Ejjaberd A Jabber instant messaging server famous because Facebook used it to built its chat application

But you didn't talk about...

Well, there are many many topics I didn't cover or mention, remember this was not going to be a book on Erlang, just a teaser. Hopefully I left a path so you can do it and you understand Erlang's phylosophy and when to use it. Some important topics I haven't mentioned are:

Records data types

ETS and DETS large date storage mechanisms

The great database Mnesia, which can be replicated through distributed nodes easily, or can run in RAM for performance.

How to officialy documment the code

How to interface with other programming languages using sockets

Much more on OTP and Erlang's shell

Error handling and supervisions techniques

Security and cookies

The last pointers

An online Erlang interpreter that comes with a hands-on tutorial

Erlang community website

Erlang Programming Another good book, that covers some topics not seen in other books

#couchdb #dets #ejjaberd #erlang #ets #mnesia #nitrogen #otp #rabbitmq #rpc

Real Erlang means OTP

After this series of posts about Erlang: 1, 2 & 3, there is too much left to say. Erlang is a wonderful language built in for distributed fault tolerant apps. But Erlang's magic needs your help. Erlang won't do anything of this if you don't use it carefuly and smart.

Erlang starts to be handy when you really know Erlang. It comes packed with a framework called OTP (Open Telecom Platform), thus Erlang is sometimes referred as Erlang/OTP. You can find a great explanation about it in What is OTP? which is part of a great online tutorial called learnYouSomeErlang that I highly recommend for new comers.

So when real Erlang hackers code Erlang, they most of the time code Erlang using OTP, they code behaviours. Behaviours are not an Erlang concept, if you know other programming languages, you will find them easy to follow. As a Djangonaut I would say behaviours are like swappable backends, you need to implement some defined functions that will get called when you change the backend module. A more general explanation, would be they are interfaces used through callbacks. Of course the best way to fully understand them is seeing a good example. You can find one in OTP design principles Behaviours overview

The most important behaviours are supervisors, gen_server and gen_fsm. Supervisors are one of the most important resources to turn your Erlang fault tolerant as the propaganda prays. The next two are interfaces to create server-client processes and finite state machines.

Erlang online docs are more than decent and they should be your first reference point. One thing I come across when I talk to someone with little knowledge of Erlang, is that they try to think and lay out and architecture for an Erlang system based on former knowledge. Don't forget Erlang is not ADA, is not LISP or Caml. You need to know Erlang in deep to really appreciate its complexity and its concurrent nature.

#OTP #behaviours #desing #erlang #gen_fsm #gen_server #principles #superviros

Getting into some basic concurrency in Erlang

Raising the bets

In last article, first baby steps in Erlang. We wrote our first module, a simple add function and our first recursive function count_list_items. The main power Erlang has comes from the concurrent part of it, so we will get into creating processes and assingning them some tasks.

Don't forget these series of articles are just an Erlang teaser, so you get to see some of Erlang's power and start reading more about it.

Erlang is structured in modules. Every module can contain public (exported) and private functions, as we have seen. Besides this we can create processes that run functions. Open Erlang's shell and create your first process, here is how:

erl > Pid = spawn(io,format,["Hello world! ~n", []]). Hello world!

spawn function is in charge of creating a process that runs the function passed. spawn/3 expects these parameters: Module, function, list_of_parameters. So what we are doing here is creating a process that prints a "Hello world!" message and then dies. Note that spawn returns the pid (process identifier).

Let's do something more complicated

You might think that Erlang has some nice features but you don't like the trade of dropping imperative languages and using locks for concurrency. Well, if you think that, I find it normal but with time you will realize how Erlang is much more powerful and useful for real solutions.

Now we are told to code a function that looks for a pattern within a string, returns atom match if it finds it or atom no_match otherwise. We will call this function find and as you can deduce it only needs 2 arguments: String and Pattern. So it's got an arity of 2. Here is a possible solution to this problem with a hidden bug (see if you can spot it).

-module(prueba). % This exports all functions so they can be used out of the module -compile(export_all). % Returns match if Pattern is part of the String, false otherwise % the find/2 is a wrapper that makes sure lists are passed % and calls find/3 which has some sort of accumulator parameter find(String, Pattern) when is_list(String), is_list(Pattern) -> find(String, Pattern, Pattern). find(String, [], []) -> match; find(String, [], Pattern) -> match; find([], [], Pattern) -> match; find([], Pattern, Pattern) -> no_match; find([H|String], [H|TPattern], Pattern) -> find(String, TPattern, Pattern); find([H|String], [PH|TPattern], Pattern) -> find(String, Pattern, Pattern).

If you are a newbie in Erlang, take some time reading the code, hopefully you will see how it works. Basically there are 3 parameters, the String, a dynamic Pattern and a static Pattern. The dynamic Pattern checks if the head of the string is equal to its head:

find([H|String], [H|TPattern], Pattern) -> find(String, TPattern, Pattern);

If heads are equal we could have for example:

find(["a"|"bcdef"], ["a"|"h"], "ah") -> find("bcdef", "h", "ah")

So next step will fall into:

find(["b"|"cdef"], ["h"|], "ah")

These times the heads are not equal, so the pattern matching falls into:

find([H|String], [PH|TPattern], Pattern) -> find(String, Pattern, Pattern).

Which kind of restarts the process:

find(["b"|"cdef"], ["h"|], "ah") -> find("cdef", "ah, "ah")

If you are an avid reader you have probably spotted a bug in the code. What happens if the String was "aah" and the Pattern "ah"?

erl > c(prueba). {ok,prueba} > prueba:find("aah", "ah") no_match

Right, the code will say there is no match. We have restarted the Pattern, but we should be looking in the whole string, not only its tail. If you are thinking about changing the clause to something like:

find([H|String], [PH|TPattern], Pattern) -> find([H|String], Pattern, Pattern).

I have to warn you that this will put you in a shoot yourself in the foot situation, an infinite loop, as Pattern never pops items, nor the String. So the solution is to add a clause and be more fine granular:

find(String, [], []) -> match; find(String, [], Pattern) -> match; find([], [], Pattern) -> match; find([], Pattern, Pattern) -> no_match; % first letter of the String matches first of the accumulator pattern % Pop bot letters and continue checking find([H|String], [H|TPattern], Pattern) -> find(String, TPattern, Pattern); % The first letter doesn't match and the acc pattern is equal to the pattern find([H|String], [PH|TPattern], [PH|TPattern]) -> find(String, [PH|TPattern], [PH|TPattern]); % The letters don't match but the acc patern does not match the pattern % This is the clause we were missing, we pass the whole string and restart find([H|String], [PH|TPattern], [PatternH|TailPattern]) -> find([H|String], [PatternH|TailPattern], [PatternH|TailPattern]).

The order of the clauses here is also very important, think about it. Of course understanding functional recursion is quite hard in the beginning, but if you think it carefuly this way of programming is a lot more close to the way humans think. Only our corrupted by imperative languages minds find it hard.

Add some concurrency, spice it up

Now we are going to create a loop function. This is used to handle a concurrent request and keep the process alive. We add to our module prueba these lines of code

% This function spawns a process and sends it a request % using rpc function find_concurrent(String, Pattern) -> Pid = start(), rpc(Pid, {find, String, Pattern}). % Creates a process running function string_worker start() -> spawn(prueba, string_worker, []). % Remote procedure call % sends a request and listens for an answer from that process rpc(Pid, Request) -> Pid ! {self(), Request}, receive {Pid, Response} -> Response end. % Encapsulates the find function in a loop string_worker() -> receive {From, {find, String, Pattern}} -> io:format("Received. String: ~p Pattern: ~p~n", [String, Pattern]), From ! {self(),find(String, Pattern)}, string_worker(); % Anything different than the first clause stops the process Any -> void end.

A lot of new code appears in these lines. But if you are in Erlang, it's because you probably know some other programming language well, so you will probably understand most of it. Even though Erlang's syntaxis looks like Prolog and it's ugly, you get used to it. The ! operator is the send message operator. It is used as:

pid_destination ! message

When we call start function, it spawns a process running string_worker, which listens for messages to arrive using receive ... end. directives. The rpc function sends this spawned new process a request conformed by a tuple of tuples {pid_sending_message, {find, String, Pattern}}. self() built-in function (BIF) returns the PID of the process that calls it. After sending the request it starts listening for a response from that single process, not any other process. Let's run it in the shell:

> c(prueba). > prueba:find_concurrent("asdfaah","sed"). Received. String: "asdfaah" Pattern: "sed" no_match > prueba:find_concurrent("asdfaah","ah"). Received. String: "asdfaah" Pattern: "ah" match

It is left for the reader to read more on Erlang's concurrency. But if you are focused, you probably have noticed, that the process spawned never dies. In general, in Erlang we will want to create just on process of every different type

#ada #caml #concurrency #erlang #haskel #lisp #recursion #recursive #tail

First baby steps in Erlang

Erlang is distributed by itself

Sure a statement like this echoes in many heads. Erlang has location transparency, which means that no matter how you setup your erlang environment, it will run. When you do concurrency using locks, even if you get the locking right, you still will run into trouble if a thread is moved out of the same machine, as the communication scheme has changed a lot. That involves refactoring concurrent code that will probably take years to refine and weed out of bugs.

Erlang code can be run the same way in a laptop than in a cluster. That's because of the actor model I talked about. Processes don't need to know where they are, where others are to run. Scalability becomes a problem so much easier to attack. Things like how I split my program into parts that can be distributed or replicated, how should they communicate and many others are resolved in Erlang built-in.

Let's start with the real deal

If you read the books recommended in my refreshing erlang article and you fully understand and master them, you will know more than me :) and you will become quite a good Erlang hacker. But mastery comes from practice. I would love to show advanced code, but we need to start with some simple examples, so people don't get scared too fast.

Tail recursion

As I said before, recursion is the path of the master. But there are two kinds of recursive functions and understanding the difference is tremendously important:

sum(N) -> sum(N-1) + N. do_sum(0, Total) -> Total. do_sum(N, Total) -> do_sum(N-1, Total+N).

First one sum is a non-tail recursive function. Second one do_sum is a tail recursive function. Tail functions do "tail calls", these ones are easy to spot by a compiler. If you know how recursion works, you are probably aware of the stack that builds up with more and more memory used, the bookeeping problem, keeping track of what to do later and what goes next. This steals efficiency and memory. If Erlang code is going to run forever, the system cannot run out of memory

Erlang does not run out of memory because it does tail call optimization which means that the compiler only keeps in the stack the last current information, as it does not need anything else, so the stack doesn't grow. This makes recursive tail functions extremely efficient. You should always rely on tail call optimization

Show me the code

Let's create our first module in which we will put some functions. The module name will be "prueba", so the file needs to be named prueba.erl

% This a comment in Erlang. % Next line defines this file as an Erlang module -module(prueba). % Next line makes these functions public, available out of the module % It's a list with the [function_names/arity] -export([add/2]). % Our first function, which adds 2 numbers, dead simple add(A, B) -> A + B.

Now let's get into Erlang's shell, which is more than a typical shell. It's also the debugger and much more. We need to run it in the same path as our module:

erl > % comments don't make sense in the shell :) > Let's compile and load our module > c(prueba). {ok,prueba} > % ok means all went right, no typos or syntax errors > % Let's call our add function module:fun(...) > prueba:add(4,5). 9 > prueba:add(2,2) - prueba:add(1,0). 3

It's time to build our first recursive tail function. We can start coding one that counts items in a list (This code would be added after add function):

% Counts items of a list % Beware the first function ends with ";" % This is a clause and it's part of the same function count_items([], Total) -> Total; count_items([H|T], Total) -> count_items(T, Total+1).

count_items is made out of two clauses separated by a semicolon. When you call the function it does a top down pattern matching, returning the first match. So we have included a base case (when the list is empty) in charge of ending recursion. The [H|T] functional operator does the rest, this is the functional magic in place.

A typical beginner mistake would be to put T (the tail) in between brackets [T]:

count_items([], Total) -> Total; count_items([H|T], Total) -> count_items([T], Total+1).

As tail is already a list, what you are really doing is [[T]] thus creating an infinite loop, boom! There is room for improvement in my code. We are assigning H the head of the list and not using it. This will prompt:

./prueba.erl:11: Warning: variable 'H' is unused

So we should be using an anonymous match operator "_":

count_items([_|T], Total) -> count_items([T], Total+1).

Try count_items function. Total is an accumulator parameter. It stores the value accumulated that in the end will be the result. Of course it needs to start being zero:

> % recompile module > c(prueba). {ok,prueba} > prueba:count_items([1,2,3,4], 0) 4

Let's wrap it up, as we only want to pass one argument: the list. We don't need to know anything about what the accumulator parameter is expected to be. So we wrap count_items in another function called count_list_items which has a guard that checks the parameter passed is a list:

% Counts items of a list % Beware that count_list_items ends with a dot % is_list(...) is a guard. There are many more: is_atom(...), is_integer(...), is_number(...) etc. count_list_items(L) when is_list(L) -> count_items(L, 0). count_items([], Total) -> Total; count_items([H|T], Total) -> count_items(T, Total+1).

Let's export our new function:

-export([add/2, count_list_items/1]).

count_items is now a private function within the module, as there is no need to export it. See how it works:

> prueba:count_list_items(5). ** exception error: no function clause matching prueba:count_list_items(5) > prueba:count_list_items({5,4}). ** exception error: no function clause matching prueba:count_list_items({5,4}) > prueba:count_list_items([1,2,3,4]). 4

#concurrency #concurrent #distributed #erlang #recursion

Refreshing Erlang

Erlang is a functional concurrent programming language. It is very different to what you probably use in your day to day as a programming language. Back in San Diego I took an Erlang programming course and I found it very interesting. Erlang programmers are like gnomes hiding in the deep of the forest, like magic unicorns, some people say they exist.

If you haven't heard of it, this is probably a good time. If you have coded concurrent languages like ADA or functional languages like Caml, Haskel or LISP, you know a mixture of the two paradigms has to be a hard mental challenge, it would be hard to make a post on how to code Erlang from scratch. But I will cover some basics, so you can know some more about this interesting language.

Why Erlang rocks

You can do fault-tolerant applications that can be modified without taking them out of service. It supports hot-swapping

It's been in use in real large-scale industrial products, like telecommunications.

Wikipedia's erlang article

Why is Erlang different?

Variables are all Uppercase and they are not really variables, stay with me and you will know why.

Every statement ends with a dot. Syntax is weird and if you love Python's syntax, you won't find it here.

Variables are like math variables in an equation. X = X + 1 you can't do this in Erlang. When you assign a value, you can't change it. So variables are not variables, and = operator is almost an assignment operator. They are called single assignment variables, they can be bound once and they are created unbound. Of course this is done in purpose, so forget imperative languages prejudices. A variable is kind of a pointer to a value that cannot change. If you know concurrent programming, you know how hard can be to debug a program, making variables static, makes your debugging pain more bearable. That's why Erlang is known as a nonmutable state language. You might be wondering, how can I code in Erlang with this sort of variables? Answer could be "be recursive my friend, find new names for your variables and control scope"

Erlang has something called atoms, atoms are non-numerical global constant values. Atoms are all lowercase, so forget using beautiful camelcase code style.

Erlang comes geared with tuples (unfortunately they use curly brackets, so Python coders like me suffer extra pain) P = {15, 20}

The = operator is considered a pattern matching operator. This is very important, because this is the way we extract values from tuples or other data structures. Code example:

> Point = {20, 10}. {20,10}. > X, Y = Point. {20, 10} > X. 20 > Y. 10

The underscore _ is called anonymous variable. It means we are not interested in this value. Code example:

>{_, {_, Who}} = {23, {name, miguel}}. {23, {name, miguel}} > Who. miguel

Lists are created with square brackets. The lists are the salsa of every functional language. [1, 2, 3, 4] the first item of a list is called head, that would be 1. The tail is the rest of the list [2, 3, 4]. If T is a list then [H|T] is also a list. H is the head and T is the tail.

> [X,Y|Z]=[a,b,c,d,e,f]. > X. a > Y. b > Z. [c, d, e, f]

Strings in Erlang are lists of integers. Besides this, Erlang is used in software like ejabberd

Of course this is only the peak of the iceberg, Erlang has many idiosyncrasies.

Concurrency made simple

It is made simple, simple as it can be made. Concurrency has never been easy and something in our brains crack when doing it. When I started learning concurrency, I was coding ADA and ADA took some time to be digestible and thank god the little book of semaphores was there to make digestion easier.

However Erlang approach to concurrency is very different and a lot simpler. It uses something called the actor model. Erlang doesn't let you create threads that share memory among them. Instead it creates ligthweight processes that communicate with each other passing messages. This is similar to how people interact in real life, they talk to each other. So you will not have to do locks or semaphores for managing concurrency.

Installing Erlang

In ubuntu you just need to do:

sudo apt-get install erlang

Then you can run the shell with erl command and the compiler with erlc. Let's do a hello world (it creates a module and defines and exports 2 functions). First create a greeting.erl file, then paste this code:

-module(greeting). -export([hello/0, print_hello/0]). hello() -> "Hello World". print_hello() -> io:format("~w ~n", [hello()]).

The way you run Erlang code is doing erl -run Module function. So in this case we would do:

erl -run greeting print_hello [72,101,108,108,111,32,87,111,114,108,100]

Good books to continue learning

Programming Erlang: Software for a concurrent world by Joe Armstrong

Erlang and OTP in action by Martin Logan, Eric Merritt, Richard Carlsson

Hopefully I will be posting some code examples in the next few days, showing some Erlang's goodies.

#ada #caml #code #concurrency #erlang #functional #haskel #lisp

New posts on Django coming

I have been a while without posting anything in the blog. I have been very busy lately working on my startup and doing some contributions to Open Source Django-apps as you can see in my Github. Today I installed a Ruby Gnome app called Grumblr that I'm testing right now, so posting becomes easier. Also I have changed CSS and added prettify code highlighting, because I'm planning on posting some articles on Django. Hopefully trying to cover some obscure parts in the docs and some interesting tips and tricks.

Little tip on how to reset ipdb

When I code Django, I leave a server running in a Linux shell.

python manage.py runserver

Sometimes when I use ipdb to debug some code, the screen input disappears and I can not see what I'm typing. Rather than ^C the server, I prefer to fix the problem. You just need to reset the screen:

import os os.system('reset')

#django #ipdb #ipython #runserver

The making of Ephemicropolis

100,000 staples arranged over 40 hours.

#vimeo #video #unbelievable #staples #amazing #acropolis

Trending Blogs

Recently Viewed Blogs

toThinkOrNotToThink