Confident And Wrong @confidentandwrong-blog - Tumblr Blog

List comprehensions break my brain, but they're fast

I asked a stackoverflow question, http://stackoverflow.com/questions/24792257/how-to-write-regular-expressions-to-match-white-space-delimited-multi-line-colum that contained these three brain cracking list comprehensions,

whitespaces = [i for l in content for i, char in enumerate(l) if char == ' '] whi_columns = [k for k, v in Counter(whitespaces).iteritems() if v == no_lines] data_columns = [[line[col].strip() for line in content] for col in columns_defs]

Which immediately punted my comprehension out the window. The worst part is that these list comprehensions is that they work as intended.

The first list comprehension,

whitespaces = [i for l in content for i, char in enumerate(l) if char == ' ']

can be broken into two for loops whitespace_indexes = []

for line in content:

for char_index, char in enumerate(line):

if char == ' ':

whitespace_indexes.append(char_index)

The second comprehension,

whi_columns = [k for k, v in Counter(whitespaces).iteritems() if v == no_lines]

can be broken into a single for loop, for num in Counter(whitespaces).iteritems():

# Counter will sum up how often a white_space index is present.

if num[1] == number_of_lines:

# If a whitespace index shows up for the total number of lines then it’s a column.

white_columns.append(num[0])

And lastly, there was this little beauty which reminded me of a nested ternary conditional

data_columns = [[line[col].strip() for line in content] for col in columns_defs]

can be expressed as two for loops,

data_columns = []

for col in columns_defs:

sub_list = []

for line in content:

sub_list.append(line[col].strip())

data_columns.append(sub_list) It turns out the comprehension portion of these statements was not the hard part. What I didn’t, ultimately, understand was the comma operator. In each of those list comprehensions there’s a section after the `for` keyword which has a comma. That comma returns a tuple. Here’s an example foo = [1, 2, 3]

bar = [4, 5, 6]

for tuple in foo, bar:

print(tuple) # tuple == ( [1, 2, 3], [4, 5, 6])

# prints

# [1, 2, 3] # [4, 5, 6] will return the tuple ([1, 2, 3], [4, 5, 6]) in one iteration. You can have mis-matched tuples which I think is super awesome.

foo = [1, 2, 3]

bar = ([4, 5, 6], [7, 8, 9])

for tuple in foo, bar:

print(tuple) # tuple == ( [1, 2, 3], ([4, 5, 6], [7, 8, 9])) # prints

#[1, 2, 3] #([4, 5, 6], [7, 8, 9])

Most of the programmers at work (C programmers who dabble in Python, mainly) Have a hard time initially understanding these comprehensions. Which to me means that these are too complex unless you’re dealing with people well versed in the language. These kinds of list comprehensions really shouldn’t be your first choice. They reduce code readability. You also can’t step through them using PDB which makes debugging them a... challenge. But they are very fast. I ran the double for loop example and it’s counter list comprehension through timeit.

alord@alord-desktop:comprehension :) $ python for_loop.py

2.46502590179

alord@alord-desktop:comprehension :) $ python comprehension.py

1.67403912544

Here’s the code I used:

------------------------------------------------- For Loop version ---------------------------------------------------

import timeit

print(timeit.timeit(

"""

content = ['------------------------------------------------------------------------------', 's200_13tb_400gb 1 +3 system, vhs_de 1:0-23, 1 53T / 218T (25% )', '-ssd_48gb-ram ny_writes, vhs 2:0-23, 3:0- ', ' _hide_spare, 1,3-19,21-25 ', ' ssd_metadata , 4:0-23, ', ' 5:0-23, ', ' 6:0-23, ', ' 7:0-23, ', ' 8:0-23, ', ' 9:0-23, ', ' 10:0-23, ', ' 11:0-23, ', ' 12:0-23, ', ' 13:0-23, ', ' 14:0-23, ', ' 15:0-23, ', ' 16:0-23, ', ' 17:0-23, ', ' 18:2-25 ', 'Unprovisioned drives: none', '']

whitespace_indexes = []

for line in content:

for char_index, char in enumerate(line):

if char == ' ':

whitespace_indexes.append(char_index)

""",

number=10000))

---------------------------------------------- Comprehension version --------------------------------------------- import timeit

print(timeit.timeit(

"""

whitespaces = [char_index for character in

content for char_index, char in enumerate(character) if char == ' ']

""",

number=10000))

so ~1.5 times slower to do the full for loop in this case. I will say that after having written this blog post I understand how the comma operator and iteration work much more comprehensively. Every time I’ve seen a comma separated iterator I’ve understood what was happening much faster than this first time. But this first time threw me for such a loop that I stand by my stance that list comprehensions like this reduce readability. So in conclusion, List comprehensions break my brain but they're also fast.

Learning to not hate python regular expressions

I have a secret to share. It’s not an airplane-sized secret but it’s also not mouse-sized either. More antelope than anything else. I like regular expressions. There, I said it. No, I won’t take that back, thank you very much. Some programmers out there will be saying, “Well of course you like regular expressions, you’re a programmer.” Others of you will hear the quote: “Some people, when confronted with a problem, think ‘I know, I'll use regular expressions.’ Now they have two problems.” - Jamie Zawinski And I used to be one of you, two problems people. Trust me, I know how you feel, but I have been shown the light. I have seen the other side of the tunnel and I have discovered that regex can, in fact, solve useful problems. I know that sounds about as appealing as eating a fist full of bark to those two problems people out there, but stick with me on this one. I got handed this mess about three weeks ago: Cluster_Totals:.*?([0-9.]*[KMGTPE])\s*/\s*([0-9.]*[KMGTPE])$\<\s[0-9.]*\%$\|\s([0-9.]*[KMGTPE])\/\s([0-9.]*[KMGTPE]) As someone who disliked regex and avoided using it in his own code, this was a nightmare. This is the stuff that Disney witches brew in cauldrons to sprinkle on unsuspecting baby dwarfs to make their beards fall off. I took one look at that mess and almost rewrote 400 lines of code just to avoid one line of regex. First of all, if you ever catch yourself saying, “I could rewrite this better from scratch,” you should read Joel Spolsky’s work in order to save yourself a lot of pain.

http://www.joelonsoftware.com/articles/fog0000000069.html

That article lays out why this is normally a really dumb idea better than I ever will. But I speak a tautology because Spolsky makes me feel as articulate as a drooling rock. But second of all (and more importantly), regular expressions are not that scary. The first thing I did that really helped out was finding an interactive source that allowed me to toy with my regular expression and see exactly what matched and didn’t match easily. In this case I lucked out and found a web application that does just that:

https://pythex.org/

I also used: http://www.pythonregex.com/ Because it showed what running several different python regex methods would produce. Next, I got a bunch of examples from the data sets that I wanted to get my matches from and put them into my unit tests. I ended up with test strings that looked like: 11|None |-A-- | 14K| 0| 14K| 32G/ 93T(< 1%)| (No SSDs)

-------------------+-----+-----+-----+-----+-----------------+-----------------

Cluster Totals: | 439K| 11M| 12M| 2.2T/ 323T(< 1%)| 2.1G/ 1.4T(< 1%)

and produced expected dictionaries that looked like:

{‘hdd_used’:’2.2T’, ‘hdd_total’:’323T’, ‘ssd_used’:’2.1G’, ‘ssd_total’:’1.4T’}

{‘hdd_used’:’2.2T’, ‘hdd_total’:’323T’, ‘ssd_used’:None, ‘ssd_total’:’None} I can not express how useful writing all of those unit tests were for learning how to properly use regular expressions, especially how regex groups work. Writing the unit tests before the code forced me to think in terms of what input lead to what output and this change in mindset made things vastly easier. The last thing that I did was to start breaking apart the original regex looking for patterns. As a refresher: Cluster_Totals:.*?([0-9.]*[KMGTPE])\s*/\s*([0-9.]*[KMGTPE])$\<\s[0-9.]*\%$\|\s([0-9.]*[KMGTPE])\/\s([0-9.]*[KMGTPE]) The first thing that really popped out was the ([0-9.]*[KMGTPE]) sections and the Cluster_Totals:.*? section. I started splitting the regular expression up into chunks. preamble = Cluster_Totals:.*? totals_grouping = ([0-9.]*[KMGTPE])\s*/\s*([0-9.]*[KMGTPE]) which lead to

regex.compile(preamble + totals_grouping + “(\<\s[0-9.]*\%\)\|\s” + totals_grouping) Which leads me to why I originally had to tackle this regex in the first place. There were many log sets that I was attempting to run this search over which contained data that looked like the example I gave but contained subtle differences in the middle formatting. If the formatting in the middle didn’t match, then the regex would just ignore the match. There were also some cases where the last grouping didn’t exist. So I changed the final result to look like

ssd_regex = regex.compile(preamble + totals_grouping + “.*?” + totals_grouping) hdd_only_regex = regex.compile(preamble + totals_grouping + “.*?” + $No SSD$) One little tip that I learned the hard way is to always include a ? at the end of a splat (*) operator unless you want that sucker to soak up every last bit of text it can find. Also, since the splat operator will cause the regex engine to attempt a greedy algorithm to match every character, you’ll significantly increase the computation time of any match or search by not including the ? operator.

I’ve also been using regex to check to see if text from web forms matches the formatting that I’ve been expecting. Example being:

re.findall("((http|https)://legacy.*/)", field.data) It’s a very simple task but it uses significantly less code than if I attempt to do multiple string.find() operations. There we have it. Regex can be about as intimidating as a fire-breathing dragon but once you get close enough, you realize it’s a tiny fire-breathing dragon.

#programming #regex

PyCon 2014

So, I went to PyCon US this year. The short of it was that PyCon is amazing. Easily the best organized convention I’ve ever been to. I haven’t been to too many tech conventions, so that may not be saying much.

PyCon US is partitioned into Tutorials, Talks, and Sprints . This year there were two days of tutorials, three days of talks, and four days of sprints. The tutorials are two blocks of three hours classes. The first two days of talks had a Keynote at 9AM until 10AM and then five different speakers every 45 minutes from 11AM to 5:40 PM. The topics covered everything from machine learning, to garbage collection, to security, to analyzing rap lyrics.

Here’s the tutorial (https://us.pycon.org/2014/schedule/tutorials/) and talk (.pycon.org/2014/schedule/talks/) schedules if you want to get a feel for the topics at the conference.

My favorite part of PyCon, however, were the sprints (https://us.pycon.org/2014/community/sprints/). Sprint leaders (mostly project leads, but some weren’t) would describe what the state of their project was and what they would like to accomplish by the end of the four days. A very abbreviated list of projects sprinted on are: CPython, PyPy, IPython, Apache Allure, Twisted, SaltStack, Mercurial, Django. Many of the core developers for these projects are present at the sprint and are willing to teach or mentor in exchange for some grease work.

I feel that working on CPython and talking to the core developers was hugely beneficial to my personal development as a programmer. Just being in the same room as the Mercurial sprint taught me a lot about how revision control works and what trade-offs Mercurial has made in comparison to other revision control systems.

One thing I found pleasantly surprising was the number of not only female attendees at PyCon US, but the number of female speakers. According to the opening statement, PyCon US had a little over 100 talks and 33% of the speakers were female. Many tech conferences boast when they have 1-3 female speakers and a 5-10% female attendance. I was really struck with how open and inviting the conference felt as a whole.

As with all endeavours in my life, I made a couple of large (and in one case, costly) mistakes. The first of which is that I didn’t have a working laptop going into PyCon. I thought (naively) that I could make it without a laptop. But I realized after my first afternoon of tutorials that in order to really learn most of what was being taught, I needed to work through and break the material given to the class. I immediately rented a laptop (which costed an arm and a leg) and never for a second felt like it was a bad investment.

The second mistake I made was not having any business cards. I had 34 business cards to go through when I returned home. Even just one week later I remember the people who gave me a card more than those who didn’t.

I had a rude awakening in Canada when it came to data usage. In America I have a smart phone that can do things. Lots of things that I have gotten very used to having. I thought I could supplement my mobile usage with Wi-Fi. The PyCon Wifi is barely fast enough to load static HTML files. It wasn’t even fast enough to pip install something. I really need to Think about my mobile data usage and Prepare my laptop going to any tech conference.

One general project tip I picked up from the development sprints was to always have the current project version as the first thing on the readme file (if not all project files). There were several times where I was working on the wrong version of code because projects either had version numbers in weird places or missing altogether. I also learned that I like it when people name their development branches after the version they’re working towards.

From here on out I’m going to cover some specific topics and things I learned at PyCon. Some of this may seem very simple to some of you while other pieces are going to be incredible specific and too detailed. These topics are self contained units, you don’t need to read them sequentially or at all to understand the rest.

CPython

One thing I picked up at PyCon is that virtual_env makes a set of symbolic links to the referenced Python installation. By installing something to the virtual_env, then it will only be installed to that virtual_env. But installing something to the linked Python instance (say the base instance), then that will also show up in the virtual_env. This can lead to a lot of confusion and make it very hard to figure out what version of a package is actually being used by Python.

Pythonbrew is a Mac exclusive solution to this problem. Python 3.4 comes with a tool called pyvenv which has support for separate Python binaries as well as site and package directories.

One thing that surprised me was that pip doesn’t install as the current user by default. If installing as the current user is desired then use the --user option.

One talk pointed out that import is, at its core, basically just a implicit namespace (the module name being the namespace unless you use the as syntax) with an exec of the files in the module. This actually helped me really understand a lot of my frustrations with import errors (or errors that happen in a module but are reported as a bad import). Also, this finally cleared up what the difference between what a Python module and package is. A module is a Python source file, a package is a directory of Python source files (modules).

Here’s a cool, though I don’t know how useful, import fact. There’s a buildin function called reload that will re-exec a module. I heard about this when someone was talking about how Django renders server side changes without a http server restart.

Python (2.7+) comes with an OrderedDict (https://docs.python.org/2/library/collections.html), which is a dictionary that remembers the order in which elements were inserted. I’m assuming that an OrderedDict is just a list of (key, value) pairs that has its access and insert methods

in a C Module. The example use cases for an OrderedDict are XML/HTML processing libraries or parsing HTTP headers.

One thing I found interesting is that Python lists have a 94% space utilization on average. Dicts on the other hand have a minimum of 33% of their allocated space unused and a maximum of ~50%, which is what is expected from a generic key hash data structure.

In newer version of CPython (2.6 with object slots, 3.2+ by default, and almost all versions of PyPy by default), objects have less time and space overhead than dicts. Use objects when an arbitrary number of references isn’t required. IE. if a struct would have worked in C then use a NamedTuple or Object. This one really caught me by surprise.

On a similar note, use the __slots__ attribute if possible (https://stackoverflow.com/questions/1336791/dictionary-vs-object-which-is-more-efficient-and-why/1336829#1336829).

__dict__ is not necessarily a dictionary even if it exposes (most) of the same methods as a dictionary.

"Every problem in computer science can be solved with another layer of indirection." - David Wheeler

IPython

A large number of presentations, classes, and open spaces at PyCon came with an IPython notebook. I can’t even begin to explain how excited I am about IPython. It’s an enhanced version of CPython that comes with a large amount of convenience functionality. The big ones that I remember off the top of my head are the ?, ??, %timeit, %%time, %run, %edit, History, interpreter shell commands, and startup files.

My favorite examples are the ? and ?? operators. ? will give present the docstring for any object, ?? will show the source code. Most of these operators are described here: http://ipython.org/ipython-doc/stable/interactive/tutorial.html

Probably one of the best features of IPython, however, is the browser-based IPython notebooks. These are Python interpreters embedded in a web browsable format. IPython notebooks come with a URL and can be hosted as a dynamic page either locally or via a Mod_wsgi enabled web server. Github will host a set of IPython notebooks on their network for free.

IPython notebooks have support for simple markdown, Latex, CPython code, and a large number of other DLS markups. These things are seriously awesome, they’re so intuitive that it’s just easier to run one on your own then listen to me prattle on about them.

http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Notebook/Running%20Code.ipynb https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks Probably my favorite part about this is that there is now work being done to have the IPython model work for many other languages like, IHaskell notebooks, IGo, IScala, IClosure, etc.

Sphinx and Hieroglyph

I didn’t work with Sphinx or Hieroglyph enough to really get a feel for either tool, but I’m very interested in how both of them work. Sphinx is a docfile generation tool which is used to build html, latex (pdf by extension), man pages, xml, or plain text documentation. By producing the first set of documentation all the other kinds of documentation come for free. From what I’ve seen, Sphinx takes a little bit of configuration and getting used to, so it seems like it’s built for decent sized project to really see the benefits. Next time I write something that needs HTML and a man page I’ll be trying out Sphinx. I’ll also be trying Hieroglyph next time I give some kind of presentation. It’s a tool that is built on top of Sphinx which generates HTML presentations.

Twisted

I have heard of the Twisted project before, but I didn’t really understand what it was or what it might be useful for. I still haven’t used the library (much) but I now understand that out of the box it’s able to parse and construct network protocol connections. It’s useful for sending and receiving standard OSI network protocols (TCIP, SMTP, POP3, IMAP) and comes with support for custom protocols.

This is not immediately useful to me personally, but given some of the projects I want to work on in the future this will become useful knowledge down the road. And no, asyncio.py does not replace Twisted... or so I was forcefully told.

Distributed (network) Messaging

This is one topic that is very immediately useful to me. I’ve been looking into distributed messaging brokers with the intent of finding the system best suited to create a distributed job broker.

I don’t know much about most of these tools, so I’m just going to list them off in the order in which I’m going to investigate them. The one message broker I’ve used before is RabbitMQ, other then that: GearMan, ZeroMQ, Celery, ActiveMQ. I would love to hear peoples thoughts on this topic.

Machine Learning

This is by far the topic that I spent the most time at PyCon learning about. I came into PyCon with almost no understanding of how machine learning works and now I feel like I know just enough to be dangerous to those sitting in close proximity. Basically, bring a large pinch of salt. If you’re interested, this is a link to a IPython Notebook which will walk you through the basics of machine learning. Absolutely wonderful tutorial. https://github.com/jakevdp/sklearn_pycon2014

To start off with let’s talk about some practical applications to machine learning, specifically projects that I think are worth pursuing. I would love to take the bugzilla database at my work as a test and training data set and then guess the chance that a new bug is a product bug, configuration bug, or any other number of useful categorizations. The next step would be for all, known and new, product bugs guess what section of code the bug is most likely found in. If I can get a program that has a reasonable amount of accuracy (say 75% for arguments sake) then this would be very useful. I find the concept of having computers accurately predict and categorize fascinating.

Cool story and all, but how does one actually turn a computer into a more accurate magic 8-ball? The basic idea behind machine learning can be summed up into 5 parts: Acquiring data, feature scaling (making data uniform), choosing an algorithm (sometimes called a model), applying the algorithm to the data (called data fitting), and validation.

The three hard parts are acquiring the data, feature scaling, and validation. Scikit-learn already comes with a large number of ML algorithms and a processes to fit the data to the algorithm. Acquiring the data is exactly what it sounds like. I’d like to answer how one does data acquisition but the instructor glossed over that detail. Someone in the class asked “how do I learn to gather data” and the instructor’s answer was “go to college.”

Feature scaling is what the database world refers to as normalization. Normally a preprocessor which makes sure the data set the algorithm is going to work on uses the same assumptions. The suggestions given in class were to make sure the data gathering has constraints on it which don’t allow too little, too much, or malformed data. The other major alternative is to take an existing database and chop off the pieces which break the code.

The example that I was given at PyCon was attempting to do image analysis on a set of thumbnails that all have different resolutions. Running a ML algorithm against a set of images with different resolutions will produce garbage. In the IPython notebook I linked to they use a PCA (principal component analysis) to drop sections of the example database.

The specific technologies we used in the tutorials were: scikit-learn, Pandas, Numpy, SciPy, and MatPlotLib. Scikit-learn is built on top of Numpy, SciPy, and MatPlotLib. It’s open source and uses the BSD license. Created make data preprocessing, classification, regression, and clustering an easier proposition. Scikit-learn is the reason why choosing an algorithm and fitting the data to that algorithm is one of the easiest parts of machine learning. The developers of scikit-learn have taken the portion of ML that used to be the hard part and trivialized it. The example page for a Support Vector Machine shows just how simple the process of using the scikit-learn library is:

from sklearn import svm X = [[0, 0], [1, 1]] y = [0, 1] clf = svm.SVC() clf.fit(X, y)

clf.predict([[2., 2.]])

Taken from: http://scikit-learn.org/stable/modules/svm.html

Numpy and SciPy are two Python libraries built to address two of CPython’s (the standard Python implementation) major faults: Speed of computation and memory use. According to these benchmarks (https://stackoverflow.com/questions/7596612/benchmarking-python-vs-c-using-blas-and-numpy) there is very little to no difference between c++ and Python. Both NumPy and SciPy use less memory than native Python data structures but I don’t know what the difference is and I’m having trouble finding documentation that shows conclusive benchmarks. Topic for another time.

MatPlotLib is a Mathlab-like 2D plotting framework. It makes it very easy to produce visualizations of all the datasets that scikit-learn and Pandas produce. I’m especially fond of the seaborn extension to MatPlotLib http://www.stanford.edu/~mwaskom/software/seaborn/examples/index.html . Seaborn is simple and produces high quality visualizations with little to no effort on the programmer's part. I view it almost as the Twitter Bootstrap (http://getbootstrap.com/) of data visualization.

There are two major types of machine learning algorithms, supervised and unsupervised. Unsupervised ML algorithms are a set of algorithms that take in a dataset and attempt to group data which is similar. This SO answer does a pretty good job of explaining the differences (https://stackoverflow.com/questions/1832076/what-is-the-difference-between-supervised-learning-and-unsupervised-learning) but I’m going to try to put it into my own words anyway.

Supervised means a dataset that contains data and some kind of classification of the data. Normally in supervised learning a human has gone through each portion of the data and set the classification by hand (thus the supervised part). The dataset is split into a “training” and “test” set, normally a 75% and 25% split respectively. The algorithm “learns” from the training set and then attempts to guess what the right classification is on the training set. The assumption is that the human guess are “correct” often enough that there is something to test against. This makes it very easy to validate a model’s fit of the data.

Unsupervised means the dataset doesn’t come pre-categorized. So the algorithm attempts to create a set of classifications or categorizes based on patterns it finds in the dataset. This requires much less effort on the part of the programmer but the results are much less trustworthy. In general unsupervised algorithms are much more susceptible to outliers, poor feature scaling, and ambiguity. They algorithms also tend to be more sensitive to certain kinds of patterns and less sensitive to others. As such the false positive and false negative rate is higher. The verification processes is also more difficult with unsupervised algorithms.

Sadly the class that was supposed to cover Model verification didn’t get to the topic. This is something that I’ll have to look into at a later time. Here’s a short list of jargon that was thrown out (in very quick succession) of things one can use to verify a model: RMSE, precision, recall score, f-score, ROC curves, cost curves. What any of that means is a mystery to me (at the time of this writing at least).

Here are some miscellaneous notes that I found useful but couldn’t quite fit into the rest of this write-up:

sklearn

_ prefix means that the method or component are uninitialized until after the data fit.

data.

supervised

Large amounts of uncertainty tend to break support vector machines.

Numpy

numpy array slicing has been overridden. Don’t expect it to act like normal array

slicing.

numpy slices return references, not views.

Don't use strings. I learned this the hard way. Convert strings to integers if possible.

MatPlotLib

I thought pyplot.Loglog and pyplot.semilog were awesome ways to display data.

Pandas

dropna() is a great way to remove null data.

Pd.get_dummies is useful for changing data to an enumerated type.

Passing mixed data to pandas then Panda will insert everything as an object.

Conda

I’ve been informed that if I want to do any amount of scientific computing in Python i should use Conda as my package manager. Having never used it myself I don’t know about the validity of this claim.

Partitioning is the heart of scaling

One thing many of the talks focused on (especially the ones that were web focused) is how important it is, for performance reasons, to be able to partition a technology stack. The classic web example of this concept is sticking a web server and database on different servers. The traditional software engineering way of stating this is that low coupling is very important.

For some reason I’ve always paid attention to the security benefits of separating a web server from the database server. I’ve also heard people talk about performance reasons but I’ve never understood that. Doesn’t putting these services on different servers now mean they have to at least deal with network latency? But the thing I never really understood is that the network is not always the system bottleneck. It’s only been my bottleneck because everything I’ve written has been so simple. It is becoming more important now that I want to do things like e-mail a list of users upon some action, change a page’s url and the thousands of references to that page, or build a categorization for the data a user is attempting to submit as a bug report.

And that made me realize that partitioning a system allows for isolation of hot spots. Partitioning allows me to focus system resources on the overloaded portions of the system. This is the same pattern I follow when I’m attempting to solve a performance problem in code (profile, isolate, refactor) just applied to a system or service rather than functions or classes.

Local outreach

There was an interesting keynote given by Jessica McKellar that talked about teaching

Python in schools. Specifically the sorry state of programming education in the United States and her opinions on how the Python programming language could help that. https://www.youtube.com/watch?v=4QOoAw6Su7M

Flask

Flask is my web framework of choice when I do web development. I went a talk called “Developing Flask Extensions” and came away with some interesting ideas (https://www.youtube.com/watch?v=OXN3wuHUBP0). Probably the biggest thing that I came away with is that the @app decorator is the heart of the Flask framework. Extensions to Flask will ultimately change the properties of the @app decorator.

Some cool extensions that I got to play with are:

Flask-Debugtoolbar : This is a flask port of the Django debug toolbar. This thing is seven

kinds of amazing. It displays a huge amount of debugging information and makes it very easy to see the state of a web application.

Flask-WTF : I’m a big fan of WTForms. This is a flask extension that makes integrating

with WTForms a very simple task. If you are writing web applications in Python, take in data, and don’t know what CSRF stands for, you should probably be using WTForms. Another option is Flask-SeaSurf.

Flask-Admin : Another port of a Django feature. This gives many (not all) of the features

that the default Django admin page. These are just the extension that I played with at PyCon. A much larger (but still not exhaustive) list can be found at (http://flask.pocoo.org/extensions/).

Subprocessing

This is going to sound dumb to anyone that’s used Python for scripting purposes, but there’s a module called subprocess and it’s awesome. The things one learns...

import subprocess subprocess.call(['git', 'clone', git_url])

Security

I’m not a security expert and am definitely not a web security expert. But these videos were interesting to me. I’m just going to leave these links here on the chance that you want to know more about these topics.

ssl

https://www.youtube.com/watch?v=SBQB_yS2K4M

http headers

https://www.youtube.com/watch?v=T-5p5ewqhVw

https://www.youtube.com/watch?v=sL_syMmRkoU

https://www.youtube.com/watch?v=nQOahpei6kw

OWASP and Qualys SSL Labs

https://www.owasp.org/index.php/Main_Page

https://www.ssllabs.com/ssltest/

Dynamic Code Analysis.

I’ve always been a fan of Pylint and coverage.py. I believe that they’re very useful tools and produce cleaner code. But I was introduced to a tool called flake8 which is a combination of pep8, pyflakes, and a cyclomatic complexity checker. Pylint freaks out at Mixins and since Flask uses so many mixin’s I’ve been having a hard time using Pylint on flask projects. I’ve been using flake8 as a fallback when Pylint is too strict.

Cool PyCon Talks

http://pyvideo.org/category/50/pycon-us-2014

Augie Fackler, Nathaniel Manista / Deliver Your Software In An Envelope

https://www.youtube.com/watch?v=mTj297sGzxw

Alex Gaynor / Fast Python, Slow Python

https://www.youtube.com/watch?v=7eeEf_rAJds

Jessica McKellar / Building and breaking a Python sandbox

https://www.youtube.com/watch?v=sL_syMmRkoU

Allison Kaptur / Import-ant Decisions

https://www.youtube.com/watch?v=aS5kXzbsLLQ

#programming #pycon2014 #machine learning #python

Lessons from my internship

This summer I did an internship at EMC Isilon and I absolutely loved it. As amazing as it was, though, it was also quite the learning experience and I’d like to share some of the things I learned with you.

First, though, some simple metrics:

Over the course of the summer, I wrote almost 13,000 lines of code, but only about 7,500 actually made it into the published beta. Of that 7,500 almost 2,000 are static HTML. In addition to that, I also wrote about 1,500 lines of docstrings. I factored out about 4,500 lines of code and nearly 1,000 straight strings (help text, descriptions, comments, docstrings). A huge chunk of that factored out code is CSS/HTML but a big part is also refactoring my Python and Javascript code.

On average, I wrote 173.33 lines of code per day, 5 days a week, for 15 weeks. In the course of that, I learned about--and now consider myself proficient in--13 new Python and Javascript libraries and learned how to profile a project that uses many different libraries, languages, and protocols. I learned how to make C modules for Python and how to manage pulling and pushing data from multiple servers and databases. Most important of all I then had people actually start using my tool. PS. deploying things is a pain in the ass.

What all that boils down to is that I built and deployed a working full stack web application. More importantly I lead a small team (another developer and myself) and owned the project.

That sounds pretty awesome, right? And it was, but part of that learning experience was doing things wrong. Really really wrong.

The following are four lessons from this summer which I consider the most important of everything I learned. I believe these will be useful to all programmers.

Never assume you will be the only programmer on a project.

When I was given my project, I was told that I would be the only developer working on it. That my department didn’t have the people with the skills or the time to work on the project with me. And I made the mistake of believing them.

If you could go and look at the revision history of my projects documentation you can actually see the day I was told that someone else was going to work on my code. Documentation goes from,

# TODO Fix this

and

“””

no return value, modifies some object variables

“””

some_call(request) # Modifies request.url variable

and

"""

This function executes all the potential actions from the management/users

page. This function modifies database values found in the user table.

NOTE: Only admins are allowed to execute user modifications.

Expected actions:

Activate: Activates a user. Deactivated users can take no actions.

Deactivate: Deactivates a user.

ChangeRole: Changes the role of a user. For more on roles go to

access_control.py

ChangePostWarning: This changes the warning text that a user sees

when they attempt to bypass the review process.

“””

I actually felt pretty ashamed of how bad my documentation was when I watched another developer struggle with my code base. I realized just how many assumptions I made and how much I relied on having created the code to read it. Names that I thought were self documenting weren’t. Docstrings that I assumed contained enough information didn’t.

This also carries over into other assumptions as well. For example, I never set a style standard for the project, go me. Even if your language comes with a set of style guides (PEP8) you still need to specify what styles to follow because not everyone is going to be familiar with the language. In fact, you should assume no one is going to be familiar with the language. Even better is when you specify an automated tool that displays breaks in style. Examples being pep8 and flake8 for python.

The biggest assumption was that the code’s workflow was clear just from function calls. When someone comes into your code base, they aren’t going to see the overall structure, side effects, and purpose of your module (package, class, file, whatever). If you don’t want to wait for them to reverse engineer your work (read, spend a looooon time) you will have to spell it out for the. No, that doesn’t make them a bad developer.

Finish the deployment early, run all of your normal tests against the deployed environment before opening it to the public.

There are actually two mistakes that I made in the process of learning this lesson. The first mistake is that I waited until the last day of my internship to try to deploy the project. The second is that I didn’t run my unit tests against the deployed project. I missed two configuration errors and a line of code that had a hard coded value which was supposed to read from a config file.

I had spent so long working on logic errors that I completely forgot that there is a class of bugs that come not from the code but from the environment that your code works in. Never assume that because something worked in the test environment that it will work in the deployment environment.

Lines of code are not a good indication of the health of a project.

I purposely started this post by talking about total lines of code followed by a breakdown of lines that actually made it into the final product. 13,000 sounds impressive until you realize how much time I spent modifying and removing bugging, unclear, complicated, and useless lines. I also have a very verbose coding style. For example, if a function like this one,

result = post_document(post, post_comments, post_history, flags)

would word wrap (PEP8 specifies that lines shouldn’t be longer than 80 characters) I would change the function to look like,

result = post_document(

post,

post_comments,

post_history,

flags)

Each of which would get counted as a line of code according to SVN. I should probably use a tool to get a more accurate account of lines of code but I spent enough time just getting SVN to not count comments and blank lines.

When my project was at it’s largest (almost 10,000 lines), it was also at it’s most bug-ridden. After refactoring, pushing common code into functions, and common problems into modules it was much easier to see the big and small picture.

Speaking of verbose code, I’ve found a characteristic of coding style that I now want to strive for in all of my projects. I’m calling it the ‘minimum number of verbose lines of code’. Not very catchy, perhaps, but it does make the code base much easier to maintain by balancing the horizontal and vertical reading requirements for the code. To me, this is how a programmer accomplishes their number one engineering goal (reduce project complexity as much as possible).

User testing (even something as simple as sitting someone down on your computer and watching them play with things) is incredibly important. You need to do this multiple times with different people.

Go read two books right now. ‘The Design Of Everyday Things’ by Donald Norman and ‘Don’t Make Me Think’ by Steve Krug. No really, I’ll wait.

You’re back? Ok, I’ll just give a quick summary. Your user facing code sucks and you have no way to tell how it sucks. It’s too intuitive for you to use tools you created.

It doesn’t matter if you’re making a console application (Can you guess what the Linux command “free” does just from the name?), a web application, or a operating system API, someone is going to use your code (hopefully). And that means you need to watch people use your system and you need to pay special attention to where they struggle. You need to do this in rounds because you’ll have UI errors that can only happen after you’ve fixed the original gaping wounds.

These four lessons were the most important to my growth as a developer, but they’re hardly all I learned this summer. Here are some more specific lessons that may not apply to all developers, but are still good things to be aware of.

In dynamic programming languages, naming things is vitally important. You don’t have a type to tell you what the variable contains. The variable names are your only lifeline. Don’t be the Titanic: have enough lifeboats.

This video explains my stance on this better than I ever could:

LINK/IMBED---> https://www.youtube.com/watch?v=YklKUuDpX5c

Basically, your name needs to be as specific as you can possibly make it to try to make the code as self documenting as possible. You don’t have a type to self document.

In Python, if you misname something and the name you accidently put in is a live variable name, you’re gonna have a bad time. My fix is to get a plugin that highlights each name that’s in scope of the current word you are typing.

This is pretty self explanatory. If you have a variable named ‘url’ and ‘uri’ in the same scope and you accidently use the wrong one, you’re going to have to run the program and have it crash (or, even worse, run correctly with bad values) before you catch the bug.

If you have:

from library import thing1

from library import thing 2

....

from library import things more than 3

Then just import the damn library. I have several files with 10-30 lines of imports that could be done with much less and I haven’t gone back to refactor that problem out because it will require me to modify thousands of lines of code. It’s just not worth the effort at this point, I have other low hanging fruit to fix. But this is still a good thing to keep in mind for future projects.

Even if your application doesn’t have performance as a primary concern, you will run into performance problems. Make sure to stress test.

This example is specific to web application, but the concept applies to all software.

Your stress test can be as simple as creating 25 VMs that all just request your website’s home page. Try putting way more data into your database(s) than you think your project will ever contain and see how each page responds. Test your software in a broken state: i.e., disconnect databases, disconnect the internet, etc.

You will need a tutorial for all non-trivial software. Users will miss giant red banners at the top of the screen.

No really, they will literally miss the giant bright red banners at the top of their screen. And not just one of your users, a large chunk. A person may be smart, but people in general tend to be pretty dumb.

Have a test server that is a exact copy of the production server. This way you can test a deployment before you actually deploy it.

...not that I would have pushed bad code to the production server or anything...

If you set up the test server in a VM all you have to do is copy over the production server and set up a new DNS configuration for the production server.

Virtual machines make life so much easier, especially when it comes to being able to spawn a new server that’s a carbon copy of your production server. Just make sure it doesn’t utilize the production database(s).

If your website is going to have more than one Javascript or CSS file, learn how to bundle them.

The number one thing you can do to increase your first visit performance is to bundle and compress Javascript and CSS files. According to Yahoo.com, the number one performance gain you should work on is the load time for a user’s first visit ( http://developer.yahoo.com/performance/rules.html ). In Flask, you can do this by using flask-assets ( http://elsdoerfer.name/docs/flask-assets/ ).

After bundling and compressing my Javascript and CSS files, my web application went from a 300ms first access under “heavy” load to 75ms first visit. Not bad considering that the Google homepage takes almost a second for first visit.

If you run into a line of code that you wrote and you still need to take a second to figure it out, you need a comment.

This is pretty self-explanatory. If you’re struggling with your own code then you should expect anyone else that reads that code to feel like their brain is being hit by a porcelain brick.

Never assume a service will be running.

I can’t go into the specifics for this one, but I believe it’s sufficient to say that you should never assume a service, a server, or even a local daemon will be running (or running correctly). Never trust, always verify. Especially because if your code breaks because another website is down, people won't care that it wasn’t your fault.

If you’ve made it this far, then I think a pat on the back is in order. Feel free to disagree with, correct, or further educate me. I’m at least good enough to be a professional programmer, but I’m not going to even pretend that I’m a true expert on this subject. If there’s one thing that truly sunk in this summer, while I was surrounded by truly smart and knowledgeable individuals, it’s that in the grand scheme of things, I am a baby in this profession still.

#programming #internship

Dirty Hacky Code

So I was reading a Gamasutra article from 2009 called “Dirty Coding Tricks.” http://www.gamasutra.com/view/feature/132500/dirty_coding_tricks.php The whole article is well worth reading for both programmers and non-codejunkies. The last of the nine stories that are covered was so interesting I wanted to quote it on here: “I was fresh out of college, still wet behind the ears, and about to enter the beta phase of my first professional game project -- a late-90s PC title. It had been an exciting rollercoaster ride, as projects often are. All the content was in and the game was looking good. There was one problem though: We were way over our memory budget.

Since most memory was taken up by models and textures, we worked with the artists to reduce the memory footprint of the game as much as possible. We scaled down images, decimated models, and compressed textures. Sometimes we did this with the support of the artists, and sometimes over their dead bodies.

We cut megabyte after megabyte, and after a few days of frantic activity, we reached a point where we felt there was nothing else we could do. Unless we cut some major content, there was no way we could free up any more memory. Exhausted, we evaluated our current memory usage. We were still 1.5 MB over the memory limit!

At this point one of the most experienced programmers in the team, one who had survived many years of development in the "good old days," decided to take matters into his own hands. He called me into his office, and we set out upon what I imagined would be another exhausting session of freeing up memory.

Instead, he brought up a source file and pointed to this line: static char buffer[1024*1024*2];

"See this?" he said. And then deleted it with a single keystroke. Done!

He probably saw the horror in my eyes, so he explained to me that he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to free up when it's really needed.

He walked out of the office and announced he had reduced the memory footprint to within budget constraints -- he was toasted as the hero of the project.

As horrified as I was back then about such a "barbaric" practice, I have to admit that I'm warming up to it. I haven't gotten into the frame of mind where I can put it to use yet, but I can see how sometimes, when you're up against the wall, having a bit of memory tucked away for a rainy day can really make a difference. Funny how time and experience changes everything.” To me this (aside; I’m not even sure this can be called a code trick, it’s more of a people trick) trick is both wonderful and awful. It’s wonderful because it’s a clear case of creativity and discipline through restrictions. It reminds me of people who have artificial deadlines for projects a week before they’re actually due. It reminds me a lot of my dad who is fond of saying “Go like hell in the beginning and hope you can coast at the end.” To my ears all of these arbitrary constraints are a kind of personal insurance. They’re risk management but the management portion is just a shot in the dark. Which is why these kind of tricks are also awful to me. All of these tricks remind me of 1850 architectes guessing how much support they’d need to build into a bridge. It reminds me of the Tay Rail Bridge built in scotland in 1878. https://en.wikipedia.org/wiki/Tay_Rail_Bridge

The Tay Rail Bridge was an early attempt to build a lighter and cheaper bridge. Up until this point most train bridges were built by guessing how much the max weight of a train would be and then multiplying by a couple hundred. This is great from the perspective of safety but terrible of the perspective of building more than one bridge... ever. This is how I feel a lot people are when they’re planning projects. Either no risk is ever accessed (Everyone plans on turning things in on the deadline) or risk is accessed by taking the estimations and multiplying by 2 if you’re very confident and more if you know better.

Enthusiasm is a resource

When you, or a team, are working on a project you must budget for enthusiasm. Drive might be a better word but the word you use is ultimately unimportant. What is important is that if you are a creator of some kind (be it programming, writing, carpentry, etc.) then you must realize what your limitations are and that drive is a limited resource.

In the last five years I have started many more projects then I have actually finished and sent out into the world yonder. My bitbucket, google drive, and gmail account are graveyards for half-baked concoctions. Someday I’ll be taken before the International Code Council for idea genocide. These projects aren’t even in a finished enough state that I can use them as a learning experience. They’ve been left malnourished and forgotten.

I’ve been making things for a while now (programs, websites, dance classes, plastic armor, headphone modifications) and I’ve run into this problem a lot. I start working on something, get half-way through it, and never finish the project. It wasn’t until I read a blog post talking about creating a game in a week or less that I realized what my problem was--or that I even had a problem.

“If your motivation is rocket-high when you start the project, but dies off in 3-4 weeks, then you can only tackle games that can be done in that timeframe!“ (http://jonbho.net/2013/08/22/can-you-write-a-game-in-one-week/ ) This quote finally forced my goat of a brain to understand most of the projects I’ve been starting and stopping are simply too large for me. I’ve been telling people “don’t bite off more than you can chew” since my first day as a tutor. My glass house was so close to my face I’m surprised I didn’t touch it. That said, now that I’ve come to this epiphany, what am I going to do with it? This is the part where I don’t follow through and correct my hypocrisy, right? Luckily I’m a bad politician, so I can acknowledge a problem and actually try to fix it.

The most effective things I’ve done so far are to experiment with self-imposed deadlines on my projects and to match up project types to my findings. It’s not enough to have a minimum project estimate, you must have a project cap as well. I’ve also just started to accept that the really great projects get more of my time. I feel less bad about starting something and leaving it to die, it’s just a thing you need to do in order to feed the beast. For example, my deadline for writing this article was four hours. My next article is probably going to have a three hour deadline. When it comes to writing I seem to have a max of 3-4 hours that I can spend on something before I lose interest and when I have an idea I need to write about it the same day that I get the idea. I’m the equivalent of a writing goldfish. I can be much more lax with programming projects then writing projects. I’d been thinking about making a resume website for half a year before I actually got around to doing it. Another coding project involves a screen filter that takes all the current onscreen colors and changes them to look like what a color-blind individual sees. My hope is to take this program and make a website that takes a URL as input and outputs what the given website looks like to a color blind individual. I’ve been working on this filter for about 40 hours now which is much longer than any of my writing projects. This is more in line with my standard goat-like tendencies. Another useful change has been compartmentalizing projects. I’ve been breaking larger projects into smaller beasts and hunting them down. In algorithm analysis classes one of the first things you drink from the fire hose is the concept of divide and conquer. Invaluable for a programmer but the bugger is damn useful for general use cases. For example, on my resume website I have a bunch of URLs which all lead to different pages. Rather than thinking about working on the entire website, I’ve started working on individual sub-pages. I’ve also been using websites like trello and pivotal tracker to break projects up into manageable chunks.

This applies to more than just tech projects. I’ve found this to work on physical objects. The most recent case is a redesign of my aluminum kite shield. The components were: strip handle, strip siding, remove paint, cut size down, attach new handle, attach new sidings. This helped in two ways. The first is that I no longer saw the project as one giant monolith that required an entire weekend to finish. I knew I could do a section in manageable chunks. The second way this helped out is that I now have evidence of a forgotten step next time I have to do something like this (paint the shield).

For me personally, this change in perspective has made me immensely more productive in the last two months.

code complete getters and setters

I've been reading Code Complete by Steve McConnell recently. The books a good read and has contained solid advice for my lowly student self.

I'm going to start this post by saying that I'm on chapter eight and my ability as a developer has grown by leaps and bounds. There's only been two other experiences which have improved my overall ability as quickly as this book combined with my current internship.

With all of that said, code complete is starting to get a little long in the tooth. I’m reading the second edition which was published in 2004 and it shows. I’m finding it deeply ironic that a book that talks about coupling in the same fashion that most people talk about government corruption is so tightly coupled to c++/Visual Basic.

In chapter six "Working Classes" under the section "Good Encapsulation" McConnell says to never expose member data to the public. He then goes on to say that you should always use getters and setters to access that data. The reasoning behind this is to isolate change (you only have to modify the getter or setter rather than all the code which uses them) and to reduce the programs complexity (The calling code shouldn’t need to know the internal workings of the getter / setter to work). The less nuts and bolts (only changing one function) to replace or modify a part the happier you’ll be when you inevitably have to change that code down the road.

This sounds perfectly reasonable (and is perfectly reasonable if you're using c++, java, c, ada, etc.). The Funny part about this advice is that you have to write much more upfront code in order to make modification heaps easier down the road (The worst case being two functions for every action / attribute). Sometimes you have to spend some time to save more in the end, right?

McConnells overall point (Encapsulation good) is amazing and very relevant. His sweeping generalization of how to Encapsulate(Don’t expose member data, use getters / setters) isn’t always relevant.

Let's use Python as a counter example to McConnell's statement. if I wanted to make a class in Python i'd say

class samurai():

def __init__ (self):

self.sword = “a freaking Katana.”

jack = samurai()

Black = samurai()

Print (jack.sword)

Print (black.sword)

Under McConnell's good encapsulation example this would be terrible because you're directly accessing member data. If this was c++ I’d have to change my print statements if I wanted to modify the .sword attribute (and if you have thousands of .sword access in c++ that’s a lot of copy-replace). And just to be clear I am accessing the public member of the samurai class in this example. But this doesn't actually break encapsulation in Python because you can change the default accessor function if you find the need to do so.

class samurai():

@property

def Sword(self):

""" getter which is used by the syntax samurai.sword"""

#add your new code here! It’s magic!

Return self.value

@sword.setter

def Sword(self, value):

""" setter which is used by the syntax samurai.sword = value"""

#I’m here just for good measure.

Self.value = value

Which means you get the best of both worlds. You don’t have the upfront cost of writing a getter and setter for every attribute (having written about 200 of those in one sitting for a java project this can be “fun”) and you also get the lower cost of modification / complexity that getters and setters gives you. Side note, this is as close as Python gets to private attributes.

class samurai():

def __init__ (self):

self._sword = “a freaking read only katana”

@property

def Sword(self):

""" getter which is used by the syntax samurai.sword"""

#add your new code here! It’s magic!

Return self._sword

The original point of getters and setters was to reduce complexity. Ironically enough sometimes they now add more complexity if you’re language is flexible enough. Did I mention that Code Complete is awesome? I think I did but I can’t remember with all the ragging I’ve done recently.

If this topic is interesting, here’s some further reading. http://tomayko.com/writings/getters-setters-fuxors http://dirtsimple.org/2004/12/python-is-not-java.html http://stackoverflow.com/questions/1554546/when-and-how-to-use-the-builtin-function-property-in-python http://stackoverflow.com/questions/6618002/python-property-versus-getters-and-setters

Trending Blogs

Recently Viewed Blogs

Confident And Wrong