#python performance

[simoluan]

I am really proud.

Yes, it was not 100% by my own work, it was thanks to the help of others...

... but I'm still the one coding it, and it finally sees the light of the day :D

From 240 seconds to 75 seconds to 41 seconds while maintaining correctness... all devs would certainly be proud :)

[simoluan]

breaking the speed record!

Currently public version:

8013 records in 76.91 seconds

Private version, self optimized:

8013 records in 63.41 seconds

Highly-optimized version with the help of the kind people at codereview.stackexchange.com:

8013 records in 40.90 seconds

Not bad. Not bad at all 😀

[simoluan]

optimization satisfaction

So, I've finished optimizing some critical paths in sims3py.

A couple of days ago, the critical paths took 266 seconds to complete on a relatively small .nhd file (47.7 MB, 8013 resources).

Now after optimization, the same critical paths take 78 seconds. Less than one-third the time they took prior to optimization.

Taken together with the earlier optimization, I still see a 4x speedup, so I'm not complaining 😀

All in all, I'm very satisfied ☺

[simoluan]

shifting into 14th gear

I'm giddy with excitement!

Thanks to the magic of the intervaltree package, a crucial codepath that originally ran in 42 seconds...

... now runs in 3 SECONDS FLAT!! Just one-fourteenth (1/14) of the original time taken! That's a 1300% SPEEDUP!!

Who says Python can't be fast? :-D

It is true that one needs the proper data structure to efficiently and effectively manipulate data. Especially with Python. And since Python comes with "Batteries Included"... if you need esoteric data structure, as long as there's a scientific/mathematic principle behind the structure, chances are someone has already coded the necessary class(es). ;-)

[simoluan]

if in doubt, profile!

So here I am, performing code profiling for sims3py PackageParser.

There will be at least two iterations: Before and After. And more if there are performance-impacting changes after the After iteration.

Every iteration, I'll collect the statistics using the (5-2) method. Or, if I have the time and/or inclination, (7-4) method.

(Explanation: "(5-2)" means take 5 observations, and discard the highest and lowest values. "(7-4)" means take 7 observations, and discard the 2 highest and 2 lowest values.)

And if I'm feeling particularly masochistic, perhaps a (9-4) observation. Hah! >:-)

[simoluan]

(possibly) significant speedup coming up

So, I was trying to find a way to optimize the slowest parts of sims3py. And serendipity found me this:

http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews

This is the first time ever I heard about "buffer protocol" and "memoryview".

Reading the article and digesting the information, I suddenly realize that these pair of Python features could very likely be the key to a performant Python program!

You see, sims3py has to do a HUGE amount of slicing and dicing. Every invocation of slicing apparently creates a copy of the buffer. And if the copy doesn't neatly fall into the size of the memory page, performance will suffer.

After finding out about these two Python features, I see how I can speedup the operations by preventing memory copy. The article mentions a 300x speedup; but honestly, I'll take it even if it promises only 2x speedup.

I'm giddy with excitement! Stay tuned on the repo for some speed boostin' commits!

[dragoni]

We've all heard it before: Python is slow.

When I teach courses on Python for scientific computing, I make this point very early in the course, and tell the students why: it boils down to Python being a dynamically typed, interpreted language, where values are stored not in dense buffers but in scattered objects. And then I talk about how to get around this by using NumPy, SciPy, and related tools for vectorization of operations and calling into compiled code, and go on from there.

But I realized something recently: despite the relative accuracy of the above statements, the words "dynamically-typed-interpreted-buffers-vectorization-compiled" probably mean very little to somebody attending an intro programming seminar. The jargon does little to enlighten people about what's actually going on "under the hood", so to speak.

So I decided I would write this post, and dive into the details that I usually gloss over. Along the way, we'll take a look at using Python's standard library to introspect the goings-on of CPython itself. So whether you're a novice or experienced programmer, I hope you'll learn something from the following exploration.