pepoluan.sims

all my posts about "the sims 3" game

[earlgreytea-for-me]

I’m getting so upset seeing Ezra preparing to fight the Inquisitor. Like he’s just a kid, he’s completely untrained, and now he has to fight not only for his life but for the lives of those he loves. He hardly even knows how to use a lightsaber but he still looks so determined. I can’t get over it.

[simoluan]

His brush with the Dark Side prepared him for this. When Kanan teaches him Vaapad, he'll be completed.

[simoluan]

on why sims3py

If you follow my Tumblog, you'll see that lately I've been focusing on the sims3py library instead of sims3.scaffolding. There are two reasons.

Reason number one, I need to be familiar with the binary data structures of Sims3 objects. As it happens, I learnt a lot on the very strange decisions the programming team made. This experience will ultimately help me in designing an efficient in-memory, file-backed object storage system.

(Perhaps contrary to many programmers, I'm from the "old school" of dataflow and DTD, where data is the most important, and everything else conforms to the data and how it is used. Thus, properly modelling the data is of utmost necessity before I can continue. Amazingly, with the help of good debuggers, code injection becomes relatively easy...)

Reason number two, I see the need of a better, cross-platform tool for the manipulation of DBPF and S3P files. The current tools seem to have been abandoned by their makers (can't blame them; after years, one would be burnt out), and the roundabout way to run then on non-Windows platforms has been more problematic. So, Sims 3 players on non-Windows platforms are in dire need of assistance.

There, my simple explanation. Once I'm finished with sims3py, I'll go full speed on sims3.scaffolding.

[simoluan]

next tool: repacker

Mac users in the MTS forums are starting to have problems with the .Net based tools, especially after upgrading to the latest OS X version. So, I'm upping the priority of this tool.

I still have to learn how to parse XML in Python, though.

But it couldn't be that hard, right?

[simoluan]

I am really proud.

Yes, it was not 100% by my own work, it was thanks to the help of others...

... but I'm still the one coding it, and it finally sees the light of the day :D

From 240 seconds to 75 seconds to 41 seconds while maintaining correctness... all devs would certainly be proud :)

[simoluan]

breaking the speed record!

Currently public version:

8013 records in 76.91 seconds

Private version, self optimized:

8013 records in 63.41 seconds

Highly-optimized version with the help of the kind people at codereview.stackexchange.com:

8013 records in 40.90 seconds

Not bad. Not bad at all 😀

[simoluan]

annndd... it's public

With the pushing of 9 commits culminating in a52b285, the optimizations I've made for sims3py are now available in the public official sims3py repository.

Have fun perusing the code!

[simoluan]

optimization satisfaction

So, I've finished optimizing some critical paths in sims3py.

A couple of days ago, the critical paths took 266 seconds to complete on a relatively small .nhd file (47.7 MB, 8013 resources).

Now after optimization, the same critical paths take 78 seconds. Less than one-third the time they took prior to optimization.

Taken together with the earlier optimization, I still see a 4x speedup, so I'm not complaining 😀

All in all, I'm very satisfied ☺

[simoluan]

shifting into 14th gear

I'm giddy with excitement!

Thanks to the magic of the intervaltree package, a crucial codepath that originally ran in 42 seconds...

... now runs in 3 SECONDS FLAT!! Just one-fourteenth (1/14) of the original time taken! That's a 1300% SPEEDUP!!

Who says Python can't be fast? :-D

It is true that one needs the proper data structure to efficiently and effectively manipulate data. Especially with Python. And since Python comes with "Batteries Included"... if you need esoteric data structure, as long as there's a scientific/mathematic principle behind the structure, chances are someone has already coded the necessary class(es). ;-)

[simoluan]

if in doubt, profile!

So here I am, performing code profiling for sims3py PackageParser.

There will be at least two iterations: Before and After. And more if there are performance-impacting changes after the After iteration.

Every iteration, I'll collect the statistics using the (5-2) method. Or, if I have the time and/or inclination, (7-4) method.

(Explanation: "(5-2)" means take 5 observations, and discard the highest and lowest values. "(7-4)" means take 7 observations, and discard the 2 highest and 2 lowest values.)

And if I'm feeling particularly masochistic, perhaps a (9-4) observation. Hah! >:-)

[simoluan]

(possibly) significant speedup coming up

So, I was trying to find a way to optimize the slowest parts of sims3py. And serendipity found me this:

http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews

This is the first time ever I heard about "buffer protocol" and "memoryview".

Reading the article and digesting the information, I suddenly realize that these pair of Python features could very likely be the key to a performant Python program!

You see, sims3py has to do a HUGE amount of slicing and dicing. Every invocation of slicing apparently creates a copy of the buffer. And if the copy doesn't neatly fall into the size of the memory page, performance will suffer.

After finding out about these two Python features, I see how I can speedup the operations by preventing memory copy. The article mentions a 300x speedup; but honestly, I'll take it even if it promises only 2x speedup.

I'm giddy with excitement! Stay tuned on the repo for some speed boostin' commits!

[simoluan]

There’s a class that might be very useful for sims3py. But using that class means sims3py is no longer independent, and now depends on an additional package.

I’m torn between wanting to use that class, or just go my way without that class but causing the tools to (probably) run...

[simoluan]

Hi! Thanks, you have very good point there.

I've found a way to sidestep the issue, thanks to the dependency's license. But thanks, anyway.

(This is a re-reblog; I stupidly forgot to check which account I was using when I reblogged earlier. Not because I'm trying to hide my identity, I just like to not intermix postings between accounts.)

[simoluan]

the license to the rescue!

As an epilogue to my earlier quandary regarding dependencies, I may have found a good solution:

Just include the source code of the library within sims3py suite!

I can legally do this because the dependency in question is released with permissive Apache License 2.0. So, as long as I properly specify the 'credits' (including proper copyright attribution), and include the necessary LICENSE/NOTICE file(s) and explicitly state what changes (if any) I do to the code, I'm legally allowed to include the source code.

Problem solved ☺

Now, I still need to ensure that there will be NO dependency on binary libraries, before I can include the dependency in the sims3py suite, but that would be relatively a simple job.

[simoluan]

to depend or not to depend...

There's a class that might be very useful for sims3py. But using that class means sims3py is no longer independent, and now depends on an additional package. I'm torn between wanting to use that class, or just go my way without that class but causing the tools to (probably) run slower. Why is depending on a separate package not preferred? There are two reasons from my perspective. One, the package might rely on a binary library, and this might cause problems with cross-platform compatibility. Two, it forces users (and devs) to install "something else" before being able to run the tools in sims3tools. Without dependencies, sims3py can simply be extracted into a folder and executed using "python -m" from within the extraction directory.

[simoluan]

sims3py development pace is lookin' good...

... even if I have to say so myself :-)

[simoluan]

sims3utils.test_corrupt

I've placed the Repacker tool on the back burner for now. It requires parsing XML, and I don't have experience doing that in Python, so I need to learn more about that.

In the meantime, and due to a pressing need, I have finished up another tool called sims3utils.test_corrupt, whose purview is to do deep sanity checks against DBPF files.

Check that the DBPF Header is not corrupt

Check that the DBPF Index is not corrupt

Check that DBPF resources are not corrupt

The last check is still being improved. Currently, it checks that compressed resources are compressed properly... And that's it. I don't yet have enough resource parsers to do a satisfactory deep-checks of each resource, but I still have some additional tests that can be done without delving into a resource's semantics.

[simoluan]

sims3utils.combiner

Next stop, an easy way to combine packages. No need to do all the mouse acrobatics with s3pe. But, I'm down with the 'flu at the moment , so that'll have to wait.

[simoluan]

The 'flu...

... has caught me.