And So It Continues... @davidbazile - Tumblr Blog

Posts

An important part of working on any technical team is human parallelism – the ability for multiple people to be able to work simultaneously on the same codebase. But it’s also important for teams to be able to work asynchronously too – I should be able to make changes to your code when you’re away, and vice versa. [...] Strong developers understand technical debt and try to limit it by designing code that is as maintainable and self-explainable as possible. Writing readable code really requires developers to change their outlook – your code should last longer in the organization that you.

Aaron Stannard on Maintainability (http://www.aaronstannard.com/post/2013/02/06/10-Reasons-Why-Youe28099re-Failing-to-Realize-Your-Potential-as-a-Developer.aspx)

#maintainability #teamwork

SOAP Opera

XML is like violence: if it doesn't solve your problem, you're not using enough of it.

xmlicious via thedailywtf

Background

So, at work I'm rewriting several data feed ingestion scripts from Perl to PHP, the majority of which contact remote servers via SOAP Web Services. The problem is that while some of those web services return data that is pristine and concise, others are less than stellar. Some contain namespaces and type definitions that yield a very predictable and well-defined data structure (i.e., arrays will appear where they should, whether only one element is there or ten). Others seem to just be random, arbitrary and of questionable value.

Anyway, I am making good progress towards my goal and have a significant number of scripts converted. Then I stumble upon a webservice whose authors have decided that they need to use upwards of 11 namespaces, that everything should be namespaced and that elements should contain children and attributes all of different namespaces. It doesn't appear to add any value, structure or predictability to the overall data set. Normally, nobody should ever have to worry about seeing this since SoapClient handles all of the unmarshalling anyway. That was not to be the case this time.

WSDL Shmisdle

So, I point my SoapClient at their WSDL and attempt a pull. I get a SOAP-ERROR complaining about not being able to load remote schema files (I forget the specific verbiage). I look at the WSDL and sure enough, there are tags pointing to several http://127.0.0.1:8080/some/path/to/file.xsd schema files. Obviously, we don't have those files because this is a remote web service, not our own.

One option is to pull the WSDL on each request, do a regular expression to change 127.0.0.1 to their domain, cache the file somewhere, then feed it to SoapClient hoping the URI is correct and won't change. Not a fan of this since (a) I don't want to be dealing with any more XML than I need to, and (b) I have no idea how deep this error goes and have no desire to find out. Would I have to fix the .XSD files the WSDL is pointing to if they are also broken? No, please.

I then take a look at what the old script was doing. Naturally, it's building a stringified SOAP Request and firing it off using Perl's LWP::UserAgent... le sigh.

Life in a post-SOAP world

So, following in the footsteps of my predecessors, I string up this SOAP request manually, fire it off via CURL and get my XML string response. Take the response string, pass it to simplexml_load_string() and print_r() the results, which was:

SimpleXMLElement Object ( )

Oh, a completely empty node. WAT.

So as it turns out, SimpleXML (rightly) will hide anything namespaced at the current element and below. Since the <soap:Envelope> node (i.e., the topmost node) of a SOAP response is namespaced, it and its children (i.e., everything we actually need) are hidden by default. Not a problem -- we can just register the soap namespace and $xml->xpath('soap:Body/getFooBarResponse'). This works to get the body node, but since everything inside our current data response is a big bowl of namespace spaghetti with interwoven namespaces serving no apparent purpose other than obfuscation, we'd need to know every node where the namespace changes and account for that with the proper "peek" operation. With 4 different end points we need to hit for this particular web service, this could potentially be 4 * N special cases that need to be hardcoded into the script. Further, this completely breaks the data adapter class we have in place for all other data sources, which assumes it's getting an object with a clean tree structure.

A Workaround

I did a bit more research at home and came up with this superhack of a workaround:

<?php $namespaces = [ 'soap' => 'http://schemas.xmlsoap.org/soap/envelope/', 'ns1' => 'com.andsoitcontinues.someentity', 'ns2' => 'com.andsoitcontinues.someotherentity', ]; $dom = new DOMDocument(); $dom->loadXml($xml_string); kill_the_namespaces($dom->childNodes, $namespaces); function kill_the_namespaces(DOMNodeList $node_list, array $namespaces) { foreach ($node_list as $current_node) { if ($current_node instanceof DOMElement) { if ($current_node->hasChildNodes()) kill_the_namespaces($current_node->childNodes, $namespaces); foreach ($namespaces as $local_name => $uri) { $current_node->removeAttributeNS($uri, $local_name); } } } }

Potential issues with this workaround

I'm well aware of the potential issues, which would include:

1. Sibling nodes with the same name will become an array

I am a firm believer in defensive programming, so I dereference all values from any SimpleXML structure using strval(), intval() or whatever data type I'm expecting. Invoking one of those methods on the foo property will yield the first element of the array anyway, which (should?) be okay for our purposes.

Before:

<ns1:foo>alpha</ns1:foo> <ns2:foo>bravo</ns2:foo> <ns3:foo>charlie</ns3:foo>

After:

<foo>alpha</foo> <foo>bravo</foo> <foo>charlie</foo>

Which SimpleXML will parse as:

[foo] => Array ( [0] => alpha [1] => bravo [2] => charlie )

2. Clobbered attributes with bias toward descendants

This one is a bit more serious than the aforementioned risk in that if it did happen, it would mask itself fairly well. I don't foresee this being a huge issue for our current use case.

Before:

After:

Which SimpleXML will parse as:

[foo] => Array ( [@attributes] => Array ( [somenum] => 333 ) )

Conclusion

I'm not particularly proud of this workaround, but given the inability to affect the behavior of the proprietors of systems we request data from, you gotta do what you gotta do.

There's some good that came of this episode -- I'm learning a ton about XPath, XML Namespaces, the good, bad and the ugly of reading and writing SOAP encoding and a ton of other things. I think I transcribed almost 100kb of XML by hand this week. So, I'll mark this down as a learning experience. Hopefully, I can write this code in a way that won't completely bewilder the next developer that has to touch it.

Update 2013-12-02:

I ended up not being able to use kill_the_namespaces() after all. Unbeknownst to me, further down the execution chain there is a need to append nodes to the exact same structure to stage files for another system's ingestion. If this was a read-only solution as I'd originally assumed, I'd be able to use the namespace remover and avoid having to put some even uglier code in this script, but alas no such luck for me (or the developer who is going to have to maintain this utility).

Reference material:

The "S" Stands for Simple

DOMElement::removeAttributeNS()

SimpleXML, Namespaces & Hair loss

#xml #php #document object model #soap

serializer.py

import datetime import os import re import sys # ----------------------------------- CONSTANTS ----------------------------------- SUPPORTED_EXTENSIONS=( 'css', 'js', 'htm', 'html', 'xhtm', 'xhtml', 'php', 'py', 'txt', 'xml', 'htaccess', 'config', 'ini' ) # --------------------------------------------------------------------------------- """ Recurses through a directory tree and serializes all supported files contained within @param string path """ def read_directory(path): # Keep some metrics files_processed = 0 bytes_read = 0 # Header path = os.path.abspath(path) print('$ serializer {0:s} {1:%Y-%m-%d %X} $'.format(path, datetime.datetime.now())) # Iterate over all directories below this for current_directory, subdirectories, files in os.walk(path): # Skip hidden directories if re.match('^\..+$', os.path.basename(current_directory)): continue # Iterate over all files at the current directory for filename in files: # Gather information about the current file absolute_path_to_file = os.path.abspath('{0}/{1}'.format(current_directory, filename)) path_to_file = os.path.relpath(absolute_path_to_file) file_extension = filename.split('.')[-1] # Clean up the relative path path_to_file = path_to_file.replace('\\', '/') path_to_file = re.sub('^\.+/', '', path_to_file) # Only process files with the proper extensions if file_extension in SUPPORTED_EXTENSIONS: fp = open(absolute_path_to_file, 'r') file_contents = fp.read().strip() fp.close() bytes_read += len(file_contents) files_processed += 1 print('#### START {0}\n{1}\n#### END {0}'.format(path_to_file, file_contents)) # Print metrics print() print('{:8d} files processed'.format(files_processed)) print('{:8d} bytes read'.format(bytes_read)) """ Main Routine """ def main(): target = '.' # Attempt to resolve any directory given as an argument if len(sys.argv) > 1: target = os.path.relpath( sys.argv[1] ) if os.path.isdir(target): read_directory(target) else: print('{} is not a valid directory'.format(target)) exit(1) # Execute Main Routine if '__main__' == __name__: main()

#portfolio #python

Much of my productivity happens after hours. This started out as an avatar and took a bit of a detour.

#finally #finding time

Who are the real heros of programming? My customers. My customers do great things. They often need my software, built and functioning properly for years to do these things. I love building stuff, but they are the real heroes. Just some of the things that they do: - get the right drugs get to the right people - get the ambulance to the right address - get the right materials purchased and delivered - get the right product built, on time and budget - get the right product shipped accurately and on time - make sure the parts going into that airplane are certified - make sure your insurance claim gets processed properly - make sure they make enough $, so they can keep doing it I can go on and on, but you kinda get the idea. I love to learn, to optimize and refactor, and to build beautiful things. But what I do pales in comparison to what they need to do. I never forget that.

Ed Weissman, 2010

#humility

HTTPTools

A subset of my PHP prototypes, I use these helper static classes whenever I need to work with form data, request methods, specialized output and session operations to avoid boilerplate isset() array_key_exists() madness.

Repository

https://github.com/spoonchucks/prototype-php/tree/master/HttpTools

Example Usage

Since no individual component depends on the existence of the other, each can be used on its own.

Request.php (source)

At its core, this is a wrapper for the $_REQUEST superglobal.

<?php if (Request::POST()) { // Contains value of $_POST['userId'], $_GET['userId'] if they exist or NULL if not $foo = Request::getParam('userId'); // The following lines are equivalent if (null === $foo) echo '$foo contains no value'; if (!Request::exists('userId')) echo 'No userId field set with Form data'; }

Session.php (source)

At its core, this is a wrapper for the $_SESSION superglobal.

<?php Session::put('somevar', 'Hello World'); // puts 'Hello World' into $_SESSION['somevar'] $somevar = Session::get('somevar'); // returns value of $_SESSION['somevar'] Session::erase('somevar'); // unsets $_SESSION['somevar'] // other methods Session::exists('somevar'); // returns true or false Session::wipe(); // unsets all Session variables Session::close(); // destroys the Session // debugging Session::dump(); // dumps a table of all the Session variables

Response.php (source)

This provides a simple way to output special data. It also eliminates the need to prepend <pre> tags or force Content-Type headers and exit() for dumping out complex data structures for debugging.

<?php // // USE CASE 1 -- JSON // $someObject = new User(); $someObject->firstName = 'Dave'; $someObject->favoriteFoods = ['hamburgers', 'pancakes']; $statusCode = Response::CREATED; Response::json($someObject, $statusCode); // // USE CASE 2 -- DOWNLOAD AS ATTACHMENT // Response::prepareDownload('image/jpeg', 'default_filename.jpg'); fpassthru($fp); // // DEBUGGING // $complexDataStructure = getSomethingComplex(); Response::text($complexDataStructure);

#portfolio #php #httptools

This script pulls and parses this blog's RSS feed, saving each article into a MongoDB database.

#python #getting comfortable #portfolio

WiFi Overcrowding

Naturally, the first instinct was to assume it's all Comcast's fault. After multiple speed tests and finally calling Comcast on Sunday to see if there was an issue on their end, I managed to narrow it down to network traffic on the router. Ping times from my laptop to my modem's IP yielded a 3-digit ping time -- Yikes. So now that I know that the issue lives on my LAN, I set out to fix the issue.

Why does my router hate me?

I ping -f the router to see that this thing is dropping packets like crazy. The router is running 802.11b/g/n on dual bands. I try changing channels, setting wireless interference mitigation mode and a whole bunch of other things. No lasting impact. While wired connections to the router would yield a ping time around 1ms, pinging the router from any wireless node ranged between 200 and 600ms.

Maybe it's interference?

There are upwards of 30 SSID's broadcasting in detectable range of our apartment, many of them clustered on the same channels. There are a vast assortment of protocols from 802.11a thru n. For a moment during one incident, I noticed that while the MacBook and iPhones were experiencing terrible latency, the Inspiron was chugging along without much issue at a much lower ping time. The Inspiron was on wireless-g and the MacBook and iPhones wireless-n. I tried moving those devices from n to g and the issue seemed to go away for a while. Ping times were consistently around 15-40ms.

Later that night...

Problem manifests itself again later that night when we're trying to watch Netflix (seriously, WiFi? Now?). Ping is through the roof on all wireless nodes and they're all on wireless-g. At this point, I start having a suspicion that the router is going bad and am considering just running to Best Buy to pick up a new one. It just so happened that they would have been closed for the night by the time I got there, so I continue fumbling around in the dark trying to fix this with every possible unscientific method.

I switched off WMM on the 2.4GHz adapter, bumped its transmit power to 200 and still no joy. I bring up the device status page to see the signal quality for all connected devices. I noticed that the MacBook -- not being used but set to automatically wake at 8PM and sleep at midnight -- was still connected to the router. I ssh in and run a quick shutdown -h now to kick it off the net. Mysteriously, ping drops to <50ms. W...T...F...?

Update (2013-09-12 19:45)

The issue seems to have ceased since then. I still have no clue as to what was actually causing the issue. I am inclined to believe that it was the MacBook's fault -- it is getting rather old and maybe the wifi card is failing (is this even a thing)? All I know is that since disabling the MacBook's scheduled wake-up, the issue has not resurfaced. Do I think my MacBook is DDoS'ing my LAN? Is this issue even resolved? Time will tell.

Further Reading:

http://www.networkworld.com/news/tech/2010/080210-wifi-interference.html

#wifi #troubleshooting #router

Adventures in Dual Booting

1. Boot from Debian Install Disc

Do it.

2. Start in Recovery Mode

I wasn't able to actually mount the partition until I took this step.

3. Switch to a Console (Alt+F2)

Once you activate a console, mount the old Debian partition at /mnt:

# mount /dev/sda5 /mnt

Then, create a symlink to the boot directory on that partition at the root of the temporary recovery file system:

# ln -s /mnt/boot /boot

4. From the mounted partition, execute grub-setup

I neglected to actually write down where this executable lives (I believe it was somewhere under lib in linux-i386 something or other). Push comes to shove, just do this:

# find /mnt -name grub-setup

You should get no error messages. Then, just reboot and the GRUB menu should reappear.

Warning: I attempted to run grub-install first which complained about not being able to find /sbin/grub-setup. This may have done something before complaining; I'm not sure.

Conclusion

Still need to figure out why I'm getting a error: no such device: {UUID} when I select windows from the list. It boots fine but still gives me the warning.

#debian #linux #dual boot #windows 7 #grub

Python Impressions

Disclaimer: I have a habit of making sweeping generalizations that are probably not universally true -- this effect is magnified when discussing things that are new to me.

Capitalization and Naming Conventions

So the Python style guide promotes the following naming convention:

class SomeClass: def some_method_that_does_something(self, other_item): some_variable = other_item.do_something() return some_variable

This is troubling for me since I'm solidly in the "camel case" camp. As a user of multiple programming languages and an advocate for making code as descriptive as possible, I give my variables meaningful names such as numberOfRecords or distanceFromCenter.

From an ergonomic perspective, the equivalent of these, number_of_records and distance_from_center takes me away from my "keystroke zone" -- I have to jump top-right to grab the underscore each time I reach a word boundary. This usually leads to more typos as on return from the voyage to the underscore key, I end up hitting the wrong key next.

From a typographical perspective, I'd argue that the use of underscores breaks up the visual silhouette a variable/function name has by introducing visual gaps which may introduce difficulty in quickly reading and comprehending long or "busy" statements, such as:

take_multiple_objects_and_do_stuff(first_object_in_recordset, second_object_in_recordset, some_property)

Lastly, from a purely aesthetic perspective, I think underscored names give the appearance of antiquity -- like "the old way of doing things" -- such as the difference between procedural mysql_fetch_object() and object-oriented $db->fetchObject(). A frivolous complaint to be sure, as nothing is stopping anyone from rewriting PDO with begin_transaction() or set_fetch_mode(), Python style.

As a realist, I understand that to other "normal" developers, there is probably no discernible difference to doing it one way or the other. As an irrationally stubborn person, I will probably continue writing camel-cased code anyway, Python convention be damned.

Namespacing via Modules

Python also presents a challenge to me when structuring modules. I like structuring my PHP with one class per file, namespaces to group like items and provide deconfliction and, if necessary, use of classes by referencing their fully qualified names thusly:

./blog/models/Story.php

namespace blog\models; class Story { private $subject; private $content; }

./blog/helpers/PermalinkHelper.php

namespace blog\helpers; class Helper { public getPermalink() { return 'something.html'; } }

./blog/index.php

require 'models/Story.php'; require 'helpers/PermalinkHelper.php'; use blog\models\Story; $story = new Story() $permalink = new blog\helpers\PermalinkHelper();

In Python, I haven't found the sweet spot yet. With keeping the exact same number of files, it would look like this:

./blog/models/Story.py

class Story: def __init__(self): self.subject = None self.content = None

./blog/helpers/PermalinkHelper.py

class PermalinkHelper: def getPermalink(self): return 'something.html'

./blog/main.py

import models.Story.Story import helpers.PermalinkHelper.PermalinkHelper story = models.Story.Story() permalink = helpers.PermalinkHelper.PermalinkHelper()

To me this is hideous. I am aware of from models import Story but say I don't want Story in the main file's global namespace? If I just wanted to be able to refer to the class by what I believe should be its fully qualified name (models.Story), I'd have to do one of the following:

Add an intermediate step of defining an __init__.py file inside of the models folder with from Story import Story

Create a models.py in the root folder and put every single class I want nested directly under models in there. (Disclaimer: I haven't tried to import asterisk from asterisk).

I'm sure that with further research, I will find an architecture that agrees with my delicate sensibilities. The quest goes on!

#python #php #complaining

Slight Change of Course

So, the plan for converting my CMS, blog.php over to Python has been shelved indefinitely. From a technical standpoint, there's not much stopping me from rewriting the entire project in Python. It might take me a few weeks and a whole mess of spit and duct tape, but I know I could do it. But for the following reasons, I have concluded that it would be the equivalent of performing rote coding exercises instead of actual learning and might not be the best use of my already splintered free time.

Covering Old Ground

Essentially, this project would just be busy work of reimplementing the same patterns from the previous build with a few "lessons learned" tweaks tossed in for good measure. And those tweaks would come at the heavy cost of increased testing time and increased time to completion. I spent an entire evening caught up in the design phase, overwhelmed by feature creep. "Oo shiny, add it to the list". "Definitely going to need that, add it to the list". Again, this is not learning.

Reinventing the Wheel

I thought about all of the distinct components required to do this properly. Low-level CRUD is of course very straightforward but for a blog tool to be anything more than Notepad.exe in the browser, there's a slightly higher level of functionality required.

One example would be that I needed some type of presentation shorthand syntax, much like Markdown. So I'd either have to write a complex and doubtlessly highly buggy string filter to split by newline where it doesn't find HTML block tags and append P tags or whatever or use a third party library. Or, HTML templating in general. Do I just use whatever Django uses? Do I write my own custom blocks?

And now that I'm already walking the dark path of outsourcing functionality by considering third party tools, why stop there? Why not pull in all kinds of other different components and just Frankenstein that mess together?

Time is Money

Okay, so I don't actually think that time == money, but time is finite and a scarce resource I can no longer afford to use unstrategically. With my oldest getting ready to go to school in two weeks, I accept that my family needs my time when I'm at home. I don't want to spend it cheaply grinding away on something I've already solved just to prove some point. blog.php is approximately at 7,000+ LOC as of today. That's a lot of boilerplate to transcribe.

Tumblr's got it Goin' On

I like it for several reasons, most notably the fact that it supports custom domains out of the box and posting different types of media. The fact that it has an iPhone app is just the bees knees. Another developer that I respect also has a tumblr blog/custom domain setup and I loves me some bandwagon!

But what does it all mean, Basil?

Once I refocused on the original point of blog.php, it all became clear. The point wasn't to write a CMS, although I did manage to delude myself into believing I would use it extensively. The point was to learn more about Model-View-Controller architecture. I think I've done that.

I realize this may seem like a betrayal of the original premise of starting a blog -- building on free land and not in the walled garden social media giants -- but the way that tumblr is set up, I'm positive that I'd never have to worry about losing my content as I plan on making rolling backups via RSS extraction. That would be something new to write and would definitely be done in Python!

#blog #python #php #refactoring

(UI Testing) Testing the Chat Log Post Type

Dave: Hi Dave

Dave: Yes, you are talking to yourself.

Dave: Indeed. But to be fair, this blog is basically me talking to myself anyway since I have no users.

Dave: Ouch.

#testing

Tester.js

A testing framework which I wrote to take the boilerplate out of writing unit tests. It provides a few basic services which come in handy for testing.

Live Demonstration

http://spoonchucks.github.io/javascript/Tester/

Repository

https://github.com/spoonchucks/Tester.js

Example Usage

The biggest utility in my opinion is the recursive print (print_r() method).

Tester.init('Tester Framework') .addCase('Assertions', function() { assert(true, 'Evaluate simple *boolean* true'); assert(false, 'Evaluate simple *boolean* false'); }) .addCase('Demonstration of :propertyDump():', function() { var collection = [ {key:'alpha-0123', name:'Derpaloo', number:2} ,{key:'bravo-4567', name:'Never gonna give you up', number:30} ,{key:'charlie-8910', name:'People are people', number:28} ]; // Dump an Array propertyDump(collection); // Dump an object propertyDump(collection[0]); }) .addCase('Demonstration of :print_r():', function() { var collection = [ {name:'Simple', obj:{foo:'bar'}, payload:['One', 'Two', [3,3,3]]} ,{name:'Nested Arrays', payload:[[[0, 1, 2], [3, 4, 5]], [6, 7, 8]]} ]; // Dump the recursive print of an array document.write(print_r(collection)); }) .render();

#portfolio #javascript

Trending Blogs

Recently Viewed Blogs

And So It Continues...