Update #11: Into the Danger Zone
I guess it's more than time. We're in December, I'm back in Tenerife for a few days, I finally have Internet both in Hamburg and here (though the latter was kind of obvious), and it's time we spoke business.
Now trust me, as much as I would rather not have to be writing this, you won't like to read this either. But we're both in this together, and this is shaping to be only Part 1 of a very, very long trip down to the day this thing makes it into running on a device. If it ever does.
I'm not really sure where to begin, or where have the changes occurred. First up, let me tell you I have been working on this; for the next 5 weeks or so after my last post on the subject, I did put my head and soul into this. It always took me a little bit of adjustment to get back into code writing every weekend, but my mind was fully trenched into pushing this thing forward. Every single part of me still believes in this, still believes in the fact that this set of APIs + Engine implementation can really make a difference and can really be used and exploited. Specially now when in these days I've gone back to my roots on the Internet reading Twitter, Facebook (even though I despise most of it), checking out Tumblr and some stuff on Instagram.
Can you guys believe how awesome it'd be to have a pluggable Engine you could hook content off these sources and have them presented to you in a nice, straightforward, clean and filtered way? I really believe in this, and there's enough gas in my heart to still power a six-cylinder 1980s Mercedes-Benz S-Class engine to the moon and back when it comes to finishing this, but it's not going to be easy, and we're still a long, long way off arriving at the finishing line.
So for today, we're going to split this post into three distinctive parts. I don't usually do this, and common sense says that since I don't blog too much, I should instead split them into three different blog posts. On the other hand, I do love sitting on my couch with an iPad and going through a lot of text, so you'll probably end up reading this in a single post, after all.
I'll try to expose the matter the way I found out something was going incredibly wrong. Must've been somewhere like 4 weeks ago, when I was about halfway getting one of the bullet-points we've discussed to eternity looking somewhat close to feature-complete, with tests for what I'd finished so far already working, that I realized if I continued, something was going to break, that the general code structure was going to give away, and that there was no way of finishing this feature without breaking the spine of the whole project.
I left whatever work I'd done in that branch, switched to master, and continued to work on something else that I should've done from the beginning, but that was the Red Alarm that signaled to me something was incredibly wrong with the codebase, and that it needed serious fixing. It is still not a nice thing to realize, since if you look back at our history, I believe things were going great until week 4 or 5 when I discovered my work was nonsense because the Engine could not deal with inconsistent streams of information. That's the moment back in time where, I believe, the whole thing started going south.
At that point, all I cared about was fixing the issue I'd found and getting back on track as soon as possible, in order to get closer and closer to the moment I'd have to stop using my TestSupplier class. So I "fixed" the Engine, made it comply with my newly found set of requirements, and moved on, not realizing instead of fixing the issue and helping the project move forward, I'd actually baked a full Xmas tree about 6-7 months ahead of time that was going to expand and define the rest of the code's architecture.
So what was the issue? What IS the issue, better said?
It's not only that the code's fundamentally broken, it's also the fact that this project began being something FUN to work on. It was fun because when I first sat down and laid the class + dependency structure for our code I tried to write something that was beautiful both on the inside and on the outside. It had to be beautiful on the inside because this is not the first project I've tried to build on my own on a path to what I believe to be a breakthrough app concept. I believed this time around I'd be much more successful because:
a) I "know how to write code better now" (whatever that meant when I started)
b) I'm going to limit every class and every file to a max size of about 600 - 800 lines, which I considered more than acceptable to keep code from spiraling out of control.
c) Before starting to write a single line of code, I sat down and built out the structure of the code on paper, believing that initial design would stand the test of time and not allow me to threaten the fun-ness of the project by spreading out class dependencies through the code like a spider-web.
d) I would write a hell of a lot of tests, attempting to follow TDD to the best of my knowledge, in order to have confidence on my own work and allow me to make any changes I deemed necessary knowing I had those tests to back me up.
So let's go over each of these one by one, and see what or why those were not enough to keep what I thought was a beautiful code architecture from rotting hideously.
On the subject of a), I didn't really have any proof that the code I would be writing now would in fact be better. But what did I mean by that, exactly? Well, it was mostly related to the fact that my previous endeavors had me playing with the Facebook SDK and I ended up making a class literally 2K lines long, which constantly required me to scroll up and down through the source file because that same class had some sort of "two-stage pipeline" hidden deep underneath through which its final results were delivered. I remember that my code felt so fragile, so out of control, that even though when it worked, I considered it nothing sort of a disappointment because it was code I did not ever want to have to look at and think "you did that". Instead, it was a lot easier for me to hide it away and believe it worked, because it did work for the things I asked it to do. I was also terribly afraid of asking it to do something I hadn't asked before. That's what bad code buys you.
But there was a reason for that hideous code: it was written during a time when I considered myself nothing sort of the term "master programmer".
I thought I could basically accomplish and finish anything I wanted to because at that point nothing software-wise had ever gone wrong for me. Every time I had a challenge set in front of me I broke through it like it were a giant wall made out of paper, which lead me to believe just because my mind could structure everything I hoped to accomplish I would be able to do it no matter what.
So why am I telling you this? I'm telling you this because it doesn't take long to discover I was fundamentally wrong about everything I did during that previous endeavor. In fact, that single failure proved to be a huge lesson in humility for me, because I discovered some things are harder to do than what the comfort-zone of college would lead you to believe. I could fail, I could make mistakes, and after that, I was ready to accept everything I did and anything I tried could be done better. In fact, since that day, I consider myself nothing more than "mediocre" when it comes to writing code, simply because after throwing days at it, I'm always seeing stuff that could be done better, which in my opinion, is the point of trying to get better at your job every single day.
So hopefully you can take something home from this little story. Because I was were you are right now: reading an article, reading about the fool that was trying to write some code, made bad mistakes and wrote about it, thinking "that's not going to happen to me because I do things Y way which would've lead me to solution Z which he didn't arrive at before he went through X." Unless you've already gone through this and know exactly what I'm talking about, this can happen to you: just remember, whatever "class" of programmer you put yourself in, nothing is above that quintessential line reading "IMPROVEMENT". Nothing.
About b), limiting files to 600-800 lines is something all of our current code complies with. Issue here? Even 600 to 800 lines of code can produce a big mess, simply because even though I heavily pursued the DRY or Don't Repeat Yourself principle, I was messing heavily around with something else called SRP, or Single Responsibility Principle. This means that even though 100% of my classes abide by the 600-800 line rule, almost all of them, except the smaller ones, do a lot more than just one thing. This is specially true if we talk about the Engine and/or Mutable Track classes. The moment I realized there was some code packed into the SEMutableTrack class which I needed somewhere else, I knew whatever measures I was taking for the code to be consistent, simple, and fun to work with, had failed in a non-spectacular fashion.
Onto c). You could say, from a higher point of view, that what this was all about was refactoring code ahead of writing the code itself. The reason being I wanted to identify all of the actors involved in the process of building the Track ahead of time before I had all of the code scattered throughout different files requiring me to later put it back together in more consistent and object-oriented way. The funny thing here is that I was doing this upside down! If you think about it, most of the times you don't know what kinds of procedures and code you're going to need to write to get something to work, and believe it or not, that's perfectly fine! But, and here's the big but, getting something to "work" is nothing but half the job, the real task lies ahead, which is cleaning up everything you've worked so when you push the code back to master the codebase has not degraded in quality. In fact, the best way to guarantee the healthiness of the code on a day to day basis is to always push yourself to check the code back in looking even better, more polished, more clean, than when you checked it out. And I believe the ultimate breaking point was when I reworked the Standard Engine's internals and got it to work with the revised specification, but did not bother splitting up all of the new responsibilities the Engine was taking upon itself.
And finally, we arrive at d). Undeniably, this has to be one of the best decisions I ever made for this project. Tests. But even my best choice did not escape from making a couple of big mistakes itself, the first being the fact that I did not write the tests in a way that they were easy to read, and thus easy to work with and/or maintain. Tests are a big point of this and any other project's core strength: because I have these tests, I can make changes to the code making sure they don't break existing functionality, and I have the confidence that the tests will always let me know when something is not going well. But the mistake I made was to think that my source code needed to be at a higher degree of quality than my tests, because after all, the tests are never going to be used in production by potential users, right? That lead me to write tests in a sloppy way, and not worry too much about having to fix them later, since I did not think tests needed to change. But it's actually the other way round: tests are there not only to verify the code's behavior, but also to guide potential collaborators through the code. Now, even though the chances of this project having additional help are fairly slim, the second part of my previous sentence still applies here, because if I leave this project for a month again (just as I currently have), I'll become a collaborator of the code the "past me" wrote, which means I'll look for an anchor point to give me an idea of how things should work, and if the tests are written poorly, I'll take a long time to return to full speed, and most importantly, I'll do it with little confidence from my subsequent changes.
Also, take into account that tests are also code, and as such, they can always be written better and/or benefit from improvements from the Engine itself, so if tests are not easy to work with, easy to change and adapt, our single most important point of trust in our work, falls down.
But it's even more than that. I found out just about a week ago that there are tools that measure how much code is actually covered by our tests. Said tools are predominantly reserved for Java, with Atlassian's Clover and IDEA's being the most prominent examples. However, they're not free and most importantly, none work with Objective-C; I think Appcode doesn't have this either. But it turns out Xcode more or less supports this through LLVM, so I was able to see which parts off the master branch are covered. Want to know the % of code covered by my tests? Around 65%, which means 1 out of every 3 lines of code are not being tested. It doesn't sound like a bad thing, but this means a couple of files have a test coverage nearing 90-95%, while others are as low as 20%. There's clearly a lot of work to be done in this department.
So now we're going to find out what did I really manage to do before arriving to all of the conclusions mentioned above, and you'll also get to know what prompted me to see that finishing what I was working on would "break the spine" of the code. You want to know what the funny thing is? Last time we spoke, I told you the codebase was "rock solid".
So, the first thing I did after fixing all of the tests was disable the Password Lock feature (which has a nasty bug, by the way), and work on updating the codebase for iOS 7 and Xcode 5's newly added support for Doxygen-style documentation. The hardest part was coding what was the height required for displaying a certain amount of text for every Ticket in the track, something on Android you get for free with ListViews; Adapters and XML, but on iOS you have to calculate this yourself. I ended up doing this by hiding a UITextView as a child of the TableView and asking it its size after setting the text of the item I have to display. Not pretty, but if I managed to share that UITextView and refactor that code out of the way of all the Suppliers, it would work well enough, and it would stand the test of time. Next, I worked on some heavy refactoring: the code for caching Tickets is something I needed and that I built into the TestSupplier class, but feeling we may be closer than I thought to writing "real" Suppliers, I refactored that code out of the TestSupplier and tried to do it in a way that allowed us to re-use it later. I originally wanted this to happen by having a barebones Supplier class that gave you this kind of functionality for free when you subclassed it, but upon trying to make this happen Objective-C wouldn't play good friends with me, so I resigned myself to creating a class to do this, called the SupplierCacheManager. At least this is off into a separate class we can re-use any way we want or need to later.
More refactoring. I gave more power to the Commander: it is now responsible for telling the Engine what's the threshold for its current Suppliers, rather than the Engine having to do this calculation by itself. This essentially meant the Commander was leaving behind the days of being simply a bag with the Suppliers the Engine had to use, to the point where it was beginning to be responsible, or accountable, for its Suppliers. A good thing, I believe.
And here's where we move on to one of my two "frozen" branches off of my master. Want to know what's the first one? State-Saving. Yup, State-Saving is the feature I couldn't fully finish, but let's go through the process slowly. The first thing I did with this branch is update the code style a bit, leaving behind a lot of defines and translating them to constant integers. Next, I had to start fighting with NSCoding. You see, NSCoding is the Serializable of Java: it is the standard when it comes to making Objects persistent in Objective-C. However, contrary to Android where you have Parcelable as a much better alternative, NSCoding really is a mirror of Serializable in the sense that it works through an extra-linguistic paradigm for creating Objects, because the Objects you create are not going through their constructor/initializer. You might not think this is a big deal, but that's because you're not understanding the importance of constructing an Object: if an Object finishes its construction process, you expect that Object to be ready to use, and understand too that any parameters you might've passed were indeed correct because, the first thing a constructor/initializer has to do is establish the Class' construction invariants are indeed met, meaning you should expect all values and operations to be valid on that Object going forward. Again, what's the big deal? We should write constructors and initializers in such a way that if they finish, we know everything's OK. If a constructor/initializer detects something's amiss, construction should fail in a bad way, not because we want the user to suffer, but because we want ourselves to notice as soon as possible that something's gone wrong. NSCoding will not trigger our constructors; like Serializable, it'll instead re-assign each of the Object's member variables one by one, which means we now have not one, but two places in our code to worry about construction invariants. So NSCoding is something I dislike, and instead of propagating it everywhere I needed, I attempted to make my own "Persistable" interface, which is not fully consistent: some classes implement it and some don't, but seeing as how that branch is off of master, it's OK for now, though it's a mess I'll have to come back and fix, and when I eventually do that I'll explain the story a bit better. Then came the fun part, though: to implement State-Saving, we not only need to save the Track itself, but also the "Users" that have a role on it. That meant Suppliers needed to be a part of the party too, so I made them join. Not a lot of fun to be honest, but I did manage to get it to work, I did manage to write up a test which filled up a Track, saved it, trashed the Engine, fired up a new one, loaded the previously saved Track and verified everything was there, users included. But this was only half of the pie: if you're using this, and you manage to fill up a Track with more than 2-3K Tickets, you don't want to save them all: you want the app to be smarter than that. So like Tweetbot, my idea was to add gaps depending on the index of the Track the user is at: if you're at the beginning, we can add a Gap at the end and kill all of those Tickets. If you're at the bottom we can do the same at the top, and if you're on the middle of the Track, we can add Gaps at the top and bottom. But how do you tell the Engine "this is the index the user is looking at"? That's the question that broke everything for me, besides the fact I was having to add Gaps in places I had not foreseen I needed to, when the code for cutting Tickets for Gaps is in fact locked away inside the SEMutableTrack.
We hit that point at the beginning of last month, on November 10th. Seeing how I did not want to face that anymore, but wanted to keep working, I switched to another branch off of master, attempting to start doing the things I should've done time ago, like start taking into account the details. In this case, I began writing a Timeframe class and began retrofitting it throughout the code, backed by the tests. That's the last thing I was working on before I came here to see you. I did manage to do something small in these two days I've been in Tenerife: I wrote a TestCase class hiding away the code required to make sure the data for picking up test code coverage and then made all of my Test Cases subclass that one instead. Weirdly I had a very hard time making that class compile in the App's target rather than in a Test Target, but in the end I decided I had to get things done and that I could take care about that small thing later. So there we go.
Painful refactoring is what we're going to do, to be honest. Beginning first with finishing the Timeframe class, merging it into master, then looking at the State-Saving code, salvaging it, merging it back to master, and then reworking the innards of the Standard Engine until everything finally makes sense, and everything is exactly where it belongs. This will be akin to taking the exterior armor off of Gipsy Danger, pulling it all apart, fix everything on the inside, and put its body parts back on.
I can assure you, realizing most of the code I've written thus far is nothing less than rubbish is perhaps one of the most hurtful admissions I've ever had to do: I essentially made the same mistakes again I had two years ago. If code is not properly organized and not properly kept, it makes itself hard to work with regarding changes, and change is something software should be designed for from the start, not as means to salvage an architecture that doesn't hold up anymore.
I'll be seeing you soon. For now, I'm going to get back to reading on my iPad. Because I love reading on my iPad.
