#hackdiary

[tommorrisdotorg]

Hackdiary: Eclipse, Lombok and code formatting

I'm writing some Java at the moment. It's lot of fun. Java is fun, right?

I'm using Lombok to make Java more fun. More magic equals more fun.

And I'm using Eclipse, because, well, Lombok works well with Eclipse. I slightly prefer IntelliJ but whatever. Eclipse works fine.

Except the Eclipse code formatter which is a pain in the ass.

So, you've created a field. Let's call it name, and it's a String.

With Lombok, you write this:

Then the Eclipse formatter turns it into this.

Which is fine for some annotations that are long, but you don't need it when it is just a getter and setter annotation from Lombok.

The solution is simple:

Open your preferences

Go to Java, Code Style, Formatter.

If the active profile is "Eclipse [built-in]", click "New", and create a new profile called something like "Eclipse and Lombok"

Click 'Edit'

Go to the 'New Lines' tab

Go down to the annotations section and make sure "Insert new line after annotations on fields" is set.

Hit OK.

Reformat your code and watch your sanity be restored.

[tommorrisdotorg]

Hackdiary: Sinatra vs. htaccess

Last week, I had a big problem. Someone hacked into my PayPal account and attempted to steal £931. Fortunately, they did so while I was at my computer, and I thus got an immediate PayPal notification. I contested it immediately, and 26 hours later, PayPal closed the dispute in my favour. I also immediately cancelled my direct debit instructions to PayPal, and have no intention to reactivate them until I need them.

It finally prompted me to sort out security on a lot of services, and to prune back a lot things I don't need. I realised that one of these was my shared hosting account. It's paid up for another year, but why bother? I'm not using it. I'm paying money for an account on a machine that I don't use. I quickly moved my email over to Google Apps, so I finally have a Google account that's tied to my proper email, rather than the bullshit Gmail username I came up with way back when.

Okay, what else was I hosting? My blog. This has moved between so many different services, I can't count. Originally, I hosted it on PHP Nuke and PostNuke (hint: stay well clear), then eventually I graduated to running a system called GLUE (no longer online), a sort of group multi-blog system. I tried WordPress and WordPress-mu. I used Dave Winer's OPML editor system, and then wrote my own version of that in PHP that used a lot of dark magic involving XSLT. Finally, I found Tumblr, and decided to outsource my blog to them. Less code to maintain. Goody.

Basically all that was left on my shared hosting site was a directory called files with general clutter I've uploaded (images mostly) and an htaccess file pointing various URLs to their new homes on Tumblr or elsewhere.

Now, to the point of this post. I bloody hate htaccess files. The syntax is unintuitive. It's basically a configuration file that has grown to the point where it's doing too much. When you have the bizarre situation of writing a rewrite rule that exists and then putting a flag on it that says "yeah, if this one is matched, skip the next rewrite rule but not the one after it", you've reached the point where... you need a goddamn if keyword. Embrace the code.

Enter Sinatra. Sinatra does the job quite magnificently.

I just create a tiny little Sinatra app and start creating URL handlers.

There's quite a few more redirect rules in there.

Pretty yawn-inducing, right?

Sure. But what is exciting is how easy to test this is.

What a revelation. Being able to test your URL routes is... eminently reasonable. Try Googling the phrase "testing htaccess files" and enjoy the clusterfuck of crappy forum posts from sites like "webmasterworld".

The amount of time I've wasted in the past writing rewrite rules in htaccess files, and thinking "why can't I just write some tests for this?" is ridiculous.

But here, when I want to add something new, I just add a new test method, write a damn test, run ruby test.rb, then fix the app.rb, and repeat. The examples above are very simple, but I've got some complicated routing where there are conditionals and so on. With this, it's a piece of piss to test. When I'm done, I check it into git and push to Heroku.

Beyond what's listed here, I just have to add a Gemfile and a Rackup file (config.ru) and it's done. I can test it locally without having to have Apache installed and running. I can test it. And it's written in actual code in an actual programming language (Ruby) rather than a terrible programming language pretending to be a configuration file.

Given how trivial this all is, why say it? Because it's such a relief how simple it all is. I first started doing Ruby because it's simple and lightweight. That's gone now. When I start a new Ruby project, I have to download a new version of Ruby and compile it (and probably complain at me that on the machine I'm using the /usr/bin/gcc symlink doesn't actually point to the GNU C Compiler but to Clang). Really.

Way back when, you could do Rails by typing sudo gem install rails; rails myapp; cd myapp; script/server. Ha. No pom.xml, no waiting for Maven to download an AbstractEnterpriseConfigurationFactoryFactory JAR. Now the level of complexity is such that you need to understand RVM and Bundler and all that shit before you can even start. Why? Because some people want to be able to run apps with different sets of RubyGems on different versions of Ruby.

I'm so glad we've all switched to Git. You know, Git, the version control system you can check your code into at 30,000 feet. Awesome. Except, of course, you can't because RVM has just realised that you haven't got Ruby 1.9.3 patch version whatever as specified in the .rvmrc file, and needs to download six versions of the blah gem. Congratulations on hacking on this while you are a mile above Kansas. Use that time that you can't code while Bundler is unable to download half of the Internet to write a blog post scoffing at Java programmers for Maven having to download half of the Internet, safe in the knoweldge that the Ruby community would never create something equivalently complex and awful...

I'm now accustomed to Ruby being a complicated clusterfuck involving RVM, Gemfiles and reinstalling and recompiling gems all the fucking time that when it all actually works, it's surpising enough that I feel the need to write a blog post about it.

[tommorrisdotorg]

Hackdiary: Redmine to Bitbucket conversion script

I'm currently in mad changeover mode with my work project. (And, yes, it's Saturday evening, and I'm writing about my work project. Don't judge me.)

We're in the process of changing a lot of things. Rails 2.3 to Rails 3. VPS slice to Heroku (the thought of never having to debug Capistrano is a nice one). I sort of realised we've got a lot of clutter that just isn't necessary these days. For instance, I'm hosting a Redmine install for this project. This is unnecessary. Bitbucket provides free private hosting for Git and Mercurial repos, and a wiki and issue tracker that goes with each one. This is more than good enough for my personal code, and just fine for this work project. It's one less thing to worry about.

But getting the issues from Redmine over to Bitbucket is one thing that needed doing. It's actually pretty easy. In Redmine, I just exported a CSV of all the issues. Then adding the issues to Bitbucket is as easy as an HTTP POST – see Atlassian's docs.

I wrote a crappy little Python script to parse the CSV (I pruned the first line off by hand) and send the stuff to Bitbucket. Here's the code.1 Consider the code licensed under the MIT License if that floats your boat. I won't save your sorry ass if something goes wrong though. It also doesn't do everything, just the stuff I care about. Again, you have a brain, you can probably edit it to do what you want.

There's lots more that needs doing, and before I shut down the Redmine server, I'm obviously going to take a backup of the Postgres database that the Redmine is running on.

Yes, I see the humour in hosting a script to put issues into Bitbucket on Github's Gist. ↩︎

[tommorrisdotorg]

Hackdiary: MediaWiki API noodling in Ruby with monkey-patching

I've been noodling with MediaWiki::Gateway today. It's a not particularly feature-rich, but otherwise awesome Ruby library for talking to MediaWiki. The not particularly feature-rich part is important: there are other libraries which do a much better job at abstracting away the complexities of MediaWiki. For instance, there's PyWikipediabot and the excellent .NET library WikiFunctions.dll.1 Both contain a lot of extremely useful functionality.

But the nice thing with MediaWiki::Gateway in Ruby is that it is an extremely simple wrapper class around the API. If you know what you are doing, you can actually get quite a lot done with it. So, obviously, I need somewhere to start.

mkdir ~/code/Ruby/wikipedia/

cd ~/code/Ruby/wikipedia/

Now, I'm just writing some throwaway scripts. Why do I need a full directory tree with lib/ and all that? No, that's too much. So, I just create a file to put all the modifications I make to the MediaWiki Gateway library.

touch mediawiki_ext.rb

I noodle around in IRB a bit. And eventually I realise there's no category function in MediaWiki Gateway. (Categories are kind of useful in encyclopedias, right?) So I open up mediawiki_ext.rb and monkey patch one in.

Now I can just fire up IRB, or write a script, and just call

require "mediawiki_ext"

And I've got MediaWiki Gateway, and my monkey patched method. Handy.

This is obviously rather an ugly way to write software. No unit tests! It's, like, old school bad-old-days Ruby. I don't care. I could instead check out the source code of MediaWiki, modify it, build my own gem. But then I have to give a shit about Bundler and Rake and all sorts of shit I don't actually care about.

The point of writing this code is I'm finding things out from the MediaWiki API. I might have a list of 20,000 articles and I need to see which ones are in which categories. I don't particularly care about modifying the library, I care about getting the results I want.

Now, I can, of course, at a later date, copy this function over into the relevant part of MediaWiki Gateway. On my own branch, obviously, and then maybe send in a pull request. That'd be useful. But I don't actually want to do that now. I'm not writing code at the moment for the sake of improving this Ruby module, I'm writing code to fix stuff on Wikipedia.

WikiFunctions.dll is the underlying library used by the super-useful but also super-unfriendly Windows application AutoWikiBrowser. ↩︎

[tommorrisdotorg]

Hackdiary: Wikihacking in Brighton

This weekend, I went down to Brighton for the MediaWiki Hackathon. I'm not a MediaWiki developer, and I only really do PHP when I need to. I do have access to the Toolserver which is a live, read-only mirror of the databases that power the Wikimedia projects. This is very useful: if I need to, I can log in via SSH, type in sql -r enwiki_p and run SQL queries against Wikipedia.

So, what did I build at the Hackathon?

Not as much as I'd like. The four regular MediaWiki developers there smashed lots of bugs1. Me? I worked on a few different things.

Open Plaques

I helped Jez from Open Plaques with the MediaWiki API, specifically Open Plaques can start using Wikimedia Commons to host images of plaques. A while back, I started pushing at Open Plaques to have Commons as an alternative file host to Flickr. Currently, Open Plaques recommends that you take a photo, CC license it, then post it on Flickr with a machine tag, and then Open Plaques pulls it in, and Flickr links to Open Plaques.

But Open Plaques could also support Commons in three ways:

By making it easy to add images from Commons.

We might be able to export all the license compatible files from Flickr to Commons, but with added metadata from Open Plaques (OP data is public domain although the photos are under various CC licenses, not all of which are Commons compatible).

Using Commons as a file hosting back-end: when people come to Open Plaques, they could upload photos directly to Open Plaques, and we'd then push them straight on to Commons and use that as a file host. They'd obviously have to agree to the relevant license and so on.

The first thing is a fairly straightforward one: being able to simply provide a Commons URL and then extract image metadata and the path to the image. That's relatively easy, and it looks like that might be something we can add.

Scala

After Jez left, I tried to work out what to do next. I decided it might be useful to have an API library in my trendier-than-thou functional programming language of choice, Scala. On the bus home, I made a few notes about the general design of such a library. After a bit of noodling with Maven, I actually got to the point where i could type mvn compile.

That I have to play these silly games every time I start a new Scala project is profoundly depressing. I basically either need to stop being a wimp and fully embrace SBT 0.10/0.11, or I need to write myself a new Maven archetype that does Scala properly and has all the libraries I want. And to wrap said archetype in a command line alias called something like "scala-new-project". Scala is supposed to be a fun, pragmatic and functional (in the sense of not-dysfunctional) language. Choosing between SBT and Maven is basically a choice between choosing a build system designed by hipsters and a build system designed by enterprise people. Whatever choice you make you will regret.

My build still has one major issue: mvn scala:console gives me errors, and JLine still buggers up my shell after I exit the damn console.

Databinder Dispatch is a breath of fresh air. It is intimidating at first to use a library where 70% of it seems to be punctuation, but the design decisions make sense. For using the MediaWiki API, it's actually very easy: you can basically do each different thing as a series of layered objects. Firstly, you construct a Request object that points to the API endpoint (MediaWiki isn't a RESTful API), then you use the <:< pseudo-operator to add all the relevant headers (specifically User-Agent), and then you can do it again to add in the authentication token (cookies rather than OAuth, but the principle is the same). Then, finally, you can simply provide a Map of the query you are sending, and then you supply an inline response handler.

I haven't yet gotten anywhere particularly interesting with the Scala library, but I've got some good ideas. It's currently in a closed repo. When it sucks less, I'll release it. That might not be for some time, sadly.

VMs, vagrants and maintenance/dev/

Over the weekend, I mentioned to one of the experienced MediaWiki developers about one of the things that puts me off MediaWiki hacking: dependencies. Hacking on MediaWiki generally means having a working MySQL install and a working Apache install, and so on. This can be a giant pain-in-the-ass on OS X, as the bazillions of tutorials on how to set them all up in the right way has shown. There's a reason why people are using things like Vagrant and other VM based systems. The same sort of quasi-VM type strategy seems to be happening with RVM, the Ruby Version Manager, which the Ruby community nicked from Python's virtualenv. Obviously, if you are in Javaland, you are running in a VM... the JVM.

But PHP is a bit more of a pain: this is the downside of being built around the very reasonable use case of "I want to be able to FTP it onto my server and have it work". What you lose in the development stage, you more than make up for in deployment.

At this point, someone pointed me to this discussion on wikitech-l.

The current bleeding edge MediaWiki basically has a script/server, and a built-in SQLite. It's almost like... Rails! You simply run maintenance/dev/install.php and it downloads PHP 5.4, and sets you up a development version of MediaWiki that uses PHP 5.4's built in web server. You can then just run maintenance/dev/start.sh and it'll boot up on port 4881.

There was one slight hiccup: the PHP 5.4 build script didn't like the fact that i had a space in one of the names of a parent directory to the SVN checkout of MediaWiki and borked. I quickly renamed it to remove the space and it compiled fine. Once the initial compile was done, I can now boot up a new, clean MediaWiki install in a few seconds. This makes it dramatically easier to start hacking on MediaWiki.

And now.

Although I didn't do any MediaWiki hacking over the weekend, I've just assigned to myself an issue. Lemme see if I can get some of my code running on Wikipedia...

After I took that photo, they finished work on a really tough bug involving GeSHi. ↩︎

[tommorrisdotorg]

Folding in Ruby and Scala

I'd like to think I'm no slouch at functional programming, but I came across a nice little example today of where Ruby's functional programming features weird me out: inject.

Consider a perfectly simple example of where functional programming would be useful: summing an array of objects of the same type that all have a property or method that returns an integer value.

Here is how one might do it in Java (or C, PHP or another language without the benefit of higher order functions):

That's easy enough. You might need to deal with what happens the length value is nil, and you might need to handle exceptions. Whatever.

But, of course, functional programming is here to make your life better. You shouldn't need to write the same foreach-accumulator code every time for every different property. So, imagine if we had an array of objects that had five different properties that are all integers: length, height, weight, smelliness. We want to sum them all, or average them all or whatever. This is where higher-order functions become so useful.

In Scala:

Okay, now imagine we've got something similar in Ruby. The code transliterates reasonably well.

Two nitpicky things here about syntax:

The map syntax is simpler in Scala: I've always found "and-colon" in Ruby to be a rather ugly and unreadable construction. I understand it alright, but I'm not confident that Ruby newbies understand it. (But, then, lotsa Ruby newbies don't grok functional programming: I certainly didn't.)

The word "inject" is a bloody strange word for a well-understood concept: every other language with fold-left calls it fold-left (or foldLeft or foldl or fold or something similar involving the word "fold"). "inject" is eminently forgettable.

I could, of course, monkey patch in "fold_left" as an alias to inject.

Here is where it gets more interesting for Scala though. The type system lets you do type-dependent monkey patching.

Consider the generic concept of folding or reducing values: it is simply a process where you turn numerous values of the same type into a unified result of a particular type. So, summing is a folding operation, as is producing a mean, a mode, a median etc.

Enter Scala implicits.

Now one can do something like this...

How does this differ from just monkey patching 'sum' into Ruby's Array class? It only applies to lists made exclusively of one particular type: Ints. Implicits let you target specific types in a way that Ruby monkey patching doesn't. This is pretty powerful, and yet another reason to learn to love static typing.

You can call sum on List[Int] but not on List[String] (unless you define a new class that implements a sum method and an implicit conversion function for List[String]). Without one of those, you'll get a type error if you call something like the following:

Think about the power of this: programming often boils down to doing some repetitive action to a list of values of the same type. In Ruby, you can monkey patch the individual class, but think about how you can make code more readable by adding methods to both the objects and the collections of objects, without those modifications spilling out and affecting collections of objects of other types.

I think this is a real example of where Scala's static type system combined with implicits and functional programming allows for a far more expressive way of coding.

[tommorrisdotorg]

Trying IronPython

For reasons I can't talk about but which may or may not become clear soon, I'm brushing up on my Python. I decided today to give IronPython a try, since of the combinations of zip (Iron, J) (Ruby, Python) it was the only one I hadn't really tried.

As I don't really do Windows or Mono, I didn't really know where to start. With most JVM-but-not-Java languages I try, the first thing I do to test the interop is to try and create a collections object from the JCF. This seems like a pretty good test of a .NET language like IronPython.

Doing so is easy enough.

from System.Collections.Generic import Dictionary foo = Dictionary[str, float]() foo.Add("bar", 5) foo.TryGetValue("bar") #=> (True, 5.0)

But what if I want to iterate through the .NET dictionary? Well, if it were a Python dictionary, I'd do something like the following. So let's try that.

for k, v in foo: print k, v

TypeError: 'KeyValuePair[str, float]' object not iterable

Okay, that makes sense. It's basically a list with a load of KeyValuePair's in the list, and a convenience method which finds the one which matches the key.

So I have to do this instead:

for i in foo: print i.Key, i.Value

This will take some getting used to.

Interestingly, this is where my lack of Python skills gets annoying. If this were Ruby and I didn't like how it was behaving, I'd reopen that class and patch it into shape (and then curse myself later on). With Scala, I could do the same, albeit with some magic and indirection.

Actually, I figured it out.

So, you define some function that does it using yield.

def dotnet_iter(coll): for p in coll: yield (p.Key, p.Value)

Now you call that to iterate the .NET collections:

for k, v in dotnet_iter(foo): print k, v

Obviously, you can wrap that into objects for abstraction.

One other slight annoyance with IronPython that slightly got me coming from Ruby and JRuby is that JRuby does some clever stuff with function/method names. So, imagine you've got a Java class that uses the standard Java way of naming things, camelCaseWithNoLeadingUppercaseCharacter(). In JRuby, you can call this as either that or as camel_case_with_no_leading_uppercase_character - the Rubyish way of naming methods.

In IronPython, no such luck. The .NET thing of giving you upper case property/method names is a bit jarring in Python (or Ruby).

The other irritation I find with trying out anything Iron- is Microsoft's bloody abysmal terminal. It truly is the worst bit about using a Windows machine. Everything else you can learn to tolerate, but the console is absolutely soul-destroying. The crappy font, mark mode, no readline. And it's everywhere. Everything you use brings up a horrible blood-curdling DOS-era terminal. Even PowerShell: Microsoft's "next-gen" shell for leet sysadmin-fu. I do not know how the Windows gang get any hacking done. (Don't tell me: Visual Studio.)

[tommorrisdotorg]

Things I'm currently quite excited about

There's a few tech things I'm pretty excited about at the moment.

Firstly, Scala is now at the point where it is actually productive. Scala 2.9.0 RC1 has just come out and it's got freaking parallel collections in the standard library. Finally, I can stop looking so longingly at Clojure. As far as I can tell, you'll be able to do something like this:

Array(1,2,3,4,5) par map (_ * 2) seq

And it'll make an array, turn it into a ParArray, map it, and turn it back into a sequential Array. That's pretty cool.

Then there's Scalatra. Scalatra is Sinatra but for Scala. And the setup is very simple. Once you've written the app, you get a WAR file, which you can upload to...

Elastic Beanstalk. Finally, Amazon have delivered a Heroku-like experience. This is really exciting for hacking stuff together. For 2¢ an hour, you can host JVM apps on an EC2 micro-slice, but without having to touch the operating system: you literally go to the website, pick a subdomain and upload a JAR file. The Hello World I wrote in Scalatra was about 5Mb.

Completely unrelated: WebGL. Holy moley is that exciting. If you are running Chrome or Firefox 4, go and ogle this. It's beautiful. I'm not nearly clever enough to do 3D programming, but just imagine a world of composable 3D objects with URIs and a universe of RDF-esque ties between them. Virtual objects roaming the one Web intermingled with thoughts and data in some kind of bizarre interlinked Platonic ether. That was kind of the dream of VRML, right? And is sort of the reality that Second Life has. But we could go further.

Imagine Wikipedia with WebGL: you click a little link in Wikipedia, and the article fades slightly only for the objects the article is representing to appear in front of you. You can rotate them, download them, throw them around in physics demonstrations, attach them to your avatar. But then each part of the WebGL model could have RDF metadata attached: this particular set of polygons is supposed to represent a particular type of material. You click and all the other virtual objects made of iron or copper or polyurethane or semi-transparent blue plastic or plutonium appear before you. As you are exploring Wikipedia, you copy a few items you find into your personal inventory, a sort of 3D version of your web bookmarks. Then you come across the Wikipedia article on nuclear reactors. And you are a curious sort, so you start seeing what happens if you chuck all sorts of strange things into the nuclear reactor. What happens if you drop this 3D model of Thomas Jefferson in to the reactor? Suddenly, our mashups have stopped being dumping crime figures on Google Maps and they've become shockingly real world and Grand Theft Auto-ish.

What else? Open Election Data. This one's a simple one: a new link in the Web of Data, joining data.gov.uk up with dbpedia. Haters gonna hate, right, but making it so that when one dataset is talking about "the Monster Raving Looney Party", they use an identifier that is linked to all the other representations of the same thing is the absolute start of the Web of Data. We may have to have these proxy projects to sit as intermediaries between governments and more direct community-created resources like Wikipedia/Dbpedia. This is an interesting start.

One final thing: Wikimedia. I've only got back into Wikipedia recently, and there's loads of fun stuff we can do with the data. I've got a list of hacks as long as my arm. I may work on one tomorrow at Rewired State.