Stream of my consciousness...

[perdalum-blog]

I am your father

[perdalum-blog]

We're heading for Venus (Venus) And still we stand tall Cause maybe they've seen us…

[perdalum-blog]

Angel of …

[perdalum-blog]

Building something to be awakened #lego #longbuild

[perdalum-blog]

#madewithpaper / fiftythree.com

[perdalum-blog]

#madewithpaper / fiftythree.com

[perdalum-blog]

Never miss another xkcd

Lately I've been forgetting to check xkcd thereby not being in on the loop at my office. Therefore:

1 #!/usr/local/bin/MathematicaScript -script 2 3 (* get today's xkcd as it is defined in it's JSON data *) 4 json=Import["http://xkcd.com/info.0.json", "JSON"]; 5 6 Export[$HomeDirectory <> "/Data/xkcd.png", 7 Labeled[ 8 Import["img" /. json,"PNG"], (* get the comic strip *) 9 Style[ 10 "alt" /. json, (* extract the alt-text *) 11 Background->LightYellow]]]

A launchd configuration, created with Lingon, to refresh the image file each day and a GeekTool setup that places the daily generated image on the top left corner of my Desktop.

[perdalum-blog]

Grasshopper

Made With Paper

[perdalum-blog]

Made With Paper

[perdalum-blog]

A simple Hadoop job using Clojure

At work we've been experimenting with MapReduce on the Hadoop platform. That means a lot of Java stuff, tinkering with both programming in Java and cluster administration. All of it rather exiting, albeit a bit frustrating at times. Luckily we've now reached a point where the Cloudera Manager is used for the cluster administration. Our first cluster consists of five 12-core Intel Blade servers each with about 50 GB of RAM and 4 TB of HDFS storage. The number of servers will probably be doubled in the coming months so we'll actually have some pretty nice hardware to experiment with. There exists a lot of different introductions to the MapReduce programming model, so I wont go into that here. Instead I'll just give you how I got introduced to it. I've been enjoying programming in functional languages for some years, primarily LISP and *Mathematica*. That interest gave a rather natural angle to MapReduce, like, yes off course that's the way to do it. Though it took me some time to get from the ordinary functional Map and Reduce/Apply model to the key, value based approach og MapReduce. In *Mathematica*, Python, LISP, etc, it's practical impossible to do any data manipulation without mapping and reducing your data. Still, there's a long way from data = Apply[reducer, Map[mapper[#], list]] to the standard word count Java example in [MapReduce Tutorial](http://hadoop.apache.org/docs/r0.20.2/mapred_tutorial.html#Example%3A+WordCount+v1.0). I really hate boiler plate code. I hate having to write the same word twice. Off course there's a lot of ways to avoid that in Java, but I like another approach to overcome that, so here enter [Clojure](http://clojure.org) and I'm about ready to show my little Hadoop job written in Clojure but before commencing the show, I'll start by showing you the code of [my-counter](https://github.com/perdalum/my-counter) — Go grab it and cast some Leinigen spells! The complete Hadoop job: (ns my-counter.core (:require [clojure-hadoop.wrap :as wrap] [clojure-hadoop.defjob :as defjob] [clojure-hadoop.imports :as imp]) (:import (java.util StringTokenizer)) (:use clojure-hadoop.job)) (imp/import-io) (imp/import-mapreduce) (defn my-map [key value] (map (fn [token] [token 1]) (enumeration-seq (StringTokenizer. value)))) (defn my-reduce [key values-fn] [[key (reduce + (values-fn))]]) (defn string-long-writer [^TaskInputOutputContext context ^String key value] (.write context (Text. key) (LongWritable. value))) (defn string-long-reduce-reader [^Text key wvalues] [(.toString key) (fn [] (map (fn [^LongWritable v] (.get v)) wvalues))]) (defjob/defjob job :map my-map :map-reader wrap/int-string-map-reader :map-writer string-long-writer :reduce my-reduce :reduce-reader string-long-reduce-reader :reduce-writer string-long-writer :output-key Text :output-value LongWritable :input-format :text :output-format :text :compress-output false :replace true) I not yet quite sure what every s-exp in the Clojure files do, but I like what I see, how concise the code is. The intention is very easy to read and the boiler plating is kept to a minimum. Apart from the implementation of the actual job I also like the project structure and the iterative development is facilitates. You can run the job and test it without a Hadoop system! Just execute the command: $ java -cp target/my-counter-0.1.0-SNAPSHOT-standalone.jar clojure_hadoop.job -job my-counter.core/job -input shakespeare.txt -output out On the Hadoop cluster the corresponding command is: $ hadoop jar my-counter-0.1.0-SNAPSHOT-standalone.jar clojure_hadoop.job -job my-counter.core/job -input shakespeare.txt -output out The next step will be to actual run the code at the REPL in a real explorative programming style.

[perdalum-blog]

Made with Paper

[perdalum-blog]

The Monkey King, a childhood hero — made with pencil, pen, and brushes

[perdalum-blog]

Made with Paper

[perdalum-blog]

Made with Paper

[perdalum-blog]

NRGi data igen

En af mine venner [](http://thrysoee.dk) byggede udfra min første NRGi artikel et Bash script som selv kan logge ind, hvilket jo passer rigtigt fint ind i denne datahøst. Jeg kunne bruge hans script direkte som skrevet eller jeg kunne, for at citere samme ven "I am rarely happier than when spending an entire day programming my computer to perform automatically a task that is completely useless to me.", skrive et lille Mathematica program baseret på hans script. Både det oprindelige Bash script og mit Mathematica program benytter Curl login funktionalitet til at oprette en https session og begynder så ellers bare at hente data. Så uden mere snak bringer jeg her lidt Mathematica kode: Først skal vi have et par URL'er url = "https://selvbetjening.nrgi.dk"; dataurl="/nrgi/am/export/day/"; Så definer jeg i hvilket tidsinterval jeg vil hente data. Begge datoer er her inklusive. startDate={2012,6,10}; endDate={2012,6,13}; For at oprette sessionen skal login-akkreditiver angives name="123456"; password="12345"; For at kunne logge ind via en input form har curl brug for et form id. Dette findes ud far dennes name som i dette tilfælde er: "form_build_id" html=Import[ url,"XMLObject"]; inputform=Cases[html,XMLElement["input",{___,"name"->"form_build_id",___},___],Infinity][[1]]; formid=(inputform[[2,2]]/.Rule->List)[[2]]; Det grundlæggende curl kald bliver med cookie jar således cookiejar = NotebookDirectory[] <> "cookiejar"; curlCommand="curl -L -s -b " <> cookiejar <>" -c " <> cookiejar; Og så skal det logges ind og et par cookies skal lægges i småkagekrukken loginCommand=curlCommand <> " -o /dev/null -d form_build_id=" <> formid <> " -d form_id=user_login -d name=" <> name <> " -d pass=" <> password <> " -d op=Log+ind " <> url; Run[loginCommand]; (* a test for exit equals zero might be nice... *) Her efter er det simpelt at lave en "Map" over en liste af datoer data=Map[ {#,Import["!"<>curlCommand <> " " <> url <> "/nrgi/am/export/day/"<>#<>".csv","Table", "HeaderLines"->1, "FieldSeparators"->";", "DateStringFormat"->{"Day","-","Month","-","Year"}, "NumberPoint"->","]}&, Table[ DateString[ DatePlus[startDate,i], {"Day","-","Month","-","Year"}], {i,0,DateDifference[startDate,endDate]}]]; Til sidst rydder jeg lige lige op DeleteFile[cookiejar] "data" indeholder nu en liste med et element for hver dag. Hver dag er en liste af to elementer: det første er datoen og det andet element er en liste af målepunkter, et for hvert kvarter døgnet igennem. Næste skridt er nu at få lavet noget som sikre at jeg ikke misser en dato fordi min laptop er slukket. Dernæst at få det hele sat op til automatisk at køre hver dag på et tidspunkt hvor der er størst sandsynlighed for at min laptop er tændt.

[perdalum-blog]

Jeg vil lege med mine strømdata fra NRGi

For et par måneder siden fik jeg installeret en ny el-måler. Det er en såkaldt "intelligent el-måler". Jeg ved dog ikke hvor intelligent den er. Den har endnu ikke svaret mig når jeg spøger den om meningen med livet, Universet og alting. Men den er hvid, meget lille og har en iPhone App. Måleren tilhører NRGi og på deres web site kan man logge ind og se hvor god man er til at bruge strøm. Vi er vist nok ret gode, da vi på en af deres grafer ligger i et meget rødt område, men det må NRGi vel være glade for. På det samme web site kan man også se en masse halv-fancy grafer over ens forbrug. F.eks. er der en døgngraf som viser hvornår vi står op og hvornår vi kommer hjem fra arbejde. Så stiger strømforbruget nemligt gevaldigt. Så er der en uge graf hvor vi kan se at vores strømforbrug er ret cyklisk. Den sidste graf er månedsgraf. De første to er liniegrafer, mens månedsgrafen er et søjlediagram med en søjle per døgn. Dvs. når man går fra døgn over uge til måned bliver detaljegraden forringet ganske betydeligt. Heldigvis, tænkte jer, er der mulighed for at eksportere ens data som CSV fil (det er noget der kan åbnes i Excel). Jeg vil nemlig gerne have disse data til at lege dataudforsker med. Men jeg var for optimistisk. Gemmer man nemlig en døgnrapport, får man fine data med et datapunkt for hvert kvarter hele døgnet igennem. Fint nok, men gemmer man en uge får man kun datapunkter for hver time og endnu værre med månedsgrafens gemme funktion. Den giver kun et sølle datapunkt for hvert døgn. Det kan man jo ikke bruge til noget! Heldigvis har jeg lært et og andet gennem de sidste 22 år hvor jeg har hacket rundt på diverse computersystemer. Så, uden mere snak, bringer jeg her et lille Applescript. Programmet får en slutdato og et antal dage. Derefter begynder det at hente døgngrafens data for hver dag fra slutdato og antal dage frem. Man skal dog lige huske at logge ind på NRGi portalen først. set enddate to (current date) - (5 * days) -- we need to go a bit back in time to ensure the data is available set numberofdays to 40 set startdate to enddate - (numberofdays * days) -- start by creating a single document to handle all the URLs tell application "Safari" make new document at end of documents end tell set myday to numberofdays repeat until myday < 1 set presentdate to startdate + (myday * days) set datestring to formatDate(presentdate) tell application "Safari" set URL of document 1 to "https://selvbetjening.nrgi.dk/nrgi/am/export/day/" & datestring & ".csv" delay 4 -- we need to wait for the request to complete [sic!] end tell set myday to myday - 1 end repeat on formatDate(thedate) set monthNo to (month of thedate) as integer set dayNo to (day of thedate) as integer set yearNo to (year of thedate) as integer set datestring to ((dayNo as string) & "-" & monthNo as string) & "-" & yearNo as string return datestring end formatDate

[perdalum-blog]

Made with Paper