blog.houseofmoran.com

(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-18104725-2', 'houseofmoran.com'); ga('send', 'pageview');

[houseofmoran]

When backend is frontend, and back again

If you're building a mostly-frontend dynamic site, but want to make it share-able on Facebook/Twitter with a nice preview image, then here's one new (slightly weird) way to do it.

I was playing around with a Sham Tartan generator because, well, why not? I started off rendering this as overlapping divs, but this is no good as an og:image or twitter:image, as Facebook/Twitter won't pick these up as images.

I changed the React front-end to instead render these on canvas with the intent that I then re-use this JS code in the back-end. However, after reading about what the recent headless mode of Chrome gives you, I realised I had another option.

The core of the idea is to run an instance of Chrome alongside your backend. When a request comes in for a preview image you ask it to take a screenshot of the page you want to use as a preview which you then proxy back to the browser.

For example, on http://shartan.houseofmoran.com/ I set the og:image in the meta tag to http://shartan.houseofmoran.com/preview.png. A request for http://shartan.houseofmoran.com/preview.png then causes a screenshot of http://shartan.houseofmoran.com/ to be taken as a PNG which I then return.

The page being used for the preview image here is the same as the homepage, but could be a different url.

My implementation of this is on github, but there are bound to be better versions out there.

You'll notice you still need a backend, so what's the benefit of this? The main benefit is that any technology that runs quickly in Chrome can be used to make a preview image. If I wanted to, I could go back to my div-based Tartan rendering method and it would all still work.

There are sure to be limitations to this, not least the stability of chrome headless itself. But, in summary, if you want to write most of your code on the front-end and want a generic method to create a preview image, then consider this pattern.

[houseofmoran]

Tautona: Visualising similarities between Slack channels

I hadn't done any lo-fi text analysis for ages, and since we've been using Slack at my job for a little while now, it was time to get munging!

My MVP was really simplistic, but actually surprisingly good. I didn't record the output of the implementation at the time, but in summary it used the Jaccard Index between channel memberships as a measure of similarity.

I then did a fairly standard thing of inverting this into a distance measure, and visualising this with a D3 Force Layout. So far, so good, without even touching any messages!

Enter: Gensim. This is a really neat little Python library that I discovered via a colleague at Skyscanner.

It focusses on doing a small number of jobs, but well. I've been playing about with its word2vec implementation in another project, but here I used TF-IDF + Latent Semantic Indexing to produce channel similarities:

The procedure is similar to my MVP, except I treat each channel as a document, consisting of a big bag of words from up to the last 1000 messages. TF-IDF is applied to boost up effect of unusual words and LSI extracts topics. I then compute all channel to channel similarities, convert to distances, and visualise as a force layout.

You can see the video of the output above, but you can also play with a visualisation of Skyscanner, lightly obfuscated.

To run this, you literally only need Python and a Slack API_TOKEN. I would love to see visualisations of other people's Slack channels. Please share them, or ping me on Twitter, and I can host them on Github!

[houseofmoran]

The (lack of) relationship between price and value

In Vienna I had an espresso-fueled discussion with Dimo and Georgi about how the price of goods relates to their value. We got through quite a few different ways in which they are indirectly connected, so I thought I’d write them all down, even if only for my future reference.

Choice paradoxes. For example, if you want people to tend to choose a product at a particular price, you can present it in order with two others: one which is far more expensive, and another which is far cheaper.

Price as independent signal of value. For example, vases, which are sometimes valued because of uniqueness, do not have that aspect as a directly visible property. If one is initially advertised at a “bargain” low price then people may not buy it because this is interpreted as a signal that the seller does not think it is rare. The seller can then raise the price to remove this effect. As long as it is still cheap enough that the buyer does not do an extensive check before purchase, then they may be able to sell a vase which is actually quite common.

Price as future resell value. Similarly to above, prices for extensive pieces of Art are hard to determine. Many collectors buy and sell on the basis of what a group of experts think a piece will sell for at a future auction. To some extent this price is then self-fulfilling, particularly since collectors are often independently rich and can wait to sell when they think the price they want can be reached.

Artificial scarcity. This is the classic "available only today" scam, where the price is reduced for a limited time. A vendor can easily do this across a large catalogue of products where any given product is given a synthetic limited availability.

Misleading discounting. Similar to above, except a product is sold at a high price for some minimum duration, before being "discounted" and advertised at the sellers desired price.

Multi-item bundles, or volume discounts, irrespective of need. This is things like "2 for the price of 1", where only 1 item is really needed. For example, you may only want a small coffee, but the per-volume cost is noticeably more than the medium size, so you are pushed into choosing it instead. You could argue this is also a form of item 1, since you often see small/medium/large as options.

Artificial correlation. For the airline industry alone, I suspect you could write a book on these. For example, typically a seller will have to pay a fee to the credit card issuer on each purchase. However this is a per-transaction cost and does not have any relation to, for example, the number of people travelling, but some airlines use that as a multiplier of the credit card charge.

Brand uplift. For two similar items in two separate catalogues, the company with the better brand is able to charge a higher price. To some extent this is reasonable, because there is value in the expectation of higher quality, reliability, better servicing and possible future resell value. However, these extras come at a cost to the seller, so there is an incentive to smear out the benefits of the brand whilst reducing costs. Brands are built on reputation, but if you can skimp on quality, whilst keeping it hard to notice it has been reduced, or get disproportionate gain in positive publicity from a loss-leading set of products, then you get the benefits of a good brand whilst reducing costs.

Brand as the product itself. There are extreme examples, like trainer brands, where wearing the same brand as your peers is more important than the shoe itself.

Negotiable prices. For example, tipping. There is not a preset price, but you must balance the shame of a too low price against not wanting to spend too much. There may be a generally accepted service charge percentage, which the the buyer just has to know. In some ways, this is a purer price because the buyer, within some constraints, can use the charge as a signal of the true value for them of the service.

Phew. There are almost certainly more, but 10 is enough for now. Suffice to say that advertised prices alone are not a valid direct indicator of true value for all the reasons highlighted above. Technically, yes, "a price is what a buyer is willing to pay", but in the context of all the above, this is just a weightless tautology which does not tell you much about value.

[houseofmoran]

The Global Minotaur: a review

I occasionally read books on economics, and they uniformly scare me.

As a parent, you look back to your own chlld-hood and realise your parents did not know what they were doing. Not because they did something bad, but because you know now that you don’t have all the answers, so they can’t have had them either. The veil of authority is removed but you can still have some degree of respect, even if not 100% agreement.

Not so with economics.

The typical economist, commentator, or politician on the Today programme will ooze confidence in their own reasoning. However, read almost anything on the fundamentals of economics, and you soon realise most of it is founded on large-scale bets, fuzzily-defended confidences and ridiculously dodgy assumptions.

I can respect practitioners, like Entrepreneurs, and to a certain extent, Bankers; more specifically, anyone who has “skin in the game”, as Taleb would say. They have to work with the horrific uncertainties of buying, selling, loaning and building, and make appropriate accommodations to get by. They support no illusions.

Go back now to the voices of Radio 4 and hear the timbre of authority, unchanged. Recent events, such as the Scottish Independence vote, and the Euro crisis, have revealed this to be nothing but bluff and bravado. As pointed out by the book, Yanis Varoufakis is disliked by the establishment because he is erudite, capable, and more than willing to point out the many flaws in the mechanisms of the modern world.

Surprisingly, this is not really a book about Greece. Against the themes of this book, Greece is at the level of roadkill. This is not a book about blame either. Many people are discussed as potent and deliberate actors but there is no desire for retribution.

This is mostly a book about balance at the macro-economic level, with a side-order of theoretical sleight-of-hand thrown in as horrific entertainment on the side.

His main thrust is that the required mechanisms of balance between exchange rates, surpluses and trade that were set up in the post-war period, based on sound fears of a repeat of the Great Depression, were deliberately manipulated in the 70s by America in a way which inevitably led to over-exploitation in the '90s, '00s, and ultimately, to the Crash of ’08.

In my reading of this, the madness of instruments like Collateralised Debt Obligations and Credit Default Swaps was just one possible way an inherently unstable system (The Global Minotaur) could have fallen.

Overall, the book is a great compacted history lesson. My knowledge of the inter-country economic history of the past 100 years is limited, so this is welcome, but this also means I can’t say what he is leaving out. For what my opinion is worth, it was enjoyable and consistent. It also makes me want to go read up on Keynes (for his part in post-war negotiations) and the '70s (for when the madness really started). Whether I’ll have the time is yet to be seen, but it has definitely provided the motivation.

The writing style, pre-minotaur period, is quite strong and easy to read. I read most of this book on the bus and/or on the walk from school to work, so needed to pick it up again repeatedly. However, I found this hard to do when he got into his explanation of The Global Minotaur, for two reasons.

If you’ve heard Yanis speak, you know he can be clear, but sometimes get distracted by florid descriptions, and over-dramatic framing. The same goes here, with repeated entwining of the real with the simile. Regardless of whether you already knew the Greek myth, he makes it work as an explanatory tool, initially. However, he returns to it too often, stretching the isomorphism to make a point when the facts themselves, once you’ve understood the framing, can stand alone.

Over and above the re-use of the simile, he generally repeats himself too often. It’s as if the later chapters are a series of blog posts which have been concatenated together. In that context, where the reader may not have seen all previous instances, it makes sense to re-state. However, here it made me wonder if I’d accidentally skipped back a page, after re-engaging with the book.

Once you get to the end, does he offer any hope, any solution?

Early in the book, he suggests that from great turmoil comes great change. These Crises (with a capital C) are expected, and are almost a built-in razing to the ground in which new seeds can grow. He offers no new ideas, at least not in this book. In a way, he just documents the large scale needs that the global economy has now, and always has had, effectively reminding us that there is no free lunch.

He also repeatedly points out that most of what is happening now, whether it is Austerity or the European Financial Stability Facility, is just more fiction; distractions and abstractions over impotence.

The latest commentary from Yanis in the book is in the preface from 2013. In it he recommends that we resist, that we say “No!” to the mechanisms that he demonstrates in this book as illogical, and that make no accommodation for the The Global Minotaur that has so recently fallen. As we’ve seen, this strategy has not done him, nor Greece, a whole lot of good, but I honestly can’t see what else would have been better.

[houseofmoran]

Looking for Bobby but found Paris instead

Go to any local attraction and you'll see people taking selfies with statues, towers, all sorts of sights. Once, walking past tourists taking pictures of Greyfriars Bobby in Edinburgh, I reckoned that some of those will be posted on Twitter. I should be able to find these, and similar pictures, automatically. How difficult could it be?

Problem: Porn. If you do a search for tweets containing "selfie" in the raw firehose, then you'll find about it's about 60% Porn. I suspect the equivalent live search on twitter is doing some sort of filtering.

Even apart from the problem of Porn, there is no gaurantee they'll even use the term "selfie", so we need a different kind of filter.

My hypothesis was that the selfies I was after would have one person in the foreground on the bottom left or right, with the object of interest in the background. I coded this up, and left it running for a bit.

Umm, ok.

ขำสัสส pic.twitter.com/aEJcf5iYjR

— ฐานิ (@Purepareal) August 2, 2015

ไม่ตอบไม่ได้แปลว่าหยิ่ง แต่มึงตั้งชื่อเฟสงี้เองนะ pic.twitter.com/DHBqWxAhS7

— งัวพีช (Official) (@PeacHZ_YYY) June 8, 2014

I seem to have written a chat app detector. What else fits my description?

Tv shows?

1 yıl önce bugün İŞiD Şengale saldırdı Binlerce Ezidi katledildi binlercesi esir alındı... http://t.co/kuIMIw8465 pic.twitter.com/ybijb5dvEV

— Nurcan Baysal (@baysal_nurcan) August 3, 2015

pic.twitter.com/9jwY1lQsZu

— خالد العوسي (@ousika) August 3, 2015

Framed quotes, inspirational or otherwise.

「村山談話」実態明らかに！ 内閣を無視だまし討ちのように発表された！ 談話の中には中国共産党「人民日報」 がよく使用するフレーズが使われている。 事前に中国と調整した疑いがある＝和田議員 http://t.co/E6DdpoLt99 pic.twitter.com/yOl4Rah2WY

— ななみ9019 (@superneoblack) August 3, 2015

Privatisation is nothing more than shifting wealth from everybody to a privileged few. @Corbyn4Leader will stop it. pic.twitter.com/sI0xK175Wk

— George Aylett (@GeorgeAylett) August 2, 2015

And headlines :-/

Larries: pic.twitter.com/zU3BUbTnHW

— liv (@drunkpaIs) August 4, 2015

With the above, it's easy to see how it matched, but some others are a bit more obscure. For each of the following, have a look at the tweet image for a bit before clicking through to the highlighted face.

ฟินเละเทะ ฟินพังพินาศ เอะอะลากเข้ามุม เอะอะกอด #ริวจิ #AF12 pic.twitter.com/gUjExo6Tn2

— หนุงหนิง (@niing_jaoka) August 3, 2015

face

It's 2 A Day Monday.Good luck to Harlandale Lady Indians Volleyball.U had a Great Camp last week. #E3P #Winning pic.twitter.com/OC3UOjuMxz

— E3Ptraining (@E3Ptraining) August 3, 2015

face

【ピヨ剣乱舞】ついに全ピヨそろいました！！！りつふぁぼ応援してくださった方々ありがとうございます！！！！！ピヨコはとってもかわいいです！！！！！次に誰かが実装されたらまたよろしくお願いします（画像のピヨの順番は愛用している一覧から） pic.twitter.com/0vkwJ7UbAP

— ミラで原稿しよう (@sauri_uto) July 29, 2015

face

You can see more of these examples and other types in my github repo.

But, did I actually find anything? It took me a about an hour of trawling through around a thousand tweets it had found, but here's a perfect example of what I wanted:

Sin lugar a dudas una de las ciudades más hermosas y mágicas ! @RestoDelMundo13 #IvanEnParís pic.twitter.com/B32iQUbFas

— Facundo Gambandé (@facugambande) August 3, 2015

A one in a thousand hit-rate is not great, but given this was barely a days worth of implementation and tweaking, it's not bad. It's also obviously improvable. For example:

I'm using the most basic face detector available in OpenIMAJ, and I'm not even filtering that by confidence.

I can easily filter by relative size as well e.g. only include those faces which take up around 25% to 20% of the image.

There is probably a text recogniser I can use to exclude images with mostly text outside the face.

I can use a more focussed source than the firehose. For example, look for geotagged tweets nearby tourist destinations, or use a larger set of keywords than just "selfie". However, the wide-open nature of the firehose is appealing, so I'd like to see how far I can get with only image analysis.

It's really a single-process app, even though I'm using Spark. If I had a bigger sample, and more limiting filters, then perhaps I could find more examples. I'm limited by the built-in firehose throttle, but I could probably scale out the image lookup, as long as that isn't then throttled.

This is a work in progress, and I'm quite happy with what I have so far, so might leave this for a bit. Always other projects on the go!

Finally, I'll end with one example I found which doesn't quite fit my original intent but which I like nonetheless:

Soon. pic.twitter.com/fuPfEfsRqh

— Arsenal Related (@ArsenalsRelated) August 2, 2015

[houseofmoran]

Drone Futures

According to BusinessWeek, long-haul trucks in America could be the first beneficiaries of self-driving technology. As far as safety and productivity goes, this should be a win.

However: self-driving cars are already arguably safer than people in urban areas, driving between states is a comparatively easy problem, and the occupant of a truck is not a necessary part of the cargo. So, it’ll be hard to argue to keep a driver behind the wheel.

Around 3% of people in the US are directly employed by the trucking industry. A large fraction of these jobs will disappear, and related jobs will also be at risk. For example, there are many small towns whose livelyhoods are dependent on truck stops for trickle-down income. When truck drivers stop less, or are removed completely, then these towns will disappear, like the towns who were decimated when the highways came along (see: Cars :-) ).

Even so, there may still be a net increase in overall happiness if more companies (e.g. Uber) get their wish of self-driving vehicles. Not least because it would stop Uber applying their apparent policy of treating their human employees like this centuries systematised labor.

http://t.co/Xu8Yw4WGbu : I would like to see some counterpoint, because I worry that Uber is just this centuries systematised slave labour

— mike_moran (@mike_moran) May 17, 2015

The changes coming from self-driving vehicles will be big, and may even be positive. However, vehicles are something we already have and mostly understand. The infrastructure we have now limits where they can go and how fast they can travel. They still have to interact with pedestrians, even if they do it more efficiently than we can. Ultimately, they will be limited by having to follow 2D constraints. Drones, however, have no such limitations.

There is a lot of hyperbole said about drones, but a lot of it is just slightly ahead of real events. Easy availability of cheap drones that can act individually or collectively in a clever way is the sort of change that affects system-level assumptions, because what used to be hard will now be easy.

On the walk to school, my son and I take turns making up imaginary machines. There is a rough theme, and one day it was construction. Inspired by youtube, I came up with a “carpet of drones”. If you wanted a new build of something you'd hire a rolled up carpet of drones for a day, and perhaps a mini fabrication machine. You’d unroll the carpet, lay it flat, attach power, start up the fab, and upload your plan from your phone. The drones would auto-organise by role, some carrying bricks, some small dollops of binder, some straightening out the bricks to cover small mistakes. Come back in a few hours, pack them all up, and you’re done.

Drones are numerous, adaptable, and already cheap enough to almost be disposable. The drone future is combinatorial and far less bounded than self-driving cars. Any prediction I make is going to be wrong or wildly unambitious, so I won’t make any right now. I’ll just leave you with something which, as a fan of Lego, I can't believe I've only just discovered now:

[houseofmoran]

The Microservices Elephant

The term “microservices” is going through a phase where everybody has their own conceptions of it, so as soon as you start talking about it in any detail you quickly realise you all mean something slightly different, and you start questioning the value of language and why you bother talking at all. However, given there is now a book out, and a conference on it next week, it feels like it’s about to become a lot more definite. I thought I’d brain-dump my own thoughts on a few of the conceptions that are out there, before my own stance gets hopelessly clouded by many others.

modules + network = microservices

This is from an article by @tef, whose opinion I generally respect and, whilst I agree that “spreading bits of state across a network is a ... world of pain and suffering” and that “keeping data in sync across a network while tolerating failure is an incredibly hard engineering problem”, I really don’t think that’s the problem that microservices are meant to solve. Note that there are plenty of other things I do agree with in that article, so please go read it.

As in so many things, if you’re faced with a very hard problem, the first thing to seriously ask yourself is whether there is actually a far easier problem lurking within, that is the one you should be solving. In this case, the problem is not keeping state in sync. Bits of state, encapsulated behind a network interface maybe, but separate bits of state, which don’t even need to always be consistent. For example, I wouldn’t build a networked filesystem using microservices techniques, but I would build an advertising system using them.

microservices = contractors for hire

Some of my contractor friends might be offended by the following, but here goes :-) Microservices collaborating within a microservices architecture don’t fully trust each other. They make reasonable demands of each other, and expect responses, based on an agreed contract, but there is always a need to be ready for that contract to be cut short, or expectations to not be met. Microservices are about wrapping unreliable actors, and making them reliable in aggregate and/or ensuring unreliability at the micro scale does not become a macro problem.

This analogy also captures the aspect of taking advantage of heterogeneity: microservices allow you to select the right tool/library/language for the job, as long as it presents itself as a semi-standard service.

A microservices architecture should treat some demands as contractually required and others as nice-to-haves. For example, all services should guarantee a low-latency response, but the quality of response can be varied, or be a partial response, so that the latency remains guaranteed. High quality should be optimised for, but low quality allowed; high latency should be survivable, but low latency expected.

microservices = a portfolio of risks

This is analogy doesn’t fit all microservice contexts, but does fit the ones I like working with. Here, you actively hedge one microservice against another, so that when one fails, or gives a low quality answer, you can fall back on another. This is actually a tactic which works within a service as well as across them. For example, within service: using an in-memory out-of-date cached copy when an external service fails; across service: use recommendations from a less-personalised service when the personalised one times out.

microservices = agent programming for the web

@robstewartuk @russmiles @boicy @russmiles no, no, and three times: no. AOP was a pie-in-the-sky dream, MS is derived from implementations.

— mike_moran (@mike_moran) February 7, 2015

(Gack; I wasted a lot of time in the 90’s thinking about agents).

As you’ll have read above, my own take on Microservices very much treats co-operating services as a network of partially trusted actors, each with their own behaviours and preferences. It’s fine for myself and others to think about them that away, and to design libraries and tools that work well together given these assumptions, but there is absolutely no need to embed this understanding in the systems themselves. I’m not saying you can't build robust, flexible, and apparently clever systems with agents, but rather that they are not necessary.

In conclusion, based on the above, my own idea of what microservices are is very much tied up with what I want to do with them, as opposed to the more wide-ranging meanings of others. No surprise there really, but now I’ve got that brain-dump out of the way, I look forward to finding out more and/or being usefully proved wrong in my assumptions about how to build systems well!

[houseofmoran]

Cost of Delay vs Cost of Failure: a balance of forces

This article presents my understanding of how the Cost of Delay model, in combination with Cost of Failure, can be used to choose between and decompose alternatives.

TLDR: A DevOps team can get long-term rewards of minimising Cost of Delay by balancing it against the stabilising force of Cost of Failure.

I went to a presentation on Cost of Delay many months ago; thanks to Ozlem Yuce for presenting it, and to Seb Rose and BCS for organising it.

There is a lack of clarity below over whether we are measuring possible cost or actual benefit, but you’ll just have to live with this until I integrate/understand this further myself! Think of this as a progress report on my understanding. Feedback appreciated, as always.

An aside on numerical interpretations (can be skipped):

I’m currently interested in using Cost of Delay as a qualitative method, partly because I don’t yet fully understand the implications of the quantitative side. However it is mostly because I am wary of numerical interpretations of methods that affect people (where “people" includes both “employees” and “customers”). As soon as a qualitative measure is represented as a number there is a strong tendency to start doing number-like things with it e.g. summing, averaging, comparing. This is often done very naively and can lead to pain for both the producer of the number and ultimately the consumer. It’s not that it is impossible to do correctly, or even necessarily hard, but it requires a bit of nuance to keep the number attached to it’s history e.g. some measure of uncertainty, like error bounds. To some this may come across as elitist or an over complication, but to me it’s about having respect for the numbers and the people they ultimately effect.

Cost of Delay

The basic idea behind Cost of Delay is to consider the loss you make as a result of not making an investment. The accumulated cost over time is some function of time. This is a very bland way of putting it, so I think it deserves a sketch:

The shapes on the right illustrate some of the different ways that the cost function can behave. To pick one example (green rectangle): say you currently have an account on a cloud provider. They have a deal where you can use their services for free for 12 months, but thereafter you have to pay a yearly fee. This highlighted graph shows the cost over the next 12 months (it would look like a step function if extended indefinitely). Another example (blue ellipse), the classic compound interest on debt: you incur some debt, and then the interest on the debt becomes part of the debt itself, cost increases exponentially.

I hope you get the gist, but let’s try an example closer to home. You have a great new idea you are working on, but you know it could take months to make it perfect, and you are not totally sure of it’s eventual worth. You think there may be a simpler idea you can ship pretty-much immediately, but it’s a bit crap in comparison. Let’s call these choices A ("definitely ok but available now”) vs B ("probably great, but not available for a while”):

Maybe I just think more in terms of functions, but to me the diagram on the left is already helpful in forcing us to think about choices. So, what are some of the possible futures, based on this formulation? Note that now we are talking about possible futures, so these costs are turned into predicted benefits. From left to right:

B_2 is a tweak on B which has higher value than A (larger gradient), not as much as B, but which is shippable now. We never come back and do B, because B_2 was good enough.

We go ahead and ship A, and let it make some money in production, whilst we work on B. B replaces A when it is ready, and it turns out to be great.

This is like 2, except another more important priority comes up and we don’t do B. Regardless, we’ll have made money on A in the mean time.

We decide to immediately do an experiment on B, but on a cut-down version which will only work for two weeks. It turns out it wasn’t so great (perhaps customers don’t like it). We end up going with A after-all.

There are many more choices or combinations than these, but I’ll leave you to come up with those yourself. My point is that thinking this way, even just diagrammatically, allows us to compare these various choices against each other, and provides a means to communicate these choices to others.

Great, is this all we need to consider? No, but before we go further, let’s call out one aspect I am explicitly not emphasising here: waste. One of the things that can hold you back from shipping an A followed by a B (future 2 above) is that option A is work that you plan to discard later. There is no way round this; it’s definitely the plan. However, we are not optimising to minimise total work done. If anything, we are optimising for expected total value accrued over time. It’s not that efficiency is unimportant, but more that it should not be considered first. I could write more on this but not here; perhaps another blog post :-)

Ok, back to Cost of Delay: there is some risk in these choices which we are not considering, which brings us to Cost of Failure.

Cost of Failure

A very simple model of the Cost of Failure consists of:

“probability”, or 'time till failure': how long from now until it fails

“impact”: when it does fail, how bad is it?

“time to recover”: once you’ve identified it has failed, how quickly can it be rectified or limited in cost?

Variation in each of these produces different models of cost, which dictates the accrued cost over time.

Here, cost may be measured in units of ‘downtime’, in actual money lost, or some other measure. These formulations are still qualitative and the ‘cost' in ‘Cost of Delay’ is not exactly the same as the ‘cost’ above.

The relevance of this comes when you see this as the flip-side of a choice that you want to make for Cost of Delay reasons. For example, consider option 4 previously, where we want to do an experiment over two weeks on B to measure value. If the failure curve looks like an escalation curve (red rectangle above, for example a small increase in site latency suddenly turns into an outage) then we won’t take the risk. However, if we can make it auto-recover (green rectangle above, for example an auto-restart of a service to clear memory leaks) then we may be happy with that for the duration of the experiment. The key aspect here is that the same team, or group of decision makers, must be willing and able to take both types of cost into account.

The great thing about DevOps teams, if you’ve set them up properly, is that you have awareness of both sides, and can choose the best compromise at all times; you see the pain and the gain. If you only see one side, then then there is an imbalance.

In summary: A DevOps team can reap the long-term rewards of minimising Cost of Delay by balancing it against the stabilising force of Cost of Failure.

[houseofmoran]

As a novel, as writing, it's definitely not Great. It is literally put-downable, as I did so, several times.

Many of his descriptions are overly wordy, and unnecessary, hence skippable. To anyone who's read Brave New World or We, the story, the framing, the plot even, is not particularly new; even down to the frequent mention of glass.

The characters, all of them, when not minimally drawn are... punchable. I could not really hook into any sort of empathy with any of them.

Having said all the above, it still gets at you. It still nags at you, makes you consider, challenges you to convince yourself: that could never happen, could it?

My context for reading this was being on holiday away in a cottage: going for a wander taking photos, attempting to share them, and being frustrated by bandwidth; trying to check Twitter, and it taking soooo long; seeing the gaping holes where photos would be if only the connection wasn't so fricking slow. This contrast shows how far my normal behaviour is from where it used to be.

Running through the book, and our lives, is the concept of easiness. Wouldn't it be easier if we were all online with full bandwidth at all times? Wouldn't it be easier if you always took your phone so you were always in touch? Ultimately, what's the easiest we could make it, even if we sacrificed everything else?

This is not a great book. But it's worth reading, and books like it, if only to point out quite how far down the slippery slope we've gone.

[houseofmoran]

Sharing applied to Healthcare

Can we ever bring the sharing economy to health care coverage? If you are not using your insurance, give it to someone else. @yeohsiewhoon

— Filip Filipov (@Filipov) May 23, 2014

The idea of sharing unused insurance has a lot of assumptions baked in, and to be frank, I am no expert on emerging economies (referenced in a later tweet). However, let's unpack this.

Let's start with our goal. Can we assume, for the sake of argument, that it is to provide both protection of the individual against ailments and collective protection of the 'herd' against transmissible diseases ? To keep this simple, let's assume this is provided in the scope of a single country i.e. we don't attempt any cross-border solution.

The choices for a solution come down largely to what degree collective versus individual responsibility is assumed. At one end of the scale we have the USA, where the individual largely expects to have to pay a company for insurance, and then claim on this per usage. The government covers some collective responsibility in the form of disease control, and Obamacare is changing things, but the assumption is largely that you have to look after yourself.

At the other end is the UK where we still pay insurance, but it is a largely flat payment and we don't tend to pay a significant amount per usage. It is a hybrid in that private insurance also exists but that is typically used as an add-on rather than for primary care. Disease control is solidly in the domain of the government.

The notion of "sharing" what you individually have only arises on the USA end of the scale. The implication is that since you've paid for something but not used it, why not enable altruism and allow it to be passed on? In this light, maybe it does make sense to enable this in some way.

At the UK end, individual sharing makes no sense, since there is nothing for you to pass on. In fact, you're already sharing it with others by giving money to the government who does it on your behalf, except you have no control over who it's shared with (unless indirectly via elections).

However does re-sharing even make sense at the USA end of the scale? To be exact, do you actually own your lack of a claim? I suspect insurance companies would say no. The insurance has been sold to you at a particular price on the basis of actuarial data about likelihood of a claim. You have been sold an option to claim healthcare at some future point. That option is not transferable because it would upset the model of expected claims.

Now there is nothing stopping an insurance company from changing the way they sell it at the beginning so that it is transferable in some way. However, I suspect you'd then pay more for that and there would still be restrictions (e.g. only shareable amongst family members). An insurance company would not allow a very open interpretation of sharing because they make a lot of their money on people who pay but don't claim.

I'm reading between the lines here a little and assuming you mean "sharing economy" in a very open sense. Given this interpretation, we're already doing it in the UK, just not under our individual control, in the form of the NHS. At the USA end of the scale, it would only be possible in a very circumscribed form. This implies that more sharing is possible in a Healthcare system like the UK.

When it comes to how this applies to emerging economies, I really don't know. I suspect the level of sharing of unallocated resource depends on where it falls on the spectrum between USA and the UK.

[houseofmoran]

The Signal and the Noise: The Art and Science of Prediction: Amazon.co.uk: Nate Silver: Books

According to the New York Times this is "Engagingly written... wholly satisfying... one of the more momentous books of the decade". This book isn't “momentous” to me because I've already read most of what he says elsewhere, and already agree with most of it: beware of binary answers to complicated questions, test your theories, assume you are wrong, then prove yourself right. In his terms, I'm a ”fox”; at least I like to think I am one. I’m certainly used to the puzzled, slightly hostile, reaction when I refuse to be drawn into an inappropriate yes/no answer.

Even if you already agree with his ideas, the first half is worth it. It covers the practicalities of prediction in politics, economics, baseball, and weather. In the second half he ventures into realms with less impartial data, like climate change and terrorism. As he himself admits, it's hard to apply statistics and bayesian reasoning here. I don't really buy his assertion that successful practitioners in these areas are ”doing bayes”; you could equally well say they are just being skeptical.

The second half is a lot more of a qualitative application of his principles to many areas. This leaves it feeling a lot weaker. I would rather have a deeper treatment of one of these difficult areas as proof that his principles can be applied more generally. However, I can see that excluding this depth means that the average reader in Waterstones can pick up the book and discover a different way of looking at the world.

Unfortunately, I'm still not too hopeful of his analytical style of thinking winning over simply declaring you've seen any old pattern in the noise. To take one example, part way through reading this book I read an article in Bloomberg about making predictions of market movements based largely on simplistic application of Fibonacci sequences. These charlatans are still around, and they get paid almost regardless of whether their predictions are correct.

So, there is very little in this book which is new, but that's ok. The important aspect is that because he has publicly demonstrated his predictive abilities, he can use this to get people to read this book. I'm not hopeful, but if even half of what he says is listened to, it will be worthwhile.

[houseofmoran]

I just finished reading this book to Nils as a pre-bedtime book. It was lovely.

It's kinda like when you go to the cinema to see a 'kids' film which turns out to have some adult-focussed content cunningly hidden. Except s/adult/geek/ : -)

[houseofmoran]

Geohash: The Worlds Address

I got a chance on Friday to give an overview of what a Geohash is, as part of the Lightning Talk series at Skyscanner. Since it was only 5 minutes long, and the follow-on questions and answers were still fresh in my head, I thought I'd turn it into a screencast.

This is my first one, like, ever, so it's a bit crap. However, it can only get better.

[houseofmoran]

This is unlikely to be totally new, but it's at least still mildly distracting. It's a continuation of my geosummary project to chunk flickr images by geohash.

It's not perfect, but I do find that deliberately chunking the interesting images from flickr into the geohash region they correspond to, as opposed to the exact place, gives it a more interesting flavour.

Contrary to what I was originally intending, I have had to add a openstreetmap+leaflet map as context, because otherwise you really have no idea where you are. I was hoping it wouldn't be necessary, particularly since I have to scale it in slightly nasty ways.

It has it's bugs, and there are a millions things I'd like to tweak, but it's not bad for version 0.0.26. Btw, surprise surprise, there are a few cats in there, starting in Amsterdam :-)

[houseofmoran]

Naked only please

Navigating a newly-discovered German swimming baths is a bit like an episode of the Crystal Maze. For those younger than me: this involved a series of rooms with dastardly puzzles which you needed to solve to get to the prize. Except the 'prize' here is men's bare arses.

If you're used to the slightly prudish British swimming baths, where full nakedness is explicitly called out as something you must not do, then it comes as a bit of a shock. It's only really made acceptable because of stereotypical German efficiency.

In a typical Edinburgh swimming bath, showers and locker areas are generally unisex with small changing cubicles along the sides in which you can hide your private parts from view. This flexibility makes the building very open, architecturally.

Unfortunately, this seems to coincide with greater responsibility put on the bather. In Portobello baths you must remember to bring a pound to put in the locker as deposit; you can choose the locker yourself, which is good, because about one in ten are broken. In contrast, today at Hofwiesenbad Gera I paid and received a key for my locker. How did I know which one is mine? Well, I read the number on the key and used that one, of course.

German baths are like a factory line, where everything has been worked out for you in advance, if only you can go the right way. Once I did, I found a very well looked after private men's shower area. Unfortunately, it came with some less well-kempt buttocks on display.

I've yet to decide if the efficiency is worth the prize.

[houseofmoran]

Looking back on my 5:2 diet

So, a year on, how's my 5:2 diet going? The good news is that, looking at the data, I've lost a whole, umm, 2kg over the past year ... hmm, wait. Ok, maybe not so good. The full graph tells a bigger story:

I reached my lowest weight of 86kg first on the 28th June 2013. If you compare that to my maximum of 97kg on the 5th of January then a reduction of 11kg over 6 months isn't too bad. However, as you can see from the graph, I started trending up again soon after.

The second axis, of total_calories expended through exercise, probably explains a large part of why. It shows that the extent and frequency of my exercise reduced soon after mid-year. Also, If I remember correctly, I kept bingeing a bit too much on off-days, as I was afraid of.

So, conclusions? All I can really say based on this is that a combination of exercise and 5:2 fasting worked for me. Now I just need to back to doing it properly again!

[houseofmoran]

It'd be dishonest to even call this a Spike. I abused D3 into showing some data that I'm collecting about tweets per geohash, over time. The video shows how the overall counts change over time, as well as diffs per minute. I was hoping to see some more obvious pattern in the variation over time, but it seems quite stable, and the small diffs look very random.