statistics and empathy @kiyototamura - Tumblr Blog

Posts

git editor setting for Acme

`git config --global core.editor ``which E```

9pfuse step-by-step

Get OSXFUSE. Then,

sudo mkdir -p /mnt/acme

sudo chown USER /mnt/acme

9 mount ``namespace``/acme /mnt/acme

Then acme -m /mnt/acme

Browsing git stuff in Acme

I use Acme. I am not a developer (anymore) but still believe in text-centric tools. Acme is perfect for that.

Because of my involvement with open source at work, I can’t escape git yet. Git is a great tool, but its Read-Only tooling away from GitHub is annoying at best. No, I am not going to use one of these heavy GUI clients.

I was going to build an acme utility myself, but then I spend not enough time to make this project worthwhile...

Enter Marius Eriksen’s gitfiles. This makes it easy to look browse git commets and such. It comes with a couple of shell scripts like gitfileshow, gitfilestat, gitfileget as well as Gitfiles which is waiting to read for a gitfileedit plumber call.

A GIF is worth a thousand words:

you can just run mk install. The current code assumes that plan9port binaries are installed in /usr/local/plan9port/bin. You might have to change that if your plan9port is installed elsewhere.

Compile-Time Fibonacci in Rust

My roommate insisted that I learn some Rust. It's incomprehensible why my programmer roommate thought his marketer roommate needs to know Rust, but oh well, I spent a bit of time learning Rust.

Rust has many curious features. One of them is (hygienic) macros. The grammar of macro_rules! is explained in prose without clear grammar definitions, making it a bit hard to grasp. But after a few false starts, I managed to write a program that calculates Fibonacci numbers at compile time. So, here it goes:

macro_rules! sum { ($x:expr, $y:expr) => { $x + $y }; ($x:expr, $y:expr, $z:expr) => { sum!($x+$y, $x+$z); }; ($x:expr, $y:expr, $z:expr, $($a:expr),*) => { sum!($x+$y, $x+$z, $($a),*); }; } fn main() { let x = sum!(1,1,0,0,0,0,0); println!("{}", x); }

The sum! is the macro that produces the n-th Fibonacci number where n is the number of arguments and the arguments start out with a pair of 1's followed by 0's.

When you run this program, you get:

$ cargo run Compiling fib v0.1.0 (file:///Users/kiyoto/rust-programs/fib) Running `target/debug/fib` 21

Yes, you see the 7th Fibonacci number (1, 1, 2, 3, 5, 8, 13, 21,...).

Here is the fully expanded macro:

$ cargo rustc -- -Z unstable-options --pretty=expanded Compiling fib v0.1.0 (file:///Users/kiyoto/rust-programs/fib) #![feature(prelude_import)] #![no_std] #[prelude_import] use std::prelude::v1::*; #[macro_use] extern crate std as std; fn main() { let x = 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0; ::std::io::_print(::std::fmt::Arguments::new_v1({ static __STATIC_FMTSTR: &'static [&'static str] = &["", "\n"]; __STATIC_FMTSTR }, &match (&x,) { (__arg0,) => [::std::fmt::ArgumentV1::new(__arg0, ::std::fmt::Display::fmt)], })); }

In particular, notice

let x = 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0;

That's a lot of additions. There's probably a better way to do this since this program takes forever to compute once n is greater than 12 or so.

#rust-lang

How Will AWS Continue Its Dominance?

Originally wrote this on Quora to see if I could win $500 for their knowledge prize thingy. I didn't. So, here's my rant worth less than $500.

Amazon Web Services's success will depend on the following:

Bringing the center of data mass onto AWS for transactional data

Going up the application stack strategically

Containers, and by that, I mean Docker and its ecosystem

1. Bringing the center of data mass onto AWS

If there is a repeating theme in the evolution of software, it must be "those who control data dominates the market." This is true both for consumer (Google, Facebook, etc.) as well as enterprise (Oracle, SAP, etc.) If you have more search data than competitors, you dominate them (Google). If you have more demographic data than competitors, you crush them (Facebook). If you power more enterprise apps than others, it's damn hard to replace you no matter how much you charge them (Oracle). Data takes time to accrue and even more time to move.

AWS made a prescient move by investing a lot of R&D on Simple Storage Service (S3) when few were even thinking about cloud. Furthermore, they made it incredibly easy to store and archive data into S3, whether it is generated on AWS or not. Fast-forward almost 10 years, and now S3 is the unassailable leader in the cloud storage space. This lock-in is so great that, even if a company decides to move the rest of their architecture off of S3, they keep using S3 (See Why does Dropbox still use Amazon S3 to store data instead of building its own data center?). At least for the new generation of data (say, anything produced after 2010 or so) is squarely inside S3.

This means that AWS must have a very low account churn rate. Sure, some people might choose alternatives as point solutions, but it's highly unlikely that an existing AWS customer completely migrates off of AWS because it's pain in the neck to migrate off of S3.

2. Going up the application stack strategically

As Geoffrey Moore points out in his seminal book "Crossing the Chasm," it's very, very tough to "go up the stack" as a platform company and much better to open up a platform anchored on a success as an application. Examples: 1. Dropbox creating a mobile app powered by its distributed file syncing technology: They tried and failed. 2. Salesforce expanding on its CRM success with Salesforce Platform: Largely a success. 3. Intel creating its own Hadoop distribution to take advantage of its hardware: They tapped out.

Amazon, of course, has its roots in being a platform. As outlined in Steve Yegge's (in)famous rant, the platform mentality is deeply ingrained in Amazon from Jeff Bezos down.

This platform, API-all-things mentality can be an issue as they go up the stack. And they do not have an option here: If they want to fight against Oracle, SAP and Microsoft head-to-head, they must enter the applications business. This means building an Office competitor, Oracle CRM competitor, SAP ERP competitor and so on. AWS is insanely good at building scalable infrastructure and designing APIs around them, but applications, especially enterprise applications, are a completely different beast. In the coming years, I expect AWS to battle against established players in the enterprise applications space, and AWS's future hinges critically on their ability to execute on this.

On the flip side, Oracle/SAP/Microsoft all need to play a catch-up game to make a dent in AWS's quickly growing fortress. In the end, applications are only useful with data in them, and data is generated much faster on AWS than on Oracle RDBMS or SQL Servers (To be fair, Microsoft Azure has made remarkable progress, and Oracle, with its deep pockets and legendary sales prowess, shouldn't be underestimated)

And of course there's Google. By far, Google is the best product and technology company. That said, as Yegge points out in his rant, they have struggled to be a platform in the past, and while hugely popular in the consumer market, their applications business is weak compared to Microsoft's even in their native habitat of cloud applications.

3. Containers/Docker

Containers are probably the most (r)evolutionary idea that happened in the last 10 years in enterprise software. Docker's speed of adoption is truly exceptional: More than 22 percent of enterprises are already using Docker in production, and it hasn't been even three years since it was open-sourced.

Literally every IT giant joined the bandwagon, most notably previously-open-source-shy Google contributing the container orchestration tool Kubernetes. AWS, of course, has skin in the game: They've been aggressively pushing their managed Docker service (ECS). But why are IBM/Google/Red Hat/Rackspace/Microsoft all behind Docker? What's so magical about it?

Because Docker gives non-AWS IaaS vendors a fighting chance by making applications universally portable.

If a company builds its infrastructure on top of AWS, from EC2 through S3 to all kinds of convenient services one API call away, it's getting slowly locked into AWS. To be sure, this is not necessarily a bad thing for said company. AWS won't go away anytime soon, and its price is always going down with more convenience added to every aspect of the service. AWS is happy with increasing usage and billing, and the customer is happy with agility and scalability.

But this is incredibly worrisome for AWS's IaaS competitors.

But how can we prevent, or at least slow down, AWS's lock-ins? Docker gave all these IaaS platforms a hint: Evangelize the philosophy of "build once, ship anywhere" atop a rapidly growing open source project. The idea of containers is nothing new. But the opportunity of every developer getting behind containers is miraculous. And all these smart people at IBM/Google/Red Hat/IBM/etc. jumped on it. So, how does this relate to AWS's dominance?

If I were AWS, I would view Docker to be the greatest frenemy: Proselytizing AWS's worldview via its deployment model but enabling it on other platforms. As long as customers deploy their applications using Docker with one of its orchestration tools, that application can be moved to other IaaS. It will still be work, but much less than if these applications were run directly on EC2 with custom scripts with dependencies on various AWS components.

This article describes the formula syntax and usage of the INDIRECT function in Microsoft Excel.

I've been looking for this function for years. A good function to do metaprogramming in Excel.

#excel

The most critical part of such a strategy must be messaging. In fighting Islamic State, we must avoid the language that it uses to promote its worldview and, at the same time, offer compelling alternative narratives. Only in this way can we deny today’s Islamists and jihadists their ability to appeal to Muslim audiences.

On ISIS and messaging. http://www.wsj.com/articles/how-to-beat-islamic-state-1449850833

Good Marketer, Bad Marketer

Good marketers are blue collar. They understand that marketing in the trenches demands hard work and are willing to learn whatever needed to get things done. Bad marketers think they are white collar. They are armchair critics who make suggestions without delivering alternatives.

Good marketers listen to and understand customers. By doing so, good marketers connect with customers, learn their pains and gains and thus succeed to generate ideas that attract more customers. Bad marketers neglect or make assumptions about customers. By doing so, bad marketers fail to understand why their work produces few leads and blame sales for “not working the leads hard enough”.

Good marketers know their market and competitors. They know the competitors’ pricing, product lines and strategic partners by heart. Salespeople trust them as well-informed advisors who can show them how to win the war most effectively. Bad marketers stay ignorant about the market and competitors and make unsubstantiated claims about them. As a result, salespeople lose faith in them.

Good marketers write well. They know that their writing reflects the quality of their ideas. They are not afraid of revising their output, hence their idea, until it’s great. Bad marketers are bad writers. They try to conceal their bad idea behind a wall of vapid text. They neither revise nor proofread carefully.

Good marketers understand that marketing exists to support sales. They sit in sales calls, build rapport with sales and try to learn what problems and bottlenecks salespeople face everyday. Bad marketers are armchair philosophers. They think that salespeople are too tactical and can’t understand their grand marketing strategy. Bad marketers’ unjustified snobbism makes them the laughing stock among salespeople during happy hours.

Good marketers are organized. They understand that delivering a marketing campaign requires delicate coordination and bold execution which in turn requires careful planning. Good marketers keep every party involved without bombarding them with favors. Bad marketers are disorganized. They procrastinate until it’s too late to prepare their campaigns and supporting collateral. They don’t inform people in a timely manner yet distract them with one-off requests at last minute.

Good marketers win with data. Bad marketers lose with excuses.

Let’s be good.

The software I use on my windows machine

Mostly for me to remember.

SRWare Iron Browser

Chrome, for Google-y stuff

Virtual Box

Vagrant

CentOS 7 in the VM Reemeber to add this to the vagrant file >config.vm.provision :shell, inline: "if [ ! $(grep single-request-reopen /etc/sysconfig/network) ]; then echo RES_OPTIONS=single-request-reopen >> /etc/sysconfig/network && systemctl restart network; fi"

RStudio

f.lux

Evernote

Gyazo

Remap ctrl/caps lock

Slack and Pidgin

Microsoft Office

Python Anaconda distro

Google IME

EmEditor

PuTTY

WinSCP

X11. Remember to install fonts with " sudo yum install dejavu-lgc-sans-fonts"

I guess I don't use that much desktop software these days.

#setup

Goodbye Elasticsearch

EDIT: Shay Banon, aka @kimchy and the original author of Elasticsearch reached out. I am copying his responses here. It's reassuring to hear from the founder (both the project and the company) that they will be zero change to the project's stewardship.

@kiyototamura also, Elasticsearch the projects remains exactly the same, nothing really changed

— Shay Banon (@kimchy) March 14, 2015

I am keeping the rest of the entry for the record, but I am much more hopeful about the future of Elasticsearch now.

This week, Elasticsearch had its first global user conference. On the first day of the event, they announced that they are dropping "search" from their name. Now the company is Elastic. They are molting out of their search-y past, going for bigger and badder things like Hadoop and Big Data. They are no longer just search but just Elastic. They are no longer Bonsai Cool but Enterprise Sexy.

Elasticsearch, ahem, Elastic, is full of fantastic people. I admire @Kimchy's technical leadership. Leslie is one of the nicest yet most effective people I have seen in the open source world. I learned a ton from Jordan Sissel and Co. over at Logstash. Hell, I don't care if Elasticsearch, the company, is called ElasticSearch or elasticsearch or Elastic or Search.

Elasticsearch, the open source project, is another matter.

With Elasticsearch.com, Elasticsearch.org died as well. Today, "elasticsearch.org" redirects to "elastic.co". I was really, really saddened by this. This, to me, was Elasticsearch the open source project's departure from its open source roots.

Contrast this, for example, with MongoDB. To this day, MongoDB has maintained both "mongodb.com" and "mongodb.org", reminding the world that IPO in the horizon does not mean that they will ditch the eponymous [1] open source project. Its their way of saying that MongoDB the company will always honor the community. Its their way of saying thank you.

One might say all of this is just bullshit antic that won't matter in several months. You might be right. But I also have an anecdote to suggest to the contrary. That this is a real change in how Elastic is different than Elasticsearch.

I am one of the maintainers of Fluentd. For those that do not know Fluentd, it is an open source log collector like Logstash, Flume and Heka. A lot of people used Fluentd with Elasticsearch because Elasticsearch is awesome, but people also liked some of Fluentd's features (tag-based routing/filed-based buffering), so they began using Fluentd to stream logs into Elasticsearch.

Fluentd used to be listed on Elasticsearch.org alongside Logstash and Apache Flume as tools to stream logs into Elasticsearch. Here is what it used to look like:

Believe it or not, for the longest time, that single image link for Fluentd referred more users to fluentd.org than anything else. Just one image link. That's how small time Fluentd was. And what a big time Elasticsearch was becoming.

We really appreciated Elasticsearch's generosity in sharing their link karma, and we responded by going out of our ways to help people use Fluentd with Elasticsearch. In this regard, I have to thank @pitr and Co. for making and maintaining Fluentd's out_elasticsearch plugin. I did my share by answering a boatload of Elasticsearch-related questions on our mailing list and writing a tutorial or two. Apparently, my DigitalOcean tutorial was useful for some Elasticsearch, Inc. staff as well:

@digitalocean @fluentd @kiyototamura this is awesome sauce. Makes it dead easy for me to do my #linux data viz project now for @lhawthorn

— Sean Kerner (@TechJournalist) September 4, 2014

But link karma no more (UPDATE: Shay Banon responded that this was a mistake). When "elasticsearch.org" died, so did their link to our little log collector project, and, as far as I can tell, Apache Flume. Actually, looking at our Google Analytics, it might have been a little earlier:

Poof. Just like that. Elastic is now all about ELK, none about third party open source projects.

In the end, it's not a huge deal. I can always go somewhere else to get linked. I am an Internet Marketer. I make a living getting linked from awesome domains.

But for me, this one was different. It had a special place in my heart beyond just traffic and SEO. The link epitomized what I love about open source: the openness and generosity that put individuals before corporations. org-to-org, not com-to-com. Whatever. I am just over-sensationalizing now.

I still wish all the best to Elastic, Elasticsearch and everyone involved in them. Hell, I just finished writing another tutorial this morning using Fluentd and Elasticsearch. But man, something died in me this week.

curiously, this was not the case until they changed they finally changed their name from 10gen to MongoDB, Inc.

Disabling the Caps-/NumLock Popup in Windows 8

Because this is so intuitive:

Type "display" in Search

Click on Change Display Settings

Click on Advanced Settings

Go to the On Screen Display tab

Uncheck "Enable on-screen display"

#windows8

Data Driven Analysis of a Data Driven Venture Capitalist

I am a regular reader of Tom Tunguz's blog. His blog has a lot of interesting insights, and he tries to anchor his argument on publicly available data. Whenever my engineering coworkers ask me about "the business side" of working at/on a startup, I point them to Tunguz's blog.

Recently, I became curious if there was any pattern to Tunguz's blog. So, I went ahead and scraped a whole bunch of his articles (whatever was accessible from his sidebar categories) and fired up Tableau and Excel to analyze the data.

Here are my findings. The data is based on [this scraper] up to the end of Q3 2014.

Becoming more and more prolific

The oldest blog entry I could find on tomtunguz.com dates back to April 4, 2012. Since then, Tunguz ramped up his pace, peaking at 49 entries in Q3 of 2013. That's more than one blog entry per every other day.

Entries are bulk-written on certain days

What is Tunguz's writing schedule like? Based on the dates that appear on individual blog entries, it seems like he bulk writes on certain days. What's interesting is that from one year to another, his "writing day" seems to shift.

The most frequent word in the titles?

Filtering out common particles and prepositions, it's..."startup". How surprising for a venture capitalist! Here is the top 10 most frequent words in his blog titles.

word count startup 121 your 57 saas 50 how 47 startups 36 market 24 marketing 22 why 20 sales 19 data 19

More articles about SaaS

As you can see, the word "SaaS" appears pretty frequently. This in fact seems to be a recent trend. Here is the graph of # of entries with "SaaS" in their titles.

A picture is worth a thousand words

Speaking of trends, I noticed that Tunguz's recent entries have a lot of pretty graphs. Here is the plot of average number of <img/> tags in his entries.

Colophon

The graphs were drawn using Tableau. Also, I wrote this script to scrape tomtunguz.com.

#data #tableau

R: apply 1/2

Because it's so obvious.

If I want to apply a function to a matrix row by row, then use apply(mat, 1, FUN=...). If column by column, use apply(mat, 2, FUN=...).

A Sum of Uncountable Positive Numbers is Infinite

The title is one of these intuitively obvious things that's a little tricky to prove.

Unpacking the Jargon

But before I get to the proof (which I thought about on my way home on Caltrain: a great way to pass time), let me convey its obviousness in less jargon-ish terms.

In real life, there are only two numerical classes: finite and infinite. Finite you can enumerate all the way through, infinite not. In math, there are different kinds of infinity. The "smallest" one is called countable, which means that if you keep counting them infinitely one by one, you can exhaust them all. The set of integers is like that, and a German dude called Cantor showed that the set of all fractions are countable, too. In general, a finite/countable union of countable sets is still countable.

But are all infinities countable? Not so. You can actually show that there are "more" real numbers than integers (or fractions): again, the proof is due to Cantor which you can read here.

Infinities that are not countable are called uncountable. Then, the title can be restated as this:

If you have more numbers than all integers combined, and if each of them is a positive number, their sum is infinite.

If you only have countably many positive numbers, their sum is NOT necessarily infinite. Here is one example: 1, 1/2, 1/4, 1/8, 1/16,...Basically, the set of fractions where the denominators are powers of 2. This set is countable (there is a 1:1 map between this set and 1, 2, 3,...) but its total sum is 2.

But somehow, when you have more numbers than countably infinite, the sum is necessarily infinite. How does this work?

Interlude: Pigeonhole Principles

Before we get to the proof. I want to go over a pretty obvious but useful idea called the Pigeonhole Principle. It's really obvious. It states that, if you have M things divided into N groups, one of them has to have M/N or more: Otherwise, the total would be less than M/N*N = M, a contradiction.

This idea also works with infinite sets, with a tweak. Remember that if you piece together a finite number of countable sets, the end result is still countable? Then, if you have an uncountable collection ℳ with countably many slots N, then at least one slot has to have infinitely many elements.

Why? Suppose otherwise. Then, each of the N countably many slots has a finite number of elements in it. Then, the whole thing should be countable, but we know ℳ is uncountable. This is a contradiction. (Actually, we can say something stronger: at least one of the N slots must be uncountably infinite because a countable union of countably sets is still countable).

This "infinite" version of the Pigeonhole Principle comes in handy later.

Proof

Here is the proof. Consider sets S1, S2, S3...defined as

S1 is the set of all positive numbers in [1, ∞)

S2 is the set of all positive numbers [1/2, 1)

S3 is the set of all positive numbers in [1/3, 1/2)...

Now, consider an uncountable set of positive numbers. Let's call this set P. Each element x of P is in Sn for some n because for any x>0, there is n such that 1/(n-1)>x≧1/n (with 1/0 defined to be ∞). So, P is in the union of Sn's.

But there are only countably many Sn's, so by the early Pigeonhole Principle, one of them has to have infinitely many elements of P. Let's say this one is Sr. Then, be definition, all elements in Sr are at least 1/r. But infinitely many elements of P are in Sr, and their sum alone is infinite. This completes the proof.

Why This Proof Works

The idea boils down to the relative "density" of countable and uncountable subsets inside the (positive) real numbers. Countably infinite sets are very small relative to the set of all positive numbers: This is why the 1, 1/2, 1/4...construction works to generate a countably infinite subset of positive reals with a finite sum. However, the moment you set foot in the uncountable world, there are a little too many elements and you start to "accumulate" somewhere on the line. The above proof takes advantage of this phenomenon.

This is totally useless for what I do day to day (I work on developer marketing at Treasure Data), but it's fun to think about this stuff once in awhile.

Memo to myself: Removing an ODBC data source from Excel

Open regedit and find ODBC.ini. Then, remove the corresponding key from "ODBC Data Sources".

Generating a Random Permutation in Excel

Excel...for what?

The other day, I was modeling something in Excel ("data scientists" out there: laugh all you want, but you too might find this post interesting) and needed to generate a random permutation.

Casual Googling resulted in pretty disappointing information: many suggested to generate a bunch of random integers in [1, N], possibly with repeats, until you cover all N integers. Then, pick the previously unseen unique numbers to extract a subsequence.

The above method actually generates a uniformly random permutation within a reasonable time: the standard coupon collector argument shows that you need ~N*logN random numbers.

Enter Fisher/Yates/Knuth

But you can actually do this with N random numbers. It's called Fisher-Yates/Knuth shuffling.

Starting with the ordered sequence 1, 2, 3...N, the shuffling algorithm goes like this:

for k = 1 to N-1 pick a random integer m from [k, N] (inclusive) swap the k-th and m-th number endfor

This is easy to verify for N = 2: You are just flipping a coin to decide if you swap 1 with 2. For N > 2, you just need to show that each of 1 through N has an equal chance of getting the k-th slot for 1 through N. In the first step, every number has a 1/N probability of getting into the first slot. For all other slots: the number has (1-1/N) chance of getting/staying there, then by induction, all slots are equally likely, hence (1-1/N)*(1/N-1) = 1/N. This completes the proof.

Implementation in Excel

Anyway, we can use Knuth's shuffle to generate random permutations in Excel.

The issue is, Excel is barely a programming language, and there is no "for loop". But that doesn't mean you can't do iterations! (Ok, we will end up using N^2 space, which kind of defeats the purpose of the algorithm...)

In the rest of this post, I will show you how to implement Knuth's shuffling in Excel. For simplicity, we assume N = 5, but this methodology works with any N (as long as Excel can hold it!)

First, you need to set up k's and m's. The trick here is we pre-generate the random m's. We are using Excel's RANDBETWEEN functions for Column B here.

Also, notice that in C1:G1, we have the sequence 1,2,3,4,5 initialized.

The next step is a bit tricky. Here is what the formula looks like:

For each row, we swap the k-th (Column A) and m-th (Column B) numbers in the previous row. This is what the gigantic IF statement is doing. Furthermore, since we want to run this calculation more than once, I have an extra condition: If the value of k is 1, then instead of the previous row, we use 1,2,3,4,5 to re-initialize the permutation (EDIT: on second thought, you actually do not have to re-initialize this and can keep running against different seed sequences without skewing the overall distribution since it's a 120-state Markov-chain with equal-probability transitions).

(Aside: INDEX function is very useful, but not that many people seem to know about its existence.)

Finally, we drag and expand the formula to populate the rest of the cells. I personally find this step most satisfying when I program in Excel.

C5:G5 is the random permutation.

Running it over and over and over

Now, let's verify this is indeed working. This step is very easy in Excel. We just keep populating Column A through G for subsequent rows. The only thing to watch out for is Column A should repeat 1,2,3,4,1,2,3,4...This can be achieved by using relative reference.

When everything is said and done, here is the distribution of 5! = 120 different permutations of [1,2,3,4,5] with 2,500 samples.

As you can see, it's fairly evenly distributed! The intercept for the line is 21, which is close to 2,500/5! = 20.833...

You can get the spreadsheet here

Obligatory promotions

My employer Treasure Data is also hiring. Most pertinent to the reader: we are looking for a data scientist to help our customers discover more value from their data. Email me at kiyoto-at-treasuredata-dot-com.

#hiring #anyperk #excel #stats

Docker Hub by the Numbers (My Docker Hackathon)

Last Saturday, I participated in the DockerCon hackathon. I am a fairly regular user of Docker (mainly to spin up new containers to verify/reproduce reported issues for Fluentd).

Docker is surely cool, and a torrent of announcements this week is a testament to its popularity and relevance. But what really fascinates me about Docker (or open source software in general) is community.

So, at the hackathon, I put on my "former data scientist" (read: a lowly quant in quantitative finance) hat, gathered data on Docker Hub (It was still called Docker Index last weekend =p) and drew a couple of not-so-sexy charts on Excel.

NB: the numbers are as of June 9th, 2014, and it's "best effort".

Data Collection: Most of the Work!

I used their API and sent a bunch of API requests to get metadata about repos and images. For example, to "crawl" repo names, I used their search API and searched for all alphabetical strings of length <= 3. To speed up the crawling process, I spun up several EC2 instances on AWS and ran scripts on them. (You can see my work here).

Yep. That's the irony. It was a Docker Hackathon, and I was spinning up EC2 instances.

Numbers

13,475: # of repos on Docker Hub (that I could find)

Docker just announced that they have over 14,000 repos, so I was pretty close...or maybe they rounded up their number =)

Here is the graph of # of repos created per month:

(June is low since we are early in the month).

The mean M2M growth is 43.99% for # of repos. The variance is still really high, so we can't really predict its future with much confidence. But if it were to growth 40% month to month, it will hit 1 million repos in Sep. 2015. (One point of reference I found is GitHub. They reached 2 million repos in 2011, three years into their operation.)

5,091: # of Docker Hub Users

This is my estimate for the number of repo owners. Which means there are 13,475/5091 = 2.64 repos per user. The median was 1, and the population standard deviation was 3.96.

Here is the bar chart of top 20 repo owners by the number of repos (or Dockerized apps):

101,477: # of images on Docker Hub

Admittedly, many of these images aren't the "tip" of the repo. I used their excellent Ancestry API to reconstruct much of their layer trees.

For example, the "deepest" path from the root to the leaf that I could find was afrantisak/ska with 120 ancestor images.

Here is the graph of the top 20 "deepest" repos.

Future Work

There are a couple of things I want to do in the future:

Dockerfile Similarity Calculations: based on my experience, we end up copy-and-modifying existing Dockerfiles a lot. Also, since many of the Dockerfile commands run sequentially, many Dockerfiles must look pretty similar to one another. Put more mathematically, if we were to view each Dockerfile as a finite sequence consisting of individual lines, if one Dockerfile is derived from another, it would be likely that they share a large subsequence. Perhaps it would be interesting to define a non-metrical distance of two Dockerfiles as the length of the maximal subsequence and run clustering over all Dockerfiles on Docker Hub.

Creating a Dockerfile tree based on the FROM inheritance: this idea came from @ewindisch. Right now, a trusted build (which means it has a Dockerfile hosted on GitHub) can inherit a non-trusted build because it's not required that a trusted build is built FROM another trusted build. It would be interesting to see how many trusted builds have a truly trusted lineage.

Thanks Docker, Inc.!

Last but not least, I had a TON of fun at Docker. As a developer marketer, I don't get to code as much anymore, and I could definitely feel that I was getting rusty. But it was really cool to roll up my sleeves once again, hang out with folks from different parts of the world, and discover a thing or two (I wrote my first Go code and learned a ton about Docker's APIs). So, a huge thanks to everyone involved in the Docker Hackathon =)

#data analytics #docker #open source

Trending Blogs

Recently Viewed Blogs

statistics and empathy