Yehia's Blog

news, views, and occasional clues

[yelkhatib-blog]

Feedback on CS Faculty Job Applications

Feedback on CS Faculty Job Applications

I’m going to volunteer some of my time to help colleagues from minority backgrounds who are on the CS faculty job market. I hope I can help even one person by giving them feedback on different elements of their application package.

If you are from a minority background and would like some feedback, please drop me an email at [email protected] with subject line “[apply-feedback]”.

D…

View On WordPress

[yelkhatib-blog]

Virtual Conferences

Well, well, well… Thanks to the global pandemic, we have learned that virtual conferences indeed are possible. Overall, the CS community has reacted fairly well. Many have moved to online settings with significantly reduced or none registration fees; well done to the committees behind EuroSys, ICLR, EuroVis, and others. On the other hand, some venues haven’t really made much reductions (I’m…

View On WordPress

[yelkhatib-blog]

Here’s a quick-and-dirty analysis of the CDNs used by the top Alexa websites that I recently knoecked together for a side project. Leaving it here in case it is useful for others interested in the subject.

CDN Discovery

Using a basic Python script, I queried the top 500 Alexa sites to identify if they are being served by a CDN. Because CNAME records are not returned for all DNS lookups, I relied on another methodology: check if the IP address in a DNS lookup ANSWER section belongs to the domain in question or not. The results are summarised in the following figure.

Insights

There are no groundbreaking insights from this plot: it shows CDN popularity amongst the top 500 websites. This contains a mix of different CDN provider types. As expected, traditional CDNs like Cloudflare, Fastly, and Akamai are among the most popular. Cloud providers are also popular, e.g. Amazon CloudFront and Aliyun (a.k.a. Alibaba Cloud). There are also telcos (e.g. China Unicom), hosting companies (e.g. OVH, Reflected Networks, and Automattic) and others (e.g. IAC Search & Media).

Upon closer look, the Chinese CDNs (such as Chinanet) are used for websites that are primarily targeting users behind the great firewall and also in Laos. This last fact is interesting as there seems to be no Internet censorship in Laos as far as I am aware, so perhaps this warrants further investigation. It might simply be due to its geographical proximity to China and, thus, Chinese CDN PoPs. Google is used by different Alphabet sites, with the exception of Evernote (perhaps a link formed by Chris O’Neill). Apart from this, other CDNs display a mix of different sites with no discernible pattern.

Limitations

There are a number of limitations to the above approach. It does not include CDNs that are serving through HTTP redirects or Anycasting. It also relies on WHOIS, so is vulnerable to any inaccuracies in the data retrieved from there. A potentially more robust alternative method would be to look into tracking CNAME chains.

CDN Response Times

I wrote another Python script to measure the response times (TTFB) of each site. These times were separated into the time needed for DNS resolution (as such, the use of DNS cache was disabled), TCP connection establishment, SSL negotiation (the local certificate bundle was used), and content transfer (RX). libcurl was used to retrieve these times. Each connection attempt was aborted if no response is received within 5 seconds. Each site’s index was first sought using HTTPS. If that timed out, the ‘www’ subdomain was sought instead, again through HTTPS. Failing these, the script reverts to HTTP at the domain and ‘www’ subdomain, respectively. Otherwise, the site is skipped.

The results are summarised in the plots below, where visualisation is again restricted to the CDNs serving 3 or more of the top 500 Alexa websites. The results are colour-coded by CDN, whose names are printed on the right hand side of each plot.

#gallery-0-6 { margin: auto; } #gallery-0-6 .gallery-item { float: left; margin-top: 10px; text-align: center; width: 50%; } #gallery-0-6 img { border: 2px solid #cfcfcf; } #gallery-0-6 .gallery-caption { margin-left: 0; } /* see gallery_shortcode() in wp-includes/media.php */

DNS

TCP

SSL

RX

Insights

This analysis gives us some general CDN comparison results. Some CDNs are better across the board than others. Fastly in particular displays low values for all times. Cloudflare is better than others for DNS, TCP, SSL, but not RX. Similarly, Line’s DNS performance is extremely good but is average for other times. One follow up study from this insight would be to look into how these CDNs operate differently from other CDNs, if at all, and if the sites they serve are of a different nature.

Another general observation is that Telco CDNs (like Chinanet and Japan NIC) seem to fare worse than traditional CDNs (like Akamai and Cloudflare).

The plots uncover other interesting variations within the performance of the sites served by each CDN. For example, there is a wide disparity in the performance of sites served by Akamai. This is particularly evident for DNS and TCP times. The same goes for Chinanet and Google across all times.

Limitations

The approach uses cURL, which returns a slightly different TTFB from that experienced by a user using a web browser. This is something I have looked into in the past (c.f. “Can SPDY Really Make the Web Faster?“) where I used the Chromium browser to retrieve webpages over SPDY and then analysed the resulting HAR and tcpdump files. An alternative is to use a headless browser like PhantomJS or Selenium, but I found these to be unreliable with varying results that fall in between those of the current methodology and the one used in the paper cited above.

Admittedly, the presented analysis is only a superficial one. More analysis is needed into the distribution of each CDN’s response times to identify where relatively extreme values arise from. An example of this is paytm.com (served by Amazon). Furthermore, a deeper comparison between CDNs across all times could be done to look into where some CDNs outperform others.

The methodology also favours sites not supporting HTTPS as their SSL negotiation times would be 0. This could be resolved by confining the search space to HTTPS results only.

Analysis of CDNs Serving Top Alexa Sites Here's a quick-and-dirty analysis of the CDNs used by the top Alexa websites that I recently knoecked together for a side project.

[yelkhatib-blog]

Domains in a HAR file

Here's some basic bash that parses a HAR file to extract a list of the unique resource domains. It uses jq

jq '.log.entries[].request | {method,url}' $1 | jq 'if .method=="GET" then .url else "" end' | grep -Eo "http(s?)://([^/]+)./" | sort | uniq

On Gist here.

P.S. You could create HAR files using a number of tools, such as PhantomJS, hdrgrab or chrome-HAR-capturer.

[yelkhatib-blog]

CrossCloud Workshop

We're organizing CrossCloud, an IEEE INFOCOM'14 workshop on cloud interoperability and federation. The aim is to bring together a congregation of systems researchers who have relevant knowledge and experience pertaining to assembling federated cloud architectures and handling the divergence in emerging cloud APIs. We'll also have an exciting keynote speaker which will be confirmed very soon, so stay tuned! Please consider submitting and participating. All details on the website:

     http://www.comp.lancs.ac.uk/~elkhatib/crosscloud

[yelkhatib-blog]

Converting JSON to CSV

A long time ago (I'm slightly embarrassed to admit) I mentioned that I wrote a script to flatten JSON to CSV and I promised to share it. I didn't fulfill this promise till today. The reason being that the script was very rough and I wanted to sharpen it before letting it loose on the interwebs. I still didn't get around to properly clean it up and probably will not for a couple more months or more. However, I decided today to release it as is. I'm sure some will find it useful enough and maybe be willing to tidy some of it up.

So here's the story. There are plenty of tools out there to convert from CSV to JSON; e.g. Mr. Data Converter, CSV to JSON, csv2couch. However, converting from JSON, a flexible self-describing format, to CSV, a much more rigid format, is not as simple. I found some attempts (like jackson-dataformat-csv) but these require a predefined schema. This did not really suit my needs as I do not want to define a schema for each JSON file I need to convert. I just know that I have a JSON with a reliably consistent structure and I want to convert it to CSV/TSV for another program that can only accept such format.

So I developed my code with such requirements in mind. So be careful: inconsistent JSON structure will result in wrong CSV. The code parses the JSON twice: once to discover the schema, and again to convert.

Here it is. Enjoy, and be gentle :-)

[yelkhatib-blog]

SPDY Adoption

The number of top 10,000 Alexa sites adopting SPDY increased between October 14 2012 and April 23 2013 from 208 to 271.

[yelkhatib-blog]

Multi-author comments in LaTeX

This is a simple command to simplify adding comments from multiple users editing a single LaTeX document, such as a manuscript shared between multiple authors.

\newcommand\notesAlice[1]{\textbf{\textcolor{blue}{Alice: #1}}} \newcommand\notesBob[1]{\textbf{\textcolor{green}{Bob: #1}}}

...and so on. Obviously edit styles to your liking. Then each author enters their notes like this:

\notesBob{This paragraph needs more text.}

To enable switching the notes on and off, use the ifthen package. Disabling all notes is handy to check document length, for instance. Here's how to do it:

\usepackage{ifthen} \newboolean{showNotes} \setboolean{showNotes}{false} \newcommand\notesAlice[1]{\ifthenelse {\boolean{showNotes}}  {\textbf{\textcolor{blue}{Alice: #1}}}  {} } \newcommand\notesBob[1]{\ifthenelse {\boolean{showNotes}}  {\textbf{\textcolor{green}{Bob: #1}}}  {} }

[yelkhatib-blog]

Get a list of unique resource domains from a HAR file

Uses jq to parse the HTTP Archive (JSON) format. More info here.

Script here: https://gist.github.com/yelkhatib/5403183

P.S. You could create HAR files using a number of tools, such as PhantomJS, hdrgrab or chrome-HAR-capturer.

[yelkhatib-blog]

Why We Need Client Hints in HTTP/2.0

Anyone who's done even a minimal web design task realizes how useful it is to know the capabilities of the client receiving the content. Chief amongst these is the User-Agent, which identifies the basic properties of the client's web browsing environment: the web browser family and version, the layout engine, and the operating system. This is important for the server to serve content that is adequate to that particular client.

There is another set of information that is very handy for ensuring optimal web page rendering, which is the display capabilities. Information such as the screen size and resolution is used by many websites to customize page layout (design and stylesheets) and content (images, buttons, etc.). This is increasingly relevant considering the following:

The proliferation of HTTP-enabled devices (mobiles, tablets, phablets, etc.) and their extremely varied display capabilities;

The never-ending race to improve the display of these devices as well as that of "traditional" web-enabled devices (i.e. desktops and laptops); and

The variance in HTTP-shipped multimedia (e.g. DASH).

To do this server-side is to save a round-trip. But previous attempts included a number of non-standard variants (UA-pixels, X-UP-devcap-screenpixels, X-Device-User-Agent, etc.) which cannot be relied upon. The alternative is to look at the User-Agent and Accept fields and try to guestimate the best screen size. This is obviously not very efficient but is surprisingly common. Relinquishing one round trip is what most responsive web designers do. Either the server replies with a 300 Alternates list [RFC2295], and/or the web site employs client-side script to report client capabilities. This incurs additional traffic and an additional round trip, something especially expensive in mobile settings. Solving this issue is not difficult. All it would take is for the user agent (i.e. browser) to provide some information about its display in a standard HTTP header. Such header information would have to be optional to avoid unnecessary overhead in cases where the user agent does not wish for the response to be optimized for it, as in the case of web services. Also, this header should refrain from giving away too much information that could compromise security or privacy (as is proposed by Device-Stock-UA which gives away detailed information about the user agent version). This is more pertinent if the next generation of HTTP is to have a Server Push feature like Google SPDY: It makes sense for the server to push content that it anticipates the client would need, but the purpose is defied if the server has to wait for more information from the client after its initial request.

Nevertheless, such proactive negotiation of client display capabilities is not naturally supported by HTTP/1.1 despite the issue being raised a long time ago. Others are also trying to solve this, albeit at a different level.

I was thus quite please to learn this week about Ilya Grigorik's proposal dubbed Client Hints [draft, slides] which is very much along the lines of what is described above. Client Hints also has measures to avoid negative implications on caching efficiency caused by inability to reuse HTTP responses. Update: Client Hints are now being implemented in Chrome. 

I think it is high time this issue is discussed as the IETF Hypertext Transfer Protocol Bis working group are actively engaged in discussion to draft HTTP/2.0 (follow progress here). I guess pivotal to this discussion is whether HTTP should be concerned with solving this issue at all, and this is probably something the community would be divided upon. There already are concerns of HTTP/2.0 being too browser-centric. My personal opinion is that this is a session and presentation issue and hence falls within the domain of HTTP.

[yelkhatib-blog]

Exciting month in the Hadoop world

January has so far been an exciting month in the Hadoop world. Three heavyweights with different offerings of the MapReduce platform commonly used in data analytics.

First up was Netflix, introducing us to Genie. Genie provides additional abstraction on top of Amazon's EMR through a RESTful API. It also allows the management of job flows on different clusters. What is not clear is whether there are any plans to make Genie available as open source or otherwise. Will we see it on GitHub soon?

In typical Netflix fashion, the blog post also provides technical insights into the Netflix kitchen this time about their data warehouse architecture. The biggest surprise to me was the reliance on Amazon S3 which I know from experience can stifle Hadoop performance. Their main reason seems to be that with S3 it is possible to spin multiple clusters simultaneously without needing to manually replicate data stores or worry about inadvertent data loss.

Next is Hortonworks Sandbox, a virtual machine packed with a couple of things that a Hadoop novice would find handy: Hortonworks' own flavor of Hadoop preinstalled, as well as a step-by-step tutorial to get started with Hadoop. I wonder if the VM is useful beyond its standalone environment, e.g. to easily manage an external Hadoop cluster. There is also an associated webinar, and a promise to provide more tutorials and exercises in the coming months.

Last but not least is Joyent, who announced an enterprise Hadoop-aaS provision that is supposed to be of much higher performance than others in the market. They claim "3x faster or at 1/3 the cost of other cloud offerings". Something's gotta give, not sure what yet.

Hadoop might indeed be old hat to some, but no one can deny that it still is an important programming framework that is widely used for crunching data.

[yelkhatib-blog]

Interface is not everything

Interface is important. It's what conveys your message to new users. It's what users interact with to use your service. However, it is not what defines your service. I think this is a message missed by many who obsess about eye candy and user experience. Don't get me wrong; these are important aspects but more important is making sure that the service does what it says on the tin.

When I think back about many of the "killer Internet services", I find it peculiar that their interfaces are not the sexiest looking ones. For instance, Google Search started and remained a pretty basic: a text box and a 'Go' button. Many other search engines that Google replaced (remember HotBot?) provided more sophisticated user interfaces at the time but failed to compete with the efficiency of Google. 

Facebook started with a ridiculous interface: it looked simple but was riddled with inconsistencies. Also, it accumulated a number of weak applications: e.g. Facebook Groups are still far inferior to old-school discussion forums, friend feeds are not as versatile as its parent the RSS feed, etc. More to the point, its interface keeps changing and we all like to complain about it. But all of this doesn't matter much and the vast majority of us happily continue to use it. Facebook promises convenient and sustained social interaction with friends and colleagues, and it delivers on this promise.

Twitter, similar tale. I must admit that upon visiting Twitter for the first time its user interface struck me as ugly and shallow. This is because I (as well as others who were quick to write it off) compared it to other feature-rich online social networks. What soon became evident is that we misunderstood their mission. Twitter is competing in a different arena offering something new, so it didn't matter if their interface looked a bit sparse.

Another example: why are there much more people using YouTube instead of Vimeo? The latter offers a far glossier interface and a slicker video playback quality, not to mention less noise through better content filtering (i.e. less webcams). But that last point is exactly why YouTube dominated from the start and still does. It promises to be a platform to deliver your video to the world, and that it does.

These examples show how many Internet users are not really bothered with getting rich, ultra-interactive user interfaces, but instead are greatly interested in things that make life easier, novel  functionality, and content that has not before been available or convenient to access.

[yelkhatib-blog]

Getting Client information in Node.js

To get information about your client in Node.JS (because it is slightly different than in PHP). Here’s an example snippet from onRequest that you could use in your server start function.