#browsermob proxy

[plightbo]

The Future of BMP

TL;DR — I’m leaving BMP but I hope the project continues under new leadership. I have some ideas about what work needs to happen and I’m here to help with the transition. Read more…

Over the last year I’ve come to two conclusions:

I am personally no longer able to commit the time necessary to maintain the BMP project, primarily because of Real Life: two young daughters (1.5yo an 3.5yo) and an increasingly-demanding day job.

The BMP codebase is somewhere between “middle-aged” (the SiteBricks / REST API layer) and “decrepit” (see history below).

Given these two facts, I’ve decided that the best thing for the project and the community is for me to (gradually) retire from the project.

I don’t want to leave the community in a lurch, so my hope is to recommend a technical direction (outlined here) and nominate a number of recent, active BMP contributors to carry the torch. 

I will, however, not be responding to any open pull requests or incoming support questions on the mailing list starting immediately. My commitment is that I support anyone else who wishes to continue this project or start up a similar-but-alternative one.

But first, a bit of history…

BMP (aka BrowserMob Proxy) dates way, way back… about ~10 years ago, when Selenium was first getting off the ground. You may not know, but Selenium wasn’t aways offered as a native “Web Driver” to various browsers. Back then, it was a simple JavaScript library that ran inside the web application.

Some of the committers (myself included) realized that this approach was limited and required that the application under test bundle Selenium inside of it. In order to bypass that requirement, we created an HTTP proxy injected the JavaScript in to each header of each page load and did all sorts of tricky stuff. Eventually we started doing really funky stuff, like doing man-in-the-middle “attacks” so that we could inject in to SSL. 

Meanwhile, I began to get in to performance monitoring and load testing and realized that the proxy could be modified to serve my desire to measure HTTP request performance. I ended up starting two separate startups and forking the Selenium code to serve my needs. While during my second startup (BrowserMob) I realized that it would be helpful if other people could get performance data from their Selenium tests, so I forked the Selenium code a third time and BrowserMob Proxy (BMP) was born.

I tell all this history partly because I’m a bit nostalgic as this is the last remnant of code I’ve dabbled with for a decade, but also partly because it helps explain why the project has the features and cruft that it has.

The Features aka The Good

Because of my participation and leadership in the web performance market, I learned a lot about the features people need when doing performance testing. Some of these features can only be done through an HTTP proxy. I won’t outline all of them (consult the docs or API if you want a full list), but the main ones I think matter are:

SSL support!

Control over DNS lookup behavior

Bandwidth simulation

Injecting headers in the request and response

Handling authentication

Backlisting individual URLs and URL patterns

And of course: extracting performance data in a useful format (HAR)

Any one of these features can be done using non-proxy technology, but it’s hard and requires multiple complex pieces of software to do it all. That’s why I believe that for general purpose usage, the approach BMP takes is a good one for most people. That said, I do discuss alternatives (proxy and non-proxy alike) in a moment.

The Cruft aka The Bad

There are two fundamental problems with BMP:

I’m a mediocre programmer :) Seriously, the code base is monolithic and tangled, making it harder to maintain than it should be.

It depends on a very old chunk of Jetty code that was created and donated by Jetty founder, Greg Wilkins, ages ago.

Issue #1 isn’t too big of an issue: we just need to get some better programmers involved, which has already been happening. 

Issue #2 is more challenging: in order to maintain SSL support and ensure that BMP works reliably with modern stuff like, say, WebSockets, we need to have a more modern proxy engine, but we’re unable to upgrade to something newer, including Jetty.

This is because the SslRelay code that does the man-in-the-middle termination is incredibly complex and hard to reproduce, especially with newer web servers like Jetty and Netty, that use async. 

The concept of the code is simple, but I will be the first to admit that I’ve never fully understood the deep, dark corners of the Jetty 5 codebase on which it depends and I’ve fortunately been able to simply tweak Greg’s code over the years.

Recommendation

My recommendation is that BMP needs a major overhaul — probably a complete rewrite. In particular, I believe that BMP could become a nice performance-oriented project that wraps around LitteProxy.

LittleProxy can do most/all of the basic proxy stuff BMP needs, including SSL, and it has most of the necessary callbacks/extensions to support the BMP features. It may be missing a few things, but I believe those can be addressed by contributing to LittleProxy and leaving BMP to focus on the clean, performance-oriented wrapper around it.

Writing it out it doesn’t sound like much, but in fact this is a considerable amount of work!

In addition to this core proxy work, I also think the project would benefit from pulling apart the core server from the client driver(s) for each programming language. On top of that, a nice web-based GUI could be incredibly useful for people just getting started or for people doing exploratory testing.

Alternatives

I believe that BMP is the best choice for most performance testing needs, but it would be prudent to call attention to some important alternatives.

On the proxy side of the house, there is Charles Proxy and Fiddler, which are both much more popular and well-known. The main downside for both of them is they don’t have all the features I outlined and they (IMO) are not quite as easily embedded inside of test automation systems.

More importantly, there are some interesting non-proxy solutions that should be considered, especially if you don’t need all the features of BMP and just want performance data.

If you’re using PhantomJS + GhostDriver, consider the getLog() API and ask for the type “har” and you’ll get back a very nice HAR file with all the timings of the assets.

If you’re using ChromeDriver, there is some really in-depth performance data available, though sadly not in the convenient HAR format.

If you’re using Firefox, with a little elbow grease you can get FireBug + the NetExport plugin to leave HAR files behind in a folder or sent to a beacon after every page load.

And lastly, many modern browsers now support the Resource Timing spec, which means you can make the JavaScript call of “window.performance.getEntries()” and get back individual timings for most assets on the page. Sadly, response code and file size are not included.

The New Guard

So who will carry the torch? I’m not sure! But the good news is that there have been a number of contributors and commercial organizations using BMP over the last few years. In particular, three individuals stand out: andyclark, jekh, and nite23.

I’ve already granted all three commit rights to the codebase and I hope that at least one of them steps up and carries forward the BMP vision. If they do, I’ll be here to help them along the way.

[elisp-solves-problems]

Playing with the new Perl bindings for the BrowserMob Proxy

Browsermob Proxy is an open source project that Patrick Lightbody split off from the main Selenium project. It works really well in tandem with Selenium Webdriver, but it was missing Perl bindings. BMP exposes a RESTish interface for interacting with the proxies, so I've strung together a Perl module to take care of it: Browsermob::Proxy. Amongst other things, you can use BMP to throttle net traffic during tests, analyze request/response pairs for things like Omniture and Google Analytics, and even alter the requests on the fly with custom headers. Here's a short run down and some basic get-started scripts for using it!

The bindings are up on CPAN with pretty limited functionality at the moment - definitely not the full BMP API. It can create proxies and HARs, and spit out the HARs back out when requested, but it doesn't do much else besides that as of v0.04. To get up and running with BMP, you'll need to download the binary from the BMP website and then execute the appropriate file in its bin/ folder to start the BMP server.

Here's an example of using Browsermob::Proxy with Selenium::Remote::Driver to capture all of the network traffic during a single request to google:

#! /usr/bin/perl use strict; use warnings; use DDP; use Browsermob::Proxy; use Selenium::Remote::Driver; my $proxy = Browsermob::Proxy->new( server_port => 8080, ); my $driver = Selenium::Remote::Driver->new( browser_name => 'chrome', proxy => $proxy->selenium_proxy ); $driver->get('https://www.google.com'); p $proxy->har;

This is assuming your BMP server is already downloaded and currently running on port 8080. But, that's all we need to create a new proxy for the Chrome browser that Selenium::Remote::Driver is about to make. The $proxy comes with a convenience method selenium_proxy() that returns the appropriate desired capabilities to configure the proxy for use with Chrome. (If you want to use a proxy with Firefox, you'll need to use a custom Firefox profile.)

Once we've instantiated our driver, having told it to use the proxy, all that's left is to create some traffic and spit out the HAR.

If you want to try out the module without having to install Selenium::Remote::Driver, you can just curl some traffic across the proxy:

#! /usr/bin/perl use strict; use warnings; use DDP; use Browsermob::Proxy; my $proxy = Browsermob::Proxy->new( server_port => 8080, ); $proxy->new_har; my $generate_traffic = 'curl -x http://127.0.0.1:' . $proxy->port . ' http://www.google.com > /dev/null 2>&1'; `$generate_traffic`; p $proxy->har;

This creates a proxy in exactly the same fashion and then uses curl to generate the traffic.

[plightbo]

BrowserMob Proxy 2.0 beta 9 released!

I can't believe it's been 6 months since the last release already! Though considering that I have a four month old infant (along with a 2 year old) it shouldn't surprise me that time seems to be flying by.

Anyway, lots of new stuff in this release:

Various improvements to make sure the HAR output matches the latest 1.2 spec

A bunch of improvements to the REST API (with docs!)

Replaced the User-Agent parser with one that won't take down the proxy if the online UA service is having trouble

Advanced feature to define which address to bind each individual proxy to

Bug fixes: concurrency problems, charset fixes, better logging, more tests, and more.

Note: in this release the request/response interceptor interfaces have been modified to take a second argument (Har). If you use the Java interfaces you'll need to update your code. If you use the JavaScript REST interface (introduced in the last release) you don't have to do anything.

As usual, you can find the latest download at http://bmp.lightbody.net.

[plightbo]

BrowserMob Proxy 2.0 beta 8 released!

It's been a few months and it was pretty obvious a new release of BrowserMob Proxy was in order. If you're impatient, you can get it here and skip this announcement. 

The most important thing to note is that I've forked the project back in to my personal Github repository. In addition, the website has been moved to bmp.lightbody.net. I did this simply because I no longer work at Neustar/Webmetrics and I'm the primary committer to this project, so I figured I should keep it where I know I'll have control over it.

Now to what's new:

A brand new test suite so I can finally make sure I'm not breaking anything too badly

Restored Java 6 compatibility, which broke in beta 7

Binary content can now be captured, resulting in base64 encoded content in the HAR body

We no longer modify the User-Agent header (it used to insert "BrowserMob RBU" in to it)

Post data (URL encoded and raw content) is now reliably captured in the HAR

This release also includes some REST API improvements:

Request/response interceptors can now be set via the REST API using a simple JavaScript scripting language (needs documentation still)

Automatic BASIC authentication can now be controlled via REST

Waiting for network activity to stop can now be accessed via REST

Network activity control, such as maxBitsPerSecond, is now accessible via REST

Control over connection timeouts is now available via REST

And more!

PS: Someday I'll get back in to more interesting blogging, but for now I'm just happy to post something :)