Original Real Time Bidding / Header Bidding writeup from 2006
Just posting this for posterity. Back in 2006 I was frustrated that our ad network clients at Strategic Data Corp were having to redirect massive volumes of ad impressions to each other while they tried to find the needles in their haystacks of valuable users in their impression streams. Each had their own set of advertisers, publishers, and user data (such as retargeting) that made different impressions valuable to each. So once they got an impression from a publisher, if it wasn’t valuable to that network, they would sell it to a competitor to recoup some of the cost. Hence the idea for what became known as Real Time Bidding as well as now Header Bidding.
I can’t say with certainty that no one else had thought of this at the time. I haven’t heard of anyone saying so though. We did file patents on specific aspects of the idea. The first patent which most resembles Header Bidding is here. The rights to that patent were later acquired by the Rubicon Project as part of the acquisition of Fox Audience Network in 2010, which was formed out of the acquisition of Strategic Data Corp in 2007. So far, the patent hasn't been granted (although a couple of downstream patents were); I have no idea why nor do I have complete picture of the decisions Rubicon has made with regard to pursuing the application.
I wrote the below document, with substantial input from my co-inventor Fabrizio Blanco (@fblanco) in 2006 as the basis to begin the patent process. It has never before been published and looking back, it’s a pretty comprehensive description of the various ideas.
In 2007, we were acquired by Fox/News Corp, and we implemented the concept in the above patent application, which is pretty much what people are now calling Header Bidding. At that time, we started requiring all of the ad networks/exchanges buying from us to switch to this method. This included Rightmedia/Yahoo, AOL/Advertising.com, and the new Appnexus. At the Fox Audience Network we also started buying this way from parties including OpenX.
Of course, Real Time Bidding took the industry by storm and became the dominant approach to trading ad impressions through intermediaries (as opposed to simple brand buys made directly with publishers) within the online advertising ecosystem. It was an amazing process to see. In 2009, working with Adam Bain (the then head of the Fox Audience Network and now COO of Twitter) and Arnie Gullov-Singh (then the Fox EVP of Product and now COO of Polyvore at Yahoo), we started proposing creating a standardized RTB ecosystem. I bought the domain openrtb.com on my personal credit card, which I then handed over to Fox and expensed. That domain later became the home of the OpenRTB consortium.
Here’s that write-up from 2006:
November 29, 2006, Santa Monica, California Jason Knapp & Fabrizio Blanco Strategic Data Corp
Summary:
This document describes a new method for creating an instantaneous auction for each advertising impression as it becomes available for display. The new method allows various advertisers and/or advertiser networks to simultaneously bid for an individual impression where the highest bidder then instantly shows their ad. This new method includes communication between various computerized advertising systems in real-time creating a dynamic real-time bidding exchange that will maximize the value of each impression to the originator of the ad impression (the publisher). This method has the further unique benefit of allowing the advertiser and/or advertiser network to communicate directly with the user’s web browser and obtain all information about that user’s computer that the advertiser otherwise can only see after they have secured the impression.
Terminology:
For reference in this document we shall use the following terminology:
Advertiser: An entity buying advertising for its own products and services directly or indirectly from the publisher.
Advertiser Network: An entity buying advertising directly or indirectly from the publisher on behalf of a number of different advertisers. An Advertiser Network is one type of Advertiser, although other Advertisers then buy these impressions from the Advertiser Network.
Publisher: An entity serving the webpage to a user that displays advertising. For example, ESPN.com is a publisher.
Impression: each individual time an ad is shown to a particular user on a webpage.
Publisher server: The ad server which makes the first decision as to what to do with an impression for an ad shown on a webpage. This server may be managed by the publisher itself using licensed or internally developed software, or this may be a server managed by another party by agreement with the publisher. In the later case, this is accomplished by the publisher redirecting impressions from its web pages to either an outside ad serving company (e.g. DoubleClick) or an outside ad network (e.g. ValueClick).
Active server: The server currently controlling what ad will be served for the impression. For any impression, this is typically first the Publisher server. In the case of redirected ads, the server that the ad is redirected to then becomes the active server for that impression and the previously active server becomes an “Upstream” server.
Upstream server: A server that was an Active server for this impression, but the impression has been redirected to another server which then became the “Active” server.
Downstream server: A server that is eligible to receive this impression via a redirect. The Active server must decide to send the impression to the Downstream server, and if it does so, that server then becomes the active server.
HTML: HyperText Markup Language, the predominant markup language for the creation of web pages.
Redirected Ad: Instead of actually displaying an ad when an impression is available, the impression can be sent to another ad server to determine the actual ad to be displayed.
URL: Uniform Resource Locator, a means of locating the resource by describing its primary access mechanism and location, example of resource could be an HTML page.
HTTP: HyperText Transfer Protocol, a protocol used to transfer or convey information over the internet, primarily used to send and retrieve HTML pages.
JavaScript: scripting language commonly used in web pages.
Cookies: Parcels of text sent by a server to a web browser and then sent back unchanged by the browser each time it accesses that server.
XML: Extensible Markup Language, a general purpose markup language that supports a wide variety of applications.
AJAX: Asynchronous JavaScript and XML, a web development technique for the creation of interactive web applications.
JSON: JavaScript Object Notation, a lightweight data-interchange format.
DOM: Document Object Model, a description of how an HTML or XML document is represented in a tree structure.
IFRAME: an HTML element which makes possible to embed another HTML document inside the main document.
Background:
Before the invention of the new method, advertisers needed to bid a fixed price for a fixed timeframe and volume of impressions based upon whatever targeting the Publisher or Active ad server allowed. The old methods did not allow Advertisers to pay different amounts based upon information they know about the user or impression that the publisher did not or would not allow targeting based on. Also Advertiser Networks need to bid a fixed price for impressions based upon how they worked on average for the advertisers they represent rather than the actual value that a specific impression would be worth at the moment it was available based upon the ads then available.
Before the invention of the new method, for any impression on the Internet, either a particular ad must be hard coded onto a particular web page such that there is no ad server, or the Active ad server must decide which advertisement to serve for that impression. This decision is made on a variety of factors, and the actual decision process varies dramatically from one system to another and one business to another, but typically may include such factors as the predicted revenues from each ad as well as, priority, the budgets, constraints and goals of the various advertisers. These ads the Active server must choose from can be grouped into two categories:
Terminal ads where only one specific image and/or text will be displayed. If a terminal ad is selected, then that particular ad is displayed.
Redirect ads where the Active server will tell the user’s browser to retrieve the ad from a Downstream server on the Internet, a process called a “redirect,” where the Downstream server is allowed to then pick from multiple ads. Once this Downstream server receives the impression, it becomes the new Active server, and it may choose to serve a Terminal ad available on its system or it may again redirect the impression to yet another Downstream party.
A key difference between these two cases is whether or not the ad decisioning process ends once the Active server selects the ad, or, in the case of the redirect, if another, Downstream server may then make a subsequent ad decision. The Active ad server will determine the Terminal or Redirect Ad based upon various different decision rules that vary based upon the individual ad servers and the business rules given to them.
The redirect case is very common in advertising on the Internet today. However, this is an extremely inefficient method of selecting which ad to serve for a particular impression. Enormous value is wasted because the Active ad server needs to make the ad serving decision without knowing the unique value of that impression to the different advertisers of the Downstream servers. Downstream servers either bid a fix price for all impressions fitting a set of general criteria, or targeting rules, or pay a revenue share of the average value they earn for the impressions they receive. This means that the Upstream server cannot observe how well a decision to redirect one particular impression to the Downstream server performs versus another impression sent to that sever.
An alternative approach being utilized today to try and alleviate this problem is to send the impression to the Downstream party, but require the Downstream party to either return the impression by a second redirect or serve an agreed upon alternative ad, if the entity owning the Downstream server (Downstream party) will not pay at least a certain amount, the threshold amount, to the entity owning the Upstream server (Upstream party) for the impression. In addition, the Active server typically gives the Downstream partly only a single threshold amount for all impressions of a particular type for a particular period, meaning that the Downstream party does not keep impressions based on their individual, relative value to the Publisher or Upstream server, but rather an overall average. This becomes particularly limiting with more Downstream parties for two reasons. First, each redirect typically results in a loss of 5-10% of impressions, as the user moves their browser on to another page or some other communications failure before an ad finally returns. Second, there is no way for the Active ad server to compare the value to be earned from the different parties for each impression. Rather they must give impressions to publishers in order based on their average performance, which will vary dramatically from individual impression to impression.
New Method
The new method described here is to contact several Downstream servers of Advertisers or Advertiser Networks simultaneously in real-time and ask them to bid on the impression based upon information about the user and impression provided by the Active ad server as well as information the advertiser may already have on the user. One particularly efficient method to accomplish this would be for the Active ad server to generate and serve a script to the user’s browser which would in turn call each of Downstream servers selected for consideration for this impression and retrieve the their bids. The script could then compare the bids including the best bid from the other Advertisers on the Active server, and then serve the highest bidder’s ad. This method also allows the Downstream servers to interact with the user’s browser in a way which provides the same information, such as from the cookies of the Downstream party, as the server would receive if the impression was redirected to the Downstream server. This allows the Downstream server to consider factors not known by the Active server in generating its bid, such as the number of times the user has already seen each ad available on the Downstream server and on what other sites the Downstream server has seen this user. Even the past purchases of this user could be considered. These factors could not be considered in a traditional bid to the Upstream entity, because the Upstream server does not know this information about the individual users.
This approach is far from obvious. First, it relies on recent improvements in HTML scripting technology that allow for such calls to be made from the users browser to the various parties, and then for the user’s browser to act on the results independently. Many technologies are involved, including those under the umbrella names “JSON”, “DOM”, and “AJAX”. Second, it represents a significant departure from traditional ways to serve advertising online, especially with regard to the interaction of multiple parties in selecting an ad to serve. Third, it also relies on the existence of sophisticated yield maximization/optimization algorithms at the parties involved, technology which is not available to most parties today.
The process is detailed as shown in the following diagram:
More detail about each of these steps follows:
Browser requests webpage: This is a typical process today, and remains unchanged.
Publisher server(s) receive request from browser, build ad script, and return webpage and ad script to browser: This step may occur in a number of ways, and may be divided between a web server and an “ad” server by including an IFRAME URL, JavaScript, or other reference to the ad server in the page returned by the web server. Or the functionality of these two servers may be integrated such that the ad server’s response is built in the web page when delivered to the user. Likewise, the publisher server may not utilize this method at all, but may instead redirect the impression to another server, which then as the Active server may generate a script. This may occur at an unlimited number of steps away from the Publisher server for a subset of that publisher’ actual impressions. Regardless, for an impression that does reach such a server, the browser eventually receives a script including calls to the other parties being asked to “bid” on this impression, as well as information about the best possible alternative ad to serve instead of an ad from one of these Downstream servers.
The script itself contains several pieces of information:
List of Downstream servers to call and technical information required, such as the Internet URL at which to call these servers
Optionally, any adjustment factors to be applied to the responses from those servers in making the decisions. These factors can change the interpretation of the responses from the servers relative to each other. These may account, for example, for past discrepancies in the final revenue figures counted between the Active server and the Downstream servers, so as to properly include predictions of future discrepancies in current ad decisions. They may also factor in issues such as the relative priority for delivery of ads from the different Downstream servers based on factors other than the bid price, such as other elements of the business relationship or the speed that ad is serving relative to a daily budget for that particular Downstream server.
URLs to call the Active server for each ad to report which ad was selected as well as any other information desired to be tracked by the Active server, such as the list of other ads that were available for use in determining the percentage of impressions where each ad was provided to the script that it was actually selected.
URLs for Downstream servers to call the Active server under various downstream events, such as a user click or conversion on the ad, to be used for tracking ad performance.
Other factors to be considered in the ad decision by the scripts, passed to the Downstream servers, or reported back to the Active server.
The code necessary to perform the ad decision, serve the appropriate content to the user, and make the various desired calls to the Active or other servers.
The script then executes in the user’s browser. The script first issues calls to each of the Downstream servers that is identified in the Active server as supporting this type of request, subsequently referred to here as a “Bid” request, to obtain their information. These requests would include information identifying to the Downstream server the source of this request, as well as, optionally, other information known to the Active server which may be useful to the Downstream servers’ decisions, such as known user profile information. Since the Downstream server is communicating directly with the user’s browser, the Downstream server will also receive any cookie information it has placed in that user’s browser from previous interactions.
Each Downstream server would receive this call, decide what ad they would serve, both or either the pricing under which they would pay the Active server or the expected amount they expect to pay, hereafter referred to as the “Bid”, and then respond with the required values, contained in JSON, XML, or any of a variety of current and future possible encapsulations. The Downstream server can also update the user’s cookie as it wishes.
The script can then wait up to a predefined period of time for a response from each of the servers. Once all responses are received, or this time is reached, the script compares the responses from each of the servers with any alternative option(s) included in the script by the Active server, selects the ad to serve.
The script would then serve two elements to the user:
the appropriate code to generate the ad or redirect it to the selected Downstream server. This code may be a URL to which the script will redirect the ad impression, for subsequent ad delivery by the Downstream server, or it may be the actual HTML code of the selected ad itself, eliminating the need for an additional redirect and thereby speeding ad delivery to the user. In either case, this URL or HTML code may also include click or conversion tracking URLs mentioned above, which would then call the Active server in the event that the user clicks, converts, or takes other action which would be useful for the Active server to know in order to properly optimize future decisions regarding the selected ad.
a counting call back to the Active server, reporting the ad that was selected as well as other information needed by the Active server to properly account for the ads considered and selected.
Modified Method: Preemptive Bidding
A Publisher may allow a similar call in order to allow a Downstream server to preempt an impression from being redirected to a competitor, even if the Publisher does not have the technology to generate the dynamic scripts described above. For example, assume a Publisher does not have an Ad Server, but instead has edited its web pages to send all Ad Impressions to a single Advertiser or Advertiser Network. A competing Advertiser Network could provide the Publisher with a fixed script, similar to that described above as generated dynamically by the Active Server, which the Publisher could embed in its web pages. The publisher would embed into this script the value typically received from the original Advertiser or Ad Network. This script would call the competing Advertiser network, request a bid, and then redirect the impression to the original Advertiser or Advertiser Network if the bid is lower than typically received from the original Advertiser or Advertiser Network, or redirect the impression to the competing Advertiser Network if the bid is higher than typically received. In this way both the Publisher and the competing Advertiser Network benefit. In addition, note that the “bid” described above could be a score based on any criteria desired by the Publisher.
Alternative Use Without an Ad Server
Furthermore, a Publisher could use such a script, in lieu of a complete ad server or dynamic script generation, to receive bids from several competing Advertisers or Advertiser networks. The Publisher could include the script, including calls to a list of Advertisers or Advertiser networks, in its web pages for serving advertising impressions. The script would request bids from each of the Advertisers or Advertiser networks, select the highest bid, and serve the corresponding advertisement or HTML. By including a counting call as described in the Primary method above, the Publisher could use a simple system to total the bids and impressions that are served to each Advertiser or Advertiser network.
Alternative Use: Server to Server Communication
A similar, alternative method would be to make a similar call to each of the Downstream parties from the Active server directly, passing the IP address of the user as well as cookie information previously received from those Downstream parties on previous such interactions. Then each Downstream party would reply with the performance information from above, as well as updated cookie information which the Active server would then store on the user’s browser. Both methods are nearly functionality identical. One difference is that in the second case, the cookie information can only be accessed by the Downstream parties when communicating through this Active server, since they would be stored under the internet address or domain of the Active server, not the Downstream server. The Downstream servers would also not be able to see or write this information when interacting with the same user elsewhere on the Internet. In addition, since the Active server is contacting the Downstream servers, there will be additional communication load on the Active server, which is handled by the user’s browser under the script design.
Method Variations:
As in these alternative examples, there are several potential variations on this design to handle various circumstances. A few of the most likely such variations include:
A query can request any or multiple of various pieces of information, such as:
Whether or not this impression will be accepted by the Downstream server: For example, the Downstream party may be paying the Upstream party on a prearranged price (such as a flat amount per impression, click, lead, or sale), in which case transmitting economic information may be irrelevant. The Downstream server may instead reject impressions because their predicted performance is too low, or because all available ads on the Downstream server are ineligible because their individual serving requirements do not fit this impression or user.
Bid price and type: The amount the Downstream party will pay for this impression, or a resulting click, lead, conversion, or other event
Score or prediction: The Downstream server could provide any or multiple of a number of different possible ways of rating this impression. For example, it could provide a prediction of the performance of this impression, such as expected revenue or payout, expected click-through or conversion rate.
URL or HTML to serve in the event that the Downstream server is selected to receive the impression
Real-time calls can be initiated from any or multiple different points in the chain
Publisher web server: The Publisher may include the script in the web page when the page is served
Publisher ad server: The Publisher may serve a web page, including code which directs the user to a separate ad server of the Publisher or a service provider working for the Publisher. This ad server may then serve the script.
Ad Network or Advertiser: After receiving an impression from the Publisher, or from another Ad Network or Advertiser, the Ad network or Advertiser may return a script to initiate a round of calls. This may occur even if a script with a round of calls already occurred at the publisher or another Upstream server.
Each server may return various types of code to the browser:
A specific creative: HTML, an image, a Flash file, or video, or other type of creative such that no further decisions need to be made about what creative to serve
A redirect to another server: The server may return code to redirect the impression to another server, which may then make any of these choices once it controls the impression
A script: A script such as the one described in this method which would make real time calls to one or more other servers, and then decide whether to serve the impression per that server’s requirements or, if included, serve the selected traditional creative or redirect.
The script may or may not be dynamically generated:
An ad server may generate the script dynamically, deciding at the time which Downstream servers to include, what, if any, traditional ad or redirect to include, and what criteria to include. Such an ad server may include sophisticated yield-management or optimization technology in order to make such decisions intelligently from a potentially large base of options.
The script may be static or “hard-coded,” returning the same code to all visitors until changed. In this way, for example, a Publisher could use such a script to make one or more real-time calls to receive bids from one or more Downstream servers, compare the bids, and send the impression to the highest bidder, without the presence of an ad server at all. Such a script could further include a static redirect or creative to send the impression to if no bids exceed a value defined in the script.
Additional Applications:
The described method also has applications outside of traditional browser based Internet advertisements. Any interactive advertising environment with the ability to communicate with multiple advertisers in real time at the time of the decision as to which ad to show could follow a similar process. Potential applications include interactive television, mobile and gaming platforms as well as even dynamic billboards or even real-time bidding for broadcast radio or television. In addition, such a method could also be used in other bidding environments, such as comparing prices for a particular item in an interactive shopping environment.
Status: [as of Nov 2006]
At this time, the inventors have completed development of a working prototype of this technology for the selection of Internet advertisements under the primary method described above. Other than this prototype, which is not being used to serve ads outside of our test environment, neither the method nor technology have been deployed anywhere as of this date nor has this concept been disclosed to any party outside of the inventors’ employer, Strategic Data Corp.