The BrowseComp: Benchmarking Web Browsing Agents
A simple yet challenging benchmark that tests AI agents' ability to discover hard-to-find information.
AI agents that learn from internet surfing are growing in importance. A performant browsing agent may locate hard-to-find information by visiting tens or hundreds of websites. Models using rapid browsing tools, such as GPT-4o, have exceeded SimpleQA criteria for extracting basic data. OpenAI is open-sourcing BrowseComp, or “Browsing Competition,” a benchmark of 1,266 challenging topics to test AI agents' ability to identify convoluted, hard-to-find internet content. You may get the study article and benchmark at OpenAI's simple evals github repository.
The BrowseComp standard
OpenAI created BrowseComp, an easy-to-verify browsing benchmark that is hard for models. One challenge to evaluating big language models is that they tend to long, open-ended replies. Questions with concise answers and one right answer are its focus. Due to this emphasis on quick replies, BrowseComp performance may not correspond with open-ended user distribution performance. OpenAI accepts this trade-off since brief responses are easier to assess and the benchmark is simple.
OpenAI required human trainers to create fact-based questions with a single, consistent, evidence-based solution, following the SimpleQA standard. BrowseComp trainers created really challenging questions. OpenAI performed three tests to ensure question difficulty:
Existing models couldn't solve the query. Trainers were asked to check that GPT-4o (with and without browsing), o1, and an early deep research model were unable to address the tasks.
Trainers were asked to perform five simple searches and ensure the solution was not on the first few pages of results.
Trainers were told to create activities that someone could not do in 10 minutes. A second trainer attempted to answer certain queries, albeit this was not fully enforced. If trainers performed tasks above 40% of the time, they were asked to amend their assignments.
OpenAI suggested trainers to start with a fact and then create a “inverted” question with a hard-to-find but easy-to-confirm response to create challenging questions Trainers would start with a “seed” (a person, event, or artefact) and explore a large space for qualities to transform into questions. We offered a sample question:
Tell me the title of the 2018–2023 EMNLP conference scientific work. The first and fourth authors attended Dartmouth College and Penn, respectively. Frequency Effects on Transformer Syntactic Rule Learning (EMNLP 2021).
While a few online searches can verify this question, a brute-force search would require looking through hundreds of publications and researching each author's history. Due to their difficulty and reliability, “asymmetry of verification” issues are useful benchmarks.
BrowseComp is simple yet evaluates AI agents' productive browsing:
Models must reason about internet content's authenticity to respond correctly.
Because BrowseComp answers are hard to find, it takes persistence and extensive browsing.
Brute-force methods are too slow or impossible to find many answers. Thus, the model must be creative to get the proper answer in a reasonable period.
BrowseComp helps browsing agents but is incomplete. BrowseComp avoids the challenges of a real user query distribution, such as long replies or ambiguity resolution, but it tests the essential ability of patience and creativity to find information. Models that win programming contests like CodeForces have great coding abilities that may apply to other technical careers, but this is not guaranteed. To solve BrowseComp, the model must be adept in finding hard-to-find information, however this isn't true for all browsing jobs.
Dataset diversity and difficulties
When establishing the BrowseComp benchmark, OpenAI asked trainees to construct questions about their own interests to provide a more interesting experience and higher-quality data.
OpenAI asked human trainees to answer BrowseComp questions to assess dataset difficulty. They were the same trainers who created the questions, but they couldn't answer them. Human trainers were asked to complete the task without an AI assistant ( ChatGPT, Claude, Perplexity, Grok, or Gemini) and were not provided the correct answer.
Certain topics are challenging, thus OpenAI lets trainers identify a subject as unsolvable and progress if they can find a solution within two hours of investigation. As seen here, trainers solved 29.2% of issues and matched the reference answer 86.4% of the time.
OpenAI model performance
OpenAI evaluated GPT-4o, GPT-4.5, OpenAI o1 (medium) models without browsing, GPT-4o with browsing, and Deep Research, a persistent web surfing agent model, on BrowseComp. As seen in the table below, GPT-4o and GPT-4.5 achieved practically 0% accuracy, highlighting the difficulties of the benchmark. BrowseComp addresses exotic, multi-hop facts that models cannot uncover without strong reasoning or tool use.
When browsing was permitted for GPT-4o, accuracy climbed from 0.6% to 1.9%, but performance remained dismal. This shows that surfing alone is insufficient; models must also think strategically, detect relevant search pathways, and analyse retrieved material. This shows that internal knowledge inference can reveal BrowseComp replies. OpenAI o1, which cannot browse but has greater reasoning, is more accurate. These data indicate that reasoning and tool use greatly impact BrowseComp performance.
Deep Research solves half the issues better than any other model. Given its ability to freely explore the web, assess and synthesise data from various sources, and adapt its search technique, it can solve previously unsolvable issues. It excels at answering specialised, non-intuitive questions that require perusing many websites, the type of challenge BrowseComp is meant to assess, by synthesising massive amounts of online information, changing course, and citing each claim.
Compute scaling during testing
OpenAI o1 on AIME and OpenAI o3‑mini low/medium/high show that agent performance scales with inference time computation. Since the queries require iteratively examining many webpages and gathering information, OpenAI should expect more inference-time computation to improve BrowseComp performance.
Aggregation methods that utilise extra computation
OpenAI tried each problem several times and used different strategies to choose the best answer to see if the Deep Research model would perform better with more compute. This experiment aggregated the 64 model outputs per question using best-of-N, weighted voting, and majority voting.
Majority votes for most typical example response.
Weighted voting uses zero-shot triggering the model to calculate a confidence score for each attempt and voting accordingly.
Best-of-N picks the most reliable.
Conclusions
BrowseComp tests a model's online search for hard-to-find information. BrowseComp measures the ability to find a single piece of information, is straightforward to examine, and challenges existing browsing agents, even if it does not measure frequent searches. By releasing BrowseComp open-source, OpenAI hopes to spur research into more reliable AI.