Build Browser Agents with Gemini + Playwright MCP

In many real-world scenarios, websites and applications do not offer APIs for programmatic access.

This is where Browser Agents become especially useful: they can interact with web pages just like a human would, by clicking buttons, filling out forms, scrolling, and extracting content.

In this notebook, we’ll explore how to build Browser Agents that can perform various tasks, mostly focused on information gathering, and even extending to image generation.

🧰 Our stack:

• Haystack Agentic framework
• Google Gemini 2.5 Flash, a capable model with generous free tier
• Playwright MCP server, which offers the browser automation tools

Setup

First, we’ll install all the necessary dependencies:

• Haystack integrations: MCP and Google GenAI
• Playwright dependencies: these are needed to successfully run it on Colab

! uv pip install mcp-haystack google-genai-haystack gdown

! npm i -D @playwright/test && npx playwright install --with-deps chrome

To get a free Gemini API key, visit Google AI Studio.

from getpass import getpass
import os

os.environ["GEMINI_API_KEY"] = getpass("Enter your Gemini API Key: ")

Start Playwright MCP server and create a Toolset

We first start the Playwright MCP server, which will give us the tools needed for Web navigation and interactions.

Note that we prepend nohup to run the server in Colab in a non-blocking way, and also some other options are required for it to work correctly in a Colab environment.

! nohup npx @playwright/mcp@latest  --headless --isolated --no-sandbox --port 8931 > playwright.log &

nohup: redirecting stderr to stdout

After a few moments, if the server started correctly, we should see a successful log message.

! cat playwright.log

Listening on http://localhost:8931
Put this in your client config:
{
  "mcpServers": {
    "playwright": {
      "url": "http://localhost:8931/mcp"
    }
  }
}
For legacy SSE transport support, you can use the /sse endpoint instead.

We then create a MCPToolset connected to the server we started.

📌 When working with tools for LLMs, it’s a good practice to only select the ones you actually need. This helps avoid confusing the LLM with too many options.

For that reason, we’ll start with just a single tool: browser_navigate, which navigates to a given URL.

You can find the full list of tools available in the Playwright MCP server documentation.

from haystack_integrations.tools.mcp import MCPToolset, StreamableHttpServerInfo

server_info = StreamableHttpServerInfo(url="http://localhost:8931/mcp")

toolset = MCPToolset(server_info=server_info, tool_names=["browser_navigate"])

print(toolset.tools)

[Tool(name='browser_navigate', description='Navigate to a URL', parameters={'type': 'object', 'properties': {'url': {'type': 'string', 'description': 'The URL to navigate to'}}, 'required': ['url'], 'additionalProperties': False, '$schema': 'http://json-schema.org/draft-07/schema#'}, function=<function MCPToolset.__init__.<locals>.create_invoke_tool.<locals>.invoke_tool at 0x79b2e079a840>, outputs_to_string=None, inputs_from_state=None, outputs_to_state=None)]

Our first Browser Agent with URL navigation

We can now build the Browser Agent, using the Haystack Agent component.

We need to specify:

• A Chat Generator, the LLM (gemini-2.5-flash).
• A good system message. This model is powerful, but it can sometimes be overly cautious and refuse to perform tasks it is capable of, so we have included a system message to encourage it.
• Tools: we pass the Toolset defined earlier.

We also specify some optional parameters:

• exit_conditions=["text"]. The Agent will exit as soon as the LLM replies only with a text response and no tool calls.
• streaming_callback=print_streaming_chunk. Using the utility function, text and Tool Calls coming from the LLM will be streamed, along with Tool Call Results from Tools.

from haystack.components.agents import Agent
from haystack.components.generators.utils import print_streaming_chunk
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.google_genai import GoogleGenAIChatGenerator

chat_generator = GoogleGenAIChatGenerator(
    model="gemini-2.5-flash",
)

system_message = """
You are an intelligent assistant equipped with tools for navigating the web.

You can use tools when appropriate, but not every task requires them — you also have strong reasoning and language capabilities.
If a request seems challenging, don't default to refusal due to perceived tool limitations. Instead, think creatively and attempt a solution using the skills you do have.
You are more capable than you might assume. Trust your abilities.
"""

agent = Agent(
        chat_generator=chat_generator,
        tools=toolset,
        system_prompt=system_message,
        exit_conditions=["text"],
        streaming_callback=print_streaming_chunk
)

📰 Find US-related news

Let’s start with a simple task: navigate to a website and filter US-related news.

messages = [
    ChatMessage.from_user("navigate to https://www.theguardian.com/world and list all US-related news")
]
result = agent.run(messages=messages)

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://www.theguardian.com/world"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://www.theguardian.com/world\nawait page.goto('https://www.theguardian.com/world');\n```\n\n### New c...

[ASSISTANT]
Here are some US-related news articles from The Guardian:

**Africa Section:**
*   US destruction of contraceptives denies 1.4m African women and girls lifesaving care, NGO says
*   US hunter reportedly killed by buffalo during expedition in South Africa
*   Rwanda agrees to take up to 250 migrants from the US

**Americas Section:**
*   Dozens more countries face higher levies on exports to US as new Trump tariffs come into effect
*   US Coast Guard releases report calling Titan disaster a ‘preventable tragedy’
*   Canada wildfires: 81 million Americans under air quality alerts as blazes rage

**Asia Pacific Section:**
*   US research station staff evacuated from Antarctica in high-risk operation

**Europe Section:**
*   ‘No clear public response from Russia’ on plans to end war, Zelenskyy says after Macron call – Europe live
*   Kremlin denies Putin agreed to trilateral meeting with Trump and Zelenskyy
*   Donald Trump says ‘everyone agrees this war must come to a close’ after Witkoff meets Putin – as it happened

**Middle East Section:**
*   Médecins Sans Frontières calls for immediate closure of US and Israeli backed aid group in Gaza – Middle East crisis live
*   Democrats sound alarm over Trump’s reported $1bn plan to retrofit gifted Qatari jet
*   US demands for Hezbollah disarmament may force Lebanon into dangerous choice

**South and Central Asia Section:**
*   Trump’s demand that India stop buying Russian oil puts Modi in tight spot
*   India to still buy oil from Russia despite Trump threats, say officials

**US Section:**
*   US treasury secretary praises ‘manufacturing renaissance’ as Trump tariffs snap into effect – live updates
*   Chris Cuomo mocked for response after falling for deepfake AOC video
*   ‘It’s not a coincidence’: after Ghislaine Maxwell’s prison move … what next?
*   Alleged killer of Minnesota politician and her husband expected to plead not guilty in court
*   James Cameron warns of ‘Terminator-style apocalypse’ if AI weaponised

Nice.

📅 Who was born on this day?

Another simple task. Let’s see how our agent behaves.

messages = [
    ChatMessage.from_user("I was born on October 17th. Find five notable people born on the same day using Wikipedia.")
]
result = agent.run(messages=messages)

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://en.wikipedia.org/wiki/October_17"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://en.wikipedia.org/wiki/October_17\nawait page.goto('https://en.wikipedia.org/wiki/October_17');\n``...

[ASSISTANT]
Here are five notable people born on October 17th:

1.  **Pope John Paul I** (1912) - Head of the Catholic Church and sovereign of the Vatican City State from August 26, 1978, until his death in September of the same year.
2.  **Arthur Miller** (1915) - American playwright and screenwriter, known for plays like *Death of a Salesman* and *The Crucible*.
3.  **Rita Hayworth** (1918) - American actress, dancer, and producer, a major glamour icon in the 1940s.
4.  **Evel Knievel** (1938) - American daredevil and entertainer, famous for his motorcycle stunts.
5.  **Eminem** (1972) - American rapper, songwriter, and record producer, widely considered one of the greatest and most influential hip-hop artists.

Browser Agent with URL navigation + accessibility snapshot

Sometimes the raw version of a web page is not easily interpretable or misses relevant information. In these cases, a textual accessibility snapshot can help the LLM.

According to the Playwright MCP server maintainers, this approach often works better than providing screenshots.

Let’s rebuild our agent.

toolset = MCPToolset(server_info=server_info, tool_names=["browser_navigate", "browser_snapshot"])

agent = Agent(
        chat_generator=chat_generator,
        tools=toolset,
        system_prompt=system_message,
        exit_conditions=["text"],
        streaming_callback=print_streaming_chunk
)

🗞️🗞️ Compare online news websites

Let’s now use our agent to find similar news stories appearing on different web sites.

messages = [ChatMessage.from_user("""
1. Visit www.nbcnews.com
2. Visit www.bbc.com
3. Identify news stories that appear on both sites.
4. Your final response must contain only a Markdown table listing the shared news stories and their respective links from each site.
""")
]
result = agent.run(messages=messages)

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "www.nbcnews.com"}



ERROR:haystack.components.tools.tool_invoker:Failed to invoke Tool `browser_navigate` with parameters {'url': 'www.nbcnews.com'}. Error: Failed to invoke tool 'browser_navigate' with args: {'url': 'www.nbcnews.com'}, got error: Tool 'browser_navigate' returned an error: [TextContent(type='text', text='Error: page.goto: Protocol error (Page.navigate): Cannot navigate to invalid URL\nCall log:\n\x1b[2m  - navigating to "www.nbcnews.com", waiting until "domcontentloaded"\x1b[22m\n', annotations=None, meta=None)]




[ASSISTANT]
I apologize, but I encountered an error when trying to navigate to www.nbcnews.com. The error indicates an invalid URL. I will try again using the full URL including "https://".



[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://www.nbcnews.com"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://www.nbcnews.com\nawait page.goto('https://www.nbcnews.com');\n```\n\n### New console messages\n- [...

[ASSISTANT]
I have successfully visited NBCNews.com. Here's a summary of the main news stories and their links:

**NBC News Stories:**

*   **"New tariffs snap into effect, putting Trump’s global trade reset into high gear"**: https://www.nbcnews.com/business/business-news/trump-tariffs-latest-round-takes-effect-thursday-august-7-2025-rcna223461
*   **"Trump demands resignation of Intel CEO over alleged China ties"**: https://www.nbcnews.com/business/business-news/trump-demands-intel-ceo-resign-rcna223594
*   **"Trump hikes India tariffs to 50% for buying Russian oil"**: https://www.nbcnews.com/business/business-news/trump-hikes-india-tariffs-50-percent-buying-russian-oil-rcna223374
*   **"Kremlin says Putin-Trump meeting will happen in 'coming days'"**: https://www.nbcnews.com/world/russia/russia-putin-trump-meeting-confirmed-ukraine-rcna223572
*   **"Fort Stewart suspect shot five coworkers with personal handgun"**: https://www.nbcnews.com/news/us-news/quornelius-radford-know-fort-stewart-shooting-suspect-rcna223529
*   **"Decade of Supreme Court rulings have given states more unfettered power in redistricting"**: https://www.nbcnews.com/politics/supreme-court/supreme-court-rulings-states-unfettered-power-redistricting-rcna223146
*   **"Scientists want to prevent AI from going rogue by teaching it to be bad first"**: https://www.nbcnews.com/tech/tech-news/ai-anthropic-researchers-predicting-dangerous-behavior-rcna223236
*   **"Surgeon says UnitedHealthcare dispute may force her into bankruptcy"**: https://www.nbcnews.com/health/health-news/texas-surgeon-says-unitedhealthcare-dispute-may-force-bankruptcy-rcna223519
*   **"New research reveals clues about memory and longevity from the brains of 'SuperAgers'"**: https://www.nbcnews.com/health/aging/superaging-memory-social-aging-northwestern-university-study-over-80-rcna222908
*   **"Ultra-processed foods make up the majority of kids' diet, CDC finds"**: https://www.nbcnews.com/health/health-news/ultra-processed-foods-make-majority-kids-diet-cdc-report-finds-rcna223481
*   **"A salty twist: Diabetes risk study says french fries are a culprit"**: https://www.nbcnews.com/health/health-news/french-fries-diabetes-risk-type-2-study-fast-food-potatoes-rcna222961
*   **"Florida woman arrested after allegedly posing as nurse and treating patients with stolen license"**: https://www.nbcnews.com/news/us-news/florida-woman-arrested-allegedly-posing-nurse-treating-4000-patients-s-rcna223580
*   **"New photos show scale of destruction in Gaza as Israel weighs expanded military operation"**: https://www.nbcnews.com/world/gaza/gaza-israel-photos-destruction-expand-war-military-operation-rcna223575
*   **"Red Sox rookie who has just 159 at-bats signs $130 million contract"**: https://www.nbcnews.com/sports/mlb/red-sox-roman-anthony-130-million-contract-rcna223591
*   **"Man accused of killing Minnesota lawmaker to appear in court on murder charges"**: https://www.nbcnews.com/news/us-news/man-accused-killing-minnesota-lawmaker-appear-court-murder-charges-rcna223576
*   **"Man accused in D.C. Jewish museum killings indicted on federal hate crime charges"**: https://www.nbcnews.com/news/us-news/man-accused-killing-israeli-embassy-staffers-indicted-federal-hate-cri-rcna223534
*   **"Assemblies of God leaders address sex abuse of boys and young men in Texas sauna"**: https://www.nbcnews.com/news/us-news/assemblies-god-leaders-sex-abuse-scandal-chi-alpha-rcna223510
*   **"United Airlines outage that caused ground stops across the U.S. is resolved"**: https://www.nbcnews..com/news/us-news/united-airlines-outage-prompts-ground-stop-major-us-hubs-rcna223550
*   **"Trump threatens to take over D.C. police and says he might deploy the National Guard"**: https://www.nbcnews.com/politics/trump-administration/trump-threatens-dc-police-takeover-might-deploy-national-guard-rcna223535
*   **"UCLA says Trump administration has suspended $584 million in grants"**: https://www.nbcnews.com/news/us-news/ucla-says-trump-administration-suspended-584-million-grants-rcna223531
*   **"Congress doesn't want to talk to Alex Acosta, Jeffrey Epstein's 'sweetheart deal' maker"**: https://www.nbcnews.com/politics/congress/congress-doesnt-want-talk-alex-acosta-epsteins-sweetheart-deal-maker-rcna223543
*   **"Tulsi Gabbard declassified documents over CIA objections, sources say"**: https://www.nbcnews.com/politics/politics-news/tulsi-gabbard-declassified-documents-objections-cia-sources-say-rcna223548
*   **"Sen. Marsha Blackburn announces she's running for governor in Tennessee"**: https://www.nbcnews.com/politics/politics-news/sen-marsha-blackburn-announces-running-governor-tennessee-rcna223338
*   **"MLB is calling up its first female umpire"**: https://www.nbcnews.com/sports/mlb/mlb-first-female-umpire-jen-pawol-rcna223446
*   **"U.S. Open singles winners will get a record $5 million each"**: https://www.nbcnews.com/sports/tennis/us-open-singles-winners-5-million-rcna223442
*   **"Seattle Storm acquire All-Star guard Brittney Sykes in trade with Washington Mystics"**: https://www.nbcsports.com/wnba/news/seattle-storm-acquire-all-star-guard-brittney-sykes-in-trade-with-washington-mystics?cid=nbcnews
*   **"Canadian teenager Mboko reaches semifinals in Montreal and will face Rybakina"**: https://www.nbcsports.com/tennis/news/canadian-teenager-mboko-reaches-semifinals-in-montreal-and-will-face-rybakina?cid=nbcnews
*   **"Drake Maye on Mike Vrabel breaking up fight: We're trying to build that intensity"**: https://www.nbcsports.com/nfl/profootballtalk/rumor-mill/news/drake-maye-on-mike-vrabel-breaking-up-fight-were-trying-to-build-that-intensity?cid=nbcnews
*   **"It takes a village"**: https://www.nbcnews.com/news/nbcblk/trump-supporting-ohio-community-beloved-haitian-immigrant-faces-deport-rcna221579
*   **"First class (lounge)"**: https://www.nbcnews.com/business/travel/airport-vip-perks-expand-rcna221997
*   **"80 years later"**: https://www.nbcnews.com/world/asia/80-years-later-hiroshima-bombing-survivors-warn-new-nuclear-warfare-rcna220177
*   **"'Walking Dead' actor Kelley Mack dies at 33"**: https://www.nbcnews.com/pop-culture/pop-culture-news/walking-dead-actress-kelley-mack-dies-33-rcna223329
*   **"Some immigrants seeking green cards may be placed in removal proceedings, USCIS says"**: https://www.nbcnews.com/news/us-news/green-card-applicants-immigrants-removal-proceedings-uscis-rcna223099
*   **"France's biggest wildfire of the summer has eclipsed the size of Paris and is still spreading"**: https://www.nbcnews.com/world/europe/frances-biggest-wildfire-summer-eclipsed-size-paris-still-spreading-rcna223559
*   **"Myanmar’s U.S.-sanctioned acting President Myint Swe dies after long illness"**: https://www.nbcnews.com/world/myanmar/myanmars-us-sanctioned-acting-president-myint-swe-dies-long-illness-rcna223568
*   **"Eddie Palmieri, pioneering Latin jazz musician, dies at 88"**: https://www.nbcnews.com/news/us-news/eddie-palmieri-pioneering-latin-jazz-musician-dies-88-rcna223561
*   **"Kelly Clarkson says her ex-husband has been ill, postpones August concerts in Las Vegas"**: https://www.nbcnews.com/pop-culture/pop-culture-news/kelly-clarkson-says-ex-husband-ill-postpones-august-concerts-las-vegas-rcna223563
*   **"Wednesday Addams is back at Nevermore after an ominous season 1 cliffhanger"**: https://www.nbcnews.com/pop-culture/pop-culture-news/wednesday-season-1-recap-everything-remember-starting-season-2-rcna223367
*   **"Apple, under pressure from Trump, says it's boosting U.S. investment"**: https://www.nbcnews.com/business/business-news/apple-trump-pressure-us-investment-rcna223444
*   **"Claire's, known for piercing millions of teens' ears, files for Chapter 11"**: https://www.nbcnews.com/business/business-news/claires-known-piercing-millions-teens-ears-files-chapter-11-2nd-time-2-rcna223447
*   **"Student loan borrowers face new challenges trying to get Public Service Loan Forgiveness"**: https://www.nbcnews.com/business/personal-finance/student-loan-borrowers-face-new-challenges-trying-get-public-service-l-rcna223448
*   **"Wall Street gains as oil ends lower in volatile trade"**: https://www.nbcnews.com/business/markets/wall-street-gains-oil-ends-lower-volatile-trade-rcna223482

Now I will navigate to www.bbc.com.

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://www.bbc.com"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://www.bbc.com\nawait page.goto('https://www.bbc.com');\n```\n\n### New console messages\n- [ERROR] F...

[ASSISTANT]
| News Story | NBC News Link | BBC News Link |
|---|---|---|
| Trump tariffs / trade with India/China | https://www.nbcnews.com/business/business-news/trump-tariffs-latest-round-takes-effect-thursday-august-7-2025-rcna223461 <br> https://www.nbcnews.com/business/business-news/trump-demands-intel-ceo-resign-rcna223594 <br> https://www.nbcnews.com/business/business-news/trump-hikes-india-tariffs-50-percent-buying-russian-oil-rcna223374 | https://www.bbc.com/news/articles/c1w83j35jjjo <br> https://www.bbc.com/news/articles/c1dxr1g4y7yo <br> https://www.bbc.com/news/articles/cx23jmvn5yzo <br> https://www.bbc.com/news/articles/cly3r664q1eo <br> https://www.bbc.com/news/articles/cdx0n7y29kdo |
| Putin-Trump meeting | https://www.nbcnews.com/world/russia/russia-putin-trump-meeting-confirmed-ukraine-rcna223572 | https://www.bbc.com/news/articles/c14gkkzvpx8o <br> https://www.bbc.com/news/articles/cr5rdl1y8ndo |
| Gaza/Israel destruction/military operation | https://www.nbcnews.com/world/gaza/gaza-israel-photos-destruction-expand-war-military-operation-rcna223575 | https://www.bbc.com/news/articles/c8ryekj1m1do |
| Florida nurse arrest/impersonation | https://www.nbcnews.com/news/us-news/florida-woman-arrested-allegedly-posing-nurse-treating-4000-patients-s-rcna223580 | https://www.bbc.com/news/articles/clyrg2z350vo |
| United Airlines outage/delays | https://www.nbcnews.com/news/us-news/united-airlines-outage-prompts-ground-stop-major-us-hubs-rcna223550 | https://www.bbc.com/news/articles/cx29q585g3go |
| Apple investment/pressure from Trump | https://www.nbcnews.com/business/business-news/apple-trump-pressure-us-investment-rcna223444 | https://www.bbc.com/news/articles/cp8zyyygxv7o <br> https://www.bbc.com/news/articles/cdx0n7y29kdo |
| France wildfires | https://www.nbcnews.com/world/europe/frances-biggest-wildfire-summer-eclipsed-size-paris-still-spreading-rcna223559 | https://www.bbc.com/news/articles/cm2vk4xl204o <br> https://www.bbc.com/news/videos/cqxg47ggj29o <br> https://www.bbc.com/news/articles/cj6y803pjkwo |
| Myanmar's acting President Myint Swe dies | https://www.nbcnews.com/world/myanmar/myanmars-us-sanctioned-acting-president-myint-swe-dies-long-illness-rcna223568 | https://www.bbc.com/news/articles/c78m0d29rvno |
| 80th anniversary of Hiroshima atomic bombing | https://www.nbcnews.com/world/asia/80-years-later-hiroshima-bombing-survivors-warn-new-nuclear-warfare-rcna220177 | https://www.bbc.com/reel/video/p0lvdqmt/what-happened-at-hiroshima- <br> https://www.bbc.com/news/videos/cwye6k0xn7zo |

from IPython.display import display, Markdown

display(Markdown(result["last_message"].text))

Definitely not bad!

If you look at streamed tool calls, you can also notice that the agent is sometimes able to correct its own actions leading to errors.

👨🏻‍💻 Find information about a GitHub contributor

This example is not original: I took it from a Hugging Face tutorial, but it’s nice.

messages = [ChatMessage.from_user("""
I'm trying to find how hard I have to work to get a repo in github.com/trending.
Can you navigate to the profile for the top author of the top trending repo,
and give me their total number of contributions over the last year?
""")
]

result = agent.run(messages=messages)

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://github.com/trending"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://github.com/trending\nawait page.goto('https://github.com/trending');\n```\n\n### Page state\n- Pag...

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://github.com/cjdsellers"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://github.com/cjdsellers\nawait page.goto('https://github.com/cjdsellers');\n```\n\n### Page state\n-...

[ASSISTANT]
The top trending repository is `nautechsystems/nautilus_trader`, and the top author is `cjdsellers`. They have made 4,203 contributions in the last year.

More tools for more advanced Browser Agents!

To unlock advanced use cases, we need more tools: the ability to click, type something, navigate back and wait.

toolset = MCPToolset(server_info=server_info,
                     tool_names=["browser_navigate", "browser_snapshot",
                                "browser_click", "browser_type",
                                 "browser_navigate_back","browser_wait_for",
                                ])

agent = Agent(
        chat_generator=chat_generator,
        tools=toolset,
        system_prompt=system_message,
        exit_conditions=["text"],
        streaming_callback=print_streaming_chunk
)

🖱️ Find a product’s price range on Amazon

I want to buy a new mouse. Let’s use our agent to discover price ranges on Amazon for this product.

prompt="""
1. Go to Amazon Italy and find all available prices for the Logitech MX Master 3S mouse.
2. Exclude any items that:
- are not computer mice;
- are not the exact Logitech MX Master 3S model;
- do not display a price;
- are bundled with other products or accessories.
3. Your final message must only contain a Markdown table with two columns: Name and Price.
"""

result = agent.run(messages=[ChatMessage.from_user(prompt)])

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://www.amazon.it"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://www.amazon.it\nawait page.goto('https://www.amazon.it');\n```\n\n### Page state\n- Page URL: https...

[TOOL CALL]
Tool: browser_type 
Arguments: {"submit": true, "element": "Search Amazon.it", "ref": "e141", "text": "Logitech MX Master 3S"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Fill \"Logitech MX Master 3S\" into \"Search Amazon.it\"\nawait page.getByRole('searchbox', { name: 'Search Amazon.it'...

[ASSISTANT]
Here are the Logitech MX Master 3S mice available on Amazon Italy with their prices:

| Name | Price |
|---|---|
| MX Master 3S - Mouse Wireless ad Alte Prestazioni, Scorrimento Ultraveloce, Ergo, 8K DPI, Tracciamento su Vetro, Clic Silenziosi, USB C, Bluetooth, Windows, Linux, Chrome - Grigio scuro | €68.53 |
| MX Master 3S - High Performance Wireless Mouse, Ultra Fast Scrolling, Ergo, 8K DPI, Glass Tracking, Silent Clicks, USB C, Bluetooth, Windows, Linux, Chrome - Light Grey | €71.16 |
| MX Master 3S for Mac - Wireless Bluetooth Mouse with Ultra Fast Scrolling, Ergonomic, 8000 DPI, Glass Tracking, Silent Clicks, USB-C, Apple, iPad - Space Grey | €65.08 |
| MX Master 3S for Business, Wireless Mouse with Silent Click, 8K DPI, USB Receiver with Logi Bolt Protected, Bluetooth, USB-C, MagSpeed, Windows/Mac/Chrome/Linux - Grey | €79.82 |
| MX Master 3S for Mac - Ultra Fast Scrolling Bluetooth Mouse, Ergonomic 8K DPI, Glass Tracking, Silent Clicks, USB-C, Apple, iPad - Light Grey | €74.02 |

from IPython.display import display, Markdown

display(Markdown(result["last_message"].text))

Here are the Logitech MX Master 3S mice available on Amazon Italy with their prices:

🖥️ GitHub exploration

prompt = "List some recent PRs merged by anakin87 on the deepset-ai/haystack repo on GitHub. Max 5."
result = agent.run(messages=[ChatMessage.from_user(prompt)])

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://github.com/deepset-ai/haystack/pulls"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://github.com/deepset-ai/haystack/pulls\nawait page.goto('https://github.com/deepset-ai/haystack/pull...

[TOOL CALL]
Tool: browser_type 
Arguments: {"element": "textbox \"Search all issues\"", "submit": true, "ref": "e115", "text": "is:pr author:anakin87 is:merged"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Fill \"is:pr author:anakin87 is:merged\" into \"textbox \"Search all issues\"\"\nawait page.getByRole('textbox', { nam...

[ASSISTANT]
Here are 5 recent PRs merged by anakin87 on the deepset-ai/haystack repo:

1.  **#9690**: chore: remove unused `type:ignore` and `cast` (merged Aug 7, 2025)
2.  **#9682**: fix: `ChatMessage` bugfixes (merged Aug 6, 2025)
3.  **#9679**: feat: add serde methods to `ImageContent` and `TextContent` (merged Aug 5, 2025)
4.  **#9662**: chore: re-export missing symbols in `__init__.py` files (merged Jul 30, 2025)
5.  **#9659**: test: fix Datadog tests and unpin `ddtrace` test dependency (merged Jul 30, 2025)

▶️ Find content creator social profiles from YouTube

Starting from a YouTube video, look for the social profiles of a creator. Since sometimes search engines block automation tools, we ask the model to find intelligent alternative ways to look for information.

prompt="""
1. Open this YouTube video: https://www.youtube.com/watch?v=axmaslLO4-4 and extract the channel author’s username.
2. Then, try to find all their social media profiles by searching the web using the username.
If you cannot perform a web search, try other available methods to find the profiles.
3. Return only the links to their social media accounts along with the platform names.
"""

result = agent.run(messages=[ChatMessage.from_user(prompt)])

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://www.youtube.com/watch?v=axmaslLO4-4"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://www.youtube.com/watch?v=axmaslLO4-4\nawait page.goto('https://www.youtube.com/watch?v=axmaslLO4-4'...

[TOOL CALL]
Tool: browser_snapshot 
Arguments: 

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### New console messages\n- [INFO] LegacyDataMixin will be applied to all legacy elements.\nSet `_legacyUndefinedCheck: true` on ...\n- [WARNING] [GroupM...

[TOOL CALL]
Tool: browser_snapshot 
Arguments: 

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### New console messages\n- [INFO] LegacyDataMixin will be applied to all legacy elements.\nSet `_legacyUndefinedCheck: true` on ...\n- [WARNING] [GroupM...

[TOOL CALL]
Tool: browser_click 
Arguments: {"element": "Mr. RIP channel link", "ref": "e220"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Click Mr. RIP channel link\nawait page.getByRole('link', { name: 'Mr. RIP' }).click();\n```\n\n### New console message...

[ASSISTANT]
The channel author's username is "Mr. RIP".


[TOOL CALL]
Tool: browser_snapshot 
Arguments: 

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### New console messages\n- [INFO] LegacyDataMixin will be applied to all legacy elements.\nSet `_legacyUndefinedCheck: true` on ...\n- [WARNING] [GroupM...

[TOOL CALL]
Tool: browser_click 
Arguments: {"element": "and 4 more links", "ref": "e1199"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Click and 4 more links\nawait page.getByRole('button', { name: 'and 4 more links' }).click();\n```\n\n### New console ...

[ASSISTANT]
Here are the social media profiles for Mr. RIP:

*   **Twitch:** https://www.twitch.tv/retireinprogress
*   **Blog:** https://retireinprogress.com/
*   **Telegram:** https://t.me/newsrip
*   **Discord:** mr.rip/discord

🗺️ Use Google Maps to find a location

result = agent.run(messages=[ChatMessage.from_user("Use Google Maps to find the deepset HQ in Berlin")])

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://www.google.com/maps"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://www.google.com/maps\nawait page.goto('https://www.google.com/maps');\n```\n\n### New console messa...

[TOOL CALL]
Tool: browser_type 
Arguments: {"ref": "e35", "element": "Search Google Maps", "text": "deepset HQ Berlin", "submit": true}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Fill \"deepset HQ Berlin\" into \"Search Google Maps\"\nawait page.getByRole('combobox', { name: 'Search Google Maps' ...

[TOOL CALL]
Tool: browser_click 
Arguments: {"element": "deepset HQ Berlin", "ref": "e182"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Click deepset HQ Berlin\nawait page.locator('#searchboxinput').click();\n```\n\n### New console messages\n- [WARNING] ...

[ASSISTANT]
The deepset HQ in Berlin is located at Zinnowitzer Str. 1, 10115 Berlin, Germany.

🚂 🚌 Find public transportation travel options

Using Google Maps, let’s ask a harder question, which requires more navigation.

prompt = """
1. Using Google Maps, find the next 3 available public transportation travel options from Paris to Berlin departing today.
2. For each option, provide a detailed description of the route (e.g., transfers, stations, duration).
3. Include a direct link to the corresponding Google Maps route for each travel option.
"""

result = agent.run(messages=[ChatMessage.from_user(prompt)])

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://www.google.com/maps"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://www.google.com/maps\nawait page.goto('https://www.google.com/maps');\n```\n\n### New console messa...

[TOOL CALL]
Tool: browser_type 
Arguments: {"ref": "e37", "element": "Search Google Maps combobox", "text": "Paris to Berlin"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Fill \"Paris to Berlin\" into \"Search Google Maps combobox\"\nawait page.getByRole('combobox', { name: 'Search Google...

[TOOL CALL]
Tool: browser_click 
Arguments: {"ref": "e39", "element": "Search button"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Click Search button\nawait page.getByRole('button', { name: 'Search' }).click();\n```\n\n### New console messages\n- [...

[TOOL CALL]
Tool: browser_click 
Arguments: {"element": "Directions button", "ref": "e42"}



ERROR:haystack.components.tools.tool_invoker:Failed to invoke Tool `browser_click` with parameters {'element': 'Directions button', 'ref': 'e42'}. Error: Failed to invoke tool 'browser_click' with args: {'element': 'Directions button', 'ref': 'e42'}, got error: Tool 'browser_click' returned an error: [TextContent(type='text', text='Error: Ref e42 not found in the current page snapshot. Try capturing new snapshot.', annotations=None, meta=None)]




[TOOL CALL]
Tool: browser_snapshot 
Arguments: 

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### New console messages\n- [WARNING] [GroupMarkerNotSet(crbug.com/242999)!:A0702B00AC0E0000]Automatic fallback to software WebG...\n- [ERROR] Failed to ...

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://www.google.com/maps/dir/Paris/Berlin/data=!4m2!4m1!3e3"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://www.google.com/maps/dir/Paris/Berlin/data=!4m2!4m1!3e3\nawait page.goto('https://www.google.com/ma...

[ASSISTANT]
Here are the next 3 available public transportation travel options from Paris to Berlin departing today:

---

**Option 1:**

*   **Duration:** 10 hours 55 minutes
*   **Departure/Arrival:** 5:10 PM from Gare de l'Est (Paris) – 4:05 AM (Friday) in Berlin
*   **Route Description:**
    *   Train ICE9555 from Gare de l'Est (Paris).
    *   Walk for 4 minutes.
    *   Bus FlixBus.
*   **Google Maps Link:** https://www.google.com/maps/dir/Paris/Berlin/data=!4m2!4m1!3e3

---

**Option 2:**

*   **Duration:** 11 hours 11 minutes
*   **Departure/Arrival:** 6:25 PM from Paris – 5:36 AM (Friday) in Berlin
*   **Route Description:**
    *   High-speed train Eurostar.
    *   Transfer to High-speed train ICE319.
    *   Transfer to High-speed train ICE100.
*   **Google Maps Link:** https://www.google.com/maps/dir/Paris/Berlin/data=!4m2!4m1!3e3

---

**Option 3:**

*   **Duration:** 10 hours 41 minutes
*   **Departure/Arrival:** 6:55 PM from Paris – 5:36 AM (Friday) in Berlin
*   **Route Description:**
    *   High-speed train Eurostar.
    *   Transfer to High-speed train ICE100.
*   **Google Maps Link:** https://www.google.com/maps/dir/Paris/Berlin/data=!4m2!4m1!3e3

---

Please note that Google Maps sometimes shows general options first, and clicking into them provides more granular details. The provided links are for the general Paris to Berlin transit route, on which these specific options are displayed.

Ready to go!

🖼️ 🤗 Image generation via Hugging Face spaces

Let’s now try something more advanced: access a Hugging Face space to generate an image; note that our agent is also asked to create a good prompt for the image generation model.

prompt="""
1. Visit Hugging Face Spaces, search the Space named exactly "FLUX.1 [schnell]" and enter it.
2. Craft a detailed, descriptive prompt using your language skills to depict: "my holiday on Lake Como".
3. Use this prompt to generate an image within the Space.
4. After prompting, wait 5 seconds to allow the image to fully generate. Repeat until the image is generated.
5. Your final response must contain only the direct link to the generated image — no additional text.
"""

result = agent.run(messages=[ChatMessage.from_user(prompt)])

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://huggingface.co/spaces/flux.1/schnell"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://huggingface.co/spaces/flux.1/schnell\nawait page.goto('https://huggingface.co/spaces/flux.1/schnel...

[TOOL CALL]
Tool: browser_navigate 
Arguments: {"url": "https://huggingface.co/spaces"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Navigate to https://huggingface.co/spaces\nawait page.goto('https://huggingface.co/spaces');\n```\n\n### Page state\n-...

[TOOL CALL]
Tool: browser_type 
Arguments: {"element": "Search models, datasets, users...", "text": "flux.1/schnell", "ref": "e11"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Fill \"flux.1/schnell\" into \"Search models, datasets, users...\"\nawait page.getByRole('textbox', { name: 'Search mo...

[TOOL CALL]
Tool: browser_click 
Arguments: {"element": "img", "ref": "e12"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Click img\nawait page.locator('.absolute').first().click();\n```\n\n### Page state\n- Page URL: https://huggingface.co...

[TOOL CALL]
Tool: browser_click 
Arguments: {"ref": "e1115", "element": "\ud83c\udfce\ufe0f\ud83d\udca8 black-forest-labs/FLUX.1-schnell"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Click 🏎️💨 black-forest-labs/FLUX.1-schnell\nawait page.getByRole('link', { name: '🏎️💨 black-forest-labs/FLUX.' }).clic...

[TOOL CALL]
Tool: browser_type 
Arguments: {"text": "A serene, sun-drenched afternoon on Lake Como. A classic Riva speedboat gently glides across the azure water, leaving a soft wake. Elegant villas with terracotta roofs and lush gardens dot the shoreline, backed by rolling green hills. In the distance, the majestic Italian Alps stand guard under a clear, cerulean sky. The air is warm and filled with the scent of blooming flowers and fresh lake water. A sense of tranquil luxury pervades the scene.", "ref": "f1e23", "element": "Prompt"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Fill \"A serene, sun-drenched afternoon on Lake Como. A classic Riva speedboat gently glides across the azure water, l...

[TOOL CALL]
Tool: browser_click 
Arguments: {"ref": "f1e24", "element": "Run"}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Ran Playwright code\n```js\n// Click Run\nawait page.locator('#iFrameResizer0').contentFrame().getByRole('button', { name: 'Run' }).click();\n```\n\n...

[TOOL CALL]
Tool: browser_wait_for 
Arguments: {"time": 5}

[TOOL RESULT]
{"meta":null,"content":[{"type":"text","text":"### Result\nWaited for 5\n\n### New console messages\n- [WARNING] Unrecognized feature: 'ambient-light-sensor'. @ https://huggingface.co/spaces/black-fo...

[ASSISTANT]
https://black-forest-labs-flux-1-schnell.hf.space/file=/tmp/gradio/3a1d55dec7f02816a136f855fafe9c754364d14f/image.webp

from PIL import Image
import requests
from io import BytesIO
from IPython.display import display

response = requests.get(result["last_message"].text)
image = Image.open(BytesIO(response.content))
display(image)

🏞️ Impressive!

The video below demonstrates a similar local setup. It’s fascinating to watch the Agent navigate the web similarly to a human.

import gdown
from ipywidgets import Video

url = "https://drive.google.com/drive/folders/1HyPdNxpzi3svmPVYGXak7mqAjVWztwsH"

gdown.download_folder(url, quiet=True, output=".")

Video.from_file('/content/browser_agent.mp4', autoplay=False, loop=False)

Video(value=b'\x00\x00\x00 ftypisom\x00\x00\x02\x00isomiso2avc1mp41\x00\x00\x00\x08free\x01SU<mdat\x00\x00\x00…

What’s next?

When I first started exploring Browser Agents, I expected to hit roadblocks quickly. But with the right framework, a solid LLM, and the Playwright MCP server (even without screenshot capabilities), you can actually get quite far. This notebook shows some fun and sometimes impressive demos that emerged from that process.

We also want to explore how to move these agents from notebooks to production: deploying in Docker environment, exposing via an API, and integrating into a simple UI. More material on this will come in the future.

Of course, there are use cases where vision capabilities or authentication handling are essential. If you are interested in support for those features, feel free to open a discussion on GitHub.

Notebook by Stefano Fiorucci