Tutorial: Human-in-the-Loop with Haystack Agents

_{Last Updated:
October 31, 2025}

Level: Advanced
Time to complete: 20 minutes
Components Used: OpenAIChatGenerator
Experimental Components Used: Agent
Prerequisites: You need an OpenAI API Key
Goal: After completing this tutorial, you’ll have learned how to implement human-in-the-loop workflows in Haystack agents using confirmation strategies, create custom confirmation policies, and control tool execution approval flows.

Overview

This tutorial introduces how to use confirmation strategies to create human-in-the-loop interactions in Haystack’s Agent component.

Why is this useful? Imagine an AI agent that can access your bank account, send emails, or make purchases. You probably want human approval before it executes sensitive operations. Confirmation strategies let you define exactly when the agent should ask for permission.

🧪 Beta Feature Notice:

The Human-in-the-Loop feature is currently in beta, available in the haystack-experimental repository. We’d love your feedback, join the conversation in this GitHub discussion and help us shape this feature!

How Confirmation Strategies Work

User sends a query to the agent
Agent determines which tool(s) to use
Before executing, the confirmation policy checks if approval is needed
If needed, the UI prompts the user for confirmation
Based on user response (confirm/reject/modify), the agent proceeds or adjusts
Agent returns the final answer to the user

Preparing the Environment

First, let’s install required packages:

%%bash

pip install -q haystack-experimental

Enter API Keys

Enter API keys required for this tutorial.

from getpass import getpass
import os

if not os.environ.get("OPENAI_API_KEY"):
    os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key:")

Setup and Imports

We begin by importing Haystack classes, UI helpers, and the experimental Agent component.

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.tools import create_tool_from_function
from rich.console import Console

from haystack_experimental.components.agents.agent import Agent
from haystack_experimental.components.agents.human_in_the_loop import (
    AlwaysAskPolicy,
    AskOncePolicy,
    BlockingConfirmationStrategy,
    NeverAskPolicy,
    RichConsoleUI,
    SimpleConsoleUI,
)

Define Agent Tools

Create three simple tools for demonstration: addition, get_bank_balance, and get_phone_number with create_tool_from_function. Alternatively, you can use @tool decorator to define your tools.

def addition(a: float, b: float) -> float:
    """Add two numbers."""
    return a + b


addition_tool = create_tool_from_function(function=addition, name="addition", description="Add two floats together.")


def get_bank_balance(account_id: str) -> str:
    """Simulate fetching a bank balance."""
    return f"Balance for account {account_id} is $1,234.56"


balance_tool = create_tool_from_function(
    function=get_bank_balance, name="get_bank_balance", description="Get the bank balance for a given account ID."
)


def get_phone_number(name: str) -> str:
    """Simulate fetching a phone number."""
    return f"The phone number for {name} is (123) 456-7890"


phone_tool = create_tool_from_function(
    function=get_phone_number, name="get_phone_number", description="Get the phone number for a given name."
)

Instantiate the Agent

Instantiate the experimental Agent with multiple tools and assign each tool a confirmation strategy.

cons = Console()

agent = Agent(
    chat_generator=OpenAIChatGenerator(model="gpt-4.1-mini"),
    tools=[balance_tool, addition_tool, phone_tool],
    system_prompt="You are a helpful financial assistant. Use the provided tools to answer user questions.",
    confirmation_strategies={
        balance_tool.name: BlockingConfirmationStrategy(
            confirmation_policy=AlwaysAskPolicy(), confirmation_ui=RichConsoleUI(console=cons)
        ),
        addition_tool.name: BlockingConfirmationStrategy(
            confirmation_policy=NeverAskPolicy(), confirmation_ui=RichConsoleUI(console=cons)
        ),
        phone_tool.name: BlockingConfirmationStrategy(
            confirmation_policy=AskOncePolicy(), confirmation_ui=RichConsoleUI(console=cons)
        ),
    },
)

Explanation

Each BlockingConfirmationStrategy defines when and how the agent asks for confirmation before executing a tool:

AlwaysAskPolicy – Always asks for approval before running.
NeverAskPolicy – Executes automatically without user confirmation.
AskOncePolicy – Asks once per tool, remembers approval for future runs.

The UI can be either:

RichConsoleUI for colorized and more aesthetic prompts.
SimpleConsoleUI for basic prompts.

Both UIs will present three options:

y: Proceed with the tool execution as planned
n: Cancel the tool execution; agent will try alternative approaches
m: Edit the tool parameters before execution

NOTE: Custom UIs and Policies can also be implemented by following the respective ConfirmationUI and ConfirmationPolicy protocols. This allows full flexibility for domain-specific workflows.

Run the Agent

Now we can run the agent with different confirmation behaviors.

Try different confirmation strategies and explore the available options returned by the UIs: “y”, “n”, “m”. This lets you test how the agent reacts to different human feedback scenarios.

Always Ask Policy (Rich UI)

result = agent.run([ChatMessage.from_user("What's the balance of account 56789?")])
cons.print(f"\n[bold green]Agent Result:[/bold green] {result['last_message'].text}")

Never Ask Policy

This is the default behavior when no confirmation strategy is defined. The tool executes without asking for confirmation.

result = agent.run([ChatMessage.from_user("What is 5.5 + 3.2?")])
cons.print(f"\n[bold green]Agent Result:[/bold green] {result['last_message'].text}")

Ask Once Policy

result = agent.run([ChatMessage.from_user("What is the phone number of Alice?")])
cons.print(f"\n[bold green]Agent Result:[/bold green] {result['last_message'].text}")

Ask Once: Cached Confirmation

If you answered “y” (yes) to the previous phone number request, the agent will not ask for confirmation again for subsequent requests to the same tool called with the same parameters.

result = agent.run([ChatMessage.from_user("What is the phone number of Alice?")])
cons.print(f"\n[bold green]Agent Result:[/bold green] {result['last_message'].text}")

Run an Agent with a Custom Policy

Now, let’s create a custom confirmation policy that asks for confirmation only when certain conditions are met. For example, you can create a policy that asks for confirmation when the tool operation involves expenses above a certain threshold.

Create a Custom Budget Based Policy

First, define a BudgetBasedPolicy that asks for confirmation if the cost exceeds a defined threshold.

from haystack_experimental.components.agents.human_in_the_loop import ConfirmationPolicy
from typing import Any

class BudgetBasedPolicy(ConfirmationPolicy):
    """Ask for confirmation when operations exceed a cost threshold."""

    def __init__(self, cost_threshold: float = 10.0):
        self.cost_threshold = cost_threshold

    def should_ask(self, tool_name: str, tool_description: str, tool_params: dict[str, Any]) -> bool:
        """Ask if the operation cost exceeds the threshold."""
        # Check for cost-related parameters
        cost = tool_params.get("cost", 0.0)
        amount = tool_params.get("amount", 0.0)
        price = tool_params.get("price", 0.0)

        return max(cost, amount, price) > self.cost_threshold

Define an Agent with Expense Tool

Define an expense tool that simulates submitting an expense report and use the BudgetBasedPolicy for confirmation.

from haystack.components.generators.utils import print_streaming_chunk

def expense(cost: float, description: str) -> float:
    """Submit an expense report that has a `cost` and `description`"""
    # This is where we would add a real submission request
    return "Expense report submitted successfully!"


expense_tool = create_tool_from_function(
    function=expense, name="expense", description="Submit an expense report that has a `cost` and `description`"
)

cons = Console()

agent = Agent(
    chat_generator=OpenAIChatGenerator(model="gpt-4.1"),
    tools=[expense_tool],
    system_prompt=(
        "You are a helpful financial assistant that can submit expense reports for users. "
        "Use the `expense` tool which only requires a cost amount and short description of the expense (e.g. 'busines lunch'). "
        "Only respond with whether the report was submitted successfully. "
    ),
    confirmation_strategies={
        expense_tool.name: BlockingConfirmationStrategy(
            confirmation_policy=BudgetBasedPolicy(cost_threshold=100.0), confirmation_ui=RichConsoleUI(console=cons)
        )
    },
)

Run the Agent

Now, submit an expense report that’s below the threshold.

result = agent.run([ChatMessage.from_user("Submit an expense report for business lunch that cost $10")])
cons.print(f"\n[bold green]Agent Result:[/bold green] {result['last_message'].text}")

Next, submit a request that’s above the threshold to get confirmation.

result = agent.run([ChatMessage.from_user("Submit an expense report for business travel that cost $200")])
cons.print(f"\n[bold green]Agent Result:[/bold green] {result['last_message'].text}")

What’s next

🎉 Congratulations! You’ve just built a Haystack Agent that incorporates human-in-the-loop workflows using confirmation strategies.

Curious to keep exploring? Here are a few great next steps:

To stay up to date on the latest Haystack developments, you can sign up for our newsletter or join Haystack discord community.

(Notebook by Sebastian Husch Lee)

Query Classification with TransformersTextRouter and TransformersZeroShotTextRouter

Creating Vision+Text RAG Pipelines