{ "cells": [ { "cell_type": "markdown", "id": "c4dec3c0", "metadata": {}, "source": [ "# \ud83d\udda5\ufe0f Computer-Use Agent with Skills\n", "\n", "*Notebook by [Kacper \u0141ukawski](https://www.linkedin.com/in/kacperlukawski/)*\n", "\n", "> \ud83d\ude80 **Part of [Haystack 3.0 Launch Week](https://haystack.deepset.ai/launch-week/haystack-3)**: five days of new drops (July 20\u201324).\n", "\n", "In this notebook, we build a fully local, fully async agent that uses a **skill** to change how it reports back, and a custom `bash` tool to actually use the machine it runs on.\n", "\n", "Haystack's [`Agent`](https://docs.haystack.deepset.ai/docs/agents) can be given **skills** - folders of instructions the agent reads on demand, in the same `SKILL.md` format used by Claude Code and Codex. A skill teaches the agent how to do something. It doesn't let the agent do anything by itself: skills teach, tools do.\n", "\n", "By the end you'll have a fully local agent that:\n", "1. Discovers a real skill via progressive disclosure (it only sees skill names + one-line descriptions until it decides to load one).\n", "2. Uses the actual machine it's running on (inspecting the OS, disk space, largest files) via an **approved** `bash` command.\n", "3. Cuts its own output tokens by loading a caveman skill built for exactly that.\n", "\n", "Every piece runs through `async`/`await` end to end - `Agent.run_async`, the `bash` tool's `async_function`, and the human-in-the-loop confirmation - with no blocking calls in the loop.\n", "\n", "\ud83d\udcfa **Watch Along**\n", "\n", "\n", "**Stack:** Haystack `Agent` + [`SkillToolset`](https://docs.haystack.deepset.ai/docs/skilltoolset) + a local Ollama model + a custom async `bash` [`Tool`](https://docs.haystack.deepset.ai/docs/tool) + human-in-the-loop confirmation." ] }, { "cell_type": "markdown", "id": "4b861943", "metadata": {}, "source": [ "## Setup\n", "\n", "Skills ship in Haystack v3:" ] }, { "cell_type": "code", "id": "fe99e4a2c45683a8", "metadata": { "ExecuteTime": { "end_time": "2026-07-21T15:28:56.101100Z", "start_time": "2026-07-21T15:28:54.878Z" } }, "source": "%pip install -q \"haystack-ai>=3.0.0\" \"ollama-haystack>=6.8.0\"", "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "execution_count": 1 }, { "cell_type": "markdown", "id": "8465f0ee1486292b", "metadata": {}, "source": [ "### Running a local model with Ollama\n", "\n", "This notebook runs fully locally with [Ollama](https://ollama.com) - no API key required. See the\n", "[Ollama integration](https://haystack.deepset.ai/integrations/ollama) for setup, then pull the\n", "tool-capable model this notebook uses:\n", "\n", "```bash\n", "ollama pull gemma4:e4b-it-qat\n", "```\n" ] }, { "cell_type": "code", "id": "e5ac4529", "metadata": { "ExecuteTime": { "end_time": "2026-07-21T15:28:56.113378Z", "start_time": "2026-07-21T15:28:56.103486Z" } }, "source": [ "OLLAMA_URL = \"http://localhost:11434\"\n", "OLLAMA_MODEL = \"gemma4:e4b-it-qat\"" ], "outputs": [], "execution_count": 2 }, { "cell_type": "markdown", "id": "49ce5cc8", "metadata": {}, "source": [ "## What is a Skill?\n", "\n", "A skill teaches an agent *how* to do something, without giving it any new capability - it's read-only knowledge (instructions, examples, reference files), not a new tool. The agent itself decides when one applies, based on the one-line description it's shown. Concretely, a **skill** is just a directory:\n", "\n", "```\n", "skills/\n", " my-skill-name/\n", " SKILL.md # YAML frontmatter (name, description) + markdown instructions\n", " README.md # bundled reference file, read on demand\n", "```\n", "\n", "[`SkillToolset`](https://docs.haystack.deepset.ai/docs/skilltoolset) exposes exactly two tools to the agent, regardless of how many skills are available:\n", "\n", "- **`load_skill(name)`** - returns a skill's full instructions, plus a manifest of any bundled files.\n", "- **`read_skill_file(name, path)`** - reads one of those bundled files on demand.\n", "\n", "This scales to a large skill library because of [**progressive disclosure**](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills#the-anatomy-of-a-skill) - loading context in stages (name/description first, full instructions only on activation, bundled files only on demand) instead of all at once. That keeps the context small even with many skills: when the toolset warms up, every skill's **name and one-line description** (not its full instructions) get added to `load_skill`'s tool description. The model sees a menu of what's available, but it only pays the token cost of a skill's full instructions when it decides to use one, and only reads bundled files it actually needs via `read_skill_file`." ] }, { "cell_type": "markdown", "id": "4a8e4ad0", "metadata": {}, "source": [ "## Getting a real skill\n", "\n", "Rather than write a toy skill, we use a real, community-authored one: [`caveman`](https://github.com/JuliusBrussee/caveman) (MIT-licensed, by [Julius Brussee](https://github.com/JuliusBrussee)), which makes an agent respond in an ultra-compressed, concise style while keeping technical accuracy. Its `SKILL.md` description says it \"auto-triggers when token efficiency is requested\" - we rely on this later to let the model decide for itself when to use it, instead of forcing it through a system prompt." ] }, { "cell_type": "code", "id": "ed68f312", "metadata": { "ExecuteTime": { "end_time": "2026-07-21T15:28:56.140754Z", "start_time": "2026-07-21T15:28:56.114945Z" } }, "source": [ "%%bash\n", "if [ ! -d \"skills_cache/caveman-skill-repo/.git\" ]; then\n", " git clone --depth 1 --filter=blob:none --sparse https://github.com/JuliusBrussee/caveman.git skills_cache/caveman-skill-repo\n", "fi\n", "git -C skills_cache/caveman-skill-repo sparse-checkout set skills/caveman" ], "outputs": [], "execution_count": 3 }, { "cell_type": "code", "id": "a7a572dd", "metadata": { "ExecuteTime": { "end_time": "2026-07-21T15:28:56.151922Z", "start_time": "2026-07-21T15:28:56.141482Z" } }, "source": [ "from pathlib import Path\n", "\n", "SKILL_NAME = \"caveman\"\n", "skills_dir = Path(\"skills_cache/caveman-skill-repo/skills\")\n", "\n", "print(\"Skill files:\")\n", "for path in sorted((skills_dir / SKILL_NAME).rglob(\"*\")):\n", " if path.is_file():\n", " print(\" \", path.relative_to(skills_dir))" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Skill files:\n", " caveman/README.md\n", " caveman/SKILL.md\n" ] } ], "execution_count": 4 }, { "cell_type": "markdown", "id": "6bfab3be", "metadata": {}, "source": [ "## Giving the skill to an agent\n", "\n", "We pass this skill to an `Agent` and watch it decide whether to load it. There's no `bash` tool yet, so we can see `load_skill` fire on its own, before we add the `bash` tool." ] }, { "cell_type": "code", "id": "b7317d8b", "metadata": { "ExecuteTime": { "end_time": "2026-07-21T15:28:56.533800Z", "start_time": "2026-07-21T15:28:56.153029Z" } }, "source": [ "from haystack.skill_stores.file_system import FileSystemSkillStore\n", "from haystack.tools import SkillToolset\n", "\n", "\n", "print(\"Discovered skills:\")\n", "skills_toolset = SkillToolset(FileSystemSkillStore(skills_dir))\n", "for info in skills_toolset.skills.values():\n", " print(f\" - {info.name}: {info.description}\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Discovered skills:\n", " - caveman: Ultra-compressed communication mode. Cuts output tokens 65% (measured) by speaking like caveman while keeping full technical accuracy. Supports intensity levels: lite, full (default), ultra, wenyan-lite, wenyan-full, wenyan-ultra. Use when user says \"caveman mode\", \"talk like caveman\", \"use caveman\", \"less tokens\", \"be brief\", or invokes /caveman. Also auto-triggers when token efficiency is requested.\n" ] } ], "execution_count": 5 }, { "cell_type": "markdown", "id": "cd931b11", "metadata": {}, "source": [ "Notice the agent hasn't loaded anything yet - `skills_toolset.skills` is metadata-only, populated purely from\n", "`SKILL.md`'s frontmatter. We ask for something that should make the agent load the skill and show its effect\n", "directly. We run it with `run_async` - Jupyter lets you `await` directly in a cell, no `asyncio.run` needed." ] }, { "cell_type": "code", "id": "9e18f860", "metadata": { "ExecuteTime": { "end_time": "2026-07-21T15:29:06.313068Z", "start_time": "2026-07-21T15:28:56.535018Z" } }, "source": [ "from haystack.components.agents import Agent\n", "from haystack.dataclasses import ChatMessage\n", "from haystack_integrations.components.generators.ollama import OllamaChatGenerator\n", "from haystack.components.generators.utils import print_streaming_chunk\n", "\n", "\n", "reader_agent = Agent(\n", " chat_generator=OllamaChatGenerator(model=OLLAMA_MODEL, url=OLLAMA_URL),\n", " system_prompt=(\n", " \"When a task matches one of your skills, actually call the \"\n", " \"`load_skill` tool first and follow its instructions exactly - \"\n", " \"never just write out what a tool call would look like as text.\"\n", " ),\n", " tools=[skills_toolset],\n", " streaming_callback=print_streaming_chunk,\n", ")\n", "\n", "result = await reader_agent.run_async(\n", " messages=[\n", " ChatMessage.from_user(\n", " \"I need my replies to be as token-efficient as possible from \"\n", " \"now on. Load whatever skill helps with that, then answer: \"\n", " \"what is Docker and why do people use containers?\"\n", " )\n", " ]\n", ")\n", "print(result[\"last_message\"].text)" ], "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "A sync streaming callback was provided at initialization for use in an async context. It will run synchronously on the event loop and may block it.\n", "A sync streaming callback was provided at runtime for use in an async context. It will run synchronously on the event loop and may block it.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[TOOL CALL]\n", "Tool: load_skill \n", "Arguments: {\"name\": \"caveman\"}\n", "\n", "[TOOL RESULT]\n", "Respond terse like smart caveman. All technical substance stay. Only fluff die.\n", "\n", "## Persistence\n", "\n", "ACTIVE EVERY RESPONSE. No revert after many turns. No filler drift. Still active if unsure. Off only: \"stop caveman\" / \"normal mode\".\n", "\n", "Default: **full**. Switch: `/caveman lite|full|ultra`.\n", "\n", "## Rules\n", "\n", "Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not \"implement a solution for\"). No tool-call narration, no decorative tables/emoji, no dumping long raw error logs unless asked \u2014 quote shortest decisive line. Standard well-known tech acronyms OK (DB/API/HTTP); never invent new abbreviations (cfg/impl/req/res/fn) \u2014 tokenizer split them same as full word: zero token saved, reader still decode. Full word cheaper AND clearer. No causal arrows (\u2192) either \u2014 own token, save nothing. Technical terms exact. Code blocks unchanged. Errors quoted exact.\n", "\n", "Preserve user's dominant language. User write Portuguese \u2192 reply Portuguese caveman. User write Spanish \u2192 reply Spanish caveman. Compress the style, not the language. No forced English openings or status phrases. ALWAYS keep technical terms, code, API names, CLI commands, commit-type keywords (feat/fix/...), and exact error strings verbatim \u2014 unless user explicitly ask for translation.\n", "\n", "No self-reference. Never name or announce the style. No \"caveman mode on\", \"me caveman think\", no third-person caveman tags. Output caveman-only \u2014 never normal answer plus \"Caveman:\" recap. Exception: user explicitly ask what the mode is.\n", "\n", "Pattern: `[thing] [action] [reason]. [next step].`\n", "\n", "Not: \"Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by...\"\n", "Yes: \"Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:\"\n", "\n", "## Intensity\n", "\n", "| Level | What change |\n", "|-------|------------|\n", "| **lite** | No filler/hedging. Keep articles + full sentences. Professional but tight |\n", "| **full** | Drop articles, fragments OK, short synonyms. Classic caveman. No tool-call narration, no decorative tables/emoji, no long raw error-log dumps unless asked. Standard acronyms OK; no invented abbreviations |\n", "| **ultra** | Strip conjunctions when cause-then-effect stay unambiguous. One word when one word enough. State each fact once. NO prose abbreviations (cfg/impl/req/res/fn/auth), NO arrows (X \u2192 Y) \u2014 measured zero token saving under tokenizer, cost decode clarity. Code symbols, function names, API names, error strings: never touch |\n", "| **wenyan-lite** | Semi-classical. Drop filler/hedging but keep grammar structure, classical register |\n", "| **wenyan-full** | Maximum classical terseness. Fully \u6587\u8a00\u6587. 80-90% character reduction. Classical sentence patterns, verbs precede objects, subjects often omitted, classical particles (\u4e4b/\u4e43/\u70ba/\u5176) |\n", "| **wenyan-ultra** | Extreme abbreviation while keeping classical Chinese feel. Maximum compression, ultra terse |\n", "\n", "Example \u2014 \"Why React component re-render?\"\n", "- lite: \"Your component re-renders because you create a new object reference each render. Wrap it in `useMemo`.\"\n", "- full: \"New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`.\"\n", "- ultra: \"Inline obj prop, new ref, re-render. `useMemo`.\"\n", "- wenyan-lite: \"\u7d44\u4ef6\u983b\u91cd\u7e6a\uff0c\u4ee5\u6bcf\u7e6a\u65b0\u751f\u5c0d\u8c61\u53c3\u7167\u6545\u3002\u4ee5 useMemo \u5305\u4e4b\u3002\"\n", "- wenyan-full: \"\u6bcf\u7e6a\u65b0\u751f\u5c0d\u8c61\u53c3\u7167\uff0c\u6545\u91cd\u7e6a\uff1b\u4ee5 useMemo \u5305\u4e4b\u5247\u514d\u3002\"\n", "- wenyan-ultra: \"\u65b0\u53c3\u7167\u5247\u91cd\u7e6a\u3002useMemo \u5305\u4e4b\u3002\"\n", "\n", "Example \u2014 \"Explain database connection pooling.\"\n", "- lite: \"Connection pooling reuses open connections instead of creating new ones per request. Avoids repeated handshake overhead.\"\n", "- full: \"Pool reuse open DB connections. No new connection per request. Skip handshake overhead.\"\n", "- ultra: \"Pool reuse open DB connections. No per-request handshake.\"\n", "- wenyan-full: \"\u6c60\u84c4\u5df2\u958b\u4e4b\u9023\uff0c\u4e0d\u9010\u8acb\u800c\u65b0\u958b\uff0c\u7701\u63e1\u624b\u4e4b\u8cbb\u3002\"\n", "- wenyan-ultra: \"\u6c60\u84c4\u9023\uff0c\u514d\u9010\u8acb\u65b0\u958b\uff0c\u7701\u63e1\u624b\u3002\"\n", "\n", "## Auto-Clarity\n", "\n", "Drop caveman when:\n", "- Security warnings\n", "- Irreversible action confirmations\n", "- Multi-step sequences where fragment order or omitted conjunctions risk misread\n", "- Compression itself creates technical ambiguity (e.g., `\"migrate table drop column backup first\"` \u2014 order unclear without articles/conjunctions)\n", "- User asks to clarify or repeats question\n", "\n", "Resume caveman after clear part done.\n", "\n", "Example \u2014 destructive op:\n", "> **Warning:** This will permanently delete all rows in the `users` table and cannot be undone.\n", "> ```sql\n", "> DROP TABLE users;\n", "> ```\n", "> Caveman resume. Verify backup exist first.\n", "\n", "## Boundaries\n", "\n", "Code/commits/PRs: write normal. \"stop caveman\" or \"normal mode\": revert. Level persist until changed or session end.\n", "\n", "Bundled files (read with `read_skill_file`):\n", "- README.md\n", "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "A sync streaming callback was provided at runtime for use in an async context. It will run synchronously on the event loop and may block it.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[ASSISTANT]\n", "Docker container technology packages application and all dependencies into a single image, which can be run consistently across different environments.\n", "\n", "People use containers because they provide **consistency** and **isolation**. They ensure \"it works on my machine\" translates to \"it works everywhere,\" regardless of host OS differences or missing library versions. Isolation prevents one app's dependency conflict from breaking another, saving resources versus full Virtual Machines (VMs).\n", "\n", "Docker container technology packages application and all dependencies into a single image, which can be run consistently across different environments.\n", "\n", "People use containers because they provide **consistency** and **isolation**. They ensure \"it works on my machine\" translates to \"it works everywhere,\" regardless of host OS differences or missing library versions. Isolation prevents one app's dependency conflict from breaking another, saving resources versus full Virtual Machines (VMs).\n" ] } ], "execution_count": 6 }, { "cell_type": "markdown", "id": "afea9891", "metadata": {}, "source": [ "## Extending with a custom bash tool\n", "\n", "A `SkillToolset` can only **read**. To let the agent act, and actually inspect the machine it's running on, it needs a real execution tool. Haystack has no built-in shell tool by design - running arbitrary commands is inherently risky and application-specific - so we write a small one ourselves as a plain [`Tool`](https://docs.haystack.deepset.ai/docs/tool) subclass. This lets the agent runthings like `uname -a`, `df -h`, or `find . -type f` for genuine computer use, not just describing what those commands would show. Perfectly, you should swap it for a sandboxed runner (a container, or a remote executor) without touching the skills side at all.\n", "\n", "The tool defines both a sync `function` and an `async_function` - Haystack picks whichever matches how you call it. Since this notebook runs fully async, we drive it through `invoke_async`, which awaits the real `asyncio.create_subprocess_shell` implementation below instead of the blocking `subprocess.run` one.\n", "\n", "> \u26a0\ufe0f **Safety note:** this tool runs whatever command string the model gives it, via `shell=True`. That's fine\n", "> for a local, single-user notebook where you approve every call. Never connect an unattended version of this\n", "> to untrusted input." ] }, { "cell_type": "code", "id": "8b2d1363", "metadata": { "ExecuteTime": { "end_time": "2026-07-21T15:29:06.370112Z", "start_time": "2026-07-21T15:29:06.337504Z" } }, "source": [ "import asyncio\n", "import subprocess\n", "from typing import Any\n", "\n", "from haystack.core.serialization import generate_qualified_class_name\n", "from haystack.tools import Tool\n", "\n", "_BASH_DESCRIPTION = (\n", " \"Execute a bash command and return its combined stdout/stderr and exit code. Use this to inspect the \"\n", " \"system, run scripts, or read/write files. Never guess or make up output - always run the real command.\"\n", ")\n", "_BASH_PARAMETERS = {\n", " \"type\": \"object\",\n", " \"properties\": {\"command\": {\"type\": \"string\", \"description\": \"The bash command to execute.\"}},\n", " \"required\": [\"command\"],\n", "}\n", "\n", "\n", "class BashTool(Tool):\n", " \"\"\"A standalone Tool that executes bash commands in a subprocess, optionally scoped to a working directory.\"\"\"\n", "\n", " def __init__(self, working_dir: str | Path | None = None, timeout: int = 60) -> None:\n", " self._working_dir = Path(working_dir) if working_dir is not None else None\n", " self._timeout = timeout\n", " super().__init__(\n", " name=\"bash\",\n", " description=_BASH_DESCRIPTION,\n", " parameters=_BASH_PARAMETERS,\n", " function=self._run,\n", " async_function=self._run_async,\n", " )\n", "\n", " def _format(self, returncode: int, stdout: str, stderr: str) -> str:\n", " parts = [f\"exit_code: {returncode}\"]\n", " if stdout:\n", " parts.append(f\"stdout:\\n{stdout.rstrip()}\")\n", " if stderr:\n", " parts.append(f\"stderr:\\n{stderr.rstrip()}\")\n", " return \"\\n\".join(parts)\n", "\n", " def _run(self, command: str) -> str:\n", " cwd = str(self._working_dir) if self._working_dir is not None else None\n", " try:\n", " proc = subprocess.run(command, shell=True, cwd=cwd, capture_output=True, text=True, timeout=self._timeout)\n", " except subprocess.TimeoutExpired:\n", " return f\"exit_code: -1\\nstderr:\\nCommand timed out after {self._timeout}s.\"\n", " return self._format(proc.returncode, proc.stdout, proc.stderr)\n", "\n", " async def _run_async(self, command: str) -> str:\n", " cwd = str(self._working_dir) if self._working_dir is not None else None\n", " proc = await asyncio.create_subprocess_shell(\n", " command, cwd=cwd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE\n", " )\n", " try:\n", " stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=self._timeout)\n", " except asyncio.TimeoutError:\n", " proc.kill()\n", " return f\"exit_code: -1\\nstderr:\\nCommand timed out after {self._timeout}s.\"\n", " return self._format(proc.returncode or 0, stdout.decode(), stderr.decode())\n", "\n", " def to_dict(self) -> dict[str, Any]:\n", " return {\n", " \"type\": generate_qualified_class_name(type(self)),\n", " \"data\": {\n", " \"working_dir\": str(self._working_dir) if self._working_dir is not None else None,\n", " \"timeout\": self._timeout,\n", " },\n", " }\n", "\n", " @classmethod\n", " def from_dict(cls, data: dict[str, Any]) -> \"BashTool\":\n", " return cls(**data[\"data\"])\n", "\n", "\n", "bash = BashTool()\n", "output = await bash.invoke_async(command=\"whoami\")\n", "print(output)" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "exit_code: 0\n", "stdout:\n", "kacper.lukawski\n" ] } ], "execution_count": 7 }, { "cell_type": "markdown", "id": "059301b6", "metadata": {}, "source": [ "## Keeping a human in the loop\n", "\n", "Before letting the agent actually run shell commands, we require approval for every `bash` call. Haystack's [human-in-the-loop](https://docs.haystack.deepset.ai/docs/human-in-the-loop) support plugs in as a `before_tool` [hook](https://docs.haystack.deepset.ai/docs/hooks): a `ConfirmationHook` intercepts pending tool calls and applies a **policy** through a **UI** before the call is allowed to run.\n", "\n", "- **Policy** - `AlwaysAskPolicy()` asks every single time (there's also `AskOncePolicy`, `NeverAskPolicy`).\n", "- **Strategy** - `BlockingConfirmationStrategy` pauses the run and waits for your answer.\n", "- **UI** - `SimpleConsoleUI()` prints the pending call and reads your `y.; `RichConsoleUI` is a fancier terminal-only alternative.\n", "\n", "We scope the hook to just the `\"bash\"` tool by name - the skill-reading tools stay unattended, since they're read-only." ] }, { "cell_type": "code", "id": "1343a014", "metadata": { "ExecuteTime": { "end_time": "2026-07-21T15:29:06.380633Z", "start_time": "2026-07-21T15:29:06.371566Z" } }, "source": [ "from haystack.hooks.human_in_the_loop import (\n", " AlwaysAskPolicy, \n", " BlockingConfirmationStrategy, \n", " ConfirmationHook, \n", " SimpleConsoleUI\n", ")\n", "\n", "confirmation_hook = ConfirmationHook(\n", " confirmation_strategies={\n", " \"bash\": BlockingConfirmationStrategy(\n", " confirmation_policy=AlwaysAskPolicy(),\n", " confirmation_ui=SimpleConsoleUI(),\n", " )\n", " }\n", ")" ], "outputs": [], "execution_count": 8 }, { "cell_type": "markdown", "id": "f65b66ee", "metadata": {}, "source": [ "## Putting it together: computer use, with and without the skill\n", "\n", "We run the **exact same** computer-use task twice: once with a plain agent, once with an agent that also has the `caveman` skill available, and compare. Both agents get the same `bash` tool and the same confirmation hook, so the only variable is whether the skill is available to load.\n", "\n", "The task is real machine inspection: OS/kernel info, disk space, the biggest files around. Approve each `bash` call with `y` + Enter." ] }, { "cell_type": "code", "id": "e7eaf2d8", "metadata": { "ExecuteTime": { "end_time": "2026-07-21T15:29:06.386962Z", "start_time": "2026-07-21T15:29:06.381929Z" } }, "source": [ "INSPECTION_QUERY = (\n", " \"Investigate this machine: get the OS/kernel info, the Python version, \"\n", " \"disk space usage, and the 5 largest files in the current directory. \"\n", " \"Report what you find.\"\n", ")" ], "outputs": [], "execution_count": 9 }, { "cell_type": "markdown", "id": "75934dc2", "metadata": {}, "source": [ "### Baseline run - no skill" ] }, { "cell_type": "code", "id": "ae76967d", "metadata": { "ExecuteTime": { "start_time": "2026-07-21T15:29:06.387911Z" } }, "source": [ "baseline_agent = Agent(\n", " chat_generator=OllamaChatGenerator(model=OLLAMA_MODEL, url=OLLAMA_URL),\n", " system_prompt=(\n", " \"You investigate the user's machine using the `bash` tool. \"\n", " \"Always run the real commands - never guess or make up any part \"\n", " \"of the output. If a command fails or gives incomplete output, \"\n", " \"run a corrected command instead of filling gaps with placeholder \"\n", " \"or hypothetical text.\"\n", " ),\n", " tools=[bash],\n", " hooks={\"before_tool\": [confirmation_hook]},\n", " streaming_callback=print_streaming_chunk,\n", " max_agent_steps=20,\n", ")\n", "\n", "baseline_result = await baseline_agent.run_async(messages=[\n", " ChatMessage.from_user(INSPECTION_QUERY),\n", "])" ], "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "A sync streaming callback was provided at initialization for use in an async context. It will run synchronously on the event loop and may block it.\n", "A sync streaming callback was provided at runtime for use in an async context. It will run synchronously on the event loop and may block it.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[TOOL CALL]\n", "Tool: bash \n", "Arguments: {\"command\": \"uname -a\\npython3 --version\\ndf -h .\\nls -lSh | head -n 5\"}\n", "\n", "\n", "--- Tool Execution Request ---\n", "Tool: bash\n", "Description: Execute a bash command and return its combined stdout/stderr and exit code. Use this to inspect the system, run scripts, or read/write files. Never guess or make up output - always run the real command.\n", "Arguments:\n", " command: uname -a\n", "python3 --version\n", "df -h .\n", "ls -lSh | head -n 5\n", "------------------------------\n" ] } ], "execution_count": null }, { "cell_type": "markdown", "id": "e3eb25bd", "metadata": {}, "source": [ "### Skill-aware run - same task, `caveman` available\n", "\n", "The query below adds one phrase, \"make this as token-efficient as possible,\" that matches `caveman`'s own trigger condition. We don't tell the agent to use the skill - it decides to, because the skill's description says that's exactly when it applies. That's progressive disclosure in action." ] }, { "cell_type": "code", "id": "b71ac7b8", "metadata": {}, "source": [ "skill_agent = Agent(\n", " chat_generator=OllamaChatGenerator(model=OLLAMA_MODEL, url=OLLAMA_URL),\n", " system_prompt=(\n", " \"You investigate the user's machine using the `bash` tool. \"\n", " \"Always run the real commands - never guess or make up any part \"\n", " \"of the output. If a command fails or gives incomplete output, \"\n", " \"run a corrected command instead of filling gaps with placeholder \"\n", " \"or hypothetical text.\"\n", " ),\n", " tools=[skills_toolset, bash],\n", " hooks={\"before_tool\": [confirmation_hook]},\n", " streaming_callback=print_streaming_chunk,\n", " max_agent_steps=20,\n", ")\n", "\n", "skill_query = INSPECTION_QUERY + \" Make this as token-efficient as possible.\"\n", "\n", "skill_result = await skill_agent.run_async(messages=[ChatMessage.from_user(skill_query)])" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "0a45f27e", "metadata": {}, "source": [ "### The comparison\n", "\n", "There are two different things worth measuring here, and they can point in different directions:\n", "\n", "- **Final-answer tokens** (`result[\"messages\"][-1].meta[\"usage\"]`): tokens spent on just the reply you read. This isolates the skill's effect on style.\n", "- **Whole-run total tokens** (`result[\"token_usage\"]`): all LLM calls in the run, including every step spent deciding which tool to call next. This depends mostly on how many separate tool calls the model made, not on the skill. Don't treat a difference here as proof the skill worked or didn't." ] }, { "cell_type": "code", "id": "641f2698", "metadata": {}, "source": [ "baseline_final_tokens = baseline_result[\"messages\"][-1].meta.get(\"usage\", {}).get(\"completion_tokens\", 0)\n", "skill_final_tokens = skill_result[\"messages\"][-1].meta.get(\"usage\", {}).get(\"completion_tokens\", 0)\n", "baseline_total_tokens = baseline_result[\"token_usage\"].get(\"completion_tokens\", 0)\n", "skill_total_tokens = skill_result[\"token_usage\"].get(\"completion_tokens\", 0)\n", "\n", "metrics = {\n", " \"final-answer tokens\": (baseline_final_tokens, skill_final_tokens),\n", " \"whole-run total tokens\": (baseline_total_tokens, skill_total_tokens),\n", "}\n", "for label, (before, after) in metrics.items():\n", " change = f\"{100 * (after - before) / before:+.0f}%\" if before else \"n/a\"\n", " print(f\"{label:<22} baseline {before:>4} with skill {after:>4} change {change:>5}\")" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "aca9dd05", "metadata": {}, "source": [ "## What's next?\n", "\n", "- **Sandbox the bash tool.** For anything beyond a local notebook, run commands in a container or a restricted shell instead of a bare `subprocess`.\n", "- **Combine with more tools.** `tools=[skills_toolset, bash, *other_tools]` - a `SkillToolset` composes with any other `Tool`/`Toolset`, as long as there's only one `SkillToolset` per agent.\n", "- **Try per-run tool selection.** `agent.run_async(tools=[...])` lets you restrict which tools are active for a given call - handy once an agent has a larger toolbox." ] }, { "cell_type": "code", "id": "8eaa0351-3870-4a1e-9dc6-e44034fceb70", "metadata": {}, "source": [], "outputs": [], "execution_count": null } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.13" } }, "nbformat": 4, "nbformat_minor": 5 }