Haystack
Get Enterprise Support
Get Started
🚀 Haystack 3.0 Office Hours — Watch the recording
Maintained by deepset

Integration: Hetzner

Use the Hetzner Inference API for open-weight LLMs hosted in Europe.

Authors
deepset
deepset-ai
GitHub Repo PyPI Package

Table of Contents

Overview

The Hetzner Inference API serves open-weight LLMs from Hetzner’s European data centers behind an OpenAI-compatible REST API. Once installed, you get access to the HetznerChatGenerator, which lets you call any of the models Hetzner serves.

Two models are available, both with a 262,144-token context window and both accepting images alongside text:

  • Qwen/Qwen3.6-35B-A3B-FP8
  • Qwen3.8-27B

In order to follow along with this guide, you’ll need a Hetzner API token for the Inference API. Add it as an environment variable, HETZNER_API_KEY.

Installation

pip install hetzner-haystack

Usage

You can use HetznerChatGenerator as a standalone component, within a pipeline or with the Agent component.

Here’s an example of using it as a standalone component:

from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.hetzner import HetznerChatGenerator

client = HetznerChatGenerator()  # defaults to "Qwen/Qwen3.6-35B-A3B-FP8"
response = client.run(
    [ChatMessage.from_user("What are Agentic Pipelines? Be brief.")]
)
print(response["replies"][0].text)

HetznerChatGenerator also supports streaming responses if you pass a streaming callback:

from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.hetzner import HetznerChatGenerator

def show(chunk):                              # simple streaming callback
    print(chunk.content, end="", flush=True)

client = HetznerChatGenerator(
    model="Qwen3.8-27B",
    streaming_callback=show,
    generation_kwargs={"max_tokens": 100, "temperature": 0.7, "top_p": 0.9},
)

client.run([ChatMessage.from_user("Summarize RAG in two lines.")])

The served models are multimodal, so you can pass images along with your prompt:

from haystack.dataclasses import ChatMessage, ImageContent
from haystack_integrations.components.generators.hetzner import HetznerChatGenerator

image = ImageContent.from_url(
    "https://cdn.hetzner.de/cdn/public/Uploads/Finnland_Luftaufnahme-v2.jpg"
)

client = HetznerChatGenerator()
response = client.run(
    [ChatMessage.from_user(content_parts=["Describe this image in one sentence.", image])]
)
print(response["replies"][0].text)

And here’s how to use it in a pipeline:

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.hetzner import HetznerChatGenerator

pipe = Pipeline()
pipe.add_component("builder", ChatPromptBuilder())
pipe.add_component("llm", HetznerChatGenerator())
pipe.connect("builder.prompt", "llm.messages")

messages = [
    ChatMessage.from_system("Give brief answers."),
    ChatMessage.from_user("Tell me about {{city}}"),
]

response = pipe.run(
    data={"builder": {"template": messages, "template_variables": {"city": "Nuremberg"}}},
)
print(response["llm"]["replies"][0].text)

License

hetzner-haystack is distributed under the terms of the Apache-2.0 license.