Maintained by deepset

Integration: Gradient

Gradient is an LLM development platform that offers simple web APIs for fine-tuning, embeddings, and inference on state-of-the-art open-source models.

Mateusz Haligowski

Table of Contents


Use pip to install the integration:

pip install gradient-haystack


Once installed, you will have access to a Generator and two Embedder objects.

  • GradientDocumentEmbedder: Use this component to create embeddings of documents. This is commonly used in indexing pipelines to store documents and their embeddings in document stores.
  • GradientTextEmbedder: Use this component to create embeddings of text, such as queries. This is commonly used as the first component of query pipelines to create embeddings of the query.
  • GradientGenerator: Use this component to query generative models with Gradient. This is commonly used in query pipelines to generate responses to queries.

Use the GradientDocumentEmbedder

You can use embedding models with `GradientDocumentEmbedder`` to create embeddings of your documents. This is commonly used in indexing pipelines to write documents and their embeddings into a document store.

import os

from haystack import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.writers import DocumentWriter

from haystack_integrations.components.embedders.gradient import GradientDocumentEmbedder


documents = [
    Document(content="My name is Jean and I live in Paris."),
    Document(content="My name is Mark and I live in Berlin."),
    Document(content="My name is Giorgio and I live in Rome."),

indexing_pipeline = Pipeline()
indexing_pipeline.add_component(instance=GradientDocumentEmbedder(), name="document_embedder")
indexing_pipeline.add_component(instance=DocumentWriter(document_store=InMemoryDocumentStore()), name="document_writer")
indexing_pipeline.connect("document_embedder", "document_writer"){"document_embedder": {"documents": documents}})

Use the GradientTextEmbedder and GradientGenerator

You can use embedding models with GradientTextEmbedder and generative models with GradientGenerator. These two are commonly used together in a query pipeline such as a retrievel-augmented generative (RAG) pipeline such as the one below.

import os

from import AnswerBuilder
from import PromptBuilder
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever

from haystack_integrations.components.embedders.gradient import GradientTextEmbedder
from haystack_integrations.components.generators.gradient import GradientGenerator

from getpass import getpass

prompt_template = """
Given these documents, answer the question.\nDocuments:
{% for doc in documents %}
    {{ doc.content }}
{% endfor %}
\nQuestion: {{question}}


retriever = InMemoryEmbeddingRetriever(document_store)
prompt_builder = PromptBuilder(template=prompt_template)
embedder = GradientTextEmbedder()
generator = GradientGenerator(base_model_slug="llama2-7b-chat")

rag_pipeline = Pipeline()
rag_pipeline.add_component(instance=embedder, name="text_embedder")
    instance=retriever, name="retriever"
rag_pipeline.add_component(instance=prompt_builder, name="prompt_builder")
rag_pipeline.add_component(instance=generator, name="llm")
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")

rag_pipeline.connect("text_embedder", "retriever")
rag_pipeline.connect("retriever", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder", "llm")
rag_pipeline.connect("llm.replies", "answer_builder.replies")
rag_pipeline.connect("retriever", "answer_builder.documents")
            "text_embedder": {"text": question},
            "prompt_builder": {"question": question},
            "answer_builder": {"query": question},


You can find a full code example showing how to use the integration in this Colab.


gradient-haystack is distributed under the terms of the Apache-2.0 license.