πŸ†• Haystack 2.11 is out with shorter import time and extended async support. Go to release notes for all highlights 🌟

Building an Interactive Feedback Review Agent with Azure AI Search and Haystack


by Amna Mubashar (Haystack), and Khye Wei (Azure AI Search)

This notebook demonstrates how you can build indexing and querying pipelines using Azure AI Search-Haystack integration. Additionally, you’ll develop an interactive feedback review agent leveraging Haystack Tools.

Install the required dependencies

# Install the required dependencies

!pip install haystack-ai==2.10.3
!pip install azure-ai-search-haystack
!pip install jq
!pip install nltk=="3.9.1"
!pip install jsonschema
!pip install kagglehub

Loading and Preparing the Dataset

We will use an open dataset consisting of approx. 28000 customer reviews for a clothing store. The dataset is available at Shopper Sentiments.

We will load the dataset and convert it into a JSON format that can be used by Haystack.


import kagglehub
path = kagglehub.dataset_download("nelgiriyewithana/shoppersentiments")
import pandas as pd
from json import loads, dumps

path = "<Path to the CSV file>"

df = pd.read_csv(path, encoding='latin1', nrows=200) # We are using 200 rows for testing purposes

df.rename(columns={'review-label': 'rating'}, inplace=True)
df['year'] = pd.to_datetime(df['year'], format='%Y %H:%M:%S').dt.year

# Convert DataFrame to JSON
json_data = {"reviews": loads(df.to_json(orient="records"))}

Once we have the JSON data, we can convert it into a Haystack Document format using the JSONConverter component. Its important to remove any documents with no content as they will not be indexed.

from haystack.components.converters import JSONConverter
from haystack.dataclasses import ByteStream
converter = JSONConverter(
  jq_schema=".reviews[]", content_key="review", extra_meta_fields={"store_location", "date", "month", "year", "rating"}
)
source = ByteStream.from_string(dumps(json_data))

documents = converter.run(sources=[source])['documents']
documents = [doc for doc in documents if doc.content is not None] # remove documents with no content

Remove any non-ASCII characters and any regex patterns that are not alphanumeric using the DocumentCleaner component.

from haystack.components.preprocessors import DocumentCleaner
cleaner = DocumentCleaner(ascii_only=True, remove_regex="i12i12i12")
cleaned_documents=cleaner.run(documents=documents)

Setting up Azure AI Search and Indexing Pipeline

We set up an indexing pipeline with AzureAISearchDocumentStore by following these steps:

  1. Configure semantic search for the index
  2. Initialize the document store with custom metadata fields and semantic search configuration
  3. Create an indexing pipeline that:
    • Generates embeddings for the documents using AzureOpenAIDocumentEmbedder
    • Writes the documents and their embeddings to the search index

The semantic configuration allows for more intelligent searching beyond simple keyword matching. Note, the metadata fields need to be declared while creating the index as the API does not allow modifying them after index creation.

from haystack import Pipeline
from haystack.components.embedders import AzureOpenAIDocumentEmbedder
from haystack.components.writers import DocumentWriter
from azure.search.documents.indexes.models import (
    SemanticConfiguration,
    SemanticField,
    SemanticPrioritizedFields,
    SemanticSearch
)

from haystack_integrations.document_stores.azure_ai_search import AzureAISearchDocumentStore


semantic_config = SemanticConfiguration(
    name="my-semantic-config",
    prioritized_fields=SemanticPrioritizedFields(
        content_fields=[SemanticField(field_name="content")]
    )
)

# Create the semantic settings with the configuration
semantic_search = SemanticSearch(configurations=[semantic_config])

document_store = AzureAISearchDocumentStore(index_name="customer-reviews-analysis", api_key="your_api_key",
                                            endpoint="your_endpoint",
    embedding_dimension=1536, metadata_fields = {"month": int, "year": int, "rating": int, "store_location": str}, semantic_search=semantic_search)

# Indexing Pipeline
indexing_pipeline = Pipeline()
indexing_pipeline.add_component(AzureOpenAIDocumentEmbedder(), name="document_embedder")
indexing_pipeline.add_component(instance=DocumentWriter(document_store=document_store), name="doc_writer")
indexing_pipeline.connect("document_embedder", "doc_writer")

indexing_pipeline.run({"document_embedder": {"documents": cleaned_documents["documents"]}})
Embedding Texts: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 6/6 [00:29<00:00,  4.91s/it]





{'document_embedder': {'meta': {'model': 'text-embedding-ada-002',
   'usage': {'prompt_tokens': 4283, 'total_tokens': 4283}}},
 'doc_writer': {'documents_written': 175}}

Creating the Query Pipeline

Here we set up the query pipeline that will retrieve relevant reviews based on user queries. The pipeline consists of:

  1. A text embedder (AzureOpenAITextEmbedder) that converts user queries into embeddings.
  2. A hybrid retriever (AzureAISearchHybridRetriever) that uses vector and semantic search to retrieve the most relevant reviews.
from haystack_integrations.components.retrievers.azure_ai_search import AzureAISearchHybridRetriever
from haystack.components.embedders import AzureOpenAITextEmbedder


# Query Pipeline
query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", AzureOpenAITextEmbedder())
query_pipeline.add_component("retriever", AzureAISearchHybridRetriever(document_store=document_store, query_type="semantic", semantic_configuration_name="my-semantic-config", top_k=10))
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
<haystack.core.pipeline.pipeline.Pipeline object at 0x10fe27610>
πŸš… Components
  - text_embedder: AzureOpenAITextEmbedder
  - retriever: AzureAISearchHybridRetriever
πŸ›€οΈ Connections
  - text_embedder.embedding -> retriever.query_embedding (List[float])
query = "Which reviews are about shipping?"

# Retrieve reviews based on the query
result = query_pipeline.run({"text_embedder": {"text": query}, "retriever": {"query": query}})
retrieved_reviews = result["retriever"]["documents"]
print(retrieved_reviews)
<iterator object azure.core.paging.ItemPaged at 0x17f829880>
[Document(id=e9f8a141855701896441cbf9fd29ad326ec5250e9263f4ea1f74a5b389d1c90c, content: 'You did everything right! Shipping was quick and reasonable, and the shirts are awesome! Colorful an...', meta: {'store_location': 'US', 'year': 2018, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=a841f950a433d05857933fb7cd46d54a6a04066f1373875359c90f096ce0bf9a, content: 'I love the shirts that I bought.Prices were great and shipping didnt take any lon', meta: {'store_location': 'US', 'year': 2024, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=0ee4d9627e9085936973126762a0aa746b5e235cf253a9f759af84fbfdd9cdae, content: 'Product was great. Love the options. anything I could imagine was available. my only issue was I pai...', meta: {'store_location': 'US', 'year': 2023, 'rating': 4, 'month': 6}, embedding: vector of size 1536), Document(id=31bb2754ad0ebf5084c90260dd45f7c9ea8f5bf5759d3caaf7e97420f8c9610b, content: 'Great shipping time. The shirts look amazing.', meta: {'store_location': 'US', 'year': 2024, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=eb580826a63a596f312f5e2757b25f53d4622fc700cb158b8f6a6b307644d61d, content: 'I love the design, quality of the shirt was great. the print was high quality, shipping was on time ...', meta: {'store_location': 'US', 'year': 2019, 'rating': 4, 'month': 6}, embedding: vector of size 1536), Document(id=351df76a4a52548725cb61803ccb0602379e465a2c0a79e05061f7d7c729054b, content: 'Great designs and quality. Items shipped quickly and correctly.', meta: {'store_location': 'US', 'year': 2024, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=b18855210319bc9b6bf3066a644114425900e95979ad77c1c1dd85e20cbac8b2, content: 'Awesome shirts ,great quality material, fantastic designs,good shipping speed and carefully packed a...', meta: {'store_location': 'US', 'year': 2024, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=6d2ad6c2991516f5c1f7793414996e763a2702ffeceb01b18a61c3225a61bc46, content: 'Once I figured out the sizing, everything was great. FYI, I am a size 10 in womens tops but prefer a...', meta: {'store_location': 'US', 'year': 2018, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=8798ccd5cde690479c764e8118f1d5423352d13b9d2b8849bc8aed801ae9a9be, content: 'A bit pricey for a tee shirt. Childs size cost $18.00 and outrageous shipping $9.99. No way this cos...', meta: {'store_location': 'US', 'year': 2024, 'rating': 3, 'month': 6}, embedding: vector of size 1536), Document(id=d58901f69050e781ca6f5c78bff61474294e9f0a4b65549bab7b9764f8eed81e, content: 'Delivered ON TIME and shirt is EXTREMELY COMFORTABLE!! You guys are THE BEST!', meta: {'store_location': 'US', 'year': 2018, 'rating': 5, 'month': 6}, embedding: vector of size 1536)]

Create Tools for Sentiment Analysis and Summarization

Install the required dependencies.

!pip install vaderSentiment
!pip install matplotlib
!pip install sumy

Create a function that will be used by review_analysis tool to visualize the sentiment distribution across customer review aspects (e.g., product quality, shipping). It compares VADER-based sentiment scores with customer ratings using color-coded bars (positive, neutral, negative).

# Function to visualize the sentiment distribution

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np


def plot_sentiment_distribution(aspects):
    # Create DataFrame from aspects data
    data = [(topic, review['sentiment']['analyzer_rating'], 
             review['review']['rating'], review['sentiment']['label'])
            for topic, reviews in aspects.items()
            for review in reviews]
    
    df = pd.DataFrame(data, columns=['Topic', 'Normalized Score', 'Original Rating', 'Sentiment'])
    
    # Calculate means
    df_means = df.groupby('Topic').agg({
        'Normalized Score': 'mean',
        'Original Rating': 'mean'
    }).reset_index()
    
    fig, ax = plt.subplots(figsize=(8, 4))  
    x = np.arange(len(df_means))
    bar_width = 0.3  
    
    # Colors for sentiment
    colors = {
        'positive': '#2ecc71',
        'neutral': '#f1c40f',
        'negative': '#e74c3c'
    }
    
    # Create bars
    sentiment_colors = [colors[df.groupby('Topic')['Sentiment'].agg(lambda x: x.mode()[0])[topic]] 
                       for topic in df_means['Topic']]
    
    bars1 = ax.bar(x - bar_width/2, df_means['Normalized Score'], 
                   bar_width, label='Normalized Score', color=sentiment_colors)
    bars2 = ax.bar(x + bar_width/2, df_means['Original Rating'], 
                   bar_width, label='Original Rating', color='gray', alpha=0.7)
    
    # Customize plot with smaller font sizes
    ax.set_ylabel('Score', fontsize=9)
    ax.set_title('Average Sentiment Scores by Topic', fontsize=10)
    ax.set_xticks(x)
    ax.set_xticklabels(df_means['Topic'], rotation=45, ha='right', fontsize=8)
    ax.tick_params(axis='y', labelsize=8)
    
    # Add value labels with smaller font size
    for bars in [bars1, bars2]:
        ax.bar_label(bars, fmt='%.2f', padding=3, fontsize=8)
    
    # Smaller legend
    ax.legend(handles=[plt.Rectangle((0,0),1,1, color=c) for c in colors.values()] + 
             [plt.Rectangle((0,0),1,1, color='gray', alpha=0.7)],
             labels=list(colors.keys()) + ['Original Rating'],
             loc='upper right',
             fontsize=8)
    
    plt.tight_layout()
    plt.show()

Create a tool to perform aspect-based sentiment analysis on customer reviews using the VADER sentiment analyzer. It involves:

  • Identifying specific aspects within reviews (e.g., product quality, shipping, customer service, pricing) using predefined keywords
  • Calculating sentiment scores for each review mentioning these aspects
  • Categorizing sentiment as ‘positive’, ’negative’, or ’neutral’
  • Normalizing sentiment scores to a scale of 1 to 5 for comparison with customer ratings
from haystack.tools import Tool
from haystack.components.tools import ToolInvoker

from typing import Dict, List
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer


def analyze_sentiment(reviews: List[Dict]) -> Dict:
    """
    Perform aspect-based sentiment analysis.
    
    For each review that mentions keywords related to a specific topic, the function computes 
    sentiment scores using VADER and categorizes the sentiment as 'positive', 'negative', or 'neutral'.
    
    """
    aspects = {
        "product_quality": [],
        "shipping": [],
        "customer_service": [],
        "pricing": []
    }
    
    # Define keywords for each topic
    keywords = {
        "product_quality": ["quality", "material", "design", "fit", "size", "color", "style"],
        "shipping": ["shipping", "delivery", "arrived"],
        "customer_service": ["service", "support", "help"],
        "pricing": ["price", "cost", "expensive", "cheap"]
    }

    
    # Initialize the VADER sentiment analyzer
    analyzer = SentimentIntensityAnalyzer()
    
    for review in reviews:
        text = review.get("review", "").lower()
        for topic, words in keywords.items():
            if any(word in text for word in words):
                # Compute sentiment scores using VADER
                sentiment_scores = analyzer.polarity_scores(text)
               
                compound = sentiment_scores['compound']
                # Normalize compound score from [-1, 1] to [1, 5]
                normalized_score = (compound + 1) * 2 + 1
                
                if compound >= 0.03:
                    sentiment_label = 'positive'
                elif compound <= -0.03:
                    sentiment_label = 'negative'
                else:
                    sentiment_label = 'neutral'
                
                # Append the review along with its sentiment analysis result
                aspects[topic].append({
                    "review": review,
                    "sentiment": {
                        "analyzer_rating": normalized_score,
                        "label": sentiment_label
                    }
                })
    plot_sentiment_distribution(aspects)

    return {
        "total_reviews": len(reviews),
        "sentiment_analysis": aspects,
        "average_rating": sum(r.get("rating", 3) for r in reviews) / len(reviews)
    }

# Use the `analyze_sentiment` function to create a tool for sentiment analysis
sentiment_tool = Tool(
    name="review_analysis",
    description="Aspect based sentiment analysis tool that compares the sentiment of reviews by analyzer and rating",
    function=analyze_sentiment,
    parameters={
        "type": "object",
        "properties": {
            "reviews": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "review": {"type": "string"},
                        "rating": {"type": "integer"},
                        "date": {"type": "string"}
                    }
                }
            },
        },
        "required": ["reviews"]
    }
)

Create a tool for summarizing customer reviews. The process involves:

  • Using the LSA (Latent Semantic Analysis) summarizer to identify and extract the most important sentences from each review
  • Creating concise summaries that capture the essence of the reviews
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.lsa import LsaSummarizer


def summarize_reviews(reviews: List[Dict]) -> Dict:
    """
    Summarize the reviews by extracting key sentences.
    """
    summaries = []
    summarizer = LsaSummarizer()
    for review in reviews:
        text = review.get("review", "")
        parser = PlaintextParser.from_string(text, Tokenizer("english"))
        summary = summarizer(parser.document, 2)  # Adjust the number of sentences as needed
        summary_text = " ".join(str(sentence) for sentence in summary)
        summaries.append({"review": text, "summary": summary_text})

    return {"summaries": summaries}

# Create the tool from the `summarize_reviews` function
summarization_tool = Tool(
    name="review_summarization",
    description="Tool to summarize customer reviews by extracting key sentences.",
    function=summarize_reviews,
    parameters={
        "type": "object",
        "properties": {
            "reviews": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "review": {"type": "string"},
                        "rating": {"type": "integer"},
                        "date": {"type": "string"}
                    }
                }
            },
        },
        "required": ["reviews"]
    }
)

Creating an Interactive Feedback Review Agent

We now have the tools to build an interactive agent for customer feedback analysis. The agent dynamically selects the appropriate tool based on user queries, gathers insights based on tool response. The agent then uses the AzureOpenAIChatGenerator to combine the query, retrieved reviews, and tool responses into a comprehensive review analysis.

from haystack.dataclasses import ChatMessage
from haystack.components.generators.chat import AzureOpenAIChatGenerator

def create_review_agent():
    """Creates an interactive review analysis agent"""
    
    chat_generator = AzureOpenAIChatGenerator(
        tools=[sentiment_tool, summarization_tool]
    )
    
    system_message = ChatMessage.from_system(
        """
        You are a customer review analysis expert. Your task is to perform aspect based sentiment analysis on customer reviews.
        You can use two tools to get insights:
        - review_analysis: to get the sentiment of reviews by analyzer and rating
        - review_summarization: to get the summary of reviews.

        Depending on the user's question, use the appropriate tool to get insights and explain them in a helpful way. 
        
        """
    )
    
    return chat_generator, system_message

tool_invoker = ToolInvoker(tools=[sentiment_tool, summarization_tool])

Let’s put our agent to the test with a sample query and see it in action! πŸš€

# Create the review assistant
chat_generator, system_message = create_review_agent()

# Initialize messages with the system message
messages = [system_message]

# Interactive loop for user input
while True:
    user_input = input("\n\nwaiting for input (type 'exit' or 'quit' to stop)\n: ")
    if user_input.lower() == "exit" or user_input.lower() == "quit":
        break
    messages.append(ChatMessage.from_user(user_input))

    print (f"\nπŸ§‘: {user_input}")
    # Build the prompt with user input and reviews
    user_prompt = ChatMessage.from_user(f"""
    {user_input}
    Here are the reviews with analysis:
    {retrieved_reviews}
    """)
    messages.append(user_prompt)

    while True:
        print("βŒ› iterating...")

        replies = chat_generator.run(messages=messages)["replies"]
        messages.extend(replies)

        # Check for tool calls and handle them
        if not replies[0].tool_calls:
            break
        tool_calls = replies[0].tool_calls

        # Print tool calls for debugging
        for tc in tool_calls:
            print("\n TOOL CALL:")
            print(f"\t{tc.tool_name}")

        tool_messages = tool_invoker.run(messages=replies)["tool_messages"]
        messages.extend(tool_messages)

    # Print the final AI response after all tool calls are resolved
    print(f"πŸ€–: {messages[-1].text}")
πŸ§‘: Whats the overall sentiment distribution?
βŒ› iterating...

 TOOL CALL:
	review_analysis








βŒ› iterating...
πŸ€–: The overall sentiment analysis of the customer reviews reveals a predominantly positive sentiment, as evidenced by the following key insights:

### Sentiment Distribution:
1. **Total Reviews Analyzed**: 10
2. **Average Rating**: 4.6 out of 5

### Breakdown by Aspects:
- **Product Quality**: All reviews regarding product quality are rated positively, with an average sentiment analyzer rating of approximately 4.37. Customers expressed satisfaction with shirt designs, quality, and options.
  
- **Shipping**: Shipping also received positive feedback, with reviews indicating quick and efficient shipping times. The sentiment analyzer rating for shipping-related comments is around 4.65, with a majority of users finding the shipping service excellent.

- **Pricing**: Pricing feedback is mixed, with some customers finding the prices reasonable, while others viewed them as somewhat high. The sentiment analyzer rating for pricing stands at approximately 1.73 for the negative feedback (related to concern over the price of children's shirts and shipping costs), indicating that while some customers are satisfied, there is room for improvement.

### Positive Feedback Highlights:
- Many reviews celebrated the quality of the shirts, the variety of options available, and the speed of shipping, with expressions like β€œawesome,” β€œfantastic designs,” and β€œgreat quality material.”

### Negative Feedback Highlights:
- The only notable negative sentiment arises around pricing complaints, particularly regarding perceived high costs for t-shirts and shipping fees.

In summary, the sentiment distribution indicates a strong overall positive reception among customers, especially regarding product quality and shipping, with a slight negative sentiment regarding pricing.

πŸ§‘: Give the summary of reviews as a paragraph.
βŒ› iterating...

 TOOL CALL:
	review_summarization
βŒ› iterating...
πŸ€–: The customer reviews highlight an overwhelmingly positive experience with the shirts purchased, praising aspects such as quick and reasonable shipping, amazing designs, and high product quality. Customers express their love for the shirts, noting great material and design options, while emphasizing that the delivery is timely. Most reviews reflect satisfaction, with one shopper mentioning that everything was great once they figured out the sizing. However, there are some concerns regarding pricing, as one reviewer found the cost of a child's shirt and shipping to be excessive. Overall, the feedback reflects excellent service and product quality, marking a strong recommendation for others.