Building an Interactive Feedback Review Agent with Azure AI Search and Haystack
Last Updated: March 13, 2025
by Amna Mubashar (Haystack), and Khye Wei (Azure AI Search)
This notebook demonstrates how you can build indexing and querying pipelines using Azure AI Search-Haystack integration. Additionally, you’ll develop an interactive feedback review agent leveraging Haystack Tools.
Install the required dependencies
# Install the required dependencies
!pip install haystack-ai==2.10.3
!pip install azure-ai-search-haystack
!pip install jq
!pip install nltk=="3.9.1"
!pip install jsonschema
!pip install kagglehub
Loading and Preparing the Dataset
We will use an open dataset consisting of approx. 28000 customer reviews for a clothing store. The dataset is available at Shopper Sentiments.
We will load the dataset and convert it into a JSON format that can be used by Haystack.
import kagglehub
path = kagglehub.dataset_download("nelgiriyewithana/shoppersentiments")
import pandas as pd
from json import loads, dumps
path = "<Path to the CSV file>"
df = pd.read_csv(path, encoding='latin1', nrows=200) # We are using 200 rows for testing purposes
df.rename(columns={'review-label': 'rating'}, inplace=True)
df['year'] = pd.to_datetime(df['year'], format='%Y %H:%M:%S').dt.year
# Convert DataFrame to JSON
json_data = {"reviews": loads(df.to_json(orient="records"))}
Once we have the JSON data, we can convert it into a Haystack Document format using the JSONConverter
component. Its important to remove any documents with no content as they will not be indexed.
from haystack.components.converters import JSONConverter
from haystack.dataclasses import ByteStream
converter = JSONConverter(
jq_schema=".reviews[]", content_key="review", extra_meta_fields={"store_location", "date", "month", "year", "rating"}
)
source = ByteStream.from_string(dumps(json_data))
documents = converter.run(sources=[source])['documents']
documents = [doc for doc in documents if doc.content is not None] # remove documents with no content
Remove any non-ASCII characters and any regex patterns that are not alphanumeric using the DocumentCleaner
component.
from haystack.components.preprocessors import DocumentCleaner
cleaner = DocumentCleaner(ascii_only=True, remove_regex="i12i12i12")
cleaned_documents=cleaner.run(documents=documents)
Setting up Azure AI Search and Indexing Pipeline
We set up an indexing pipeline with AzureAISearchDocumentStore
by following these steps:
- Configure semantic search for the index
- Initialize the document store with custom metadata fields and semantic search configuration
- Create an indexing pipeline that:
- Generates embeddings for the documents using
AzureOpenAIDocumentEmbedder
- Writes the documents and their embeddings to the search index
- Generates embeddings for the documents using
The semantic configuration allows for more intelligent searching beyond simple keyword matching. Note, the metadata fields need to be declared while creating the index as the API does not allow modifying them after index creation.
from haystack import Pipeline
from haystack.components.embedders import AzureOpenAIDocumentEmbedder
from haystack.components.writers import DocumentWriter
from azure.search.documents.indexes.models import (
SemanticConfiguration,
SemanticField,
SemanticPrioritizedFields,
SemanticSearch
)
from haystack_integrations.document_stores.azure_ai_search import AzureAISearchDocumentStore
semantic_config = SemanticConfiguration(
name="my-semantic-config",
prioritized_fields=SemanticPrioritizedFields(
content_fields=[SemanticField(field_name="content")]
)
)
# Create the semantic settings with the configuration
semantic_search = SemanticSearch(configurations=[semantic_config])
document_store = AzureAISearchDocumentStore(index_name="customer-reviews-analysis", api_key="your_api_key",
endpoint="your_endpoint",
embedding_dimension=1536, metadata_fields = {"month": int, "year": int, "rating": int, "store_location": str}, semantic_search=semantic_search)
# Indexing Pipeline
indexing_pipeline = Pipeline()
indexing_pipeline.add_component(AzureOpenAIDocumentEmbedder(), name="document_embedder")
indexing_pipeline.add_component(instance=DocumentWriter(document_store=document_store), name="doc_writer")
indexing_pipeline.connect("document_embedder", "doc_writer")
indexing_pipeline.run({"document_embedder": {"documents": cleaned_documents["documents"]}})
Embedding Texts: 100%|ββββββββββ| 6/6 [00:29<00:00, 4.91s/it]
{'document_embedder': {'meta': {'model': 'text-embedding-ada-002',
'usage': {'prompt_tokens': 4283, 'total_tokens': 4283}}},
'doc_writer': {'documents_written': 175}}
Creating the Query Pipeline
Here we set up the query pipeline that will retrieve relevant reviews based on user queries. The pipeline consists of:
- A text embedder (
AzureOpenAITextEmbedder
) that converts user queries into embeddings. - A hybrid retriever (
AzureAISearchHybridRetriever
) that uses vector and semantic search to retrieve the most relevant reviews.
from haystack_integrations.components.retrievers.azure_ai_search import AzureAISearchHybridRetriever
from haystack.components.embedders import AzureOpenAITextEmbedder
# Query Pipeline
query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", AzureOpenAITextEmbedder())
query_pipeline.add_component("retriever", AzureAISearchHybridRetriever(document_store=document_store, query_type="semantic", semantic_configuration_name="my-semantic-config", top_k=10))
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
<haystack.core.pipeline.pipeline.Pipeline object at 0x10fe27610>
π
Components
- text_embedder: AzureOpenAITextEmbedder
- retriever: AzureAISearchHybridRetriever
π€οΈ Connections
- text_embedder.embedding -> retriever.query_embedding (List[float])
query = "Which reviews are about shipping?"
# Retrieve reviews based on the query
result = query_pipeline.run({"text_embedder": {"text": query}, "retriever": {"query": query}})
retrieved_reviews = result["retriever"]["documents"]
print(retrieved_reviews)
<iterator object azure.core.paging.ItemPaged at 0x17f829880>
[Document(id=e9f8a141855701896441cbf9fd29ad326ec5250e9263f4ea1f74a5b389d1c90c, content: 'You did everything right! Shipping was quick and reasonable, and the shirts are awesome! Colorful an...', meta: {'store_location': 'US', 'year': 2018, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=a841f950a433d05857933fb7cd46d54a6a04066f1373875359c90f096ce0bf9a, content: 'I love the shirts that I bought.Prices were great and shipping didnt take any lon', meta: {'store_location': 'US', 'year': 2024, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=0ee4d9627e9085936973126762a0aa746b5e235cf253a9f759af84fbfdd9cdae, content: 'Product was great. Love the options. anything I could imagine was available. my only issue was I pai...', meta: {'store_location': 'US', 'year': 2023, 'rating': 4, 'month': 6}, embedding: vector of size 1536), Document(id=31bb2754ad0ebf5084c90260dd45f7c9ea8f5bf5759d3caaf7e97420f8c9610b, content: 'Great shipping time. The shirts look amazing.', meta: {'store_location': 'US', 'year': 2024, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=eb580826a63a596f312f5e2757b25f53d4622fc700cb158b8f6a6b307644d61d, content: 'I love the design, quality of the shirt was great. the print was high quality, shipping was on time ...', meta: {'store_location': 'US', 'year': 2019, 'rating': 4, 'month': 6}, embedding: vector of size 1536), Document(id=351df76a4a52548725cb61803ccb0602379e465a2c0a79e05061f7d7c729054b, content: 'Great designs and quality. Items shipped quickly and correctly.', meta: {'store_location': 'US', 'year': 2024, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=b18855210319bc9b6bf3066a644114425900e95979ad77c1c1dd85e20cbac8b2, content: 'Awesome shirts ,great quality material, fantastic designs,good shipping speed and carefully packed a...', meta: {'store_location': 'US', 'year': 2024, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=6d2ad6c2991516f5c1f7793414996e763a2702ffeceb01b18a61c3225a61bc46, content: 'Once I figured out the sizing, everything was great. FYI, I am a size 10 in womens tops but prefer a...', meta: {'store_location': 'US', 'year': 2018, 'rating': 5, 'month': 6}, embedding: vector of size 1536), Document(id=8798ccd5cde690479c764e8118f1d5423352d13b9d2b8849bc8aed801ae9a9be, content: 'A bit pricey for a tee shirt. Childs size cost $18.00 and outrageous shipping $9.99. No way this cos...', meta: {'store_location': 'US', 'year': 2024, 'rating': 3, 'month': 6}, embedding: vector of size 1536), Document(id=d58901f69050e781ca6f5c78bff61474294e9f0a4b65549bab7b9764f8eed81e, content: 'Delivered ON TIME and shirt is EXTREMELY COMFORTABLE!! You guys are THE BEST!', meta: {'store_location': 'US', 'year': 2018, 'rating': 5, 'month': 6}, embedding: vector of size 1536)]
Create Tools for Sentiment Analysis and Summarization
Install the required dependencies.
!pip install vaderSentiment
!pip install matplotlib
!pip install sumy
Create a function that will be used by review_analysis
tool to visualize the sentiment distribution across customer review aspects (e.g., product quality, shipping). It compares VADER-based sentiment scores with customer ratings using color-coded bars (positive, neutral, negative).
# Function to visualize the sentiment distribution
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
def plot_sentiment_distribution(aspects):
# Create DataFrame from aspects data
data = [(topic, review['sentiment']['analyzer_rating'],
review['review']['rating'], review['sentiment']['label'])
for topic, reviews in aspects.items()
for review in reviews]
df = pd.DataFrame(data, columns=['Topic', 'Normalized Score', 'Original Rating', 'Sentiment'])
# Calculate means
df_means = df.groupby('Topic').agg({
'Normalized Score': 'mean',
'Original Rating': 'mean'
}).reset_index()
fig, ax = plt.subplots(figsize=(8, 4))
x = np.arange(len(df_means))
bar_width = 0.3
# Colors for sentiment
colors = {
'positive': '#2ecc71',
'neutral': '#f1c40f',
'negative': '#e74c3c'
}
# Create bars
sentiment_colors = [colors[df.groupby('Topic')['Sentiment'].agg(lambda x: x.mode()[0])[topic]]
for topic in df_means['Topic']]
bars1 = ax.bar(x - bar_width/2, df_means['Normalized Score'],
bar_width, label='Normalized Score', color=sentiment_colors)
bars2 = ax.bar(x + bar_width/2, df_means['Original Rating'],
bar_width, label='Original Rating', color='gray', alpha=0.7)
# Customize plot with smaller font sizes
ax.set_ylabel('Score', fontsize=9)
ax.set_title('Average Sentiment Scores by Topic', fontsize=10)
ax.set_xticks(x)
ax.set_xticklabels(df_means['Topic'], rotation=45, ha='right', fontsize=8)
ax.tick_params(axis='y', labelsize=8)
# Add value labels with smaller font size
for bars in [bars1, bars2]:
ax.bar_label(bars, fmt='%.2f', padding=3, fontsize=8)
# Smaller legend
ax.legend(handles=[plt.Rectangle((0,0),1,1, color=c) for c in colors.values()] +
[plt.Rectangle((0,0),1,1, color='gray', alpha=0.7)],
labels=list(colors.keys()) + ['Original Rating'],
loc='upper right',
fontsize=8)
plt.tight_layout()
plt.show()
Create a tool to perform aspect-based sentiment analysis on customer reviews using the VADER sentiment analyzer. It involves:
- Identifying specific aspects within reviews (e.g., product quality, shipping, customer service, pricing) using predefined keywords
- Calculating sentiment scores for each review mentioning these aspects
- Categorizing sentiment as ‘positive’, ’negative’, or ’neutral’
- Normalizing sentiment scores to a scale of 1 to 5 for comparison with customer ratings
from haystack.tools import Tool
from haystack.components.tools import ToolInvoker
from typing import Dict, List
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
def analyze_sentiment(reviews: List[Dict]) -> Dict:
"""
Perform aspect-based sentiment analysis.
For each review that mentions keywords related to a specific topic, the function computes
sentiment scores using VADER and categorizes the sentiment as 'positive', 'negative', or 'neutral'.
"""
aspects = {
"product_quality": [],
"shipping": [],
"customer_service": [],
"pricing": []
}
# Define keywords for each topic
keywords = {
"product_quality": ["quality", "material", "design", "fit", "size", "color", "style"],
"shipping": ["shipping", "delivery", "arrived"],
"customer_service": ["service", "support", "help"],
"pricing": ["price", "cost", "expensive", "cheap"]
}
# Initialize the VADER sentiment analyzer
analyzer = SentimentIntensityAnalyzer()
for review in reviews:
text = review.get("review", "").lower()
for topic, words in keywords.items():
if any(word in text for word in words):
# Compute sentiment scores using VADER
sentiment_scores = analyzer.polarity_scores(text)
compound = sentiment_scores['compound']
# Normalize compound score from [-1, 1] to [1, 5]
normalized_score = (compound + 1) * 2 + 1
if compound >= 0.03:
sentiment_label = 'positive'
elif compound <= -0.03:
sentiment_label = 'negative'
else:
sentiment_label = 'neutral'
# Append the review along with its sentiment analysis result
aspects[topic].append({
"review": review,
"sentiment": {
"analyzer_rating": normalized_score,
"label": sentiment_label
}
})
plot_sentiment_distribution(aspects)
return {
"total_reviews": len(reviews),
"sentiment_analysis": aspects,
"average_rating": sum(r.get("rating", 3) for r in reviews) / len(reviews)
}
# Use the `analyze_sentiment` function to create a tool for sentiment analysis
sentiment_tool = Tool(
name="review_analysis",
description="Aspect based sentiment analysis tool that compares the sentiment of reviews by analyzer and rating",
function=analyze_sentiment,
parameters={
"type": "object",
"properties": {
"reviews": {
"type": "array",
"items": {
"type": "object",
"properties": {
"review": {"type": "string"},
"rating": {"type": "integer"},
"date": {"type": "string"}
}
}
},
},
"required": ["reviews"]
}
)
Create a tool for summarizing customer reviews. The process involves:
- Using the LSA (Latent Semantic Analysis) summarizer to identify and extract the most important sentences from each review
- Creating concise summaries that capture the essence of the reviews
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.lsa import LsaSummarizer
def summarize_reviews(reviews: List[Dict]) -> Dict:
"""
Summarize the reviews by extracting key sentences.
"""
summaries = []
summarizer = LsaSummarizer()
for review in reviews:
text = review.get("review", "")
parser = PlaintextParser.from_string(text, Tokenizer("english"))
summary = summarizer(parser.document, 2) # Adjust the number of sentences as needed
summary_text = " ".join(str(sentence) for sentence in summary)
summaries.append({"review": text, "summary": summary_text})
return {"summaries": summaries}
# Create the tool from the `summarize_reviews` function
summarization_tool = Tool(
name="review_summarization",
description="Tool to summarize customer reviews by extracting key sentences.",
function=summarize_reviews,
parameters={
"type": "object",
"properties": {
"reviews": {
"type": "array",
"items": {
"type": "object",
"properties": {
"review": {"type": "string"},
"rating": {"type": "integer"},
"date": {"type": "string"}
}
}
},
},
"required": ["reviews"]
}
)
Creating an Interactive Feedback Review Agent
We now have the tools to build an interactive agent for customer feedback analysis. The agent dynamically selects the appropriate tool based on user queries, gathers insights based on tool response. The agent then uses the AzureOpenAIChatGenerator
to combine the query, retrieved reviews, and tool responses into a comprehensive review analysis.
from haystack.dataclasses import ChatMessage
from haystack.components.generators.chat import AzureOpenAIChatGenerator
def create_review_agent():
"""Creates an interactive review analysis agent"""
chat_generator = AzureOpenAIChatGenerator(
tools=[sentiment_tool, summarization_tool]
)
system_message = ChatMessage.from_system(
"""
You are a customer review analysis expert. Your task is to perform aspect based sentiment analysis on customer reviews.
You can use two tools to get insights:
- review_analysis: to get the sentiment of reviews by analyzer and rating
- review_summarization: to get the summary of reviews.
Depending on the user's question, use the appropriate tool to get insights and explain them in a helpful way.
"""
)
return chat_generator, system_message
tool_invoker = ToolInvoker(tools=[sentiment_tool, summarization_tool])
Let’s put our agent to the test with a sample query and see it in action! π
# Create the review assistant
chat_generator, system_message = create_review_agent()
# Initialize messages with the system message
messages = [system_message]
# Interactive loop for user input
while True:
user_input = input("\n\nwaiting for input (type 'exit' or 'quit' to stop)\n: ")
if user_input.lower() == "exit" or user_input.lower() == "quit":
break
messages.append(ChatMessage.from_user(user_input))
print (f"\nπ§: {user_input}")
# Build the prompt with user input and reviews
user_prompt = ChatMessage.from_user(f"""
{user_input}
Here are the reviews with analysis:
{retrieved_reviews}
""")
messages.append(user_prompt)
while True:
print("β iterating...")
replies = chat_generator.run(messages=messages)["replies"]
messages.extend(replies)
# Check for tool calls and handle them
if not replies[0].tool_calls:
break
tool_calls = replies[0].tool_calls
# Print tool calls for debugging
for tc in tool_calls:
print("\n TOOL CALL:")
print(f"\t{tc.tool_name}")
tool_messages = tool_invoker.run(messages=replies)["tool_messages"]
messages.extend(tool_messages)
# Print the final AI response after all tool calls are resolved
print(f"π€: {messages[-1].text}")
π§: Whats the overall sentiment distribution?
β iterating...
TOOL CALL:
review_analysis
β iterating...
π€: The overall sentiment analysis of the customer reviews reveals a predominantly positive sentiment, as evidenced by the following key insights:
### Sentiment Distribution:
1. **Total Reviews Analyzed**: 10
2. **Average Rating**: 4.6 out of 5
### Breakdown by Aspects:
- **Product Quality**: All reviews regarding product quality are rated positively, with an average sentiment analyzer rating of approximately 4.37. Customers expressed satisfaction with shirt designs, quality, and options.
- **Shipping**: Shipping also received positive feedback, with reviews indicating quick and efficient shipping times. The sentiment analyzer rating for shipping-related comments is around 4.65, with a majority of users finding the shipping service excellent.
- **Pricing**: Pricing feedback is mixed, with some customers finding the prices reasonable, while others viewed them as somewhat high. The sentiment analyzer rating for pricing stands at approximately 1.73 for the negative feedback (related to concern over the price of children's shirts and shipping costs), indicating that while some customers are satisfied, there is room for improvement.
### Positive Feedback Highlights:
- Many reviews celebrated the quality of the shirts, the variety of options available, and the speed of shipping, with expressions like βawesome,β βfantastic designs,β and βgreat quality material.β
### Negative Feedback Highlights:
- The only notable negative sentiment arises around pricing complaints, particularly regarding perceived high costs for t-shirts and shipping fees.
In summary, the sentiment distribution indicates a strong overall positive reception among customers, especially regarding product quality and shipping, with a slight negative sentiment regarding pricing.
π§: Give the summary of reviews as a paragraph.
β iterating...
TOOL CALL:
review_summarization
β iterating...
π€: The customer reviews highlight an overwhelmingly positive experience with the shirts purchased, praising aspects such as quick and reasonable shipping, amazing designs, and high product quality. Customers express their love for the shirts, noting great material and design options, while emphasizing that the delivery is timely. Most reviews reflect satisfaction, with one shopper mentioning that everything was great once they figured out the sizing. However, there are some concerns regarding pricing, as one reviewer found the cost of a child's shirt and shipping to be excessive. Overall, the feedback reflects excellent service and product quality, marking a strong recommendation for others.