Haystack docs home page


The following sections show you how to get from a raw file to a cleaned Document that is ready to be indexed in a DocumentStore.

Haystack includes a suite of tools to:

  • extract text from different file types,
  • normalize white space
  • split text into smaller pieces to optimize retrieval

These data preprocessing steps can have a big impact on the systems performance and effective handling of data is key to getting the most out of Haystack.

Check out our preprocessing tutorial if you'd like to start working with code examples already!

Document Format

You can index Document objects (recommended) or plain dictionaries into a Haystack DocumentStore. The output of your preprocessing should comply with one of these two formats. To learn more about the Document class see Primitives.

# Option 1: Native Haystack Documents
docs = [
meta={'name': DOCUMENT_NAME, ...}
), ...
# Option 2: Plain dict
docs = [
'content': 'DOCUMENT_TEXT_HERE',
'meta': {'name': DOCUMENT_NAME, ...}
}, ...

File Conversion

There are a range of different file converters in Haystack that can extract text from files and cast them into the unified dictionary format shown above. Haystack features support for txt, pdf and docx files and there is even a converter that leverages Apache Tika. Please refer to the API docs to see which converter best suits you.

Converters are Haystack Nodes and can be added to Pipelines. See our Advanced Indexing Pipeline Tutorial for an example.

Web Crawler

In Haystack, you will find a web crawler that will help you scrape text from websites and save it to file. See the API documentation for more details.

from haystack.nodes import Crawler
crawler = Crawler(output_dir="crawled_files")
docs = crawler.crawl(


While each of the above conversion methods produce documents that are already in the format expected by the Document Store, it is recommended that they are further processed in order to ensure optimal Retriever and Reader performance. The PreProcessor takes one of the documents created by the converter as input, performs various cleaning steps and splits them into multiple smaller documents.

For suggestions on how best to split your documents, see Optimization

from haystack.nodes import PreProcessor
doc = converter.convert(file_path=file, meta=None)
processor = PreProcessor(
docs = processor.process(doc)
  • clean_empty_lines will normalize 3 or more consecutive empty lines to be just a two empty lines
  • clean_whitespace will remove any whitespace at the beginning or end of each line in the text
  • clean_header_footer will remove any long header or footer texts that are repeated on each page
  • split_by determines what unit the document is split by: 'word', 'sentence' or 'passage'
  • split_length sets a maximum number of 'word', 'sentence' or 'passage' units per output document
  • split_respect_sentence_boundary ensures that document boundaries do not fall in the middle of sentences
  • split_overlap sets the amount of overlap between two adjacent documents after a split. Setting this to a positive number essentially enables the sliding window approach.

PreProcessors are Haystack Nodes and can be added to Pipelines. See our Advanced Indexing Pipeline Tutorial for an example.