Document Search and Chat

In this use case, we will implement a Chat with PDF. The process involves searching for a pertinent document within the collection based on the user query and subsequently utilizing the Chat Interface

Step 1- Load the Data and Create chunks

The initial step involves selecting the suitable loader depending on your specific use case. This can include options like PDF, Document, URL, YouTube, subtitles, etc., depending on your requirements. After loading the content, we divide the document into smaller segments. This segmentation is necessary because it enables us to provide users with relevant information from the appropriate chunk when they query any prompt.

Check all the loader list: https://docs.aiplanet.com/components/document-loaders and more information on text splitters: https://docs.aiplanet.com/components/text-splitters

Step 2- Index your document

To index a document, it's essential to generate an embedding for each chunk and store it in a vector database. When utilizing a drag to any vector store, ensure the usage of a unique collection name. Enabling the persist option as true eliminates the necessity to recreate indexes for existing content.

We use vector storage to store the embeddings of the document and retrieve them based on search techniques for the given prompt. The Vector Store accommodates various search types, with similarity search being utilized in this instance.

Check the Vector Store documentation: https://docs.aiplanet.com/components/vector-store

Step 3- Hybrid Search using Ensemble Retriever- Define your Retriever

To implement an advanced RAG concept, we use a Hybrid Search approach incorporating our custom component, the Ensemble Retriever. This retriever seamlessly combines both the Keyword Retriever and the Vector Store Semantic Retriever, representing a critical step. The Ensemble Retriever plays a pivotal role in retrieving context relevant to the user query.

The Ensemble Retriever necessitates the document to conduct internal keyword searches. Additionally, it incorporates a Vector Store Retriever to enable the connection of both components for Hybrid Search functionality.

We have provided detailed documentation outlining the necessity of Hybrid Search: https://docs.aiplanet.com/terminologies/hybrid-search-ensemble-retriever

Step 4- Generator Model - Chain your LLM with Retriever

The Retrieval-Augmented Generation (RAG) pipeline involves two main parts: the Retriever and the Generator. The Retriever finds useful information from the index based on the user's query. Then, this relevant information, along with the prompt, is given to the Large Language Model (LLM), which acts as the generator in the RAG pipeline.

That's it. You can now use Chat Interface to chat with your document.