# Document Search and Chat

### Step 1- Load the Data and Create chunks

The initial step involves selecting the suitable loader depending on your specific use case. This can include options like PDF, Document, URL, YouTube, subtitles, etc., depending on your requirements. After loading the content, we divide the document into smaller segments. This segmentation is necessary because it enables us to provide users with relevant information from the appropriate chunk when they query any prompt.

Check all the loader list: <https://docs.aiplanet.com/components/document-loaders> and more information on text splitters: <https://docs.aiplanet.com/components/text-splitters>

<figure><img src="https://lh7-us.googleusercontent.com/HkfOHoF7KWbe41pmBLqVMFE-ZExvCR-kCB6fi5V3QIWqngr3c58k9O66W5kQHSKL9nQClt6fRJc67nU4sHqHMheXJ-lumYzqkF2bMUt_lw9CAOGPH2D6P77fkQEtetguFil3HqbAeO2g80EYXDZ4fkA" alt=""><figcaption></figcaption></figure>

### **Step 2- Index your document**

To index a document, it's essential to generate an embedding for each chunk and store it in a vector database. When utilizing a drag to any vector store, ensure the usage of a unique collection name. Enabling the persist option as true eliminates the necessity to recreate indexes for existing content.

We use vector storage to store the embeddings of the document and retrieve them based on search techniques for the given prompt. The Vector Store accommodates various search types, with similarity search being utilized in this instance.

Check the Vector Store documentation: <https://docs.aiplanet.com/components/vector-store>

<figure><img src="https://lh7-us.googleusercontent.com/NkkI9o7eSF3UmafVS4x8tHF2sp6it_ZcXMo5T5iAnB4Rwi4AwTf2_bJ30Dw9gtNbNqBlHJyc9vuVZzxS1v4FpFMo22qqVYKZlTZSbRVrJQAZsr0I7HdcTqrkoa3kpe-0gmyxhmww4a69vfgsla42Kqk" alt=""><figcaption></figcaption></figure>

### **Step 3- Hybrid Search using Ensemble Retriever- Define your Retriever**

To implement an advanced RAG concept, we use a Hybrid Search approach incorporating our custom component, the Ensemble Retriever. This retriever seamlessly combines both the Keyword Retriever and the Vector Store Semantic Retriever, representing a critical step. The Ensemble Retriever plays a pivotal role in retrieving context relevant to the user query.

The Ensemble Retriever necessitates the document to conduct internal keyword searches. Additionally, it incorporates a Vector Store Retriever to enable the connection of both components for Hybrid Search functionality.&#x20;

We have provided detailed documentation outlining the necessity of Hybrid Search: <https://docs.aiplanet.com/terminologies/hybrid-search-ensemble-retriever>

<figure><img src="https://lh7-us.googleusercontent.com/KJxBHQ3Vq6TJxjhoJNMvGpPkDAZ2JRje87O9SEFeK8k1AhQ1b-HJkHJgABKiVSPfAjWI3DyN1vm-aSS62oQLlzJ8J7ZX54kE7gdAx3zWBPLj0cCNxGR5CsHgiVsq9PvQv5FZyIQp9XhLvdgIs8wxwkI" alt=""><figcaption></figcaption></figure>

### **Step 4- Generator Model - Chain your LLM with Retriever**

The Retrieval-Augmented Generation (RAG) pipeline involves two main parts: the Retriever and the Generator. The Retriever finds useful information from the index based on the user's query. Then, this relevant information, along with the prompt, is given to the Large Language Model (LLM), which acts as the generator in the RAG pipeline.

<figure><img src="https://lh7-us.googleusercontent.com/OGDOnR5JUWN9_mEYo6xPLKpHiCxuYa8MVZSL9rqAH6_1eFt6Hl-QeO9deUQwTZrNskL4Q3DhL41OzMoRbOcHDSwdP7ocIocaPu8ntetsootp8odQ-4Oyo12VbPGEn5HELpsejDsMHd5CN-CEUQ1vqfo" alt=""><figcaption></figcaption></figure>

That's it. You can now use Chat Interface to chat with your document.&#x20;