# Document Search and Chat

### Step 1- Load the Data and Create chunks

The initial step involves selecting the suitable loader depending on your specific use case. This can include options like PDF, Document, URL, YouTube, subtitles, etc., depending on your requirements. After loading the content, we divide the document into smaller segments. This segmentation is necessary because it enables us to provide users with relevant information from the appropriate chunk when they query any prompt.

Check all the loader list: <https://docs.aiplanet.com/components/document-loaders> and more information on text splitters: <https://docs.aiplanet.com/components/text-splitters>

<figure><img src="https://lh7-us.googleusercontent.com/HkfOHoF7KWbe41pmBLqVMFE-ZExvCR-kCB6fi5V3QIWqngr3c58k9O66W5kQHSKL9nQClt6fRJc67nU4sHqHMheXJ-lumYzqkF2bMUt_lw9CAOGPH2D6P77fkQEtetguFil3HqbAeO2g80EYXDZ4fkA" alt=""><figcaption></figcaption></figure>

### **Step 2- Index your document**

To index a document, it's essential to generate an embedding for each chunk and store it in a vector database. When utilizing a drag to any vector store, ensure the usage of a unique collection name. Enabling the persist option as true eliminates the necessity to recreate indexes for existing content.

We use vector storage to store the embeddings of the document and retrieve them based on search techniques for the given prompt. The Vector Store accommodates various search types, with similarity search being utilized in this instance.

Check the Vector Store documentation: <https://docs.aiplanet.com/components/vector-store>

<figure><img src="https://lh7-us.googleusercontent.com/NkkI9o7eSF3UmafVS4x8tHF2sp6it_ZcXMo5T5iAnB4Rwi4AwTf2_bJ30Dw9gtNbNqBlHJyc9vuVZzxS1v4FpFMo22qqVYKZlTZSbRVrJQAZsr0I7HdcTqrkoa3kpe-0gmyxhmww4a69vfgsla42Kqk" alt=""><figcaption></figcaption></figure>

### **Step 3- Hybrid Search using Ensemble Retriever- Define your Retriever**

To implement an advanced RAG concept, we use a Hybrid Search approach incorporating our custom component, the Ensemble Retriever. This retriever seamlessly combines both the Keyword Retriever and the Vector Store Semantic Retriever, representing a critical step. The Ensemble Retriever plays a pivotal role in retrieving context relevant to the user query.

The Ensemble Retriever necessitates the document to conduct internal keyword searches. Additionally, it incorporates a Vector Store Retriever to enable the connection of both components for Hybrid Search functionality.&#x20;

We have provided detailed documentation outlining the necessity of Hybrid Search: <https://docs.aiplanet.com/terminologies/hybrid-search-ensemble-retriever>

<figure><img src="https://lh7-us.googleusercontent.com/KJxBHQ3Vq6TJxjhoJNMvGpPkDAZ2JRje87O9SEFeK8k1AhQ1b-HJkHJgABKiVSPfAjWI3DyN1vm-aSS62oQLlzJ8J7ZX54kE7gdAx3zWBPLj0cCNxGR5CsHgiVsq9PvQv5FZyIQp9XhLvdgIs8wxwkI" alt=""><figcaption></figcaption></figure>

### **Step 4- Generator Model - Chain your LLM with Retriever**

The Retrieval-Augmented Generation (RAG) pipeline involves two main parts: the Retriever and the Generator. The Retriever finds useful information from the index based on the user's query. Then, this relevant information, along with the prompt, is given to the Large Language Model (LLM), which acts as the generator in the RAG pipeline.

<figure><img src="https://lh7-us.googleusercontent.com/OGDOnR5JUWN9_mEYo6xPLKpHiCxuYa8MVZSL9rqAH6_1eFt6Hl-QeO9deUQwTZrNskL4Q3DhL41OzMoRbOcHDSwdP7ocIocaPu8ntetsootp8odQ-4Oyo12VbPGEn5HELpsejDsMHd5CN-CEUQ1vqfo" alt=""><figcaption></figcaption></figure>

That's it. You can now use Chat Interface to chat with your document.&#x20;


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.aiplanet.com/genai-stack-1/usecases/document-search-and-chat.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.