RAG Index Operator

The RAG Index operator is designed to create and manage vector store indices from various document sources. These indices are crucial for Retrieval Augmented Generation (RAG) workflows, allowing LLMs to retrieve relevant information from a knowledge base before generating responses. It supports different input modes (document tables, folders) and embedding models, and provides options for chunking, saving, and loading indices.

Requirements

• Python Packages:
  • llama-index
  • tiktoken
  • torch (with CUDA support for local embeddings on GPU)
• ChatTD Operator: Required for asynchronous task execution and embedding model management.

Input/Output

Inputs

• Input 1 (Document Table, optional): A Table DAT containing document data with columns like doc_id, filename, content, metadata. Used when Input Mode is Doc Table or Auto Detect.

Outputs

• Documents Table (Table DAT): Stores information about the processed documents, including their content, metadata, and hash.
• Index Info Table (Table DAT): Provides details about created and loaded indices, such as name, number of nodes, creation timestamp, and status.

Parameters

Page: Index

Create Index (Createindex) op('rag_index').par.Createindex Pulse

Default:

None

Index Name (Indexname) op('rag_index').par.Indexname Str

Default:

None

Input Mode (Inputmode) op('rag_index').par.Inputmode Menu

Default:

auto

Document Folder (Documentfolder) op('rag_index').par.Documentfolder Folder

Default:

None

File Pattern (Filepattern) op('rag_index').par.Filepattern Str

Default:

None

Current Status (Status) op('rag_index').par.Status Str

Default:

None

Active Index (Activeindex) op('rag_index').par.Activeindex Toggle

Default:

false

Embedding Model (Embedmodel) op('rag_index').par.Embedmodel Menu

Default:

local

Ollama Model (Ollamamodel) op('rag_index').par.Ollamamodel StrMenu

Default:

nomic-embed-text

Chunk Size (Chunksize) op('rag_index').par.Chunksize Int

Default:

1024

Chunk Overlap (Chunkoverlap) op('rag_index').par.Chunkoverlap Int

Default:

20

Sync to File (Savetofile) op('rag_index').par.Savetofile Toggle

Default:

false

Index Folder (Indexfolder) op('rag_index').par.Indexfolder Folder

Default:

None

Save Index (Saveindex) op('rag_index').par.Saveindex Pulse

Default:

None

Load Index (Loadindex) op('rag_index').par.Loadindex Pulse

Default:

None

Load on Start (Loadonstart) op('rag_index').par.Loadonstart Toggle

Default:

false

Clear All (Clearall) op('rag_index').par.Clearall Pulse

Default:

None

Header

Progress (Progress) op('rag_index').par.Progress Float

Default:

0.0

Stop Index Creation (Stopindex) op('rag_index').par.Stopindex Pulse

Default:

None

Page: About

ChatTD (Chattd) op('rag_index').par.Chattd OP

Default:

None

Show Built In Pars (Showbuiltin) op('rag_index').par.Showbuiltin Toggle

Default:

false

Bypass (Bypass) op('rag_index').par.Bypass Toggle

Default:

false

Version (Version) op('rag_index').par.Version Str

Default:

None

Last Updated (Lastupdated) op('rag_index').par.Lastupdated Str

Default:

None

Creator (Creator) op('rag_index').par.Creator Str

Default:

None

Website (Website) op('rag_index').par.Website Str

Default:

None

Usage Examples

Creating an Index from a Folder of Documents

1. Set Document Folder to the path containing your text or markdown files.
2. Set File Pattern to *.txt *.md (or other relevant patterns).
3. Choose your Embedding Model (e.g., Local Embed).
4. Adjust Chunk Size and Chunk Overlap as needed.
5. Pulse Create Index.
6. Monitor the Status and Progress parameters.

Saving and Loading an Index

1. After creating an index, ensure Sync to File is enabled or pulse Save Index.
2. Set Index Folder to your desired save location.
3. To load an existing index, set Index Folder and pulse Load Index (or enable Load on Start).

Technical Notes

• This operator leverages the llama-index library for RAG functionalities.
• Index creation can be resource-intensive, especially for large document sets or complex embedding models.
• The ChatTD operator is essential for managing Python dependencies and providing access to embedding models.
• Chunking parameters (Chunk Size, Chunk Overlap) significantly impact the quality of retrieval. Experiment with these values based on your document content.

Related Operators

• RAG Retriever: Used to query the created index.
• ChatTD: Provides core services and embedding models.
• Source Ops: Can be used to generate document tables for input.