Skip to content
LOPS Docs

Rag Index Operator

Overview

Section titled “Overview”

The Rag Index operator is designed to create and manage vector store indices from documents. It allows you to ingest data from various sources, chunk the data, generate embeddings using different models, and store the index for later use. This operator is essential for building Retrieval Augmented Generation (RAG) pipelines within TouchDesigner, enabling you to query your documents efficiently.

Note: This operator requires the llama-index Python library and its dependencies.

Parameters

Section titled “Parameters”

Parameters are organized into pages.

Create Index (Createindex) op('rag_index').par.Createindex Pulse
Default:
None
Stop Index Creation (Stopindex) op('rag_index').par.Stopindex Pulse
Default:
None
Progress (Progress) op('rag_index').par.Progress Float
Default:
0
Current Status (Status) op('rag_index').par.Status Str
Default:
"" (Empty String)
Clear All (Clearall) op('rag_index').par.Clearall Pulse
Default:
None
Input Mode (Inputmode) op('rag_index').par.Inputmode Menu
Default:
doctable
Options:
auto, doctable, folder
Document Folder (Documentfolder) op('rag_index').par.Documentfolder Folder
Default:
None
File Pattern (Filepattern) op('rag_index').par.Filepattern Str
Default:
*.txt *.md
Index Name (Indexname) op('rag_index').par.Indexname Str
Default:
local_docs
Embedding Model (Embedmodel) op('rag_index').par.Embedmodel Menu
Default:
local
Options:
local, openai, ollama
Chunk Size (Chunksize) op('rag_index').par.Chunksize Int
Default:
1024
Range:
100 to 8192
Slider Range:
512 to 2048
Chunk Overlap (Chunkoverlap) op('rag_index').par.Chunkoverlap Int
Default:
20
Range:
0 to 1024
Slider Range:
0 to 200
Sync to File (Savetofile) op('rag_index').par.Savetofile Toggle
Default:
Off
Index Folder (Indexfolder) op('rag_index').par.Indexfolder Folder
Default:
None
Save Index (Saveindex) op('rag_index').par.Saveindex Pulse
Default:
None
Load Index (Loadindex) op('rag_index').par.Loadindex Pulse
Default:
None
Load on Start (Loadonstart) op('rag_index').par.Loadonstart Toggle
Default:
Off
ChatTD (Chattd) op('rag_index').par.Chattd OP
Default:
/dot_lops/ChatTD
Active Index (Activeindex) op('rag_index').par.Activeindex Toggle
Default:
Off
Show Built In Pars (Showbuiltin) op('rag_index').par.Showbuiltin Toggle
Default:
Off
Bypass (Bypass) op('rag_index').par.Bypass Toggle
Default:
Off

Callbacks

Section titled “Callbacks”
Available Callbacks:
  • onIndexCreated
  • onIndexLoaded
  • onIndexSaved
  • onIndexCleared
  • onError
  • onStatusUpdate

Usage Examples

Section titled “Usage Examples”

Creating an Index from a Folder

Section titled “Creating an Index from a Folder”
1. Set 'Input Mode' to `folder`.
2. Drag desired folder onto 'Document Folder'.
3. Set 'File Pattern' (e.g., `*.txt *.md`).
4. Select 'Embedding Model' source.
5. Adjust 'Chunk Size' / 'Chunk Overlap' if needed.
6. Pulse 'Create Index'. Monitor 'Progress' and 'Status'.

Creating an Index from a DAT Table

Section titled “Creating an Index from a DAT Table”
1. Prepare a DAT Table with columns: `doc_id`, `filename`, `content`, `metadata`.
2. Connect DAT to Input 1.
3. Set 'Input Mode' to `doctable` or `auto`.
4. Select 'Embedding Model' source.
5. Adjust 'Chunk Size' / 'Chunk Overlap' if needed.
6. Pulse 'Create Index'.

Saving and Loading

Section titled “Saving and Loading”
1. Create index as above.
2. (Optional) Turn 'Sync to File' On.
3. Set 'Index Folder'.
4. Pulse 'Save Index' to save manually.
5. To load: Set 'Index Folder' correctly. Pulse 'Load Index'.
6. For auto-load: Turn 'Sync to File' and 'Load on Start' On. Index loads when project starts.

Technical Notes

Section titled “Technical Notes”
  • Index creation can be computationally intensive, especially with large document sets. Local embedding models may require significant RAM/VRAM.
  • The actual embedding process relies on configurations within the linked ChatTD component. Ensure API keys or local model paths are correctly set up there.
  • Saved indices are stored in a subfolder named after the Index Name within the specified Index Folder.
  • Clearing the index (Clear All) removes it from memory but does not delete the saved files on disk.
Section titled “Related Operators”