Context Grabber Operator

Overview

The Context Grabber LOP serves as a flexible aggregator for collecting various types of contextual information within a TouchDesigner network. It can gather images from TOPs, text content from DATs, custom text snippets, and links to external media files (PDF, Video, Audio). This collected context is then formatted into a structure suitable for injection into prompts used by other LOPs like the Agent Operator or Gemini Image Gen.

How it Works

This operator doesn’t have direct inputs or outputs for data flow. Instead, other LOPs (like Agent) typically reference this operator via an OP parameter and call its internal GetContext() method to retrieve the formatted context information just before making an API call.

Requirements

• Python Packages:
  • opencv-python (cv2): Used internally for image processing (converting TOPs to base64).
  • numpy: Required by OpenCV.
  • Install these via the ChatTD Python Manager.
• ChatTD Operator: Required for some base utilities and logging. Ensure the ChatTD Operator parameter on the ‘About’ page points to your configured ChatTD instance.

Input/Output

Inputs

None (Context sources are specified via parameters).

Outputs

None (Context is retrieved programmatically via its GetContext() method).

GetContext() Return Structure

When another LOP calls op('context_grabber1').ext.GrabOpContextEXT.GetContext(), it returns a dictionary containing:

• formatted_context (list): A list of dictionaries, each representing a message to be inserted into the prompt (e.g., {"role": "system", "content": "..."}). The role and content depend on the Context Insert Mode parameter and the collected text/DAT/message content.
• additional_images (list or None): If Add Images to User is enabled, this is a list of dictionaries formatted for LLM image input (e.g., {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,...", "detail": "auto"}}). Otherwise, it’s None.
• additional_files (list or None): If PDF, Video, or Audio URLs are added, this is a list of dictionaries formatted for file input (e.g., {"type": "file", "file": {"file_id": "https://...", "format": "application/pdf"}}). Otherwise, it’s None.

Parameters

Page: Grab Context From Ops

Context Insert Mode (Contextinsertmode) op('context_grabber').par.Contextinsertmode Menu

Default:

User Message

Options:

System Message, User Message, Assistant Message, Split

Split Context Location (Splitcontextlocation) op('context_grabber').par.Splitcontextlocation Menu

Default:

Start

Options:

Start, End, Both

Print Context (Printcontext) op('context_grabber').par.Printcontext Pulse

Default:

None

Add Images to User (Addimagestouser) op('context_grabber').par.Addimagestouser Toggle

Default:

0

Options:

off, on

Image Detail Level (Imagedetail) op('context_grabber').par.Imagedetail Menu

Default:

auto

Options:

auto, low, high

Images (Images) op('context_grabber').par.Images Sequence

Default:

None

TOP (Images0top) op('context_grabber').par.Images0top TOP

Default:

None

Label (Images0label) op('context_grabber').par.Images0label Str

Default:

None

Add DATs to Context (Adddattocontext) op('context_grabber').par.Adddattocontext Toggle

Default:

0

Options:

off, on

Text or Table DAT (Datop) op('context_grabber').par.Datop Sequence

Default:

None

DAT (Datop0dat) op('context_grabber').par.Datop0dat OP

Default:

None

Label (Datop0label) op('context_grabber').par.Datop0label Str

Default:

None

Add Text to Context (Addtexttocontext) op('context_grabber').par.Addtexttocontext Toggle

Default:

0

Options:

off, on

Messages (Text) op('context_grabber').par.Text Sequence

Default:

None

Text (Text0text) op('context_grabber').par.Text0text Str

Default:

None

Label (Text0label) op('context_grabber').par.Text0label Str

Default:

None

Add PDF to User (Addpdftouser) op('context_grabber').par.Addpdftouser Toggle

Default:

0

Options:

off, on

PDF (Pdf) op('context_grabber').par.Pdf Sequence

Default:

None

PDF URL (Pdf0url) op('context_grabber').par.Pdf0url Str

Default:

None

Add Video to User (Addvideotouser) op('context_grabber').par.Addvideotouser Toggle

Default:

0

Options:

off, on

Video (Video) op('context_grabber').par.Video Sequence

Default:

None

Video URL (Video0url) op('context_grabber').par.Video0url Str

Default:

None

Add Audio to User (Addaudiotouser) op('context_grabber').par.Addaudiotouser Toggle

Default:

0

Options:

off, on

Audio (Audio) op('context_grabber').par.Audio Sequence

Default:

None

Audio URL (Audio0url) op('context_grabber').par.Audio0url Str

Default:

None

Page: About

Bypass (Bypass) op('context_grabber').par.Bypass Toggle

Default:

0

Options:

off, on

Show Built-in Parameters (Showbuiltin) op('context_grabber').par.Showbuiltin Toggle

Default:

0

Options:

off, on

Version (Version) op('context_grabber').par.Version Str

Default:

1.0.0

Last Updated (Lastupdated) op('context_grabber').par.Lastupdated Str

Default:

2025-02-23

Creator (Creator) op('context_grabber').par.Creator Str

Default:

dotsimulate

Website (Website) op('context_grabber').par.Website Str

Default:

https://dotsimulate.com

ChatTD Operator (Chattd) op('context_grabber').par.Chattd OP

Default:

/dot_lops/ChatTD

Clear Log (Clearlog) op('context_grabber').par.Clearlog Pulse

Default:

None

Usage Examples

Linking to Agent Operator

1. Create a Context Grabber operator (e.g., named 'context_grabber1').
2. Configure the Context Grabber:
   - Add an image sequence block pointing to a TOP (e.g., `moviefilein1`).
   - Add a DAT sequence block pointing to a Table DAT (e.g., `table1`).
   - Enable 'Add Images to User' and 'Add DATs to Context'.
   - Set 'Context Insert Mode' (e.g., 'System Message').
3. Create an Agent operator (e.g., 'agent1').
4. On the Agent operator's 'Context' parameter page, drag 'context_grabber1' onto the 'Context Op' parameter.
5. When 'agent1' makes an LLM call, it will automatically call 'context_grabber1.GetContext()' and include the formatted context (system message with table data) and the image (added to the user message).

Debugging Context Output

1. Configure the Context Grabber with desired images, DATs, text, etc.
2. Set the 'Context Insert Mode' and other formatting options.
3. Pulse the 'Print Context' parameter.
4. Check the TouchDesigner textport (Alt+T) to see the exact structure of messages and file/image lists that would be generated.

Technical Notes

• Context Retrieval: This operator is passive; other LOPs must actively call its GetContext() method to retrieve the formatted data.
• Image Encoding: Images from TOPs are converted to base64 encoded JPEG data URIs for inclusion in prompts.
• Context Formatting: The Context Insert Mode and Split Context Location parameters control how textual/DAT/message context is wrapped in message dictionaries (role/content pairs).
• Media URLs: URLs provided for PDF, Video, and Audio must be publicly accessible via HTTPS for the LLM to potentially retrieve them. Model support for processing these file types varies.
• Sequences: Parameters like Images, Datop, Text, Pdf, Video, Audio are sequences. Use the + button on the parameter dialog to add more blocks (e.g., Images1top, Images1label, etc.) to include multiple items of that type.

Related Operators

• Agent Operator: Often consumes the context generated by this operator.
• Gemini Image Gen: Can use this operator via its Context Grabber (Optional) parameter.
• ChatTD: Provides base utilities.