Trinetra: Why I Built a Proactive Video Architecture

The Genesis: The Agentic Browser Hurdle

It started with a vision for the ultimate productivity tool: an agentic browser. I spent two months building a highly capable Chromium-based agent—an elevated Python extension designed to navigate the web, analyze screen content, and fill out complex forms autonomously. It worked beautifully on static pages, but I quickly ran into a brick wall: video.

This is the same fundamental challenge industry giants like Google encountered with Project Mariner. To build a truly agentic browser, the system must be able to “watch” a video—whether it’s a software tutorial or a product demo—and extract actionable steps. But I realized that shipping this with a standard Vision-Language Model (VLM) was a non-starter.

Current VLMs follow a reactive paradigm. They re-process the entire video every single time you ask a question. Because of the quadratic $\mathcal{O}(T^{2})$ attention complexity, asking a simple question about an hour-long video costs upwards of $15-$20 and takes minutes of compute. Even the most advanced models struggle with “Lost in the Middle” phenomena and extreme hallucinations when the context gets too long.

The Evolution of Temporal Attention Shift (TAS)

Before building the full engine, I had to solve the “blindness” of 2D vision encoders to time. Standard encoders like SigLIP treat frames as isolated islands. They see a “man with a screwdriver” at $t=1$ and $t=2$, but they don’t see the act of turning. Inspired by MIT’s Temporal Shift Module (TSM), I developed Temporal Attention Shift. While TSM hard-shifts channels, TAS uses a learnable gate $\alpha$ to decide which specific features need temporal context, moving information from the past and future into the current embedding without 3D convolution costs.

$$ TAS_{k}(x) = \alpha_{k} \odot Shift_{k}(x) + (1 - \alpha_{k}) \odot x, \quad \alpha_{k} = \sigma(W_{k}x) $$

A single temporal shift has a fixed “receptive field.” If you shift by 1 frame, you see a finger twitch (gesture), but you miss the 5-second assembly step (action) or the 2-minute scene change. To bridge this gap, I treated the video embedding stream like a multi-resolution signal. By running three TAS streams in parallel—$TAS_2$ (High-freq), $TAS_8$ (Mid-freq), and $TAS_{32}$ (Low-freq)—Trinetra perceives the video through three different “eyes” simultaneously. This wavelet-like decomposition captures everything from micro-movements to narrative shifts in a single $\mathcal{O}(N)$ pass.

graph LR
    In[Input Frame t] --> Split{Split Channels}
    Split --> S1[Shift +1]
    Split --> S2[Shift -1]
    Split --> Ident[Identity]
    S1 & S2 & Ident --> Merge[Merge & Gate]
    Merge --> Out[Temporal Embedding]

The 8-Module Temporal Engine: Solving Long-Form Video

To move from “seeing” to “understanding,” I built a fused engine of eight modules that process the embedding stream during ingest.

graph TD
    In[Embedding Stream] --> M1[1. Discrete Temporal Calculus]
    In --> M2[2. Continuous Time Encoding]
    M1 & M2 --> M3[3. Temporal Dilated Attention]
    M3 --> M4[4. Temporal Memory Tokens]
    M4 --> M5[5. Persistent GRU State]
    M5 --> M6[6. Cross-Frame Gating]
    M6 --> M7[7. Motion-Adaptive Pooling]
    M7 --> M8[8. Audio-Visual Cross-Gating]
    M8 --> Out[Temporal Event Graph]

1. Discrete Temporal Calculus ($\Delta_t$)

Attention often gets bogged down in static scenes. If a camera sits still for 10 minutes, the embeddings are nearly identical, but a reactive model treats them all as equally important. I implemented a first-order derivative that acts as a Temporal Edge Detector. Just as a Sobel operator finds edges in a photo, $\Delta_t$ finds “causal edges” in time. When the magnitude of $\Delta_t$ spikes, the system knows a state-change has occurred, flagging it for the Temporal Event Graph.

$$ \Delta_{t} = MLP([e_{t} || e_{t-1}]) $$

2. Continuous Time Encoding

Frame-based systems lose the “rhythm” of time if the FPS changes or if frames are dropped. I used 384 sinusoidal frequency bands to create a high-resolution “clock” for the embeddings. This allows the model to understand the precise sub-second distance between events, regardless of the sampling density.

graph LR
    T[Time t] --> Sin[sin/cos Freqs]
    Sin --> PE[Positional Embedding]
    E[Frame Embedding] --> Add[Element-wise Add]
    PE --> Add
    Add --> Out[Time-Aware Embedding]

$$ PE(t, 2i) = \sin(t / 10000^{2i/D}), \quad PE(t, 2i+1) = \cos(t / 10000^{2i/D}) $$

3. Temporal Dilated Attention (TDA)

Standard attention is $\mathcal{O}(T^2)$, making it impossible to look back at the beginning of a 2-hour video. TDA applies attention at exponentially increasing dilations ($1, 4, 8, 16$). This mimics human memory: we remember the last few seconds vividly, the last few minutes generally, and the last hour only by its “key frames,” reducing complexity to $\mathcal{O}(T \log T)$.

graph TD
    subgraph TDA [Temporal Dilated Attention]
    F1[Frame t] --- D1((Dilation 1))
    F1 --- D4((Dilation 4))
    F1 --- D16((Dilation 16))
    D1 --- P1[Immediate Past]
    D4 --- P4[Recent History]
    D16 --- P16[Distant Context]
    end

4. Temporal Memory Tokens

Long videos eventually overflow even the best attention windows. I integrated 8 learnable “Memory Tokens” that act as a low-rank summary of the entire video. These tokens use cross-attention to “distill” the history as the video streams by. They provide $\mathcal{O}(1)$ space compression—meaning the memory footprint stays the same whether the video is 5 minutes or 5 hours.

graph LR
    Stream[Embedding Stream] --> CA[Cross Attention]
    MT[Memory Tokens] --> CA
    CA --> MT_new[Updated Summary]
    MT_new -.-> Final[Context Pool]

5. Persistent GRU State ($h_t$)

Attention is an “All-or-Nothing” mechanism; it often lacks a “gut feeling” for narrative flow. A persistent Gated Recurrent Unit (GRU) runs alongside the attention modules. It acts as an Infinite Impulse Response (IIR) filter, maintaining a hidden state $h_t$ that carries the “vibe” or context of the video across every single frame, ensuring narrative continuity.

graph LR
    E[Frame Embedding] --> GRU[GRU Unit]
    H_prev[Hidden State t-1] --> GRU
    GRU --> H_next[Hidden State t]
    H_next --> Narrative[Global Context]

$$ h_{t} = GRU(e_{t}, h_{t-1}) $$

6. Cross-Frame Gating

A common issue in video understanding is “Semantic Flicker”—where the model’s understanding of a scene jumps around due to noise in individual frames. This module calculates a learned gate $g_t$ that controls how much the previous embedding influences the current one. This stabilizes the signal for the LLM, ensuring a coherent narrative output.

graph LR
    Et[Embedding t] --> Gate[Gate Gen]
    Et_1[Embedding t-1] --> Gate
    Gate --> Gt[Gate g_t]
    Et --> Mix[Weighted Sum]
    Et_1 --> Mix
    Gt --> Mix
    Mix --> Stable[Stabilized Signal]

$$ \hat{e}{t} = g{t} \odot e_{t} + (1 - g_{t}) \odot (e_{t-1} + i_{t}), \quad g_{t} = \sigma(MLP([e_{t} || e_{t-1}])) $$

7. Motion-Adaptive Pooling

Processing every frame of a 1-hour video is wasteful, as most frames are essentially “empty.” Using optical flow and the $\Delta_t$ signal, this module scores the “informativeness” of every frame. It drops up to 99% of frames, keeping only the “peaks” of action. This ensures the 500-token context sent to the LLM is high-density signal.

graph TD
    Frames[Video Frames] --> Delta[Delta_t Detector]
    Delta --> Score[Informativeness Score]
    Score --> Filter{Threshold?}
    Filter -- No --> Drop[Discard Frame]
    Filter -- Yes --> Keep[Key Event Frame]
    Keep --> Pool[Semantic Context]

8. Audio-Visual Cross-Gating

In tutorials, the visual (“pointing at a chip”) and the audio (“this is the CPU”) are inseparable. This module projects Whisper and CLAP embeddings into the visual space. It “grounds” the visual events in what is being said, allowing the system to retrieve an event by searching for the meaning of the dialogue associated with it.

graph LR
    V[Visual Stream] --> G[Cross-Gating]
    A[Audio/Whisper Stream] --> P[Projection]
    P --> G
    G --> Out[Grounded Event]

The Result: The Temporal Event Graph (TEG)

The engine outputs its data into a TEG, a “Semantic Compression Ladder.” This structure is built on the principle that information-theoretic entropy must decrease as we move up the hierarchy, mirroring human episodic memory.

$$ H(Frames) \gg H(Events) > H(Chapters) \gg H(Video) $$

graph BT
    F["Level 1: Frames - Max Entropy"] --> E["Level 2: Events - Reduced Entropy"]
    E --> C["Level 3: Chapters - Low Entropy"]
    C --> V["Level 4: Video - Min Entropy"]

Why This Wins

When we tested this on a consumer RTX 3050 (4GB), the results were structural, not just incremental. By moving from reactive “re-watching” to proactive “indexing,” we achieved 317,500x fewer query-time FLOPs and a 9x improvement in temporal awareness keywords. Trinetra proves that for the multi-query regime, the proactive paradigm is the only way forward. Ingest once, query forever.

For the full technical details, including FLOP measurements and the formal complexity proof, refer to the Trinetra Preprint.