Arun Baby

Becoming Unlabelable

7 minute read

“Time: The 4th Dimension of Vision.”

1. Introduction: From Static to Dynamic

So far, we have looked at Computer Vision as a series of snapshots. “Here is a screenshot—find the button.” “Here is a photo—find the cat.” But the real world is Temporal. Reality does not exist in a single JPEG; it flows.

For an agent to operate in the physical world (Robotics, Autonomous Driving, Security) or the digital video world (YouTube Analysis), it must understand Time.

  • Action Recognition: A static image of a person with their hand raised is ambiguous. Are they waving hello? Are they hailing a taxi? Are they stretching? You need the previous 10 frames to know the motion vector.
  • Physics Prediction: If a ball is in the air, where will it land? You need to observe its trajectory across frames to estimate velocity and gravity.
  • Causality: “The cup fell because the cat pushed it.” Causality requires observing the sequence Event A -> Event B.

In this final post of the Vision module, we will explore the architecture of Video Agents: How to process massive streams of visual data efficiently, track objects across time, and perceive depth in a 3D world.

2. The Computational Cost of Time

Video is the heaviest modality in AI.

  • Math:
    • 1 Image (1080p) $\approx$ 6MB uncompressed.
    • 1 Second of Video (30fps) $\approx$ 180MB raw data.
    • 1 Hour of Video $\approx$ 600GB raw data.
    • Even compressed (H.264), it is massive.

We cannot simply feed a raw 30fps video stream into a Vision Transformer or GPT-4V.

  • Cost: GPT-4V charges ~$0.01 per frame. Processing 1 second (30 frames) would cost $0.30. Processing 1 minute would cost $18. This is economically impossible for most use cases.
  • Latency: Processing 30 images takes ~30 seconds of compute. The agent would lag behind reality instantly.

2.1 Strategy: Keyframe Extraction (Smart Sampling)

The most common technique is to not watch the video. We watch a summary.

  • Uniform Sampling: Only take 1 frame every second (1 FPS). For most human actions (walking, cooking), 1 FPS captures enough semantics.
  • Content Adaptive Sampling: We compare Frame $T$ and Frame $T-1$. We calculate the Pixel Difference (or Histogram Difference).
    • If Diff < Threshold: The scene is static (e.g., an empty hallway). Discard Frame $T$.
    • If Diff > Threshold: Something moved. Keep Frame $T$.
    • Result: A Security Camera Agent might process 0 frames for 2 hours, then burst into processing 10 FPS when a person walks in.

2.2 Strategy: Visual Embeddings (Video-LLaMA)

Instead of Tokenizing pixels (which is verbose), we use specialized Video Encoders (like VideoMAE or X-CLIP).

  • Input: A 2-second clip (60 frames).
  • Mechanism: These models use “Spatiotemporal Attention”. They compress the redundancy. (The wall background didn’t change for 60 frames, so don’t encode it 60 times).
  • Output: A single dense vector representing “A man walking a dog”.
  • The LLM reasoning happens on these high-level vectors, allowing it to “watch” hours of video by processing just a sequence of compressed vectors.

3. Object Tracking (DEVA / SORT)

In a single image, we detect “Car”. In the next image, we detect “Car”. How does the agent know it is the same car? This is the Identity Persistence problem. Without it, an agent counting cars on a highway would count the same car 30 times as it drives across 30 frames.

3.1 The Logic: Detection vs Tracking

  • Detection (YOLO): “There is a car at (100, 100).” (Independent per frame).
  • Tracking: “Object ID #45 (Car) moved from (100, 100) to (110, 100).”

3.2 Algorithms: SORT and DEVA

  1. SORT (Simple Online and Realtime Tracking):
    • IOU Matching: If Box A in Frame 1 overlaps 90% with Box B in Frame 2, they are likely the same object.
    • Kalman Filter: This is the magic. It uses physics. If the car was moving Right at 10px/frame, the filter predicts where the box should be in Frame 3.
    • Occlusion Handling: If the car drives behind a tree (detection fails), the Kalman filter still “hallucinates” the predicted box for a few frames. When the car re-emerges, the tracker re-locks onto it, preserving ID #45.
  2. DEVA (Decoupled Video Segmentation):
    • A modern foundation model that propagates “Masks” across time.
    • Usage: You click a person in Frame 1 (SAM prompt). DEVA automatically segments that person in the next 1,000 frames using optical flow, even as they turn around or change shape.

4. 3D Perception and Depth

Our cameras detect 2D pixels (x, y, color). But the world is 3D (x, y, z). Information is lost during this projection.

  • Illusion: A small toy car 1 meter away looks identical to a real car 50 meters away.
  • Danger: An autonomous drone agent needs to know the difference, or it will crash.

4.1 Monocular Depth Estimation

Humans use two eyes (Stereo) for depth. Can a single camera see depth? Yes, using context. If a car is “floating” above the horizon line, it’s far away. Models like Depth Anything or MiDaS are trained to predict a Depth Map (a heatmap where color = distance) from a single RGB image.

  • Agent Input: RGB Image.
  • Process: Run Depth Anything.
  • Agent Logic: “The Pixels for the cup have a depth value of 0.5m. The pixels for the wall have 3.0m. Therefore the cup is reachable.”

4.2 SLAM (Simultaneous Localization and Mapping)

For moving agents (drones/robots), sensing isn’t enough. They need a Map. vSLAM (Visual SLAM) algorithms process the video stream to build a persistent 3D Point Cloud of the environment.

  • The agent builds a mental model: “I am at coordinate (x,y,z) in the Kitchen.”
  • This allows “Memory of Place”. If you tell the agent “Go to the fridge”, it knows the 3D coordinates of the fridge from a previous conceptual map building session.

5. Deployment: The Edge vs. Cloud Trade-off

Video Agents represent the extreme edge of “Edge AI”.

  • Bandwidth: You physically cannot stream 6 cameras of 4K video to AWS. The uplink bandwidth doesn’t exist.
  • Latency: A self-driving car cannot wait 500ms for the cloud to say “Stop”.
  • Privacy: People do not want their home security feeds or factory floor footage sent to OpenAI.

The Hybrid Architecture:

  • The Edge Device (NVIDIA Jetson / Orin / Raspberry Pi):
    • Runs the high-frequency loops.
    • Runs YOLO (Detection) and SORT (Tracking) at 30 FPS.
    • Runs simple logic: “If Person enters Red Zone…”
  • The Cloud (LLM):
    • Receives only Events and Snapshots.
    • Edge Trigger: “Person detected.” -> Sends 1 JPG to Cloud.
    • Cloud Reasoning: “Is this person wearing a uniform? Yes. Ignore.”

This split—Perception on Edge, Reasoning in Cloud—is the only viable way to build scalable Video Agents today.

6. Summary

Video Analysis adds the dimension of Time to our agents, enabling them to understand Cause, Effect, and Physics.

  • Sampling: Smart keyframing is essential to manage data volume.
  • Tracking: Assigns identities to objects across time using IOU and Kalman Filters.
  • Depth: Recovers the missing Z-axis using Monocular Depth estimation.
  • Edge AI: The necessary infrastructure to run these heavy perception loops locally.

This concludes the Vision module of our curriculum. We have given our agents Eyes (ViT), Semantic Understanding (CLIP), UI Reading (OmniParser), Detection (YOLO), and Temporal Perception (Video). Our agents can now Talk, Listen, and See. In the next section, we move to Orchestration & Reliability (Days 26-30), focusing on how to make these complex systems robust in production.

Related across topics

Model Monitoring Systems

15 minute read

The silent killer of ML models is not a bug in the code, but a change in the world.