Multimodal
Chunk video, embed clips, search by visual-semantic meaning
The baseline path: split a video into fixed-length segments, embed each segment as multimodal video, and retrieve by natural language.
Use multimodal when you are not sure what end queries will target, or when you want deeper clip-level meaning from the video itself (scene, action, and audiovisual context together) rather than speech or a single frame alone.
Pipeline
POST …/process { chunkDurationSec? }
→ chunk-video (ffmpeg, fixed-length segments)
→ embed-chunk (per segment, via AI Gateway)
→ store vectors in PostgresChunk duration
chunkDurationSec defaults to 15. The start request accepts 5, 10, 15, or 30. Shorter chunks produce more embed requests and finer timing; dollar cost still tracks wall-clock length. Gemini Embedding 2 samples 1 fps for clips up to 32s (then caps at 32 frames), so 5–30s chunks stay in the linear band.
Search
- Admin UI: multimodal search screen
- Phase 2:
POST /search - SDK:
client.search({ query, … })
Hits include startSec, endSec, chunkIndex, similarity score, and sourceObject (bucket + key) for resolving the original file.