Multimodal
Chunk video, embed clips, search by visual-semantic meaning
The baseline path: split a video into fixed-length segments, embed each segment as multimodal video, and retrieve by natural language.
Use multimodal when you are not sure what end queries will target, or when you want deeper clip-level meaning from the video itself (scene, action, and audiovisual context together) rather than speech or a single frame alone.
Pipeline
POST …/process { chunkDurationSec? }
→ chunk-video (ffmpeg, fixed-length segments)
→ embed-chunk (per segment, via AI Gateway)
→ store vectors in PostgresChunk duration
chunkDurationSec defaults to 15. The start request accepts 5, 10, 15, or 30. Shorter chunks cost more embeds and catch finer timing; longer chunks (up to 30s) stay within Gemini Embedding 2’s 1 fps sampling band (max 32 frames per clip).
Search
- Admin UI: multimodal search screen
- Phase 2:
POST /search - SDK:
client.search({ query, … })
Hits include startSec, endSec, chunkIndex, similarity score, and sourceObject (bucket + key) for resolving the original file.