Modalities
Independent indexes plus a hybrid fused path
v0.1 ships three separate indexes and a hybrid path. Each independent modality has its own process path, search endpoint, cost tracking, and admin UI routes. Starting one pipeline does not run the others. Hybrid is a fourth path: one shared segment grid, then fused search.
| Path | Good for | Index unit | Timing |
|---|---|---|---|
| Multimodal | Broader clip meaning when query intent is mixed or unknown | Video chunk (5–30s) | startSec-endSec |
| Speech | Spoken words and dialogue | Transcript segment | startSec-endSec |
| Vision | What's visible on screen | Still frame | timestampSec |
| Hybrid | One index for mixed queries; fuse video + speech + vision with RRF | Shared segment (5–30s) | startSec-endSec |
Independent modalities share uploads and collections but each builds its own embeddings and ranking. Hybrid builds a separate shared-grid index; it does not replace the three single-modality indexes.
End-to-end testing so far covers the opinionated defaults (Gemini embeddings and Whisper). For other models, confirm equivalent behavior in that model's docs and via the AI Gateway before you depend on them.
Practical guidance
- Speech-heavy interview → index speech.
- Silent B-roll or product shots → index vision.
- Mixed or unknown query intent, or deeper scene meaning → index multimodal.
- Want one process that covers all three signals with fused ranking → index hybrid.
- For demos, index the same file under more than one path and compare hits / costs.
Shared product surfaces
Independent modalities:
- Operate on the same uploads and optional collections
- Block a second concurrent start for the same file with 409 (one active parent task)
- Record usage / costs when the gateway returns cost metadata
- Support re-run (re-process replaces or rebuilds that modality's index for the file)
Hybrid shares uploads/collections and the one-active-parent guard, but operator retry is a full re-segment (see Hybrid).