Book a 30-min call
← cd ../blogs
$ cat posts/streaming-video-when-not-to-respond.mdx

Streaming video models: the hard part is knowing when not to respond

June 19, 2026 · ImmovableTech Team

  • Computer Vision
  • Real-Time Systems
  • Evaluation

Silence is an output, and almost nothing measures it

A model watching a camera feed makes one decision at every tick that matters more than any other: whether to say anything at all. Get that wrong in the permissive direction and the system is worse than useless. A model that narrates continuously produces a wall of text about an empty corridor, and the one event you cared about is buried in it. Nobody reads the wall. The alert that fires on every frame is the alert nobody has configured a pager for.

This is not the capability the field measures. Streaming video-language models are graded almost entirely on whether the answer is correct once a question has been asked, and that framing quietly deletes the part that makes a live system usable. We have been building computer-vision systems for a while, mostly batch — aerial imagery, dense segmentation, throughput problems. Moving to a live feed changes which questions are hard, and the honest summary is that accuracy stopped being the interesting number quite early.

Offline video QA cannot measure this, structurally

Hand a model a clip and a question and you have already made the decision under test. The benchmark designer chose the moment. All that remains is whether the model gets the answer right, which is a real capability and the wrong one to optimise for if the deployment is a camera rather than a video library.

StreamingBench (November 2024) bills itself as the first comprehensive benchmark for streaming video understanding — 900 videos, 4,500 human-curated QA pairs, 18 tasks, with five questions per video posed at different time points to simulate a continuous stream. That last design choice is the tell. The questions arrive on a schedule the benchmark controls; the model is never asked to decide that now is the moment to speak.

OVO-Bench (January 2025, CVPR 2025) went further and is the closest thing to an honest attempt in a widely-used benchmark. It comprises 12 tasks over 644 videos with roughly 2,800 human-curated meta-annotations carrying precise timestamps, split across three modes: backward tracing, real-time visual perception and — the interesting one — forward active responding, where the model must withhold its answer until enough future evidence has arrived. That genuinely tests restraint. It still starts from a user query at a known timestamp, so what it measures is how long to wait before answering a question someone asked, not whether anything worth mentioning just happened. Those are different capabilities, and only the second one is what a surveillance or assistance deployment needs.

The paper is also blunt about the result: existing online video models frequently collapse on forward active responding, and offline models prompted into the task tend to guess randomly when the query contains words like “currently” or “ongoing”.

The metrics that actually matter

Three numbers determine whether a streaming model is deployable, and none of them is accuracy.

The first is response timing relative to the event. LiveStarPro (arXiv, 16 June 2026) formalises this as TimDiff — the temporal deviation between each response and the ground-truth semantic clip it belongs to, with missed clips penalised by the full clip duration and multiple responses to the same clip accumulating latency penalties. Their headline gain over the previous best online video-LLM is an 18.2% reduction in TimDiff alongside a 28.9% improvement in semantic correctness, which is the right shape of claim: correctness and timing reported as separate axes rather than folded into one score.

The second is redundancy, which is the false-alarm problem wearing a different hat. LiveStarPro’s TimRedun measures how far the model deviates from producing exactly one response per event, and TimCover measures the fraction of events that got any response at all. The pair is what makes the evaluation honest, and their own results table shows why you need both. VideoLLM-online and VideoLLM-MoD score the highest TimCover of anything measured — they respond to nearly every frame, so of course they cover every event — while performing worse on everything else. A model that never shuts up has perfect recall by construction. Report coverage alone and it looks like the state of the art.

The third is the cost of speaking when nothing happened, and this is where even the good work hedges. The StreamingHarness paper (arXiv, 7 June 2026) proposes SW-F1, a streaming weighted F1 that explicitly exists to penalise “always-respond” behaviour. But the weights it ships with are 2.0 for true positives, 2.0 for false negatives and 0.2 for false positives. A false alarm is worth one-tenth of a miss. The authors are upfront that this is deliberate — they treat answer correctness as the primary concern — and for a benchmark that is a defensible choice. For an operator watching sixteen cameras overnight it is exactly inverted. Their quiet hours are almost entirely quiet, so false positives are the only thing that will accumulate, and a metric that discounts them by 10× will rank models in an order that has nothing to do with which one you can leave running.

What nobody publishes, and what we ended up building ourselves, is the number that decides procurement: false alarms per hour of genuinely quiet footage. It is not hard to measure. Take an hour of a feed where nothing happens, run the model, count the utterances. It is absent from the streaming benchmarks because they are built from event-annotated sources and scored per annotated clip, which means quiet footage is largely sampled out before evaluation begins.

The frame budget is the actual engineering constraint

Live inference gives you a fixed compute budget per second of wall-clock time, and that budget buys you a frame rate. The frame rate determines which events are physically visible to the model. Everything else is downstream of that arithmetic, and it is arithmetic worth doing before choosing a model.

Assume the sampler takes an instantaneous frame every T seconds and the event you care about is visible for d seconds, with d under T. Whether a frame lands inside the event depends on phase, which you do not control, so across many events the fraction you see is d / T:

def visibility(event_duration_s: float, sample_interval_s: float) -> float:
    """Fraction of short events captured by uniform sampling at a fixed interval."""
    return min(1.0, event_duration_s / sample_interval_s)


visibility(0.4, 1.0)  # 0.40 — a 400 ms event at 1 FPS
visibility(0.4, 0.5)  # 0.80 — the same event at 2 FPS

At 1 FPS, a 400 ms event — a door closing, a hand entering and leaving frame, a fall — is invisible six times out of ten. Not misclassified. Invisible. No amount of model quality recovers it, and no accuracy metric computed on the frames you did sample will ever show you the gap.

Now price the frame rate. StreamingHarness reports stable sub-second latency at 1 FPS on a single NVIDIA H200 across a two-hour broadcast, and gets there through vLLM prefix caching — their own comparison shows a sliding-window baseline stabilising at roughly 4 seconds per step, four times over the real-time threshold, because consecutive steps share no common prefix and prefix caching gives it nothing. AURA (arXiv, 5 April 2026) reports a real-time demo with ASR and TTS running at 2 FPS on two 80 GB accelerators. LiveStarPro sustains around 3 FPS on hour-long streams.

So the going rate for streaming video understanding in mid-2026 is roughly one to three frames per second per accelerator. Sixteen cameras is a rack, not a box. And the StreamingHarness authors say the quiet part in their own limitations section: they run at 1 FPS to balance memory length against efficiency, and their framework is not optimised for domains needing higher temporal resolution — fast athletic movement, brief decisive moments. Trading memory length for frame rate is possible in principle and they leave it to future work.

That trade is the whole design space. Frames per second, seconds of retained context and dollars per camera-hour pick two, and the choice of which events exist gets made in that trade rather than in model selection.

Cheap trigger, expensive model — and what it costs you

The architecture that follows is obvious and correct: do not run the VLM continuously. Put something cheap and always-on in front of it — frame differencing, an audio threshold or a small detector — and spend the VLM only on segments the trigger flags. Event-VStream (arXiv, 22 January 2026) is the same idea pushed into the model itself, detecting state transitions from motion, semantic and predictive cues and triggering language generation only at those boundaries rather than on a fixed decoding interval.

camera → decode (1 FPS keyframes)
       → cheap trigger: motion / audio / small detector   [always on, ~ms]
       → if fired: buffered clip → VLM                    [~1 accelerator]
       → response gate: timing + dedupe
       → alert

This is the same shape as a pattern we have used elsewhere: on a real-time fraud system we put cheap deterministic pattern matching ahead of the expensive model precisely so there was no reason to spend a graph neural network forward pass on velocity spikes (the write-up is here). The economics are compelling and the failure mode is different in video, in a way that took us longer to internalise than it should have.

The trigger’s recall becomes the ceiling on the whole system’s recall. Not approximately — exactly. An event the trigger does not fire on is an event the VLM never sees, so the VLM’s quality is irrelevant to it. If the motion trigger catches 80% of the events that matter, the system catches at most 80%, and swapping in a better VLM moves that number by zero.

We learned this the way you would expect. On an early always-on prototype we tuned the trigger for precision, because every spurious trigger was a VLM invocation we were paying for, and precision is the knob that visibly reduces the bill. Then we evaluated the system on the clips the trigger produced and the numbers were excellent. They were excellent because we had defined the test set as the trigger’s output. The events the trigger silently dropped were not in the denominator. We only found the gap when someone watched raw footage against the alert log by hand and found things that had plainly happened and generated nothing — slow movement well under the motion threshold, mostly, which is a category the trigger was structurally blind to rather than occasionally wrong about.

The fix was not a better threshold. It was measuring the trigger separately against hand-labelled raw footage, treating its recall as a hard system-level budget, and accepting a much higher false-trigger rate — and therefore a materially higher VLM bill — to buy it back. The right knob for cost is the VLM’s frame rate on triggered segments, not the trigger’s sensitivity.

What we’d do differently

We would build the quiet-hours evaluation set first, before choosing a model. Ours came third, after the model and after the trigger, which meant every earlier decision was made against a metric that could not see the failure we ended up caring about most. An hour of boring footage with zero ground-truth events is trivial to collect and is the single most informative test artefact in the whole system, because it is the only one where the correct output is known exactly and in advance: nothing.

We would also stop reporting coverage without redundancy alongside it. Our first internal dashboard showed event coverage prominently and response count nowhere, which is precisely the presentation that makes a model that narrates constantly look like the best one available. LiveStarPro’s tables make the same point with published numbers, and we had rediscovered it the expensive way a few weeks earlier.

The thing we would not change is the cheap-trigger architecture. It is the right shape. It just needs to be evaluated as two systems with two recall numbers rather than one system with one.

References

We build always-on computer-vision systems as part of our AI & Machine Learning Engineering practice. Talk to us if you are trying to work out what a live camera deployment will actually cost per accelerator.