Find when a word shows on screen in monitoring footage
0:30CC-BY-SA-4.0 · Datarhei
Proof frames
Tool calls
count_occurrences
Arguments
{
"source": "$CLIP",
"text": "OFFLINE",
"max_spans": 5
}
Result
{
"ok": true,
"data": {
"status": "in_progress",
"retry_after_seconds": 5
}
}
count_occurrences
Arguments
{
"source": "$CLIP",
"text": "OFFLINE",
"max_spans": 5
}
Result
{
"ok": true,
"data": {
"status": "in_progress",
"retry_after_seconds": 5,
"message": "Analyzed 28 of 151 sampled frames; repeat this call to poll."
}
}
count_occurrences
Arguments
{
"source": "$CLIP",
"text": "OFFLINE",
"max_spans": 5
}
Result
{
"ok": true,
"data": {
"status": "in_progress",
"retry_after_seconds": 5,
"message": "Analyzed 52 of 151 sampled frames; repeat this call to poll."
}
}
count_occurrences
Arguments
{
"source": "$CLIP",
"text": "OFFLINE",
"max_spans": 5
}
Result
{
"ok": true,
"data": {
"status": "in_progress",
"retry_after_seconds": 5,
"message": "Analyzed 79 of 151 sampled frames; repeat this call to poll."
}
}
count_occurrences
Arguments
{
"source": "$CLIP",
"text": "OFFLINE",
"max_spans": 5
}
Result
{
"ok": true,
"data": {
"status": "in_progress",
"retry_after_seconds": 5,
"message": "Analyzed 111 of 151 sampled frames; repeat this call to poll."
}
}
count_occurrences
Arguments
{
"source": "$CLIP",
"text": "OFFLINE",
"max_spans": 5
}
Result
{
"ok": true,
"data": {
"status": "in_progress",
"retry_after_seconds": 5,
"message": "Analyzed 148 of 151 sampled frames; repeat this call to poll."
}
}
count_occurrences
Arguments
{
"source": "$CLIP",
"text": "OFFLINE",
"max_spans": 5
}
Result
{
"ok": true,
"data": {
"status": "complete",
"count": 1,
"spans": [
{
"index": 1,
"start_sec": 9.6,
"end_sec": 12.6,
"duration_sec": 3,
"score": 1,
"representative_sec": 11.2,
"frame_badge": "#1",
"weak": false,
"region": {
"x": 0.381,
"y": 0.477,
"w": 0.241,
"h": 0.085
},
"travel": "stationary"
}
],
"spans_returned": 1,
"spans_omitted": 0,
"window": {
"start_sec": 0,
"end_sec": 30.03
},
"fps_used": 5,
"frames_sampled": 151,
"frames_analyzed": 146,
"checks": {
"text": "OFFLINE",
"description": null,
"color": null,
"shape": null
},
"contact_sheet": {
"file_path": "<cache>/bc2a8cd1fdeb20004c3a/contact_sheets/occurrences_3113c7625cd56079597a.jpg",
"sample_count": 1,
"rows": 1,
"cols": 1,
"frame_width": 288,
"frame_height": 192,
"labels": [
"#1 11.2s"
],
"tool_file": {
"_tag": "ToolFile",
"name": "occurrences_3113c7625cd56079597a.jpg",
"mimeType": "image/jpeg",
"encoding": "base64",
"data": "<15952 base64 chars dropped>",
"byteLength": 11964
}
},
"note": "Spans are sample-based (+/- 1/fps); verify each representative_sec with inspect_window. contact_sheet is one image with a tile per listed span, in span order: a zoomed frame of the span's best match, boxed, labeled with the span number and time. Look at each tile and drop any span whose tile is not the asked-for thing; spans marked * (`weak`) are the likeliest false ones. The answer is the number of tiles that match, plus any omitted spans and respecting any merged gaps noted above.",
"verify": "Rough count: spans can be false, merge across scene cuts, split one appearance, or miss crossings. Before answering, look at each tile of the contact_sheet image (one per span, in span order, a zoomed frame of its best match, boxed) and drop any span whose tile is not the asked-for thing; spans with weak=true (tile label *) are the likeliest false ones. The answer is the number of tiles that match, plus any spans the sheet does not show; if you also count by eye and differ, trust your own."
}
}
frame_at
Arguments
{
"source": "$CLIP",
"time_sec": 11
}
Result
{
"ok": true,
"data": {
"frame_time_sec": 10.977633,
"width": 640,
"height": 360,
"source_width": 640,
"source_height": 360,
"crop": null,
"scale": 1,
"file_path": "<cache>/bc2a8cd1fdeb20004c3a/frame_at/frame_at_6f7e0e1c5929f9a1.png",
"tool_file": {
"_tag": "ToolFile",
"name": "frame_at_6f7e0e1c5929f9a1.png",
"mimeType": "image/png",
"encoding": "base64",
"data": "<194576 base64 chars dropped>",
"byteLength": 145931
}
}
}
Question and answer
- Question
- When does the word OFFLINE show up on the screen in this video?
- Answer
- The tool finds the word OFFLINE on screen from 9.6 to 12.6 seconds.
Check passedIoU 0.94, minimum 0.5
How it was checked
- Truth from
- constructed
- Fixed at
Planted by this recipe before any tool ran: a Pillow-drawn card reading 'OFFLINE' (white letters on a dark box) was overlaid with ffmpeg on the first 30 s of 'Lake Constance harbour webcam: boats moving past the lighthouse' (licence CC-BY-SA-4.0) from 9.5 s to 12.5 s, the picture otherwise unchanged. The truth is that interval, 9.5 to 12.5 s, checked by an IoU of at least 0.5; frame_at asks for the middle of it, 11 s.
A script drew the word OFFLINE on a card over this monitoring footage for 3 seconds, at a time it chose; that planted 3-second span is the truth. The answer is the longest span the tool listed, and frame_at was asked for the middle of the planted span, not of the tool's span. Other spans, if any, are not shown on the card.
Try it yourself
Use agentic-video on my video. Find when the exact word OFFLINE is on the screen: run count_occurrences on the whole video with text OFFLINE, then run frame_at at the middle of the longest span it lists. Tell me the start and end seconds of that span. Use my own file.
- Get the .mcpb file and open it in Claude Desktop.
- Or:
pip install agentic-video
Clip and licence
- Licence
- CC-BY-SA-4.0
- Author
- Datarhei
- Source
- commons.wikimedia.org
This file is licensed under the Creative Commons Attribution-Share Alike 4.0 International license.