Find when small text in one corner of a video appears

Read the screenCookingVerified

0:20CC-BY-SA-2.0 · Clay Gilliland (Flickr user 26781577@N07)

Proof frames

Click a frame to seek

Tool calls

4 calls · agentic-video 0.3.1

count_occurrencesimac · 635 ms

Arguments

{
  "source": "$CLIP",
  "text": "BOOKMARK",
  "start_sec": 2.8,
  "end_sec": 12.8,
  "max_spans": 5
}

Result

{
  "ok": true,
  "data": {
    "status": "in_progress",
    "retry_after_seconds": 5
  }
}

count_occurrencesimac · 1 ms

Arguments

{
  "source": "$CLIP",
  "text": "BOOKMARK",
  "start_sec": 2.8,
  "end_sec": 12.8,
  "max_spans": 5
}

Result

{
  "ok": true,
  "data": {
    "status": "in_progress",
    "retry_after_seconds": 5,
    "message": "Analyzed 34 of 50 sampled frames; repeat this call to poll."
  }
}

count_occurrencesimac · 1 ms

Arguments

{
  "source": "$CLIP",
  "text": "BOOKMARK",
  "start_sec": 2.8,
  "end_sec": 12.8,
  "max_spans": 5
}

Result

{
  "ok": true,
  "data": {
    "status": "complete",
    "count": 1,
    "spans": [
      {
        "index": 1,
        "start_sec": 5.8,
        "end_sec": 9.8,
        "duration_sec": 4,
        "score": 1,
        "representative_sec": 7.4,
        "frame_badge": "#1",
        "weak": false,
        "region": {
          "x": 0.869,
          "y": 0.028,
          "w": 0.116,
          "h": 0.033
        },
        "travel": "stationary"
      }
    ],
    "spans_returned": 1,
    "spans_omitted": 0,
    "window": {
      "start_sec": 2.8,
      "end_sec": 12.8
    },
    "fps_used": 5,
    "frames_sampled": 50,
    "frames_analyzed": 50,
    "checks": {
      "text": "BOOKMARK",
      "description": null,
      "color": null,
      "shape": null
    },
    "contact_sheet": {
      "file_path": "<cache>/2647be56ed16ceeafe99/contact_sheets/occurrences_cbb486fdf7109e4b658b.jpg",
      "sample_count": 1,
      "rows": 1,
      "cols": 1,
      "frame_width": 288,
      "frame_height": 192,
      "labels": [
        "#1 7.4s"
      ],
      "tool_file": {
        "_tag": "ToolFile",
        "name": "occurrences_cbb486fdf7109e4b658b.jpg",
        "mimeType": "image/jpeg",
        "encoding": "base64",
        "data": "<21012 base64 chars dropped>",
        "byteLength": 15757
      }
    },
    "note": "Spans are sample-based (+/- 1/fps); verify each representative_sec with inspect_window. contact_sheet is one image with a tile per listed span, in span order: a zoomed frame of the span's best match, boxed, labeled with the span number and time. Look at each tile and drop any span whose tile is not the asked-for thing; spans marked * (`weak`) are the likeliest false ones. The answer is the number of tiles that match, plus any omitted spans and respecting any merged gaps noted above.",
    "verify": "Rough count: spans can be false, merge across scene cuts, split one appearance, or miss crossings. Before answering, look at each tile of the contact_sheet image (one per span, in span order, a zoomed frame of its best match, boxed) and drop any span whose tile is not the asked-for thing; spans with weak=true (tile label *) are the likeliest false ones. The answer is the number of tiles that match, plus any spans the sheet does not show; if you also count by eye and differ, trust your own."
  }
}

frame_atimac · 559 ms

Arguments

{
  "source": "$CLIP",
  "time_sec": 7.8,
  "crop": {
    "x": 440,
    "y": 0,
    "width": 200,
    "height": 60
  },
  "scale": 3
}

Result

{
  "ok": true,
  "data": {
    "frame_time_sec": 7.8,
    "width": 600,
    "height": 180,
    "source_width": 640,
    "source_height": 360,
    "crop": {
      "x": 440,
      "y": 0,
      "width": 200,
      "height": 60
    },
    "scale": 3,
    "file_path": "<cache>/2647be56ed16ceeafe99/frame_at/frame_at_2d722e30ab943f7c.png",
    "tool_file": {
      "_tag": "ToolFile",
      "name": "frame_at_2d722e30ab943f7c.png",
      "mimeType": "image/png",
      "encoding": "base64",
      "data": "<137956 base64 chars dropped>",
      "byteLength": 103467
    }
  }
}

Question and answer

Question: When does the word BOOKMARK appear in the corner of the video?
Answer: The tool finds the word in one span from 5.8 to 9.8 seconds.

Check passedIoU 1.00, minimum 0.7

How it was checked

Truth from: constructed
Fixed at: 2026-10-30 19:41

Planted by this recipe: ffmpeg overlaid the word 'BOOKMARK' as 12 px white text on a dark plate, in the top-right corner of 'Making lavash flatbread in a clay oven in Goris, Armenia' (its first 20 s), from 5.80 s to 9.80 s. The truth is that interval, 5.80 to 9.80 s, matched with IoU tolerance 0.7. The calls give count_occurrences the window 2.80 to 12.80 s (the text padded 3 s either side, clamped to the clip) and frame_at the middle of the text, 7.80 s, cropped to the 200 x 60 px corner box. The answer is the longest span count_occurrences lists in that window (the first on a tie), or its count when it lists none.

A script drew the word BOOKMARK as 12 px text in the top-right corner for 4 seconds, so the planted interval is the one the script set. The answer is the longest span the tool lists in a window 3 seconds wider on each side; small text can be missed, and then the check fails.

Try it yourself

Use agentic-video on my video. Count how many times the word BOOKMARK appears as on-screen text from 2.8 to 12.8 seconds, then take one frame from the middle of the text and crop the corner where it sits. Tell me when the word starts and when it ends on screen.

Get the .mcpb file and open it in Claude Desktop.
Or: pip install agentic-video

Clip and licence

Licence: CC-BY-SA-2.0
Author: Clay Gilliland (Flickr user 26781577@N07)
Source: commons.wikimedia.org

This file is licensed under the Creative Commons Attribution-Share Alike 2.0 Generic license.