agentic-video Gallery

Find the exact start of a spliced-in shot

Find a momentLectureVerified
0:24CC-BY-3.0 · Susan Dean, Barbara Illowsky

Proof frames

Click a frame to seek

Tool calls

2 calls · agentic-video 0.3.1
refine_temporal_beamimac · 505 ms

Arguments

{
  "source": "$CLIP",
  "windows": [
    {
      "start_sec": 8.36,
      "end_sec": 19.36
    }
  ],
  "fanout": 4,
  "samples_per_child": 3
}

Result

{
  "ok": true,
  "data": {
    "parents": [
      {
        "parent_index": 1,
        "input_start_sec": 8.36,
        "input_end_sec": 19.36,
        "start_sec": 8.36,
        "end_sec": 19.36,
        "children": [
          {
            "child_index": 1,
            "start_sec": 8.36,
            "end_sec": 11.11,
            "sample_timestamps": [
              8.818333333333333,
              9.735,
              10.651666666666666
            ],
            "contact_sheet_path": "/private<cache>/db17bf4a17071de80431/contact_sheets/sheet_3samples_3cols_6a146d.jpg",
            "sample_count": 3,
            "tool_file": {
              "_tag": "ToolFile",
              "name": "sheet_3samples_3cols_6a146d.jpg",
              "mimeType": "image/jpeg",
              "encoding": "base64",
              "data": "<240664 base64 chars dropped>",
              "byteLength": 180497
            }
          },
          {
            "child_index": 2,
            "start_sec": 11.11,
            "end_sec": 13.86,
            "sample_timestamps": [
              11.568333333333333,
              12.485,
              13.401666666666666
            ],
            "contact_sheet_path": "/private<cache>/db17bf4a17071de80431/contact_sheets/sheet_3samples_3cols_902890.jpg",
            "sample_count": 3,
            "tool_file": {
              "_tag": "ToolFile",
              "name": "sheet_3samples_3cols_902890.jpg",
              "mimeType": "image/jpeg",
              "encoding": "base64",
              "data": "<172492 base64 chars dropped>",
              "byteLength": 129369
            }
          },
          {
            "child_index": 3,
            "start_sec": 13.86,
            "end_sec": 16.61,
            "sample_timestamps": [
              14.318333333333333,
              15.235,
              16.151666666666667
            ],
            "contact_sheet_path": "/private<cache>/db17bf4a17071de80431/contact_sheets/sheet_3samples_3cols_af186f.jpg",
            "sample_count": 3,
            "tool_file": {
              "_tag": "ToolFile",
              "name": "sheet_3samples_3cols_af186f.jpg",
              "mimeType": "image/jpeg",
              "encoding": "base64",
              "data": "<172624 base64 chars dropped>",
              "byteLength": 129466
            }
          },
          {
            "child_index": 4,
            "start_sec": 16.61,
            "end_sec": 19.36,
            "sample_timestamps": [
              17.06833333333333,
              17.985,
              18.901666666666667
            ],
            "contact_sheet_path": "/private<cache>/db17bf4a17071de80431/contact_sheets/sheet_3samples_3cols_f33cd5.jpg",
            "sample_count": 3,
            "tool_file": {
              "_tag": "ToolFile",
              "name": "sheet_3samples_3cols_f33cd5.jpg",
              "mimeType": "image/jpeg",
              "encoding": "base64",
              "data": "<240612 base64 chars dropped>",
              "byteLength": 180457
            }
          }
        ]
      }
    ],
    "total_samples": 12,
    "estimated_vision_tokens": 3096
  }
}
detect_cutsimac · 195 ms

Arguments

{
  "source": "$CLIP",
  "start_sec": 8.36,
  "end_sec": 19.36
}

Result

{
  "ok": true,
  "data": {
    "cuts": [
      {
        "timestamp_sec": 12.36,
        "score": 0.502269,
        "kind": "cut"
      },
      {
        "timestamp_sec": 15.36,
        "score": 0.500017,
        "kind": "cut"
      }
    ],
    "total": 2
  }
}

Question and answer

Question
Exactly when does the new shot start in this clip?
Answer
The first cut in the window, into the new shot, is at 12.36 seconds.
Check passedanswer 12.36, expected 12.36 ±0.5

How it was checked

Truth from
constructed
Fixed at

Planted by this recipe before any tool ran: ffmpeg cut 24 s of the source from 39.192 s, and from 12.360 s for 3 s the picture is a red still with the words INSERTED SHOT (the sound is kept). Before the cut, the whole 24 s bed was checked with an ffmpeg scene filter (score above 0.15): no cut in it, so the first cut is the planted one. The truth is the planted start 12.360 s, with tolerance 0.5 s. Both tools get the coarse window 8.360 s to 19.360 s, fixed before the run as 4 s before and 7 s after the planted start.

A script put a 3-second red still with the words INSERTED SHOT over the picture at a set time, so the answer is that planted time, and refine_temporal_beam only zooms in and does not say where the change is, so the time comes from detect_cuts.

Try it yourself

Use agentic-video to find when a new shot starts in my video. Run refine_temporal_beam on my video over the window from 8.36 to 19.36 seconds, with fanout 4 and 3 samples per child. Then run detect_cuts on that same window and tell me the time of the first cut it lists. Use my own file.
  1. Get the .mcpb file and open it in Claude Desktop.
  2. Or: pip install agentic-video

Clip and licence

Licence
CC-BY-3.0
Author
Susan Dean, Barbara Illowsky
Source
commons.wikimedia.org
This file is licensed under the Creative Commons Attribution 3.0 Unported license.