agentic-video Gallery

Find the exact start of a spliced-in shot

Find a momentSportsVerified
0:24CC-BY-4.0 · Alexis Jazz

Proof frames

Click a frame to seek

Tool calls

2 calls · agentic-video 0.3.1
refine_temporal_beamimac · 587 ms

Arguments

{
  "source": "$CLIP",
  "windows": [
    {
      "start_sec": 4.933,
      "end_sec": 15.933
    }
  ],
  "fanout": 4,
  "samples_per_child": 3
}

Result

{
  "ok": true,
  "data": {
    "parents": [
      {
        "parent_index": 1,
        "input_start_sec": 4.933,
        "input_end_sec": 15.933,
        "start_sec": 4.933,
        "end_sec": 15.933,
        "children": [
          {
            "child_index": 1,
            "start_sec": 4.933,
            "end_sec": 7.683,
            "sample_timestamps": [
              5.391333333333333,
              6.308,
              7.224666666666666
            ],
            "contact_sheet_path": "/private<cache>/0c69826f688ec7ed218c/contact_sheets/sheet_3samples_3cols_5cfd3d.jpg",
            "sample_count": 3,
            "tool_file": {
              "_tag": "ToolFile",
              "name": "sheet_3samples_3cols_5cfd3d.jpg",
              "mimeType": "image/jpeg",
              "encoding": "base64",
              "data": "<280128 base64 chars dropped>",
              "byteLength": 210094
            }
          },
          {
            "child_index": 2,
            "start_sec": 7.683,
            "end_sec": 10.433,
            "sample_timestamps": [
              8.141333333333334,
              9.058,
              9.974666666666666
            ],
            "contact_sheet_path": "/private<cache>/0c69826f688ec7ed218c/contact_sheets/sheet_3samples_3cols_9b7c4.jpg",
            "sample_count": 3,
            "tool_file": {
              "_tag": "ToolFile",
              "name": "sheet_3samples_3cols_9b7c4.jpg",
              "mimeType": "image/jpeg",
              "encoding": "base64",
              "data": "<174756 base64 chars dropped>",
              "byteLength": 131066
            }
          },
          {
            "child_index": 3,
            "start_sec": 10.433,
            "end_sec": 13.183,
            "sample_timestamps": [
              10.891333333333334,
              11.808,
              12.724666666666666
            ],
            "contact_sheet_path": "/private<cache>/0c69826f688ec7ed218c/contact_sheets/sheet_3samples_3cols_80f114.jpg",
            "sample_count": 3,
            "tool_file": {
              "_tag": "ToolFile",
              "name": "sheet_3samples_3cols_80f114.jpg",
              "mimeType": "image/jpeg",
              "encoding": "base64",
              "data": "<174452 base64 chars dropped>",
              "byteLength": 130839
            }
          },
          {
            "child_index": 4,
            "start_sec": 13.183,
            "end_sec": 15.933,
            "sample_timestamps": [
              13.641333333333334,
              14.558,
              15.474666666666666
            ],
            "contact_sheet_path": "/private<cache>/0c69826f688ec7ed218c/contact_sheets/sheet_3samples_3cols_92a662.jpg",
            "sample_count": 3,
            "tool_file": {
              "_tag": "ToolFile",
              "name": "sheet_3samples_3cols_92a662.jpg",
              "mimeType": "image/jpeg",
              "encoding": "base64",
              "data": "<284812 base64 chars dropped>",
              "byteLength": 213609
            }
          }
        ]
      }
    ],
    "total_samples": 12,
    "estimated_vision_tokens": 3096
  }
}
detect_cutsimac · 231 ms

Arguments

{
  "source": "$CLIP",
  "start_sec": 4.933,
  "end_sec": 15.933
}

Result

{
  "ok": true,
  "data": {
    "cuts": [
      {
        "timestamp_sec": 8.933326,
        "score": 0.482232,
        "kind": "cut"
      },
      {
        "timestamp_sec": 11.933326,
        "score": 0.485249,
        "kind": "cut"
      }
    ],
    "total": 2
  }
}

Question and answer

Question
Exactly when does the new shot start in this clip?
Answer
The first cut in the window, into the new shot, is at 8.93 seconds.
Check passedanswer 8.933326, expected 8.933333 ±0.5

How it was checked

Truth from
constructed
Fixed at

Planted by this recipe before any tool ran: ffmpeg cut 24 s of the source from 18.741 s, and from 8.933 s for 3 s the picture is a red still with the words INSERTED SHOT (the sound is kept). Before the cut, the whole 24 s bed was checked with an ffmpeg scene filter (score above 0.15): no cut in it, so the first cut is the planted one. The truth is the planted start 8.933 s, with tolerance 0.5 s. Both tools get the coarse window 4.933 s to 15.933 s, fixed before the run as 4 s before and 7 s after the planted start.

A script put a 3-second red still with the words INSERTED SHOT over the picture at a set time, so the answer is that planted time, and refine_temporal_beam only zooms in and does not say where the change is, so the time comes from detect_cuts.

Try it yourself

Use agentic-video to find when a new shot starts in my video. Run refine_temporal_beam on my video over the window from 4.933 to 15.933 seconds, with fanout 4 and 3 samples per child. Then run detect_cuts on that same window and tell me the time of the first cut it lists. Use my own file.
  1. Get the .mcpb file and open it in Claude Desktop.
  2. Or: pip install agentic-video

Clip and licence

Licence
CC-BY-4.0
Author
Alexis Jazz
Source
commons.wikimedia.org
This file is licensed under the Creative Commons Attribution 4.0 International license.