agentic-video Gallery

Name the sound in one 5-second window of game footage

SoundGamingVerified
0:30CC-BY-SA-3.0 · Wildfire Games

Proof frames

Click a frame to seek

Tool calls

1 calls · agentic-video 0.3.1
label_soundsmacbook · 1947 ms

Arguments

{
  "source": "$CLIP",
  "start_sec": 10,
  "end_sec": 15,
  "window_sec": 5,
  "labels": [
    "siren",
    "church bells",
    "car horn",
    "dog barking",
    "clapping",
    "train",
    "helicopter",
    "glass breaking",
    "thunderstorm",
    "rooster crowing"
  ]
}

Result

{
  "ok": true,
  "data": {
    "source_path": "~/Code/agentic-video-gallery/cards/label-gaming/clip.mp4",
    "start_sec": 10,
    "end_sec": 15,
    "window_sec": 5,
    "model": "clef-omni",
    "labels": [
      "siren",
      "church bells",
      "car horn",
      "dog barking",
      "clapping",
      "train",
      "helicopter",
      "glass breaking",
      "thunderstorm",
      "rooster crowing"
    ],
    "windows": [
      {
        "start_sec": 10,
        "end_sec": 15,
        "top_label": "siren",
        "top_chance": 0.7834,
        "none_chance": 0.0197,
        "chances": {
          "siren": 0.7834,
          "church bells": 0.0168,
          "car horn": 0.0301,
          "dog barking": 0.0209,
          "clapping": 0.0163,
          "train": 0.024,
          "helicopter": 0.0332,
          "glass breaking": 0.019,
          "thunderstorm": 0.0181,
          "rooster crowing": 0.0185
        }
      }
    ],
    "input_tokens": 385,
    "complete": true,
    "next_start_sec": null,
    "note": "Audio was sent to Cloudflare Workers AI. Chances are the model's own and sum to 1 with none_chance; top_label is null when 'none of these' is more likely than every label. Sounds outside `labels` are forced onto the nearest label or none, so give every sound that may be present."
  }
}

Question and answer

Question
What sound is heard from 10 to 15 seconds?
Answer
The tool names the sound in the 5-second window from 10 to 15 seconds as siren.
Check passedrequired labels present, no forbidden labels

How it was checked

Truth from
constructed
Fixed at

Planted by this recipe: ffmpeg cut the first 30 s of the source and mixed a synthetic siren (pitch sweep 550 to 1250 Hz, 3.1 s long, starting at 10.8 s) over the source's own audio at gain 0.2. The siren lies inside the 5 s window from 10 s to 15 s, the window that holds its start. The truth is the label siren for that window; the call covers that same window.

A script mixed a synthetic siren over this game clip's own music at a time it chose, and the answer is the top label of that one 5-second window, so the card does not show every sound in the clip.

Try it yourself

Use agentic-video to name the sound in my video from 10 to 15 seconds. Call label_sounds on that range with 5-second windows and this list of sounds: siren, church bells, car horn, dog barking, clapping, train, helicopter, glass breaking, thunderstorm, rooster crowing. Tell me the sound it names. Use my own file.
  1. Get the .mcpb file and open it in Claude Desktop.
  2. Or: pip install agentic-video

Clip and licence

Licence
CC-BY-SA-3.0
Author
Wildfire Games
Source
commons.wikimedia.org
This file is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license.