agentic-video Gallery

Name the sound in one 5-second window of space footage

SoundSpaceTool missed
0:30PD · NASA Johnson Space Center

Proof frames

Click a frame to seek

Tool calls

1 calls · agentic-video 0.3.1
label_soundsmacbook · 3627 ms

Arguments

{
  "source": "$CLIP",
  "start_sec": 5,
  "end_sec": 10,
  "window_sec": 5,
  "labels": [
    "siren",
    "church bells",
    "car horn",
    "dog barking",
    "clapping",
    "train",
    "helicopter",
    "glass breaking",
    "thunderstorm",
    "rooster crowing"
  ]
}

Result

{
  "ok": true,
  "data": {
    "source_path": "~/Code/agentic-video-gallery/cards/label-space/clip.mp4",
    "start_sec": 5,
    "end_sec": 10,
    "window_sec": 5,
    "model": "clef-omni",
    "labels": [
      "siren",
      "church bells",
      "car horn",
      "dog barking",
      "clapping",
      "train",
      "helicopter",
      "glass breaking",
      "thunderstorm",
      "rooster crowing"
    ],
    "windows": [
      {
        "start_sec": 5,
        "end_sec": 10,
        "top_label": null,
        "top_chance": 0.0362,
        "none_chance": 0.8519,
        "chances": {
          "siren": 0.0311,
          "church bells": 0.0362,
          "car horn": 0.0159,
          "dog barking": 0.0072,
          "clapping": 0.0093,
          "train": 0.0145,
          "helicopter": 0.007,
          "glass breaking": 0.0033,
          "thunderstorm": 0.0175,
          "rooster crowing": 0.0061
        }
      }
    ],
    "input_tokens": 385,
    "complete": true,
    "next_start_sec": null,
    "note": "Audio was sent to Cloudflare Workers AI. Chances are the model's own and sum to 1 with none_chance; top_label is null when 'none of these' is more likely than every label. Sounds outside `labels` are forced onto the nearest label or none, so give every sound that may be present."
  }
}

Question and answer

Question
What sound is heard from 5 to 10 seconds?
Answer
The tool names the sound in the 5-second window from 5 to 10 seconds as nothing (no label beat 'none of these').
Tool returned
[null]
Truth
["siren"]
Check failedanswer value is not a list of labels for kind labels

How it was checked

Truth from
constructed
Fixed at

Planted by this recipe: ffmpeg cut the first 30 s of the source and mixed a synthetic siren (pitch sweep 550 to 1250 Hz, 3.2 s long, starting at 5.9 s) over the source's own audio at gain 0.2. The siren lies inside the 5 s window from 5 s to 10 s, the window that holds its start. The truth is the label siren for that window; the call covers that same window.

A script mixed a synthetic siren over this space clip's own sound at a time it chose, and the answer is the top label of that one 5-second window, so the card does not show every sound in the clip.

Try it yourself

Use agentic-video to name the sound in my video from 5 to 10 seconds. Call label_sounds on that range with 5-second windows and this list of sounds: siren, church bells, car horn, dog barking, clapping, train, helicopter, glass breaking, thunderstorm, rooster crowing. Tell me the sound it names. Use my own file.
  1. Get the .mcpb file and open it in Claude Desktop.
  2. Or: pip install agentic-video

Clip and licence

Licence
PD
Author
NASA Johnson Space Center
Source
commons.wikimedia.org
This file is in the public domain in the United States because it was solely created by NASA.