Demo Page
All clips are 5 s, 256×256, 16 fps. Every example comes from the held-out test set; the tag on each card (R = removal, A = addition, D = dataset, F = failure) identifies it across sections. Input is what a method receives, Reference is the ground-truth edit.
Supplementary results for the responses: full tables, protocols and examples that the responses point to. Each section is tagged with the reviewer weakness (W) / question (Q) it answers, in the numbering of each response; the index below goes from each reviewer's points to the sections. Everything below this section is unchanged from the original submission; the numbering of sections 1–6 is kept so that references in the paper and the responses still match.
Reviewer r1hr
W3: §7 InstructAV2AV fine-tuned on our data, §9 Split by YouTube id; rare and unseen objects
Q1: §1 Human study, §2 Audio separation
Q2: §3 Updated judge
Q3: §5 Reference frame
Reviewer hmxL
W1: §1 Human study
W2: §3 Updated judge
W3: §5 Reference frame, §6 Zero-shot performance
Q1: §1 Human study, §2 Audio separation
Q2: §3 Updated judge, §4 Paired tests for the ablation
Q3: §7 InstructAV2AV fine-tuned on our data, §8 Prompts given to each method
Reviewer Rmu9
W1: §4 Paired tests for the ablation
W2: §7 InstructAV2AV fine-tuned on our data
W3: §1 Human study, §2 Audio separation
W4: §10 Other prompts
Q1: §7 InstructAV2AV fine-tuned on our data
Q2: §6 Zero-shot performance
Q3: §11 Backbone
Definitions. Core success means that, on one clip, the target is removed / added both visually and acoustically, the unrelated visual and audio content is preserved, and there is no major artifact. Confidence intervals are 95 % cluster bootstrap over source videos. Paired tests are exact McNemar. Pseudo-target is the reference output built by our data pipeline (called “oracle” in parts of the submission).
Protocol. We collected 11 questionnaires from 10 raters. Each screen shows the source clip, the instruction and one anonymised output; the order is shuffled per rater and the raters do not know which system produced an output. For each output they answer:
Questions 1–4 are answered Pass / Fail / Uncertain. We report the percentage of ratings that are not Pass. Core success = Pass on 1–4 and no major artifact.
Pseudo-targets and our outputs.
| % of ratings | removal: pseudo-target | removal: ours | addition: pseudo-target | addition: ours |
|---|---|---|---|---|
| sound edit fails | 72.7 | 70.0 | 7.4 | 40.9 |
| visual edit fails | 21.8 | 52.0 | 1.9 | 18.2 |
| visual background damaged | 27.3 | 42.0 | 1.9 | 20.5 |
| other sounds damaged | 25.5 | 14.0 | 1.9 | 15.9 |
| major artifact | 32.7 | 48.0 | 7.4 | 31.8 |
| core success [95 % CI] | 16.4 [5, 29] | 14.0 [2, 27] | 83.3 [69, 95] | 40.9 [23, 57] |
Addition pseudo-targets are reliable. For removal pseudo-targets, three of the four checks pass in 73–78 % of ratings; the weak point is the sound, which is not removed in 72.7 % of ratings. On the full test set the judge gives the same picture: the sound is not removed in 81.7 % of removal pseudo-targets.
To test whether incomplete sound removal is specific to our pipeline, we paired the same pseudo-target video with removal audio from other methods (200 test clips, updated judge). Audio success = target sound removed, other sounds kept, no major audio artifact.
| removal audio from | sound removed % | other sounds kept % | audio success % |
|---|---|---|---|
| our pseudo-target (SAM-Audio with best-of-seed selection) | 21.0 | 84.5 | 16.0 |
| SAM-Audio [3], single pass | 25.5 | 72.5 | 17.0 |
| CoherentAVEdit [4] | 7.5 | 74.5 | 4.5 |
| AUDIT [5] | 44.5 | 9.0 | 1.0 |
| ZEUS [6] | 4.5 | 86.0 | 2.5 |
| no separation (source audio) | 0.5 | 99.5 | 0.5 |
Our pseudo-targets are on par with single-pass SAM-Audio on audio success and keep the other sounds better (+12 pp, p = 0.002). They are significantly better than the three audio editors (p < 0.001 each). Incomplete sound removal is therefore a limit of current audio separation.
Verified subset. We will release a per-clip “verified” flag for the 261 test clips whose pseudo-target passes the sound check, and report removal on this subset. On it, our model removes the sound in 38.7 % of clips (11.5 % on the other clips), against 20.7 % for InstructAV2AV (p < 0.001) and 11.5 % for CoherentAVEdit.
The judge is updated to the new Gemini model. It now receives the source clip together with the edited clip (and the reference frame for addition), and answers the same six questions as the raters. For consistency it first decides whether the clip contains a checkable audio-visual event.
Why a checkable event matters. Synchronisation can only be judged when the clip contains an event that is both seen and heard at an identifiable moment, such as a dog barking or a hammer striking. A clip with only continuous ambient sound has nothing to check. In the submission such clips counted as consistent, and they accounted for 55–82 % of the “Yes” verdicts. The updated judge reports them separately, and they never count as synchronised.
Full test set (1,425 clips).
| task | system | core success % | has checkable event % | synchronised, among checkable % | core success and verified consistency % |
|---|---|---|---|---|---|
| removal | pseudo-target | 4.8 | 69.1 | 11.0 | 0.7 |
| removal | Ours | 0.9 | 71.7 | 19.7 | 0.0 |
| removal | InstructAV2AV | 1.1 | 82.9 | 30.6 | 0.1 |
| removal | AVI-Edit (pseudo-target audio given) | 0.0 | 82.0 | 46.5 | 0.0 |
| addition | pseudo-target | 43.6 | 95.9 | 89.5 | 41.4 |
| addition | Ours | 5.5 | 87.7 | 35.4 | 5.0 |
| addition | InstructAV2AV | 2.4 | 89.5 | 17.7 | 1.8 |
| addition | CoherentAVEdit (pseudo-target video given) | 37.4 | 93.9 | 82.1 | 35.5 |
Consistency is reported together with core success because on its own it rewards leaving the clip unchanged: AVI-Edit has the highest consistency on removal but edits the object in only 9 % of clips.
Degraded cross-attention outputs (AVIO-XAttn). In the submission these outputs scored above the main model (removal VES 68 vs 40). Under the updated judge they receive visual preservation 1 %, major artifact 100 % and core success 0 %.
Agreement with the raters (86 rated outputs; human label = strict majority of the raters, ties dropped):
| question | κ removal / addition | raw agreement % removal / addition | judge passes an output the raters fail, % removal / addition |
|---|---|---|---|
| visual edit | 0.60 / 0.29 | 80 / 74 | 18 / 0 |
| acoustic edit | 0.52 / 0.32 | 82 / 74 | 10 / 33 |
| visual preservation | 0.70 / 0.27 | 85 / 82 | 7 / 50 |
| major visual artifact | 0.80 / 0.27 | 90 / 80 | 0 / 33 |
| core success | 0.63 / 0.42 | 93 / 70 | 3 / 12 |
The judge agrees well with the raters on removal. On addition it is stricter than the raters: it passes 67 % of the visual additions where the raters pass 92 %. Its errors are mostly missed successes; it rarely passes an output that the raters fail.
The raters also confirm the pattern of the submission’s Table 1(c): for removal pseudo-targets the object is visually removed far more often than its sound (raters 78 % vs 27 %; Table 1(c) VES 55.4 % vs AES 11.1 %).
The raters did not label consistency, so the consistency columns come from the judge only. Agreement between raters on audio-artifact severity is low, so audio quality is reported with FAD / KL / LSD.
Same 200 test clips, updated judge, each variant against the main model on the same clip. Every difference against the cross-attention variant is significant (p < 0.001); removing the reference frame costs the visual addition but not the acoustic one.
| comparison | task | measure | main model % | variant % | difference [95 % CI] | McNemar p |
|---|---|---|---|---|---|---|
| main model without the reference frame | addition | object added visually | 18.0 | 0.5 | −17.5 pp [−22.9, −12.4] | < 0.001 |
| main model without the reference frame | addition | sound added | 33.0 | 29.5 | −3.5 pp (CI includes 0) | not significant |
| cross-attention conditioning | addition | background preserved | 45.5 | 3.0 | −42.5 pp [−49.5, −35.5] | < 0.001 |
| cross-attention conditioning | addition | core success | 6.0 | 0.0 | −6.0 pp [−9.5, −3.0] | < 0.001 |
| cross-attention conditioning | removal | background preserved | 27.0 | 2.0 | −25.0 pp [−31.2, −19.1] | < 0.001 |
| cross-attention conditioning | removal | other sounds preserved | 79.0 | 38.5 | −40.5 pp [−48.5, −32.5] | < 0.001 |
The purpose of this component is to train one model on a mixture of reference-conditioned and reference-free samples, so that it handles removal (no reference) and addition (reference given). For the mixing we followed the curriculum of Ditto [7], which anneals the probability of providing the reference frame.
We trained an otherwise identical model with a constant keep-probability of 0.5 and compared both at the same step (33.6k, the furthest the new run reached) on 200 test clips.
| 200 clips, step 33.6k | constant 0.5 | progressive | paired difference (constant − progressive) |
|---|---|---|---|
| removal PSNR (dB) ↑ | 20.01 | 19.18 | +0.83 [+0.52, +1.15] |
| removal mel-L1 ↓ | 9.61 | 9.23 | +0.39 [+0.18, +0.61] |
| addition PSNR (dB) ↑ | 18.51 | 18.58 | −0.07 [−0.16, +0.02] |
| removal core success % ↑ | 0.5 | 1.5 | −1.0 [−2.5, 0.0], p = 0.50 |
| addition core success % ↑ | 3.5 | 2.0 | +1.5 [−0.5, +4.0], p = 0.38 |
The two schedules perform comparably overall. We will present mixed reference / reference-free training as the component and the annealed schedule as one way to implement it. The sentence in the paper will read: “During training the reference frame is randomly dropped, so that a single model learns editing both with and without visual guidance. In our implementation the keep-probability is annealed from 1 to 0 over the first 100k steps, following Ditto; a constant probability of 0.5 performs comparably at equal training steps.”
No retraining is needed, because training already drops the reference in the same way. Same 200 clips and seeds, updated judge.
| addition | object added visually % | sound added % |
|---|---|---|
| Ours, with the reference frame | 18.0 | 33.0 |
| Ours, without a reference | 0.5 | 29.5 |
The sound is still added from the text alone. The visual addition needs the reference, because the text does not determine what the object looks like or where it is.
Examples (the same 11 clips are used in 5.3). Same seed with and without the reference.
Add acoustic guitar

Add percussion instrument

Add stringed instrument

Add frogs

Add male voice

Add birds

Add guinea fowl

Add vehicle engine

Add birds

Add engine

Add birds

On 11 test clips we replaced the reference frame with a modified one and kept everything else fixed (same seed).
| reference given to the model | what happens |
|---|---|
| exact first frame | the object is added |
| a frame from a later moment (4.5 s) | works almost as well as the exact first frame |
| grayscale | the object is added in some clips; colour returns within about a second |
| JPEG at quality 5, blur, 16×16 up-sampled | the object degrades or disappears |
| mirrored frame | the layout returns to the source and the object fades |
| first frame of a different clip | fails |
A reference that keeps the object identifiable and in place works; heavily damaged frames fail.
Examples. Top: the reference frame given to the model. Bottom: our output. Same seed in every column.
acoustic guitar— reference given (top) and our output (bottom) for each variant








percussion instrument— reference given (top) and our output (bottom) for each variant








stringed instrument— reference given (top) and our output (bottom) for each variant








frogs— reference given (top) and our output (bottom) for each variant







male voice— reference given (top) and our output (bottom) for each variant








birds— reference given (top) and our output (bottom) for each variant








guinea fowl— reference given (top) and our output (bottom) for each variant







vehicle engine— reference given (top) and our output (bottom) for each variant








birds— reference given (top) and our output (bottom) for each variant








engine— reference given (top) and our output (bottom) for each variant








birds— reference given (top) and our output (bottom) for each variant








The paper checkpoint is run without any retraining on data outside AVIOBench. The clips in 6.1 and 6.2 come from an external pool (AV-SFT). Their YouTube ids appear in no AVIOBench split (train, validation or test), and the clips are not inpainted or edited in any way. The clips in 6.3 are source clips of JAVEdit-100k.
No reference frame. Prompt: a video without {object}.
Remove dog
the dog is removed, the toy and the window are kept; the audio level drops by 12.8 dB
Remove frog
the frog is removed, the pond is preserved; the audio level drops by 5.3 dB
Remove rooster
the rooster is removed, the ground is filled plausibly
Remove train
the whole train is removed, rails and background are filled plausibly
Removal works when the object is compact and the background is simple.
In the paper, the addition source is an inpainted video and the reference frame is taken from the target video. Here neither is true.
a video with {object}.| Examples: real source video | edited first frame given to the model | our output (with audio). |
Add birds

a bird flies across the sky in front of the cliff; the sea is preserved
Add car

the car stays on the quay next to the canal; boats, water and lights are preserved
Add dog

the dog stays on the shore for the whole clip while the waves keep moving; the sea is preserved
Add dog

the dog stays next to the wind chimes and a bark is added to the original chime sound
Source clips from the subject-removal part of JAVEdit-100k [2]. No reference frame. The object name is taken from the JAVEdit instruction and inserted into our prompt, a video without {object}.
Remove person
JAVEdit instruction: "Remove the person jogging on the pathway and eliminate the sounds of their footsteps and breathing."
the jogger is removed for the whole clip; the lit garden path is preserved
Remove person
JAVEdit instruction: "Remove the person from the video."
the person is removed; the filled region is blurred
Remove saxophone player
JAVEdit instruction: "Remove the person playing the saxophone and the instrument from the video."
the player and the instrument are removed; the background lights are preserved
We fine-tuned InstructAV2AV [1] on the AVIOBench training split, starting from its released checkpoint. It is fine-tuned separately for removal and for addition, which gives two specialist models (152 + 151 = 303 H100-hours). Ours is one model for both tasks (359 H100-hours). Both are evaluated on the same 200 test clips.
Human core success (11 clips per task; 50 ratings on removal, 44 on addition):
| core success % [95 % CI] | Ours (one model) | InstructAV2AV fine-tuned (two models) | paired difference |
|---|---|---|---|
| removal | 14.0 [2, 27] | 20.0 [4, 40] | −6.0 pp [−17.7, +7.7], not significant |
| addition | 40.9 [23, 57] | 45.5 [20, 70] | −4.5 pp [−34.1, +27.3], not significant |
Raters find that ours preserves the other sounds better in removal (86 % vs 70 %, +16 pp [+2, +32]).
The raters also scored a third system on the same clips: a cascade that keeps our video and replaces the audio with the output of CoherentAVEdit fine-tuned on AVIOBench. Its core success is 20.0 % [4, 37] on removal and 43.2 % [25, 61] on addition, in the same range as the two systems above.
Objective metrics (against the pseudo-target):
| removal: Ours | removal: fine-tuned | addition: Ours | addition: fine-tuned | |
|---|---|---|---|---|
| video: PSNR ↑ | 20.04 | 21.15 | 18.79 | 17.44 |
| video: SSIM ↑ | 0.655 | 0.715 | 0.619 | 0.622 |
| video: LPIPS ↓ | 0.282 | 0.237 | 0.228 | 0.256 |
| video: FVD ↓ | 482.8 | 274.5 | 610.9 | 373.9 |
| audio: FAD ↓ | 2.91 | 2.54 | 4.29 | 4.09 |
| audio: KL ↓ | 1.49 | 1.59 | 1.41 | 1.79 |
| audio: LSD ↓ | 12.84 | 17.31 | 11.64 | 16.73 |
Performance is comparable. Ours is better on KL and LSD on both tasks and on addition PSNR and LPIPS; the fine-tuned baseline is better on removal video, FVD and FAD. The advantage of our model is that one set of weights handles both removal and addition, with or without a reference frame, while InstructAV2AV provides a dedicated checkpoint for each type of edit, with separate ones for insertion and removal. One checkpoint is preferable because each training pair supervises both directions, so an object and its sound are learned once and serve both tasks, and because only one model has to be trained, stored and run.
Fine-tuning also shows the value of the dataset: under the updated judge it raises InstructAV2AV’s visual removal rate from 24.0 % to 47.2 %.
Every method receives the same object name for a clip, in that method’s own input format. No prompt is tuned per clip.
| method | removal prompt | addition prompt | other inputs |
|---|---|---|---|
| Ours | a video without {object} |
a video with {object} |
source video and audio; first frame for addition |
| CoherentAVEdit | a video without {object} |
a video with {object} |
pseudo-target video (it edits the audio only) |
| AVI-Edit | a video without {object} |
a video with {object} |
object mask; pseudo-target audio |
| InstructAV2AV (released and fine-tuned) | Remove the {object}. |
Add the {object}. |
source video and audio; first frame for addition |
InstructAV2AV is an instruction-following model, so it receives the same content phrased as an instruction. The AVI-Edit paper does not prescribe a removal prompt; on 16 clips we also tried the example prompt shown in its paper together with a box mask, and the ranking did not change.
YouTube identity. We split the data by 5-second segment, not by source video, so two different segments of the same YouTube video can fall in train and in test. No segment appears in two splits, but 162 of the 1,425 test clips (11.4 %) come from a video that also has a different segment in train.
We re-evaluated every system on the 1,254 test clips whose source video appears in neither train nor validation. No metric moves by more than 0.1 FAD, 0.12 dB PSNR or 1.0 pp core success, and no ranking changes. Our model is not better on the 162 overlapping clips than on the rest (core success 1.2 % vs 0.9 % on removal, 4.9 % vs 5.6 % on addition). We will report the results on the 1,254 clips and release the benchmark with a split by YouTube id.
Rare and unseen objects. Test clips grouped by how often their object occurs in training (1,713 distinct object names in train). Ours, updated judge, % of clips:
| never seen (78 clips) | 1–5 times (88) | 6–50 times (250) | > 50 times (1,009) | |
|---|---|---|---|---|
| removal: object removed visually | 41.0 | 43.2 | 36.4 | 38.8 |
| removal: sound removed | 14.1 | 19.3 | 17.6 | 16.2 |
| addition: object added visually | 17.9 | 11.4 | 19.2 | 14.7 |
| addition: sound added | 50.0 | 27.3 | 34.0 | 38.0 |
Our model performs similarly on rare, unseen and frequent objects. Train and test share the same object categories, so we do not claim generalisation to unseen categories.
The model is trained with a single template, a video with {object} / a video without {object}. Here the paper checkpoint is run, without retraining, with three phrasings it never saw in training. Everything else is unchanged (same 40 test clips, same seeds, same reference frames for addition).
| phrasing | removal prompt | addition prompt |
|---|---|---|
| training template | a video without {object} |
a video with {object} |
| instruction | Remove the {object}. |
Add the {object}. |
| descriptive sentence | the same scene, but the {object} is gone and can no longer be heard |
the same scene, now with {object} that can be seen and heard |
| paraphrase | a clip in which there is no {object} |
a clip that includes {object} |
Results under the updated judge (40 clips, % of clips):
| phrasing | removal: object removed | removal: sound removed | removal: other sounds preserved | addition: object added | addition: sound added | addition: other sounds preserved |
|---|---|---|---|---|---|---|
| training template | 35.0 | 7.5 | 77.5 | 12.5 | 35.0 | 90.0 |
| instruction | 22.5 | 7.5 | 92.5 | 7.5 | 35.0 | 90.0 |
| descriptive sentence | 30.0 | 10.0 | 77.5 | 10.0 | 40.0 | 87.5 |
| paraphrase | 30.0 | 5.0 | 80.0 | 12.5 | 42.5 | 87.5 |
The unseen phrasings behave close to the training template. The object name carries the edit; the surrounding wording matters little.
Examples. Same clip, same seed and (for addition) same reference frame; only the prompt changes.
Remove birds
Remove vehicle
Remove birds
Add frogs
Add piano
Add vehicle engine
The design is not tied to JavisDiT++. Source-conditioned modulation acts through the normalization layers, and the reference frame enters through the input sequence. Larger audiovisual diffusion transformers such as LTX-2 have both, so the method can be applied to them. Because of resource constraints we tested it on JavisDiT++, a lightweight joint audio-video backbone: our model has 2.26 B parameters and trains in 359 H100-hours. The paper will state that the results are demonstrated on JavisDiT++ and that applying the design to larger backbones is the next step.
What the experiments on JavisDiT++ show about the mechanism itself (Section 4): replacing source-conditioned modulation with cross-attention lowers background preservation by 42.5 pp on addition and 25.0 pp on removal, and removing the reference frame lowers visual addition by 17.5 pp. One backbone supports inputs with and without a reference and both tasks.
[1] Zheng et al., InstructAV2AV: Instruction-Guided Audio-Video Joint Editing, arXiv:2605.18467.
[2] Chen et al., JAVEdit: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation, arXiv:2606.03168.
[3] Shi et al., SAM Audio: Segment Anything in Audio, arXiv:2512.18099.
[4] Ishii et al., Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits, arXiv:2512.07209.
[5] Wang et al., AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models, NeurIPS 2023.
[6] Manor and Michaeli, Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion, ICML 2024.
[7] Ditto, arXiv:2510.15742.
Each pair is built from one real clip: the named object is segmented with a text-prompted video segmenter and inpainted out, and its sound is separated out of the soundtrack. A pair is used in both directions — original → edited for removal, edited → original for addition. 24,444 training / 1,389 validation / 1,425 test pairs; the test set covers 374 object descriptions.
Object: flute
Object: brass instrument
Object: train engine
Object: birds chirping
Object: animal
Object: white-tailed eagle
Object: individual
Object: helicopter
Our model edits the picture and the soundtrack jointly. "Best" refers to the edit itself: in these clips the object named in the prompt and its sound are both removed or added, and the edit holds for the whole clip. The outputs can still show blur and deformation, which we state as a limitation. These showcase clips reappear in the baseline comparisons of sections 3 and 4 under the same tags.
Text prompt only. The object is clearly visible and audible in the input and is gone from both the picture and the soundtrack in our output.
Remove brass instrument
Remove birds chirping
Remove animal
Remove individual
Remove dogs
Text prompt plus the first frame of the target. The input contains neither the object nor its sound; our output adds both, in sync.
Add flute
Add train engine
Add running water
Add white-tailed eagle
Add helicopter
Unimodal editors change one modality and leave the other untouched, and none of them supports addition, so this comparison is on removal (clips R1–R5). Each baseline is judged in its own modality; one cascade of a video editor and an audio editor follows.
ROSE, VideoPainter and LGVI edit the picture only. All clips in this block are silent, so only the visual edit is compared.
Remove brass instrument
Remove birds chirping
Remove animal
Remove individual
Remove dogs
ZEUS and AUDIT edit the soundtrack only. Audio players only, so only the audio edit is compared.
Remove brass instrument
Remove birds chirping
Remove animal
Remove individual
Remove dogs
The video edited by ROSE combined with the audio edited by ZEUS — two independent models, neither aware of the other's edit.
Remove brass instrument
Remove birds chirping
Remove animal
Remove individual
Remove dogs
InstructAV2AV and ours generate both streams. AVI-Edit generates the video and is given the target audio track; CoherentAVEdit generates the audio and is given the edited video.
Clips R1–R5. All methods receive the input clip and the text prompt.
Remove brass instrument
Remove birds chirping
Remove animal
Remove individual
Remove dogs
Clips A1–A5. Every method that generates video is additionally given the first frame of the target.
Add flute
Add train engine
Add running water
Add white-tailed eagle
Add helicopter
Further results of our model, selected automatically: an audio-visual judge (a multimodal LLM) must confirm the edit in the video, in the audio, and their synchronisation.
Text prompt only (R6–R29).
Remove birds
Remove engine
Remove a person
Remove high-revving engine
Remove birds
Remove birds
Remove birds
Remove trumpets
Remove running water
Remove birds
Remove knife
Remove clarinet
Remove chirping birds
Remove birds
Remove gunshots
Remove birds
Remove animal
Remove dishes
Remove owl
Remove worker
Remove idling engine
Remove livestock
Remove birds
Remove man
Text prompt plus first frame (A6–A30).
Add didgeridoo
Add didgeridoo
Add clarinet
Add birds
Add bass guitar
Add pianist
Add male performer
Add accordion
Add person
Add drum kit
Add saxophonist
Add bald man
Add individual
Add birds
Add birds
Add birds
Add individual
Add cow
Add birds
Add crowd
Add brass instrument
Add birds
Add individual
Add individual
Add helicopter
Typical failures of the dataset and of our model, grouped by type. They were found with the audio-visual judge, which is asked whether the object is visible, whether it is audible, and whether sound and picture are in sync; its explanation is quoted under each example.
Errors in how the pairs were built; they bound what any model trained or scored on this data can reach.
The separated soundtrack of the edited clip still contains the object's sound.
Object: idling engine
Judge: The idling engine sound is audibly present and remains consistent without interruption across the clip.
Object: drumming
Judge: The sound of drumming is clearly audible and remains consistent throughout the clip.
The inpainted picture of the edited clip still shows the object or a recognisable remnant.
Object: truck engine
Judge: The engine is clearly visible and remains consistent in its appearance and operation across the entire clip.
Object: toilet
Judge: The toilet is consistently visible as the camera pulls back and focuses.
The prompt names a sound source that is off-screen, or does not match what the picture shows.
Object: motorcycle engine
Judge: The video shows a car engine, not a motorcycle engine; the target is entirely missing.
Object: engine
Judge: The target engine is not visually present at any point in the clip.
Our model, text prompt only, on clips whose input clearly shows and sounds the object.
The output is close to the input: the object is still fully visible.
Remove truck engine
Judge: The engine remains visible throughout the entire clip.
Remove toilet
Judge: The target object (toilet) remains visible as a close-up surface for the entire duration.
The object is only partly erased and leaves a blurred or ghost-like shape.
Remove train
Judge: A massive, moving artifact retaining the shape and color of the train is visible for the entire clip.
Remove train
Judge: A large, moving ghost image of the train remains visible on the tracks for the entire clip.
The visual removal succeeds while the soundtrack keeps the object's sound, although the reference audio is clean.
Remove electronic device
Judge: The buzzing sound of the electronic device remains clearly audible for the entire duration.
Remove motorcycle engine
Judge: The sound of a motorcycle engine is clearly audible continuously from the beginning to the end of the clip.
Our model, text prompt plus first frame, on clips whose input has neither the object nor its sound.
The object starts from the given first frame but its shape or identity changes over time.
Add cargo ship
Judge: The ship structure is highly distorted and morphs continuously, lacking consistency.
Add animal
Judge: The animal severely distorts and loses its initial identity, becoming an unrecognizable shape starting around 0.5s.
The object is present at the first frame and then disappears.
Add male vocalist
Judge: No singing action is visible, and the person's head disappears entirely at 00:01.000.
Add canine
Judge: The dog arbitrarily disappears and blends into the floor texture at 00:01.000 and 00:04.000.
Both streams contain the object, but sound events do not line up with the visible action.
Add barking dog
Judge: The audible barks do not match the visible actions of the dog, which appears to be sniffing or eating rather than barking, especially noticeable around 00:03 and 00:04.
Add rooster
Judge: The rooster crowing sound occurs between 00:01.8 and 00:03.0, but the visible rooster's beak remains closed and it does not perform the physical action of crowing.