ICLR 2027 Submission

Demo Page

Contents

All clips are 5 s, 256×256, 16 fps. Every example comes from the held-out test set; the tag on each card (R = removal, A = addition, D = dataset, F = failure) identifies it across sections. Input is what a method receives, Reference is the ground-truth edit.

Rebuttal

Supplementary results for the responses: full tables, protocols and examples that the responses point to. Each section is tagged with the reviewer weakness (W) / question (Q) it answers, in the numbering of each response; the index below goes from each reviewer's points to the sections. Everything below this section is unchanged from the original submission; the numbering of sections 1–6 is kept so that references in the paper and the responses still match.

Definitions. Core success means that, on one clip, the target is removed / added both visually and acoustically, the unrelated visual and audio content is preserved, and there is no major artifact. Confidence intervals are 95 % cluster bootstrap over source videos. Paired tests are exact McNemar. Pseudo-target is the reference output built by our data pipeline (called “oracle” in parts of the submission).

1. Human study r1hr Q1 · hmxL W1, Q1 · Rmu9 W3

Protocol. We collected 11 questionnaires from 10 raters. Each screen shows the source clip, the instruction and one anonymised output; the order is shuffled per rater and the raters do not know which system produced an output. For each output they answer:

  1. Is the object visually removed / added?
  2. Is its sound removed / added?
  3. Is the visual background preserved?
  4. Are the other sounds preserved?
  5. Visual artifact: none / minor / major.
  6. Audio artifact: none / minor / major.

Questions 1–4 are answered Pass / Fail / Uncertain. We report the percentage of ratings that are not Pass. Core success = Pass on 1–4 and no major artifact.

Pseudo-targets and our outputs.

% of ratings removal: pseudo-target removal: ours addition: pseudo-target addition: ours
sound edit fails 72.7 70.0 7.4 40.9
visual edit fails 21.8 52.0 1.9 18.2
visual background damaged 27.3 42.0 1.9 20.5
other sounds damaged 25.5 14.0 1.9 15.9
major artifact 32.7 48.0 7.4 31.8
core success [95 % CI] 16.4 [5, 29] 14.0 [2, 27] 83.3 [69, 95] 40.9 [23, 57]

Addition pseudo-targets are reliable. For removal pseudo-targets, three of the four checks pass in 73–78 % of ratings; the weak point is the sound, which is not removed in 72.7 % of ratings. On the full test set the judge gives the same picture: the sound is not removed in 81.7 % of removal pseudo-targets.

2. Audio separation: our pipeline against other methods r1hr Q1 · hmxL Q1 · Rmu9 W3

To test whether incomplete sound removal is specific to our pipeline, we paired the same pseudo-target video with removal audio from other methods (200 test clips, updated judge). Audio success = target sound removed, other sounds kept, no major audio artifact.

removal audio from sound removed % other sounds kept % audio success %
our pseudo-target (SAM-Audio with best-of-seed selection) 21.0 84.5 16.0
SAM-Audio [3], single pass 25.5 72.5 17.0
CoherentAVEdit [4] 7.5 74.5 4.5
AUDIT [5] 44.5 9.0 1.0
ZEUS [6] 4.5 86.0 2.5
no separation (source audio) 0.5 99.5 0.5

Our pseudo-targets are on par with single-pass SAM-Audio on audio success and keep the other sounds better (+12 pp, p = 0.002). They are significantly better than the three audio editors (p < 0.001 each). Incomplete sound removal is therefore a limit of current audio separation.

Verified subset. We will release a per-clip “verified” flag for the 261 test clips whose pseudo-target passes the sound check, and report removal on this subset. On it, our model removes the sound in 38.7 % of clips (11.5 % on the other clips), against 20.7 % for InstructAV2AV (p < 0.001) and 11.5 % for CoherentAVEdit.

3. Updated judge r1hr Q2 · hmxL W2, Q2

The judge is updated to the new Gemini model. It now receives the source clip together with the edited clip (and the reference frame for addition), and answers the same six questions as the raters. For consistency it first decides whether the clip contains a checkable audio-visual event.

Why a checkable event matters. Synchronisation can only be judged when the clip contains an event that is both seen and heard at an identifiable moment, such as a dog barking or a hammer striking. A clip with only continuous ambient sound has nothing to check. In the submission such clips counted as consistent, and they accounted for 55–82 % of the “Yes” verdicts. The updated judge reports them separately, and they never count as synchronised.

Full test set (1,425 clips).

task system core success % has checkable event % synchronised, among checkable % core success and verified consistency %
removal pseudo-target 4.8 69.1 11.0 0.7
removal Ours 0.9 71.7 19.7 0.0
removal InstructAV2AV 1.1 82.9 30.6 0.1
removal AVI-Edit (pseudo-target audio given) 0.0 82.0 46.5 0.0
addition pseudo-target 43.6 95.9 89.5 41.4
addition Ours 5.5 87.7 35.4 5.0
addition InstructAV2AV 2.4 89.5 17.7 1.8
addition CoherentAVEdit (pseudo-target video given) 37.4 93.9 82.1 35.5

Consistency is reported together with core success because on its own it rewards leaving the clip unchanged: AVI-Edit has the highest consistency on removal but edits the object in only 9 % of clips.

Degraded cross-attention outputs (AVIO-XAttn). In the submission these outputs scored above the main model (removal VES 68 vs 40). Under the updated judge they receive visual preservation 1 %, major artifact 100 % and core success 0 %.

Agreement with the raters (86 rated outputs; human label = strict majority of the raters, ties dropped):

question κ removal / addition raw agreement % removal / addition judge passes an output the raters fail, % removal / addition
visual edit 0.60 / 0.29 80 / 74 18 / 0
acoustic edit 0.52 / 0.32 82 / 74 10 / 33
visual preservation 0.70 / 0.27 85 / 82 7 / 50
major visual artifact 0.80 / 0.27 90 / 80 0 / 33
core success 0.63 / 0.42 93 / 70 3 / 12

The judge agrees well with the raters on removal. On addition it is stricter than the raters: it passes 67 % of the visual additions where the raters pass 92 %. Its errors are mostly missed successes; it rarely passes an output that the raters fail.

The raters also confirm the pattern of the submission’s Table 1(c): for removal pseudo-targets the object is visually removed far more often than its sound (raters 78 % vs 27 %; Table 1(c) VES 55.4 % vs AES 11.1 %).

The raters did not label consistency, so the consistency columns come from the judge only. Agreement between raters on audio-artifact severity is low, so audio quality is reported with FAD / KL / LSD.

4. Paired tests for the ablation hmxL Q2 · Rmu9 W1

Same 200 test clips, updated judge, each variant against the main model on the same clip. Every difference against the cross-attention variant is significant (p < 0.001); removing the reference frame costs the visual addition but not the acoustic one.

comparison task measure main model % variant % difference [95 % CI] McNemar p
main model without the reference frame addition object added visually 18.0 0.5 −17.5 pp [−22.9, −12.4] < 0.001
main model without the reference frame addition sound added 33.0 29.5 −3.5 pp (CI includes 0) not significant
cross-attention conditioning addition background preserved 45.5 3.0 −42.5 pp [−49.5, −35.5] < 0.001
cross-attention conditioning addition core success 6.0 0.0 −6.0 pp [−9.5, −3.0] < 0.001
cross-attention conditioning removal background preserved 27.0 2.0 −25.0 pp [−31.2, −19.1] < 0.001
cross-attention conditioning removal other sounds preserved 79.0 38.5 −40.5 pp [−48.5, −32.5] < 0.001

5. Reference frame r1hr Q3 · hmxL W3

5.1 Constant vs progressive reference dropout

The purpose of this component is to train one model on a mixture of reference-conditioned and reference-free samples, so that it handles removal (no reference) and addition (reference given). For the mixing we followed the curriculum of Ditto [7], which anneals the probability of providing the reference frame.

We trained an otherwise identical model with a constant keep-probability of 0.5 and compared both at the same step (33.6k, the furthest the new run reached) on 200 test clips.

200 clips, step 33.6k constant 0.5 progressive paired difference (constant − progressive)
removal PSNR (dB) ↑ 20.01 19.18 +0.83 [+0.52, +1.15]
removal mel-L1 ↓ 9.61 9.23 +0.39 [+0.18, +0.61]
addition PSNR (dB) ↑ 18.51 18.58 −0.07 [−0.16, +0.02]
removal core success % ↑ 0.5 1.5 −1.0 [−2.5, 0.0], p = 0.50
addition core success % ↑ 3.5 2.0 +1.5 [−0.5, +4.0], p = 0.38

The two schedules perform comparably overall. We will present mixed reference / reference-free training as the component and the annealed schedule as one way to implement it. The sentence in the paper will read: “During training the reference frame is randomly dropped, so that a single model learns editing both with and without visual guidance. In our implementation the keep-probability is annealed from 1 to 0 over the first 100k steps, following Ditto; a constant probability of 0.5 performs comparably at equal training steps.”

5.2 Main model without a reference frame

No retraining is needed, because training already drops the reference in the same way. Same 200 clips and seeds, updated judge.

addition object added visually % sound added %
Ours, with the reference frame 18.0 33.0
Ours, without a reference 0.5 29.5

The sound is still added from the text alone. The visual addition needs the reference, because the text does not determine what the object looks like or where it is.

Examples (the same 11 clips are used in 5.3). Same seed with and without the reference.

N1

Add acoustic guitar

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference
N2

Add percussion instrument

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference
N3

Add stringed instrument

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference
N4

Add frogs

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference
N5

Add male voice

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference
N6

Add birds

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference
N7

Add guinea fowl

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference
N8

Add vehicle engine

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference
N9

Add birds

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference
N10

Add engine

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference
N11

Add birds

Input
Reference frame
Reference frame
first frame of the target
Ours
with the reference
Ours
no reference

5.3 Imperfect reference frames

On 11 test clips we replaced the reference frame with a modified one and kept everything else fixed (same seed).

reference given to the model what happens
exact first frame the object is added
a frame from a later moment (4.5 s) works almost as well as the exact first frame
grayscale the object is added in some clips; colour returns within about a second
JPEG at quality 5, blur, 16×16 up-sampled the object degrades or disappears
mirrored frame the layout returns to the source and the object fades
first frame of a different clip fails

A reference that keeps the object identifiable and in place works; heavily damaged frames fail.

Examples. Top: the reference frame given to the model. Bottom: our output. Same seed in every column.

N1 Add acoustic guitar — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
reference: frame of another clip
frame of another clip
N2 Add percussion instrument — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
reference: frame of another clip
frame of another clip
N3 Add stringed instrument — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
reference: frame of another clip
frame of another clip
N4 Add frogs — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
N5 Add male voice — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
reference: frame of another clip
frame of another clip
N6 Add birds — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
reference: frame of another clip
frame of another clip
N7 Add guinea fowl — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
N8 Add vehicle engine — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
reference: frame of another clip
frame of another clip
N9 Add birds — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
reference: frame of another clip
frame of another clip
N10 Add engine — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
reference: frame of another clip
frame of another clip
N11 Add birds — reference given (top) and our output (bottom) for each variant
reference: exact first frame
exact first frame
reference: later frame (4.5 s)
later frame (4.5 s)
reference: grayscale
grayscale
reference: JPEG q = 5
JPEG q = 5
reference: blur
blur
reference: 16×16 up-sampled
16×16 up-sampled
reference: mirrored
mirrored
reference: frame of another clip
frame of another clip

6. Zero-shot performance hmxL W3 · Rmu9 Q2

The paper checkpoint is run without any retraining on data outside AVIOBench. The clips in 6.1 and 6.2 come from an external pool (AV-SFT). Their YouTube ids appear in no AVIOBench split (train, validation or test), and the clips are not inpainted or edited in any way. The clips in 6.3 are source clips of JAVEdit-100k.

6.1 Removal on real videos

No reference frame. Prompt: a video without {object}.

Z1

Remove dog

Input
real video, original audio
Ours
text only

the dog is removed, the toy and the window are kept; the audio level drops by 12.8 dB

Z2

Remove frog

Input
real video, original audio
Ours
text only

the frog is removed, the pond is preserved; the audio level drops by 5.3 dB

Z3

Remove rooster

Input
real video, original audio
Ours
text only

the rooster is removed, the ground is filled plausibly

Z4

Remove train

Input
real video, original audio
Ours
text only

the whole train is removed, rails and background are filled plausibly

Removal works when the object is compact and the background is simple.

6.2 Addition on real videos

In the paper, the addition source is an inpainted video and the reference frame is taken from the target video. Here neither is true.

  • Source video. A real external clip with its original audio.
  • Reference frame. An independent image editor (GPT image editing) receives the first frame of the real clip and the instruction “add one {object} in the foreground”. The object name is chosen by the same model from a fixed list of sounding objects, given only the frame. The editor never sees any AVIOBench data. No target video exists.
  • Model. The paper checkpoint with the main inference settings. Prompt: a video with {object}.
Examples: real source video edited first frame given to the model our output (with audio).
Y1

Add birds

Input
real video, original audio
Reference frame
Reference frame
first frame edited by an image editor
Ours
text + edited first frame

a bird flies across the sky in front of the cliff; the sea is preserved

Y2

Add car

Input
real video, original audio
Reference frame
Reference frame
first frame edited by an image editor
Ours
text + edited first frame

the car stays on the quay next to the canal; boats, water and lights are preserved

Y3

Add dog

Input
real video, original audio
Reference frame
Reference frame
first frame edited by an image editor
Ours
text + edited first frame

the dog stays on the shore for the whole clip while the waves keep moving; the sea is preserved

Y4

Add dog

Input
real video, original audio
Reference frame
Reference frame
first frame edited by an image editor
Ours
text + edited first frame

the dog stays next to the wind chimes and a bark is added to the original chime sound

6.3 JAVEdit-100k

Source clips from the subject-removal part of JAVEdit-100k [2]. No reference frame. The object name is taken from the JAVEdit instruction and inserted into our prompt, a video without {object}.

J1

Remove person

Input
JAVEdit-100k source clip
Ours
text only

JAVEdit instruction: "Remove the person jogging on the pathway and eliminate the sounds of their footsteps and breathing."
the jogger is removed for the whole clip; the lit garden path is preserved

J2

Remove person

Input
JAVEdit-100k source clip
Ours
text only

JAVEdit instruction: "Remove the person from the video."
the person is removed; the filled region is blurred

J3

Remove saxophone player

Input
JAVEdit-100k source clip
Ours
text only

JAVEdit instruction: "Remove the person playing the saxophone and the instrument from the video."
the player and the instrument are removed; the background lights are preserved

7. InstructAV2AV fine-tuned on our data r1hr W3 · hmxL Q3 · Rmu9 W2, Q1

We fine-tuned InstructAV2AV [1] on the AVIOBench training split, starting from its released checkpoint. It is fine-tuned separately for removal and for addition, which gives two specialist models (152 + 151 = 303 H100-hours). Ours is one model for both tasks (359 H100-hours). Both are evaluated on the same 200 test clips.

Human core success (11 clips per task; 50 ratings on removal, 44 on addition):

core success % [95 % CI] Ours (one model) InstructAV2AV fine-tuned (two models) paired difference
removal 14.0 [2, 27] 20.0 [4, 40] −6.0 pp [−17.7, +7.7], not significant
addition 40.9 [23, 57] 45.5 [20, 70] −4.5 pp [−34.1, +27.3], not significant

Raters find that ours preserves the other sounds better in removal (86 % vs 70 %, +16 pp [+2, +32]).

The raters also scored a third system on the same clips: a cascade that keeps our video and replaces the audio with the output of CoherentAVEdit fine-tuned on AVIOBench. Its core success is 20.0 % [4, 37] on removal and 43.2 % [25, 61] on addition, in the same range as the two systems above.

Objective metrics (against the pseudo-target):

  removal: Ours removal: fine-tuned addition: Ours addition: fine-tuned
video: PSNR ↑ 20.04 21.15 18.79 17.44
video: SSIM ↑ 0.655 0.715 0.619 0.622
video: LPIPS ↓ 0.282 0.237 0.228 0.256
video: FVD ↓ 482.8 274.5 610.9 373.9
audio: FAD ↓ 2.91 2.54 4.29 4.09
audio: KL ↓ 1.49 1.59 1.41 1.79
audio: LSD ↓ 12.84 17.31 11.64 16.73

Performance is comparable. Ours is better on KL and LSD on both tasks and on addition PSNR and LPIPS; the fine-tuned baseline is better on removal video, FVD and FAD. The advantage of our model is that one set of weights handles both removal and addition, with or without a reference frame, while InstructAV2AV provides a dedicated checkpoint for each type of edit, with separate ones for insertion and removal. One checkpoint is preferable because each training pair supervises both directions, so an object and its sound are learned once and serve both tasks, and because only one model has to be trained, stored and run.

Fine-tuning also shows the value of the dataset: under the updated judge it raises InstructAV2AV’s visual removal rate from 24.0 % to 47.2 %.

8. Prompts given to each method hmxL Q3

Every method receives the same object name for a clip, in that method’s own input format. No prompt is tuned per clip.

method removal prompt addition prompt other inputs
Ours a video without {object} a video with {object} source video and audio; first frame for addition
CoherentAVEdit a video without {object} a video with {object} pseudo-target video (it edits the audio only)
AVI-Edit a video without {object} a video with {object} object mask; pseudo-target audio
InstructAV2AV (released and fine-tuned) Remove the {object}. Add the {object}. source video and audio; first frame for addition

InstructAV2AV is an instruction-following model, so it receives the same content phrased as an instruction. The AVI-Edit paper does not prescribe a removal prompt; on 16 clips we also tried the example prompt shown in its paper together with a box mask, and the ranking did not change.

9. Split by YouTube id; rare and unseen objects r1hr W3

YouTube identity. We split the data by 5-second segment, not by source video, so two different segments of the same YouTube video can fall in train and in test. No segment appears in two splits, but 162 of the 1,425 test clips (11.4 %) come from a video that also has a different segment in train.

We re-evaluated every system on the 1,254 test clips whose source video appears in neither train nor validation. No metric moves by more than 0.1 FAD, 0.12 dB PSNR or 1.0 pp core success, and no ranking changes. Our model is not better on the 162 overlapping clips than on the rest (core success 1.2 % vs 0.9 % on removal, 4.9 % vs 5.6 % on addition). We will report the results on the 1,254 clips and release the benchmark with a split by YouTube id.

Rare and unseen objects. Test clips grouped by how often their object occurs in training (1,713 distinct object names in train). Ours, updated judge, % of clips:

  never seen (78 clips) 1–5 times (88) 6–50 times (250) > 50 times (1,009)
removal: object removed visually 41.0 43.2 36.4 38.8
removal: sound removed 14.1 19.3 17.6 16.2
addition: object added visually 17.9 11.4 19.2 14.7
addition: sound added 50.0 27.3 34.0 38.0

Our model performs similarly on rare, unseen and frequent objects. Train and test share the same object categories, so we do not claim generalisation to unseen categories.

10. Other prompts Rmu9 W4

The model is trained with a single template, a video with {object} / a video without {object}. Here the paper checkpoint is run, without retraining, with three phrasings it never saw in training. Everything else is unchanged (same 40 test clips, same seeds, same reference frames for addition).

phrasing removal prompt addition prompt
training template a video without {object} a video with {object}
instruction Remove the {object}. Add the {object}.
descriptive sentence the same scene, but the {object} is gone and can no longer be heard the same scene, now with {object} that can be seen and heard
paraphrase a clip in which there is no {object} a clip that includes {object}

Results under the updated judge (40 clips, % of clips):

phrasing removal: object removed removal: sound removed removal: other sounds preserved addition: object added addition: sound added addition: other sounds preserved
training template 35.0 7.5 77.5 12.5 35.0 90.0
instruction 22.5 7.5 92.5 7.5 35.0 90.0
descriptive sentence 30.0 10.0 77.5 10.0 40.0 87.5
paraphrase 30.0 5.0 80.0 12.5 42.5 87.5

The unseen phrasings behave close to the training template. The object name carries the edit; the surrounding wording matters little.

Examples. Same clip, same seed and (for addition) same reference frame; only the prompt changes.

P1

Remove birds

Input
training template
“a video without birds”
instruction
“Remove the birds.”
descriptive sentence
“the same scene, but the birds is gone and can no longer be heard”
paraphrase
“a clip in which there is no birds”
P2

Remove vehicle

Input
training template
“a video without vehicle”
instruction
“Remove the vehicle.”
descriptive sentence
“the same scene, but the vehicle is gone and can no longer be heard”
paraphrase
“a clip in which there is no vehicle”
P3

Remove birds

Input
training template
“a video without birds”
instruction
“Remove the birds.”
descriptive sentence
“the same scene, but the birds is gone and can no longer be heard”
paraphrase
“a clip in which there is no birds”
P4

Add frogs

Input
training template
“a video with frogs”
instruction
“Add the frogs.”
descriptive sentence
“the same scene, now with frogs that can be seen and heard”
paraphrase
“a clip that includes frogs”
P5

Add piano

Input
training template
“a video with piano”
instruction
“Add the piano.”
descriptive sentence
“the same scene, now with piano that can be seen and heard”
paraphrase
“a clip that includes piano”
P6

Add vehicle engine

Input
training template
“a video with vehicle engine”
instruction
“Add the vehicle engine.”
descriptive sentence
“the same scene, now with vehicle engine that can be seen and heard”
paraphrase
“a clip that includes vehicle engine”

11. Backbone Rmu9 Q3

The design is not tied to JavisDiT++. Source-conditioned modulation acts through the normalization layers, and the reference frame enters through the input sequence. Larger audiovisual diffusion transformers such as LTX-2 have both, so the method can be applied to them. Because of resource constraints we tested it on JavisDiT++, a lightweight joint audio-video backbone: our model has 2.26 B parameters and trains in 359 H100-hours. The paper will state that the results are demonstrated on JavisDiT++ and that applying the design to larger backbones is the next step.

What the experiments on JavisDiT++ show about the mechanism itself (Section 4): replacing source-conditioned modulation with cross-attention lowers background preservation by 42.5 pp on addition and 25.0 pp on removal, and removing the reference frame lowers visual addition by 17.5 pp. One backbone supports inputs with and without a reference and both tasks.

References

[1] Zheng et al., InstructAV2AV: Instruction-Guided Audio-Video Joint Editing, arXiv:2605.18467.
[2] Chen et al., JAVEdit: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation, arXiv:2606.03168.
[3] Shi et al., SAM Audio: Segment Anything in Audio, arXiv:2512.18099.
[4] Ishii et al., Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits, arXiv:2512.07209.
[5] Wang et al., AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models, NeurIPS 2023.
[6] Manor and Michaeli, Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion, ICML 2024.
[7] Ditto, arXiv:2510.15742.

1Dataset

Each pair is built from one real clip: the named object is segmented with a text-prompted video segmenter and inpainted out, and its sound is separated out of the soundtrack. A pair is used in both directions — original → edited for removal, edited → original for addition. 24,444 training / 1,389 validation / 1,425 test pairs; the test set covers 374 object descriptions.

D1

Object: flute

Original clip
object present
Object mask
text-prompted segmentation
Edited clip
object inpainted out, its sound separated out
D2

Object: brass instrument

Original clip
object present
Object mask
text-prompted segmentation
Edited clip
object inpainted out, its sound separated out
D3

Object: train engine

Original clip
object present
Object mask
text-prompted segmentation
Edited clip
object inpainted out, its sound separated out
D4

Object: birds chirping

Original clip
object present
Object mask
text-prompted segmentation
Edited clip
object inpainted out, its sound separated out
D5

Object: animal

Original clip
object present
Object mask
text-prompted segmentation
Edited clip
object inpainted out, its sound separated out
D6

Object: white-tailed eagle

Original clip
object present
Object mask
text-prompted segmentation
Edited clip
object inpainted out, its sound separated out
D7

Object: individual

Original clip
object present
Object mask
text-prompted segmentation
Edited clip
object inpainted out, its sound separated out
D8

Object: helicopter

Original clip
object present
Object mask
text-prompted segmentation
Edited clip
object inpainted out, its sound separated out

2Best results

Our model edits the picture and the soundtrack jointly. "Best" refers to the edit itself: in these clips the object named in the prompt and its sound are both removed or added, and the edit holds for the whole clip. The outputs can still show blur and deformation, which we state as a limitation. These showcase clips reappear in the baseline comparisons of sections 3 and 4 under the same tags.

2.1Text-guided removal

Text prompt only. The object is clearly visible and audible in the input and is gone from both the picture and the soundtrack in our output.

R1

Remove brass instrument

Input
Ours
text only
Reference
R2

Remove birds chirping

Input
Ours
text only
Reference
R3

Remove animal

Input
Ours
text only
Reference
R4

Remove individual

Input
Ours
text only
Reference
R5

Remove dogs

Input
Ours
text only
Reference

2.2Text + first-frame addition

Text prompt plus the first frame of the target. The input contains neither the object nor its sound; our output adds both, in sync.

A1

Add flute

Input
Ours
text + first frame
Reference
A2

Add train engine

Input
Ours
text + first frame
Reference
A3

Add running water

Input
Ours
text + first frame
Reference
A4

Add white-tailed eagle

Input
Ours
text + first frame
Reference
A5

Add helicopter

Input
Ours
text + first frame
Reference

3Comparison with unimodal editing baselines

Unimodal editors change one modality and leave the other untouched, and none of them supports addition, so this comparison is on removal (clips R1–R5). Each baseline is judged in its own modality; one cascade of a video editor and an audio editor follows.

3.1Video-only baselines

ROSE, VideoPainter and LGVI edit the picture only. All clips in this block are silent, so only the visual edit is compared.

R1

Remove brass instrument

Input
ROSE
video-only
VideoPainter
video-only
LGVI
video-only
Ours
joint, text only
Reference
R2

Remove birds chirping

Input
ROSE
video-only
VideoPainter
video-only
LGVI
video-only
Ours
joint, text only
Reference
R3

Remove animal

Input
ROSE
video-only
VideoPainter
video-only
LGVI
video-only
Ours
joint, text only
Reference
R4

Remove individual

Input
ROSE
video-only
VideoPainter
video-only
LGVI
video-only
Ours
joint, text only
Reference
R5

Remove dogs

Input
ROSE
video-only
VideoPainter
video-only
LGVI
video-only
Ours
joint, text only
Reference

3.2Audio-only baselines

ZEUS and AUDIT edit the soundtrack only. Audio players only, so only the audio edit is compared.

R1

Remove brass instrument

Input
ZEUS
audio-only
AUDIT
audio-only
Ours
joint, text only
Reference
R2

Remove birds chirping

Input
ZEUS
audio-only
AUDIT
audio-only
Ours
joint, text only
Reference
R3

Remove animal

Input
ZEUS
audio-only
AUDIT
audio-only
Ours
joint, text only
Reference
R4

Remove individual

Input
ZEUS
audio-only
AUDIT
audio-only
Ours
joint, text only
Reference
R5

Remove dogs

Input
ZEUS
audio-only
AUDIT
audio-only
Ours
joint, text only
Reference

3.3Cascade: ROSE + ZEUS

The video edited by ROSE combined with the audio edited by ZEUS — two independent models, neither aware of the other's edit.

R1

Remove brass instrument

Input
ROSE + ZEUS
video by ROSE, audio by ZEUS
Ours
joint, text only
Reference
R2

Remove birds chirping

Input
ROSE + ZEUS
video by ROSE, audio by ZEUS
Ours
joint, text only
Reference
R3

Remove animal

Input
ROSE + ZEUS
video by ROSE, audio by ZEUS
Ours
joint, text only
Reference
R4

Remove individual

Input
ROSE + ZEUS
video by ROSE, audio by ZEUS
Ours
joint, text only
Reference
R5

Remove dogs

Input
ROSE + ZEUS
video by ROSE, audio by ZEUS
Ours
joint, text only
Reference

4Comparison with joint audio-visual editing baselines

InstructAV2AV and ours generate both streams. AVI-Edit generates the video and is given the target audio track; CoherentAVEdit generates the audio and is given the edited video.

4.1Removal

Clips R1–R5. All methods receive the input clip and the text prompt.

R1

Remove brass instrument

Input
InstructAV2AV
generates both streams
AVI-Edit
generates the video; audio track is given
CoherentAVEdit
generates the audio; edited video is given
Ours
generates both streams
Reference
R2

Remove birds chirping

Input
InstructAV2AV
generates both streams
AVI-Edit
generates the video; audio track is given
CoherentAVEdit
generates the audio; edited video is given
Ours
generates both streams
Reference
R3

Remove animal

Input
InstructAV2AV
generates both streams
AVI-Edit
generates the video; audio track is given
CoherentAVEdit
generates the audio; edited video is given
Ours
generates both streams
Reference
R4

Remove individual

Input
InstructAV2AV
generates both streams
AVI-Edit
generates the video; audio track is given
CoherentAVEdit
generates the audio; edited video is given
Ours
generates both streams
Reference
R5

Remove dogs

Input
InstructAV2AV
generates both streams
AVI-Edit
generates the video; audio track is given
CoherentAVEdit
generates the audio; edited video is given
Ours
generates both streams
Reference

4.2Addition

Clips A1–A5. Every method that generates video is additionally given the first frame of the target.

A1

Add flute

Input
InstructAV2AV
generates both streams
AVI-Edit
generates the video; audio track is given
CoherentAVEdit
generates the audio; edited video is given
Ours
generates both streams
Reference
A2

Add train engine

Input
InstructAV2AV
generates both streams
AVI-Edit
generates the video; audio track is given
CoherentAVEdit
generates the audio; edited video is given
Ours
generates both streams
Reference
A3

Add running water

Input
InstructAV2AV
generates both streams
AVI-Edit
generates the video; audio track is given
CoherentAVEdit
generates the audio; edited video is given
Ours
generates both streams
Reference
A4

Add white-tailed eagle

Input
InstructAV2AV
generates both streams
AVI-Edit
generates the video; audio track is given
CoherentAVEdit
generates the audio; edited video is given
Ours
generates both streams
Reference
A5

Add helicopter

Input
InstructAV2AV
generates both streams
AVI-Edit
generates the video; audio track is given
CoherentAVEdit
generates the audio; edited video is given
Ours
generates both streams
Reference

5More examples

Further results of our model, selected automatically: an audio-visual judge (a multimodal LLM) must confirm the edit in the video, in the audio, and their synchronisation.

5.1Removal

Text prompt only (R6–R29).

R6

Remove birds

Input
Ours
text only
Reference
R7

Remove engine

Input
Ours
text only
Reference
R8

Remove a person

Input
Ours
text only
Reference
R9

Remove high-revving engine

Input
Ours
text only
Reference
R10

Remove birds

Input
Ours
text only
Reference
R11

Remove birds

Input
Ours
text only
Reference
R12

Remove birds

Input
Ours
text only
Reference
R13

Remove trumpets

Input
Ours
text only
Reference
R14

Remove running water

Input
Ours
text only
Reference
R15

Remove birds

Input
Ours
text only
Reference
R16

Remove knife

Input
Ours
text only
Reference
R17

Remove clarinet

Input
Ours
text only
Reference
R18

Remove chirping birds

Input
Ours
text only
Reference
R19

Remove birds

Input
Ours
text only
Reference
R20

Remove gunshots

Input
Ours
text only
Reference
R21

Remove birds

Input
Ours
text only
Reference
R22

Remove animal

Input
Ours
text only
Reference
R23

Remove dishes

Input
Ours
text only
Reference
R24

Remove owl

Input
Ours
text only
Reference
R25

Remove worker

Input
Ours
text only
Reference
R26

Remove idling engine

Input
Ours
text only
Reference
R27

Remove livestock

Input
Ours
text only
Reference
R28

Remove birds

Input
Ours
text only
Reference
R29

Remove man

Input
Ours
text only
Reference

5.2Addition

Text prompt plus first frame (A6–A30).

A6

Add didgeridoo

Input
Ours
text + first frame
Reference
A7

Add didgeridoo

Input
Ours
text + first frame
Reference
A8

Add clarinet

Input
Ours
text + first frame
Reference
A9

Add birds

Input
Ours
text + first frame
Reference
A10

Add bass guitar

Input
Ours
text + first frame
Reference
A11

Add pianist

Input
Ours
text + first frame
Reference
A12

Add male performer

Input
Ours
text + first frame
Reference
A13

Add accordion

Input
Ours
text + first frame
Reference
A14

Add person

Input
Ours
text + first frame
Reference
A15

Add drum kit

Input
Ours
text + first frame
Reference
A16

Add saxophonist

Input
Ours
text + first frame
Reference
A17

Add bald man

Input
Ours
text + first frame
Reference
A18

Add individual

Input
Ours
text + first frame
Reference
A19

Add birds

Input
Ours
text + first frame
Reference
A20

Add birds

Input
Ours
text + first frame
Reference
A21

Add birds

Input
Ours
text + first frame
Reference
A22

Add individual

Input
Ours
text + first frame
Reference
A23

Add cow

Input
Ours
text + first frame
Reference
A24

Add birds

Input
Ours
text + first frame
Reference
A25

Add crowd

Input
Ours
text + first frame
Reference
A26

Add brass instrument

Input
Ours
text + first frame
Reference
A27

Add birds

Input
Ours
text + first frame
Reference
A28

Add individual

Input
Ours
text + first frame
Reference
A29

Add individual

Input
Ours
text + first frame
Reference
A30

Add helicopter

Input
Ours
text + first frame
Reference

6Failure cases

Typical failures of the dataset and of our model, grouped by type. They were found with the audio-visual judge, which is asked whether the object is visible, whether it is audible, and whether sound and picture are in sync; its explanation is quoted under each example.

6.1Dataset failures

Errors in how the pairs were built; they bound what any model trained or scored on this data can reach.

6.1.1Sound separation leaves the object audible

The separated soundtrack of the edited clip still contains the object's sound.

F1.1.1

Object: idling engine

Original clip
object present
Edited clip
removal target / addition input

Judge: The idling engine sound is audibly present and remains consistent without interruption across the clip.

F1.1.2

Object: drumming

Original clip
object present
Edited clip
removal target / addition input

Judge: The sound of drumming is clearly audible and remains consistent throughout the clip.

6.1.2Inpainting leaves the object (or a remnant) visible

The inpainted picture of the edited clip still shows the object or a recognisable remnant.

F1.2.1

Object: truck engine

Original clip
object present
Edited clip
removal target / addition input

Judge: The engine is clearly visible and remains consistent in its appearance and operation across the entire clip.

F1.2.2

Object: toilet

Original clip
object present
Edited clip
removal target / addition input

Judge: The toilet is consistently visible as the camera pulls back and focuses.

6.1.3The named object is heard but never seen in the original clip

The prompt names a sound source that is off-screen, or does not match what the picture shows.

F1.3.1

Object: motorcycle engine

Original clip
object present
Edited clip
removal target / addition input

Judge: The video shows a car engine, not a motorcycle engine; the target is entirely missing.

F1.3.2

Object: engine

Original clip
object present
Edited clip
removal target / addition input

Judge: The target engine is not visually present at any point in the clip.

6.2Removal failures

Our model, text prompt only, on clips whose input clearly shows and sounds the object.

6.2.1The object is left in place

The output is close to the input: the object is still fully visible.

F2.1.1

Remove truck engine

Input
Ours
text only
Reference

Judge: The engine remains visible throughout the entire clip.

F2.1.2

Remove toilet

Input
Ours
text only
Reference

Judge: The target object (toilet) remains visible as a close-up surface for the entire duration.

6.2.2A ghost or blurred remnant stays behind

The object is only partly erased and leaves a blurred or ghost-like shape.

F2.2.1

Remove train

Input
Ours
text only
Reference

Judge: A massive, moving artifact retaining the shape and color of the train is visible for the entire clip.

F2.2.2

Remove train

Input
Ours
text only
Reference

Judge: A large, moving ghost image of the train remains visible on the tracks for the entire clip.

6.2.3The object leaves the picture but its sound stays

The visual removal succeeds while the soundtrack keeps the object's sound, although the reference audio is clean.

F2.3.1

Remove electronic device

Input
Ours
text only
Reference

Judge: The buzzing sound of the electronic device remains clearly audible for the entire duration.

F2.3.2

Remove motorcycle engine

Input
Ours
text only
Reference

Judge: The sound of a motorcycle engine is clearly audible continuously from the beginning to the end of the clip.

6.3Addition failures

Our model, text prompt plus first frame, on clips whose input has neither the object nor its sound.

6.3.1The object drifts away from the first frame (identity or shape changes)

The object starts from the given first frame but its shape or identity changes over time.

F3.1.1

Add cargo ship

Input
Ours
text + first frame
Reference

Judge: The ship structure is highly distorted and morphs continuously, lacking consistency.

F3.1.2

Add animal

Input
Ours
text + first frame
Reference

Judge: The animal severely distorts and loses its initial identity, becoming an unrecognizable shape starting around 0.5s.

6.3.2The object fades out after the first frame

The object is present at the first frame and then disappears.

F3.2.1

Add male vocalist

Input
Ours
text + first frame
Reference

Judge: No singing action is visible, and the person's head disappears entirely at 00:01.000.

F3.2.2

Add canine

Input
Ours
text + first frame
Reference

Judge: The dog arbitrarily disappears and blends into the floor texture at 00:01.000 and 00:04.000.

6.3.3Object and sound are both added, but out of sync

Both streams contain the object, but sound events do not line up with the visible action.

F3.3.1

Add barking dog

Input
Ours
text + first frame
Reference

Judge: The audible barks do not match the visible actions of the dog, which appears to be sniffing or eating rather than barking, especially noticeable around 00:03 and 00:04.

F3.3.2

Add rooster

Input
Ours
text + first frame
Reference

Judge: The rooster crowing sound occurs between 00:01.8 and 00:03.0, but the visible rooster's beak remains closed and it does not perform the physical action of crowing.