Mask Fields in Video Diffusion Transformers

Diagnosing and Calibrating Internal Localization for Training-Free Editing
Anonymous
Under review, ICLR 2027
📄 Paper 💻 Code 🎬 Results 📚 BibTeX

TL;DR

Training-free video editors use the diffusion transformer's cross-attention as a mask that decides where an edit lands. We show this signal is not a mask at all but an unstable latent field — fragmented into dozens of islands per frame, flickering between consecutive frames, extracted from a hand-picked layer that turns out to be the worst of the network. We diagnose the field layer-by-layer, split its defects into repairable (fragmentation, jitter) and unrepairable (missing figure–ground separation), and calibrate accordingly: select the layer for the property no operator can recover, repair the rest with two axis-matched, training-free operators.

117/118
videos with improved temporal mask stability on FiVE
+32%
relative mask mIoU over FlowDirector, full calibration
3.4×
faster: calibration replaces brute-force velocity averaging
0
extra network evaluations, weights modified, or training

The problem: attention is used as a mask, but is not one

Teaser: per-block fragmentation, edit gates, gate stability
(a) Per-block spatial fragmentation of subject-token cross-attention in Wan2.1 — FlowDirector's hand-picked block (red) is the most fragmented of all 30. (b) The actual edit gate: FlowDirector's pooling (top) fragments and reshuffles across frames; calibration (bottom) yields one coherent, temporally aligned region. (c) Pooling's gate stability collapses exactly when video content crystallizes; the calibrated gate holds.

Probing every block of Wan2.1-T2V-1.3B (1,200 subject-token maps) shows the pathology is structural: the median map fragments into dozens of connected components where a mask has one; consecutive frames share less than half their mask area; and layer choice is a third failure surface — the coherent-looking block attends diffusely to half the scene (the "coherence trap"), while the hand-picked block is the most fragmented of all. An identical probe on CogVideoX-2b reproduces every finding.

The framework: select for the unrepairable, repair the rest

Calibration framework diagram
All signals are read from the single forward pass FlowDirector already computes — no extra network evaluation, no training.

Substitution edits same seed, same prompts — FlowDirector vs. full calibration

FlowDirector's pooling (middle) vs. full calibration (right: selected extraction layer + repair operators). Each caption says where to look — FlowDirector's fragmented, flickering gate lets the edit spill outside the object; the calibrated gate keeps it contained.

plane → helicopter watch the crew's clothes

FlowDirector repaints the foreground crewman's yellow shirt into a green hi-viz vest and rewrites his headset — collateral damage far outside the object. Calibration preserves both crew members.

Source
FlowDirector (pooling)
+ Full calibration

bus → jeep watch the opening frames

Under FlowDirector's pooling, a translucent bus ghost stays fused to the jeep's front in the opening frames — a half-transformed hybrid. The calibrated edit is clean from the first frame.

Source
FlowDirector (pooling)
+ Full calibration

hockey player → batman watch the goal net

FlowDirector's gate bleeds onto the pop-up goal net behind the player and half-erases it; calibration leaves the net standing while making the same character edit.

Source
FlowDirector (pooling)
+ Full calibration

tennis player → spider-suit watch the racket

FlowDirector mangles the racket into a dark branch-like artifact mid-swing; calibration keeps the racket intact through the stroke.

Source
FlowDirector (pooling)
+ Full calibration

car → silver jeep watch the jeep body shimmer

In stills the two edits look identical — in motion FlowDirector's jeep shimmers frame-to-frame (the gate flickers), while the calibrated jeep holds steady. The traces below quantify it.

Source
FlowDirector (pooling)
+ Full calibration

car → black jeep, burnout watch through the smoke

Calibration renders a more defined jeep (tires, wheel arch, windshield) and holds it steady through the smoke; FlowDirector's vehicle blurs and pulses.

Source
FlowDirector (pooling)
+ Full calibration
Output flicker traces
Output-level flicker (mean frame-to-frame change in excess of the source's own motion) on the three car edits above: FlowDirector's edited region shimmers through the whole video; full calibration sits at the source's motion level.

Source-absent localization the setting only internal masks can serve

Inserting an object that does not exist in the source — a cap on a bear, a hat on a musician — leaves every external segmenter with nothing to segment. Queried for the absent object, an open-vocabulary detector doesn't fail gracefully: it returns a confident box on every video, usually the entire subject. The generator's own cross-attention is the only signal that responds to the query at all — which makes calibrating it the load-bearing problem.

bear + a cap

Source
FlowDirector (pooling)
+ Calibration

musician + a hat

Source
FlowDirector (pooling)
+ Calibration

What the gate actually looks like

Gate overlay, bus
The edit gate applied to the flow, FlowDirector pooling vs. calibration.
Layer comparison
Layer choice is the dominant spatial lever: the same operators at the selected layer vs. the hand-picked one.
Gate stability traces
Gate stability across denoising steps: pooling decays exactly as content crystallizes; the calibrated gate holds throughout.

Calibration replaces brute-force compute

FlowDirector stabilizes its noisy mask field by averaging the edit velocity over four independent noise resamples — 16 DiT forwards per denoising step. That averaging exists because the field is unstable. Calibration replaces it with structure: as the resample count drops to one, pooling's stability collapses while calibrated masks barely move — so full calibration at a quarter of the forwards (3.4× faster measured end-to-end) still beats FlowDirector's default configuration on both axes.

Drop-in usage

# FlowDirector baseline
python edit.py ... --mask_arm pool

# operator mode: swap pooling for the two repair operators
python edit.py ... --mask_arm full

# full calibration: also move extraction to the selected layer
python edit.py ... --mask_arm full --mask_block 8

The operators are ~150 lines of tensor ops on the attention heat; conductances are read from the forward pass FlowDirector already computes.

BibTeX

@inproceedings{maskfield2027,
  title     = {Mask Fields in Video Diffusion Transformers: Diagnosing and
               Calibrating Internal Localization for Training-Free Editing},
  author    = {Anonymous},
  booktitle = {Submitted to ICLR},
  year      = {2027},
}