Diffusion-forcing world model rollouts. Each sample is ~2.7 s (16 raw frames @ 6 fps). Side-by-side videos show ground truth on the left, model prediction on the right. First half of each clip = history (GT), second half = autoregressively sampled future. Click a sample header to expand its videos (default-collapsed to keep the page light).
_s1 = ar_stride=1 (finer step). Short 16-frame rollouts live on the sister page: yxma/vtwm-ablation-v2.Click a run name to jump to its rollout videos. ✓ = feature on, ✗ = off. Lower val_loss is better.
| run | fusion | views | tactile | shift16 | delta-ref | cam-pose | gate | val_loss |
|---|---|---|---|---|---|---|---|---|
| stride2 (window ref) · 40s AR SOTA (val_loss) | mmdit_mv | 3 | 2 | ✓ | window | ✗ | — | 0.013 |
| stride2 + h4/f12 window · 40s AR | mmdit_mv | 3 | 2 | ✓ | window | ✗ | — | 0.013 |
| stride2 + p01 · 40s AR | mmdit_mv | 3 | 2 | ✓ | episode_p01 | ✗ | — | 0.0159 |
| mv1 256-res (window ref) · 40s AR | mmdit_mv | 3 | 2 | ✓ | window | ✗ | — | 0.0143 |
| stride2 + p01 + hybrid (dual6+single7) · 40s AR | mmdit_mv_hybrid | 3 | 2 | ✓ | episode_p01 | — | — | 0.0161 |
| stride2 + p01 + hybrid (dual6+single7) · 40s AR (ar_stride=1) | mmdit_mv_hybrid | 3 | 2 | ✓ | episode_p01 | — | — | 0.0161 |
| stride2 + p01 + hybrid + flow matching · 40s AR | mmdit_mv_hybrid | 3 | 2 | ✓ | episode_p01 | — | — | 0.1445 |
| stride2 + p01 + hybrid (deeper, dual6+single8) · 40s AR | mmdit_mv_hybrid | 3 | 2 | ✓ | episode_p01 | — | — | 0.0159 |
| stride2 + p01 + hybrid (deeper, dual6+single8) · 40s AR (ar_stride=1) | mmdit_mv_hybrid | 3 | 2 | ✓ | episode_p01 | — | — | 0.0159 |
Latent-space MSE of predicted future vs ground truth. Contact/no-contact split uses tactile latent-to-reference energy (threshold 0.05).
| run | view MSE | TL MSE | TR MSE | tactile contact MSE | tactile no-contact MSE |
|---|---|---|---|---|---|
| stride2 (window ref) · 40s AR | n/a | n/a | n/a | n/a | n/a |
| stride2 + h4/f12 window · 40s AR | n/a | n/a | n/a | n/a | n/a |
| stride2 + p01 · 40s AR | n/a | n/a | n/a | n/a | n/a |
| mv1 256-res (window ref) · 40s AR | n/a | n/a | n/a | n/a | n/a |
| stride2 + p01 + hybrid (dual6+single7) · 40s AR | n/a | n/a | n/a | n/a | n/a |
| stride2 + p01 + hybrid (dual6+single7) · 40s AR (ar_stride=1) | n/a | n/a | n/a | n/a | n/a |
| stride2 + p01 + hybrid + flow matching · 40s AR | n/a | n/a | n/a | n/a | n/a |
| stride2 + p01 + hybrid (deeper, dual6+single8) · 40s AR | n/a | n/a | n/a | n/a | n/a |
| stride2 + p01 + hybrid (deeper, dual6+single8) · 40s AR (ar_stride=1) | n/a | n/a | n/a | n/a | n/a |
# Loss curve analysis — multi-view + cam-pose ablations | Run | Type | Best val_loss_visual | Best val_loss_tactile | Final val_loss | |----------------------|-------------------------------------------------|----------------------|-----------------------|----------------| | vo_left | single view (left) | 0.0096 @ ep84 | — | 0.0097 | | vo_middle | single view (middle/top) | 0.0128 @ ep97 | — | 0.0128 | | **vo_right** | single view (right) | **0.0093 @ ep97** | — | 0.0093 | | p2_mv | 3-view shared weights, no tactile | 0.0099 @ ep96 | — | 0.0099 | | p3_mv2t | 3-view + 2-tactile, shared weights | 0.0100 @ ep97 | 0.0040 | 0.0139 | | p4_mv2t_extr (no gate) | p3 + cam-pose extrinsics | 0.0102 @ ep98 | **0.3244 (blow-up)** | 0.3346 | | **p4_gate** | p4 + scalar gate (this fix) | **0.0096 @ ep134** | 0.2621 (still bad) | 0.2751 | | **p5_gate** | vision-only p2 + cam-pose + gate | **0.0095 @ ep149** | — | 0.0095 | | vo_v0 (legacy) | single view (old 1v0t pipeline) | 0.0167 @ ep53 | — | 0.0171 | | mm_v0 (legacy) | 1-view + 2-tactile channel-stack (old mm_v0) | 0.0162 @ ep56 | 0.0032 | 0.0198 | ## Conclusions 1. **The gate fix worked for the VISUAL branch.** Both `p4_gate` (0.0096) and `p5_gate` (0.0095) now **match the best single-view baseline `vo_right` (0.0093)** — for the first time, multi-view + cam-pose conditioning beats the best single-view model on visual fidelity. Without the gate (`p4_mv2t_extr`) the visual stream was already OK (0.0102) but tactile blew up. 2. **The gate did NOT save the tactile branch.** `p4_gate` val_loss_tactile = 0.262 (vs `p3_mv2t` baseline 0.0040, ~65× worse). The single scalar gate delays the cam-pose perturbation by one step but does not prevent it from injecting noise into the tactile token stream once it opens. Likely fix for a future run: route cam-pose ONLY through view tokens (gate visual cam-pose embedding, but never add it to tactile-stream AdaLN). 3. **Plateau.** `p4_gate` and `p5_gate` reach their best around ep122–150 and have plateaued for >1h; safe to inference now even though target was 200 epochs. 4. **Multi-view rank** (visual val_loss, lower=better): `p5_gate (0.0095) ≈ p4_gate (0.0096) ≈ vo_right (0.0093) < vo_left (0.0096) < p2_mv (0.0099) ≈ p3_mv2t (0.0100) < vo_middle (0.0128)` 5. **`vo_middle` is the worst single view** by a margin — top-down camera has the least useful geometry for predicting future frames.
Stride=2 re-encoded latents → 4x training data. Same recipe as mv1_3v2t. Ties pre-stride2 SOTA at ep39. 40-second autoregressive rollout at test time. Showing 3 of 3 samples.
Shorter history (4), longer future (12) — 1.5x more train windows per episode. Ties stride2 SOTA at ep39. 40-second autoregressive rollout at test time. Showing 3 of 3 samples.
Same mv1 recipe at 256x256 input (2x spatial). val_loss not directly comparable (different latent dims), kept for visual quality comparison. 40-second autoregressive rollout at test time. Showing 1 of 1 samples.
Stride2 with tactile_ref_source=episode_p01. Closes only ~halfway the gap vs window (Δ=+0.003), confirming window ref is tighter. 40-second autoregressive rollout at test time. Showing 3 of 3 samples.
Dual→Single hybrid fusion: first 6 blocks are MMDiTBlockDFMV (per-modality QKV+MLP), last 7 blocks are SingleStreamHybridBlock (shared QKV+MLP with per-modality adaLN, forces deep V-T fusion). Depth=12, 85M params (0.84x mmdit_mv baseline). Warm-started from mv1_v2_stride2_p01 (loaded 282/298 tensors bit-exact). 40-second autoregressive rollout at test time. Showing 3 of 3 samples.
Dual→Single hybrid fusion: first 6 blocks are MMDiTBlockDFMV (per-modality QKV+MLP), last 7 blocks are SingleStreamHybridBlock (shared QKV+MLP with per-modality adaLN, forces deep V-T fusion). Depth=12, 85M params (0.84x mmdit_mv baseline). Warm-started from mv1_v2_stride2_p01 (loaded 282/298 tensors bit-exact). 40-second autoregressive rollout at test time. (ar_stride=1) Showing 3 of 3 samples.
Hybrid backbone + rectified flow matching training objective (SD3/FLUX/HunyuanVideo). Same 85M backbone. Linear-interpolation schedule + velocity target + Euler ODE sampling (50 steps instead of 100). Val_loss numerically incomparable to pred_v runs — compare via decoded LPIPS + tactile MSE. 40-second autoregressive rollout at test time. Showing 3 of 3 samples.
Deeper hybrid: depth=14 (+2 single-stream blocks) for more cross- modal fusion capacity. Same dual_depth=6 to preserve warm-start. ~100M params — matches original mmdit_mv budget. 40-second autoregressive rollout at test time. Showing 3 of 3 samples.
Deeper hybrid: depth=14 (+2 single-stream blocks) for more cross- modal fusion capacity. Same dual_depth=6 to preserve warm-start. ~100M params — matches original mmdit_mv budget. 40-second autoregressive rollout at test time. (ar_stride=1) Showing 3 of 3 samples.
