← VideoGenX

Guide

How we score the quality of a lossy speedup

Cross-step caching and sparse attention speed up video generation by skipping work, so the output changes. Each lossy config will be scored against the same engine’s lossless output on the same GPU, and the dashboard will show the scores without a pass line.

Updated

PSNR against the official Diffusers reference, lossless clips onlyMeasured
B200median 19.3 dB
H100median 20.9 dB

PSNR against the official Diffusers reference (dB)

Run 41280191475 · Lossless SGLang, four GPUs, no lossy optimization. Every decoded frame is scored.
B200 clips below 17 dB, with no lossy optimization
7 / 20
H100 clips below 17 dB, with no lossy optimization
5 / 20
From the lowest to the highest lossless clip
15.1 dB
All 40 scores
PromptB200 (dB)H100 (dB)
ping-pong12.8612.88
shoe-laces14.7915.08
zipper-close16.1016.90
paper-fold16.8616.80
umbrella-open17.0216.94
book-pages17.1017.10
leaves-rake16.9617.71
station-clock13.0823.76
kettle-pour19.4619.65
sewing-machine19.0920.90
paint-brush16.9424.33
dog-shake21.0520.88
ice-cubes21.2421.16
wind-chimes22.8620.81
basketball-bounce22.5821.53
window-rain23.5523.49
ceramic-wheel26.1725.39
rolling-ball26.5926.46
coffee-grinder27.5427.56
candle-light27.3927.92

Why there is no PSNR cut-off

Today the dashboard scores each clip by PSNR against the official Diffusers reference, with the same prompt and seed, over every decoded frame. A simple rule would grade lossy configs on that scale, for example by treating 10 to 17 dB as poor.

The chart above shows why that rule fails. No clip in it uses a lossy optimization, yet many fall inside the example band. Both GPU models ran the same engine, SGLang, and each one alone spans most of the range. Most prompts also land at nearly the same score on both GPUs. The spread follows the prompt more than the hardware.

Different engines and parallel layouts also produce different pixels from the same prompt and seed. A fixed cut-off against the reference would grade the prompt and the implementation, not the optimization.

The method

Pair each lossy run with the same engine's lossless run. The GPU, engine, server layout, prompt and seed stay fixed and only the lossy options change, so any difference in the output comes from the optimization.

Held fixed
  • GPU
  • Engine
  • Server layout
  • Prompt
  • Seed
Two runs
Lossless runlossy options off
Lossy runlossy options on
Score every decoded frame
PSNR
dB · higher is closer
SSIM
≤ 1 · higher is closer
LPIPS
0 = identical · lower is closer
Report per config

The median and the worst clip over the scored prompts (up to 20).

Each config reports PSNR, SSIM and LPIPS, each as two numbers: the median clip and the worst clip over the scored prompts. The lossless config also runs twice. The gap between those two runs is the noise floor for the pair, so a difference smaller than that gap is not read as a change from the optimization.

Two rules apply when the scores launch:

  • Scores, no pass line. Each lossy config shows its scores next to its speed, and readers weigh the trade.
  • A threshold comes later, if at all. Any threshold would first be calibrated against a lossy change people already accept, such as FP8.

Finding the sweet spot

Each lossy config becomes one point: its speedup over the lossless twin against how close its clips stay to that twin. The twin anchors the chart at 1.0× and identical. Configs from one engine join into a line, so an engine that submits several cache thresholds draws a curve. Points further up and to the right are faster and closer to lossless.

Illustrative, not measured

Speedup over the lossless twin ↑

1.0×1.5×2.0×2.5×3.0×
↗ faster and closer to losslessLossless twinCross-step cacheCache + sparse attention

PSNR against the same engine’s lossless output (dB) · right is closer

The chart uses placeholder numbers to show how it will read. Switch the score to compare PSNR, SSIM and LPIPS. The LPIPS axis is reversed, so right is always closer. Hover over or select a point to see its numbers.

Where the scores show up

On the VideoGenX dashboard, the Lossy optimizations option draws lossy configs as dashed lines, each with the lossless line of the same run, engine and layout to compare with. "See what changes" plays a lossy clip against its lossless twin. Once paired scores exist, they will appear in the lossy line's legend, the point tooltip and the clip readout. Until then, the readout's Paired scores row shows Planned.

Data. The strip chart reads the published reference PSNR of the run linked under it when the page is generated. The video-generation README describes the scorer and how each score is verified. The sweet-spot chart is illustrative.