NVIDIA GPU Benchmark
NVIDIA L20
Z-Image-ControlNet: Hybrid Cache + Parallelism
| Z-Image-ControlNet |
Context Parallel: Ulysses 2 |
Context Parallel: Ulysses 4 |
+ ControlNet Parallel |
| Base L20x1: 22s |
15.7s |
12.7s |
🚀7.71s |
 |
 |
 |
 |
| + Hybrid Cache |
+ Torch Compile |
+ Async Ulyess CP |
+ FP8 All2All + CUDNN ATTN |
| 🚀6.85s |
6.45s |
6.38s |
🚀6.19s, 5.47s |
 |
 |
 |
 |
FLUX.1-dev: Hybrid Cache + Parallelism
| Baseline(L20x1) |
F1B0 (0.08) |
F1B0 (0.20) |
F8B8 (0.15) |
F12B12 (0.20) |
| 24.85s |
15.59s |
8.58s |
15.41s |
15.11s |
 |
 |
 |
 |
 |
| Baseline(L20x1) |
F1B0 (0.08) |
F8B8 (0.12) |
F8B12 (0.12) |
F8B16 (0.20) |
| 27.85s |
6.04s |
5.88s |
5.77s |
6.01s |
 |
 |
 |
 |
 |
UAA: Ulysses Anything Attention
Qwen-Image & FLUX.1-dev
✅Any Sequence Length
U*: Ulysses Attention, UAA: Ulysses Anything Attenton, UAA*: UAA + Gloo, Device: NVIDIA L20
FLUX.1-Dev w/o CPU Offload, 28 steps; Qwen-Image w/ CPU Offload, 50 steps; Gloo: Extra All Gather w/ Gloo
| CP2 w/ U* |
CP2 w/ UAA* |
CP2 w/ UAA |
L20x1 |
CP2 w/ UAA* |
CP2 w/ U* |
L20x1 |
CP2 w/ UAA* |
| FLUX, 13.87s |
🎉13.88s |
14.75s |
23.25s |
🎉13.75s |
Qwen, 132s |
181s |
🎉133s |
 |
 |
|
 |
 |
 |
 |
 |
| 1024x1024 |
1024x1024 |
1024x1024 |
1008x1008 |
1008x1008 |
1312x1312 |
1328x1328 |
1328x1328 |
| ✔️U* ✔️UAA |
✔️U* ✔️UAA |
✔️U* ✔️UAA |
NO CP |
❌U* ✔️UAA |
✔️U* ✔️UAA |
NO CP |
❌U* ✔️UAA |
Z-Image-Turbo
✅Any Head Num
Ulysses: Ulysses Attention, FP8 Ulysses: Ulysses w/ FP8 All2All, Device: NVIDIA L20
🔥Z-Image (Head=30, ❌CAN NOT divisible by 4), 1024x1024, 9 steps.
| Ulysses 2, L20 |
Ulysses 4 |
FP8 Ulysses 4 |
+ Cache |
+ FP8 DQ |
| 1024x1024, 3.19s |
1024x1024, 1.98s |
1024x1024, 1.89s |
1024x1024, 1.63s |
1024x1024, 1.23s |
 |
 |
 |
 |
 |
Async Ulysses QKV Projection
FLUX.1-dev
Ulysses: Standard Ulysses Attention, Async Ulysses: Ulysses Attenton with Async QKV Projection
| L20x2 w/ Ulysses |
w/ Async Ulysses |
w/ Ulysses + compile |
w/ Async Ulysses + compile |
| FLUX.1, 13.87s |
🎉13.20s |
12.21s |
🎉11.97s |
 |
 |
 |
 |
Async FP8 Ulysses Attention
FLUX.1-dev
| L20x2 w/ Ulysses |
w/ Ulysses FP8 |
w/ Ulysses + compile |
w/ Ulysses FP8 + compile |
| FLUX.1, 13.87s |
🎉13.36s |
12.21s |
🎉11.54s |
|
 |
|
 |
NVIDIA H100
| Model |
Baseline H100x1 |
Ulysses 2 |
+ FA3 |
+ cache |
+ compile |
| FLUX.1-dev: 50 steps |
9.30s |
6.04s |
5.99s |
2.60s |
1.92s |
| Qwen-Image: 50 steps |
18.49s |
12.81s |
12.75s |
5.67s |
4.20s |
Reproduce commands:
# FLUX.1-dev: 50 steps
python3 -m cache_dit.generate flux --steps 50
torchrun --nproc_per_node=2 -m cache_dit.generate flux --steps 50 --parallel ulysses
torchrun --nproc_per_node=2 -m cache_dit.generate flux --steps 50 --parallel ulysses --attn _flash_3
torchrun --nproc_per_node=2 -m cache_dit.generate flux --steps 50 --parallel ulysses --attn _flash_3 --cache
torchrun --nproc_per_node=2 -m cache_dit.generate flux --steps 50 --parallel ulysses --attn _flash_3 --cache --compile
# Qwen-Image: 50 steps
python3 generate.py qwen_image --steps 50
torchrun --nproc_per_node=2 -m cache_dit.generate qwen_image --steps 50 --parallel ulysses
torchrun --nproc_per_node=2 -m cache_dit.generate qwen_image --steps 50 --parallel ulysses --attn _flash_3
torchrun --nproc_per_node=2 -m cache_dit.generate qwen_image --steps 50 --parallel ulysses --attn _flash_3 --cache
torchrun --nproc_per_node=2 -m cache_dit.generate qwen_image --steps 50 --parallel ulysses --attn _flash_3 --cache --compile