Frequently Asked Questions (FAQ)¶

Installation & Dependencies¶

How to install Flash Attention 3 (FA3)?¶

Flash Attention 3 provides optimized attention kernels for better performance. To install:

git clone git@github.com:Dao-AILab/flash-attention.git
cd flash-attention/hopper
python setup.py install

After installation, you need to modify the attention dispatch file:

vi /usr/local/lib/python3.12/dist-packages/diffusers/models/attention_dispatch.py

Find _diffusers_flash_attn_3::_flash_attn_forward and add return_attn_probs=True:

return_attn_probs=True

Install Flash Attention 2 (FA2)?¶

Flash Attention 2 provides optimized attention kernels for better performance on Ampere/Ada GPUs. To install:

git clone git@github.com:Dao-AILab/flash-attention.git
cd flash-attention
# Defaults: "80;90;100;110;120" Adjust based on your GPU architecture
export FLASH_ATTN_CUDA_ARCHS=80 # 80 for Ampere, 90 for Hopper, ...
MAX_JOBS=8 pip install flash-attn --no-build-isolation
# or, build from local source for more control:
# export MAX_JOBS=32 for faster compilation with more CPU cores
python setup.py install

How to install Sage Attention?¶

Sage Attention is an efficient attention implementation. To install:

git clone https://github.com/thu-ml/SageAttention.git
cd SageAttention
export EXT_PARALLEL=4 NVCC_APPEND_FLAGS="--threads 8" MAX_JOBS=32  # Optional
export TORCH_CUDA_ARCH_LIST=9.0 # 9.0 for Hopper, 8.9 for Ada
python setup.py install

Common Issues¶

torch.compile errors when running examples¶

If you encounter errors with torch.compile when running cache-dit examples, try the following solutions:

Clear the torch inductor cache:

rm -rf /tmp/torchinductor_root/

Then retry running your example.

Upgrade PyTorch to the latest version:

pip install --upgrade torch torchvision

If the issue persists, please open an issue with:
- Your PyTorch version (python -c "import torch; print(torch.__version__)") - The complete error traceback - Your system configuration (GPU model, CUDA version, etc.)

NCCL errors during distributed inference¶

Please consider to upgrade to the latest PyTorch and NCCL versions, as they may contain important bug fixes and performance improvements. You can upgrade PyTorch and NCCL using the following commands:

pip install --upgrade torch torchvision torchaudio triton
pip install --upgrade nvidia-nccl-cu12 # or, for CUDA 13: pip install --upgrade nvidia-nccl-cu13

Performance Optimization¶

Which attention backend should I use?¶

Cache-DiT supports multiple attention backends for different use cases. For a complete overview of attention backends in diffusers, see the Attention Backends. The main attention backends supported by cache-dit are:

flash: Flash Attention 2 - Good performance on Ampere/Ada GPUs
_flash_3: Flash Attention 3 - Best for Hopper architecture GPUs (H100, H200)
native: Native PyTorch SDPA - Default, works on all devices
_native_cudnn: cuDNN-based native attention
_sdpa_cudnn: SDPA with cuDNN (cache-dit specific, supports context parallelism with attention masks)
sage: Sage Attention - Good balance between performance and compatibility

Recommendation:

H100/H200: Use _flash_3 for best performance
A100/A6000: Use flash or sage
Other GPUs: Use native or sage