Installation Instructions

Note: The most up-to-date installation instructions are embedded in a set of scripts bundled in the FBGEMM repo under setup_env.bash.

The general steps for installing FBGEMM GenAI are as follows:

1. Set up an isolated runtime environment.

2. Set up the toolchain for either a CPU-only, CUDA, or ROCm runtime.

3. Install PyTorch.

4. Install the FBGEMM GenAI package.

5. Run post-installation checks.

Before installing FBGEMM GenAI, please check FBGEMM Releases Compatibility Table to ensure that prerequisite hardware and software you are using is compatible with the version of FBGEMM GenAI you plan to install.

Set Up Runtime Environment

Follow the instructions for setting up the runtime environment:

1. Set Up CUDA Environment

2. Install Python Libraries

3. Install PyTorch

4. Install Triton

Install the FBGEMM GenAI Package

Install through PyTorch PIP

PyTorch PIP is the preferred channel for installing FBGEMM GenAI:

# !! Run inside the Conda environment !!

# CUDA Nightly
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu126/
pip install --pre fbgemm-genai --index-url https://download.pytorch.org/whl/nightly/cu126/

# CUDA Release
pip install torch --index-url https://download.pytorch.org/whl/cu126/
pip install fbgemm-genai --index-url https://download.pytorch.org/whl/cu126/

# Test the installation
python -c "import torch; import fbgemm_gpu.experimental.gen_ai"

Install through Public PyPI

# !! Run inside the Conda environment !!

# CUDA Nightly
pip install fbgemm-gpu-nightly-genai

# CUDA Release
pip install fbgemm-gpu-genai

Post-Installation Checks

After installation, run an import test to ensure that the library is correctly linked and set up.

# !! Run inside the Conda environment !!

python -c "import torch; import fbgemm_gpu.experimental.gen_ai; print(torch.ops.fbgemm.quantize_fp8_per_row)"

Please refer to Post-Installation Checks for information on additional post-install checks.