144 lines
6.5 KiB
Markdown
144 lines
6.5 KiB
Markdown
# ComfyUI ROCm Fix — nimo-cachy (Strix Halo / gfx1151)
|
|
|
|
## Machine
|
|
|
|
- **Host:** nimo-cachy (nimo.loc)
|
|
- **GPU:** AMD Radeon 8060S (Strix Halo APU, gfx1151 / RDNA 3.5)
|
|
- **OS:** CachyOS (Arch-based), kernel 7.0.12-1-cachyos
|
|
- **ComfyUI:** `~/git/ComfyUI` (v0.24.0), served via systemd on port 8188
|
|
|
|
---
|
|
|
|
## Problems & Fixes (in order applied)
|
|
|
|
### 1. SQLAlchemy not found
|
|
|
|
**Symptom:** ComfyUI crashed on start with `ModuleNotFoundError: No module named 'sqlalchemy'`.
|
|
|
|
**Root cause:** The venv at `~/git/ComfyUI/venv` was created with Python 3.13, but `/usr/bin/python3` now points to Python 3.14. The systemd service was calling `/usr/bin/python3.13 main.py` (bare, outside the venv). SQLAlchemy was only installed inside the venv, not system-wide. PEP 668 also blocks system-wide pip installs.
|
|
|
|
**Fix:** Update systemd service `ExecStart` to use the venv Python:
|
|
```
|
|
ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py --listen 0.0.0.0 --enable-manager
|
|
```
|
|
|
|
---
|
|
|
|
### 2. CUDA torch in venv — no NVIDIA GPU
|
|
|
|
**Symptom:** After fixing the venv path, crashed with `RuntimeError: Found no NVIDIA driver on your system`.
|
|
|
|
**Root cause:** The venv contained `torch 2.12.0+cu130` (CUDA/NVIDIA build). Someone had run `pip install` which pulled the default CUDA torch from PyPI.
|
|
|
|
**Fix:** Recreate the venv with Python 3.14 (system Python, which has the pacman-installed ROCm pytorch), enable system site-packages, and install ROCm torchvision/torchaudio:
|
|
|
|
```sh
|
|
rm -rf ~/git/ComfyUI/venv
|
|
python3 -m venv ~/git/ComfyUI/venv
|
|
sed -i 's/include-system-site-packages = false/include-system-site-packages = true/' ~/git/ComfyUI/venv/pyvenv.cfg
|
|
|
|
# Install ROCm torchvision/torchaudio (--no-deps to prevent pip pulling CUDA torch as dependency)
|
|
~/git/ComfyUI/venv/bin/pip install torchvision torchaudio \
|
|
--index-url https://download.pytorch.org/whl/rocm7.2 --no-deps
|
|
|
|
# Install everything else except torch/torchvision/torchaudio/torchsde
|
|
grep -viE "^torch|^torchvision|^torchaudio" requirements.txt > /tmp/comfy_reqs.txt
|
|
~/git/ComfyUI/venv/bin/pip install -r /tmp/comfy_reqs.txt
|
|
~/git/ComfyUI/venv/bin/pip install torchsde
|
|
~/git/ComfyUI/venv/bin/pip install comfyui-frontend-package==1.45.15
|
|
```
|
|
|
|
The system ROCm torch is provided by `python-pytorch-rocm` (pacman). Verify:
|
|
```sh
|
|
~/git/ComfyUI/venv/bin/python3 -c "import torch; print(torch.__version__, torch.version.rocm)"
|
|
# Expected: 2.12.0 7.2.3
|
|
```
|
|
|
|
---
|
|
|
|
### 3. ROCm has no precompiled kernels for gfx1151
|
|
|
|
**Symptom:** Service started, GPU detected, but `rocm-smi` showed 0% GPU utilization during inference. All compute fell through to CPU (200% CPU, generation took 30+ minutes).
|
|
|
|
**Root cause:** The rocBLAS library only ships precompiled kernels up to gfx1103 (RDNA 3). gfx1151 (Strix Halo / RDNA 3.5) has no matching kernel files, so every GPU operation silently fell back to CPU.
|
|
|
|
**Fix:** Set `HSA_OVERRIDE_GFX_VERSION=11.0.0` to tell the ROCm runtime to use gfx1100 (RDNA 3) kernels as a fallback. Add to `/etc/systemd/system/comfyui.service`:
|
|
|
|
```ini
|
|
[Service]
|
|
Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0
|
|
```
|
|
|
|
This makes `torch.cuda.get_device_properties(0).gcnArchName` report `gfx1100` instead of `gfx1151`.
|
|
|
|
---
|
|
|
|
### 4. Sub-quadratic attention (CPU fallback) used instead of pytorch attention
|
|
|
|
**Symptom:** Logs showed `Using sub quadratic optimization for attention` — the slowest CPU-only attention path.
|
|
|
|
**Root cause:** ComfyUI's `model_management.py` enables pytorch attention for AMD only if `aotriton_supported(arch)` returns True. It looks for precompiled aotriton kernel images at `torch.__path__[0]/lib/aotriton.images/`. On Arch, `python-aotriton` installs these to `/usr/lib/aotriton.images/` instead — a different path. The directory lookup raised `FileNotFoundError`, caught silently by a bare `except: pass`, leaving `ENABLE_PYTORCH_ATTENTION = False`.
|
|
|
|
Additionally, even if the path was found, there are no `gfx1151`-specific aotriton images. With `HSA_OVERRIDE_GFX_VERSION=11.0.0`, the arch reports as `gfx1100`, which matches the `gfx11xx` wildcard pattern in the aotriton package.
|
|
|
|
**Fix:** Two symlinks (requires sudo):
|
|
|
|
```sh
|
|
# Point torch to the system aotriton images
|
|
sudo ln -s /usr/lib/aotriton.images \
|
|
/usr/lib/python3.14/site-packages/torch/lib/aotriton.images
|
|
|
|
# Add gfx1151 entry pointing to gfx120x kernels as fallback
|
|
sudo ln -s /usr/lib/aotriton.images/amd-gfx120x \
|
|
/usr/lib/aotriton.images/amd-gfx1151
|
|
```
|
|
|
|
After restart, logs should show `Using pytorch attention` instead of `Using sub quadratic optimization`.
|
|
|
|
---
|
|
|
|
### 5. AIMDO async weight offloading deadlocks on AMD
|
|
|
|
**Symptom:** Even with pytorch attention fixed, ComfyUI hung indefinitely on "Load Checkpoint". RSS grew ~5MB/3s (model streaming from disk to CPU RAM) but never completed. GPU stayed at 0%.
|
|
|
|
**Root cause:** ComfyUI's AIMDO (Async Inference Model Data Offloading) system uses CUDA streams for async weight prefetching from CPU to GPU. At startup it logs `Could not autodetect AIMDO implementation, assuming Nvidia` — it falls back to an Nvidia-specific CUDA stream implementation that deadlocks on AMD/ROCm.
|
|
|
|
**Fix:** Add `--disable-smart-memory` to bypass AIMDO entirely:
|
|
|
|
```ini
|
|
ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py \
|
|
--listen 0.0.0.0 --enable-manager --disable-smart-memory
|
|
```
|
|
|
|
---
|
|
|
|
## Final Service File
|
|
|
|
`/etc/systemd/system/comfyui.service`:
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=ComfyUI Service
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
User=schmeeve
|
|
WorkingDirectory=/home/schmeeve/git/ComfyUI
|
|
Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0
|
|
ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py --listen 0.0.0.0 --enable-manager --disable-smart-memory
|
|
Restart=always
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
---
|
|
|
|
## Gotchas for Future Upgrades
|
|
|
|
- **Never run `pip install torch` or `pip install -r requirements.txt` in the venv** — pip will pull the CUDA build from PyPI, overwriting the ROCm setup. Always use `--no-deps` for torch-family packages or install from `--index-url https://download.pytorch.org/whl/rocm7.2`.
|
|
- **The aotriton symlinks live in `/usr/lib/python3.14/site-packages/torch/lib/`** — a pacman upgrade of `python-pytorch-rocm` will recreate that directory and wipe the symlink. Re-run the `ln -s` command after any pytorch upgrade.
|
|
- **`HSA_OVERRIDE_GFX_VERSION=11.0.0`** makes the GPU advertise as gfx1100. If ROCm ever ships proper gfx1151 support (rocBLAS kernels + aotriton images), remove this override.
|
|
- **`include-system-site-packages = true`** in `venv/pyvenv.cfg` is required. The ROCm torch lives at `/usr/lib/python3.14/site-packages/torch`, not inside the venv.
|