schmeeve/schmeeve-toolz
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d60b61262b1d683b6697a084145f05553d78dc61
schmeeve-toolz/docs/NIMO-COMFYUI-ROCM.md
schmeeve d60b61262b Trying to fix comfyUI
2026-06-14 08:49:29 -07:00

6.5 KiB
Raw Blame History

ComfyUI ROCm Fix — nimo-cachy (Strix Halo / gfx1151)

Machine

  • Host: nimo-cachy (nimo.loc)
  • GPU: AMD Radeon 8060S (Strix Halo APU, gfx1151 / RDNA 3.5)
  • OS: CachyOS (Arch-based), kernel 7.0.12-1-cachyos
  • ComfyUI: ~/git/ComfyUI (v0.24.0), served via systemd on port 8188

Problems & Fixes (in order applied)

1. SQLAlchemy not found

Symptom: ComfyUI crashed on start with ModuleNotFoundError: No module named 'sqlalchemy'.

Root cause: The venv at ~/git/ComfyUI/venv was created with Python 3.13, but /usr/bin/python3 now points to Python 3.14. The systemd service was calling /usr/bin/python3.13 main.py (bare, outside the venv). SQLAlchemy was only installed inside the venv, not system-wide. PEP 668 also blocks system-wide pip installs.

Fix: Update systemd service ExecStart to use the venv Python:

ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py --listen 0.0.0.0 --enable-manager

2. CUDA torch in venv — no NVIDIA GPU

Symptom: After fixing the venv path, crashed with RuntimeError: Found no NVIDIA driver on your system.

Root cause: The venv contained torch 2.12.0+cu130 (CUDA/NVIDIA build). Someone had run pip install which pulled the default CUDA torch from PyPI.

Fix: Recreate the venv with Python 3.14 (system Python, which has the pacman-installed ROCm pytorch), enable system site-packages, and install ROCm torchvision/torchaudio:

rm -rf ~/git/ComfyUI/venv
python3 -m venv ~/git/ComfyUI/venv
sed -i 's/include-system-site-packages = false/include-system-site-packages = true/' ~/git/ComfyUI/venv/pyvenv.cfg

# Install ROCm torchvision/torchaudio (--no-deps to prevent pip pulling CUDA torch as dependency)
~/git/ComfyUI/venv/bin/pip install torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/rocm7.2 --no-deps

# Install everything else except torch/torchvision/torchaudio/torchsde
grep -viE "^torch|^torchvision|^torchaudio" requirements.txt > /tmp/comfy_reqs.txt
~/git/ComfyUI/venv/bin/pip install -r /tmp/comfy_reqs.txt
~/git/ComfyUI/venv/bin/pip install torchsde
~/git/ComfyUI/venv/bin/pip install comfyui-frontend-package==1.45.15

The system ROCm torch is provided by python-pytorch-rocm (pacman). Verify:

~/git/ComfyUI/venv/bin/python3 -c "import torch; print(torch.__version__, torch.version.rocm)"
# Expected: 2.12.0 7.2.3

3. ROCm has no precompiled kernels for gfx1151

Symptom: Service started, GPU detected, but rocm-smi showed 0% GPU utilization during inference. All compute fell through to CPU (200% CPU, generation took 30+ minutes).

Root cause: The rocBLAS library only ships precompiled kernels up to gfx1103 (RDNA 3). gfx1151 (Strix Halo / RDNA 3.5) has no matching kernel files, so every GPU operation silently fell back to CPU.

Fix: Set HSA_OVERRIDE_GFX_VERSION=11.0.0 to tell the ROCm runtime to use gfx1100 (RDNA 3) kernels as a fallback. Add to /etc/systemd/system/comfyui.service:

[Service]
Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0

This makes torch.cuda.get_device_properties(0).gcnArchName report gfx1100 instead of gfx1151.

4. Sub-quadratic attention (CPU fallback) used instead of pytorch attention

Symptom: Logs showed Using sub quadratic optimization for attention — the slowest CPU-only attention path.

Root cause: ComfyUI's model_management.py enables pytorch attention for AMD only if aotriton_supported(arch) returns True. It looks for precompiled aotriton kernel images at torch.__path__[0]/lib/aotriton.images/. On Arch, python-aotriton installs these to /usr/lib/aotriton.images/ instead — a different path. The directory lookup raised FileNotFoundError, caught silently by a bare except: pass, leaving ENABLE_PYTORCH_ATTENTION = False.

Additionally, even if the path was found, there are no gfx1151-specific aotriton images. With HSA_OVERRIDE_GFX_VERSION=11.0.0, the arch reports as gfx1100, which matches the gfx11xx wildcard pattern in the aotriton package.

Fix: Two symlinks (requires sudo):

# Point torch to the system aotriton images
sudo ln -s /usr/lib/aotriton.images \
    /usr/lib/python3.14/site-packages/torch/lib/aotriton.images

# Add gfx1151 entry pointing to gfx120x kernels as fallback
sudo ln -s /usr/lib/aotriton.images/amd-gfx120x \
    /usr/lib/aotriton.images/amd-gfx1151

After restart, logs should show Using pytorch attention instead of Using sub quadratic optimization.

5. AIMDO async weight offloading deadlocks on AMD

Symptom: Even with pytorch attention fixed, ComfyUI hung indefinitely on "Load Checkpoint". RSS grew ~5MB/3s (model streaming from disk to CPU RAM) but never completed. GPU stayed at 0%.

Root cause: ComfyUI's AIMDO (Async Inference Model Data Offloading) system uses CUDA streams for async weight prefetching from CPU to GPU. At startup it logs Could not autodetect AIMDO implementation, assuming Nvidia — it falls back to an Nvidia-specific CUDA stream implementation that deadlocks on AMD/ROCm.

Fix: Add --disable-smart-memory to bypass AIMDO entirely:

ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py \
    --listen 0.0.0.0 --enable-manager --disable-smart-memory

Final Service File

/etc/systemd/system/comfyui.service:

[Unit]
Description=ComfyUI Service
After=network-online.target
Wants=network-online.target

[Service]
User=schmeeve
WorkingDirectory=/home/schmeeve/git/ComfyUI
Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0
ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py --listen 0.0.0.0 --enable-manager --disable-smart-memory
Restart=always

[Install]
WantedBy=multi-user.target

Gotchas for Future Upgrades

  • Never run pip install torch or pip install -r requirements.txt in the venv — pip will pull the CUDA build from PyPI, overwriting the ROCm setup. Always use --no-deps for torch-family packages or install from --index-url https://download.pytorch.org/whl/rocm7.2.
  • The aotriton symlinks live in /usr/lib/python3.14/site-packages/torch/lib/ — a pacman upgrade of python-pytorch-rocm will recreate that directory and wipe the symlink. Re-run the ln -s command after any pytorch upgrade.
  • HSA_OVERRIDE_GFX_VERSION=11.0.0 makes the GPU advertise as gfx1100. If ROCm ever ships proper gfx1151 support (rocBLAS kernels + aotriton images), remove this override.
  • include-system-site-packages = true in venv/pyvenv.cfg is required. The ROCm torch lives at /usr/lib/python3.14/site-packages/torch, not inside the venv.
Reference in New Issue View Git Blame Copy Permalink