From d60b61262b1d683b6697a084145f05553d78dc61 Mon Sep 17 00:00:00 2001 From: schmeeve Date: Sun, 14 Jun 2026 08:49:29 -0700 Subject: [PATCH] Trying to fix comfyUI --- docs/NIMO-COMFYUI-ROCM.md | 143 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 docs/NIMO-COMFYUI-ROCM.md diff --git a/docs/NIMO-COMFYUI-ROCM.md b/docs/NIMO-COMFYUI-ROCM.md new file mode 100644 index 0000000..77c5d7b --- /dev/null +++ b/docs/NIMO-COMFYUI-ROCM.md @@ -0,0 +1,143 @@ +# ComfyUI ROCm Fix — nimo-cachy (Strix Halo / gfx1151) + +## Machine + +- **Host:** nimo-cachy (nimo.loc) +- **GPU:** AMD Radeon 8060S (Strix Halo APU, gfx1151 / RDNA 3.5) +- **OS:** CachyOS (Arch-based), kernel 7.0.12-1-cachyos +- **ComfyUI:** `~/git/ComfyUI` (v0.24.0), served via systemd on port 8188 + +--- + +## Problems & Fixes (in order applied) + +### 1. SQLAlchemy not found + +**Symptom:** ComfyUI crashed on start with `ModuleNotFoundError: No module named 'sqlalchemy'`. + +**Root cause:** The venv at `~/git/ComfyUI/venv` was created with Python 3.13, but `/usr/bin/python3` now points to Python 3.14. The systemd service was calling `/usr/bin/python3.13 main.py` (bare, outside the venv). SQLAlchemy was only installed inside the venv, not system-wide. PEP 668 also blocks system-wide pip installs. + +**Fix:** Update systemd service `ExecStart` to use the venv Python: +``` +ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py --listen 0.0.0.0 --enable-manager +``` + +--- + +### 2. CUDA torch in venv — no NVIDIA GPU + +**Symptom:** After fixing the venv path, crashed with `RuntimeError: Found no NVIDIA driver on your system`. + +**Root cause:** The venv contained `torch 2.12.0+cu130` (CUDA/NVIDIA build). Someone had run `pip install` which pulled the default CUDA torch from PyPI. + +**Fix:** Recreate the venv with Python 3.14 (system Python, which has the pacman-installed ROCm pytorch), enable system site-packages, and install ROCm torchvision/torchaudio: + +```sh +rm -rf ~/git/ComfyUI/venv +python3 -m venv ~/git/ComfyUI/venv +sed -i 's/include-system-site-packages = false/include-system-site-packages = true/' ~/git/ComfyUI/venv/pyvenv.cfg + +# Install ROCm torchvision/torchaudio (--no-deps to prevent pip pulling CUDA torch as dependency) +~/git/ComfyUI/venv/bin/pip install torchvision torchaudio \ + --index-url https://download.pytorch.org/whl/rocm7.2 --no-deps + +# Install everything else except torch/torchvision/torchaudio/torchsde +grep -viE "^torch|^torchvision|^torchaudio" requirements.txt > /tmp/comfy_reqs.txt +~/git/ComfyUI/venv/bin/pip install -r /tmp/comfy_reqs.txt +~/git/ComfyUI/venv/bin/pip install torchsde +~/git/ComfyUI/venv/bin/pip install comfyui-frontend-package==1.45.15 +``` + +The system ROCm torch is provided by `python-pytorch-rocm` (pacman). Verify: +```sh +~/git/ComfyUI/venv/bin/python3 -c "import torch; print(torch.__version__, torch.version.rocm)" +# Expected: 2.12.0 7.2.3 +``` + +--- + +### 3. ROCm has no precompiled kernels for gfx1151 + +**Symptom:** Service started, GPU detected, but `rocm-smi` showed 0% GPU utilization during inference. All compute fell through to CPU (200% CPU, generation took 30+ minutes). + +**Root cause:** The rocBLAS library only ships precompiled kernels up to gfx1103 (RDNA 3). gfx1151 (Strix Halo / RDNA 3.5) has no matching kernel files, so every GPU operation silently fell back to CPU. + +**Fix:** Set `HSA_OVERRIDE_GFX_VERSION=11.0.0` to tell the ROCm runtime to use gfx1100 (RDNA 3) kernels as a fallback. Add to `/etc/systemd/system/comfyui.service`: + +```ini +[Service] +Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0 +``` + +This makes `torch.cuda.get_device_properties(0).gcnArchName` report `gfx1100` instead of `gfx1151`. + +--- + +### 4. Sub-quadratic attention (CPU fallback) used instead of pytorch attention + +**Symptom:** Logs showed `Using sub quadratic optimization for attention` — the slowest CPU-only attention path. + +**Root cause:** ComfyUI's `model_management.py` enables pytorch attention for AMD only if `aotriton_supported(arch)` returns True. It looks for precompiled aotriton kernel images at `torch.__path__[0]/lib/aotriton.images/`. On Arch, `python-aotriton` installs these to `/usr/lib/aotriton.images/` instead — a different path. The directory lookup raised `FileNotFoundError`, caught silently by a bare `except: pass`, leaving `ENABLE_PYTORCH_ATTENTION = False`. + +Additionally, even if the path was found, there are no `gfx1151`-specific aotriton images. With `HSA_OVERRIDE_GFX_VERSION=11.0.0`, the arch reports as `gfx1100`, which matches the `gfx11xx` wildcard pattern in the aotriton package. + +**Fix:** Two symlinks (requires sudo): + +```sh +# Point torch to the system aotriton images +sudo ln -s /usr/lib/aotriton.images \ + /usr/lib/python3.14/site-packages/torch/lib/aotriton.images + +# Add gfx1151 entry pointing to gfx120x kernels as fallback +sudo ln -s /usr/lib/aotriton.images/amd-gfx120x \ + /usr/lib/aotriton.images/amd-gfx1151 +``` + +After restart, logs should show `Using pytorch attention` instead of `Using sub quadratic optimization`. + +--- + +### 5. AIMDO async weight offloading deadlocks on AMD + +**Symptom:** Even with pytorch attention fixed, ComfyUI hung indefinitely on "Load Checkpoint". RSS grew ~5MB/3s (model streaming from disk to CPU RAM) but never completed. GPU stayed at 0%. + +**Root cause:** ComfyUI's AIMDO (Async Inference Model Data Offloading) system uses CUDA streams for async weight prefetching from CPU to GPU. At startup it logs `Could not autodetect AIMDO implementation, assuming Nvidia` — it falls back to an Nvidia-specific CUDA stream implementation that deadlocks on AMD/ROCm. + +**Fix:** Add `--disable-smart-memory` to bypass AIMDO entirely: + +```ini +ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py \ + --listen 0.0.0.0 --enable-manager --disable-smart-memory +``` + +--- + +## Final Service File + +`/etc/systemd/system/comfyui.service`: + +```ini +[Unit] +Description=ComfyUI Service +After=network-online.target +Wants=network-online.target + +[Service] +User=schmeeve +WorkingDirectory=/home/schmeeve/git/ComfyUI +Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0 +ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py --listen 0.0.0.0 --enable-manager --disable-smart-memory +Restart=always + +[Install] +WantedBy=multi-user.target +``` + +--- + +## Gotchas for Future Upgrades + +- **Never run `pip install torch` or `pip install -r requirements.txt` in the venv** — pip will pull the CUDA build from PyPI, overwriting the ROCm setup. Always use `--no-deps` for torch-family packages or install from `--index-url https://download.pytorch.org/whl/rocm7.2`. +- **The aotriton symlinks live in `/usr/lib/python3.14/site-packages/torch/lib/`** — a pacman upgrade of `python-pytorch-rocm` will recreate that directory and wipe the symlink. Re-run the `ln -s` command after any pytorch upgrade. +- **`HSA_OVERRIDE_GFX_VERSION=11.0.0`** makes the GPU advertise as gfx1100. If ROCm ever ships proper gfx1151 support (rocBLAS kernels + aotriton images), remove this override. +- **`include-system-site-packages = true`** in `venv/pyvenv.cfg` is required. The ROCm torch lives at `/usr/lib/python3.14/site-packages/torch`, not inside the venv.