From 204880508b1f889853007350d9a960c823e89255 Mon Sep 17 00:00:00 2001 From: schmeeve Date: Sun, 14 Jun 2026 11:20:41 -0700 Subject: [PATCH] the struggle (with ComfyUI) is real --- docs/NIMO-COMFYUI-ROCM.md | 177 +++++++++++++++++++------------------- 1 file changed, 88 insertions(+), 89 deletions(-) diff --git a/docs/NIMO-COMFYUI-ROCM.md b/docs/NIMO-COMFYUI-ROCM.md index 77c5d7b..3662620 100644 --- a/docs/NIMO-COMFYUI-ROCM.md +++ b/docs/NIMO-COMFYUI-ROCM.md @@ -5,114 +5,54 @@ - **Host:** nimo-cachy (nimo.loc) - **GPU:** AMD Radeon 8060S (Strix Halo APU, gfx1151 / RDNA 3.5) - **OS:** CachyOS (Arch-based), kernel 7.0.12-1-cachyos -- **ComfyUI:** `~/git/ComfyUI` (v0.24.0), served via systemd on port 8188 +- **ComfyUI:** `~/git/ComfyUI-bak` (v0.13.0), served via systemd on port 8188 +- **System ROCm:** 7.2.4 (pacman: `rocm-hip-sdk`, `python-pytorch-rocm` 2.12.0-4) --- -## Problems & Fixes (in order applied) +## Current State (as of 2026-06-14) -### 1. SQLAlchemy not found +ComfyUI v0.13.0 (old codebase) is deployed because v0.24.0 (current master) has a **native gfx1151 execution deadlock** — it hangs after "VAE load device" when the GPU reports as gfx1151 natively, regardless of torch version, attention mode, or async offload settings. See [Known Issues](#known-issues) below. -**Symptom:** ComfyUI crashed on start with `ModuleNotFoundError: No module named 'sqlalchemy'`. - -**Root cause:** The venv at `~/git/ComfyUI/venv` was created with Python 3.13, but `/usr/bin/python3` now points to Python 3.14. The systemd service was calling `/usr/bin/python3.13 main.py` (bare, outside the venv). SQLAlchemy was only installed inside the venv, not system-wide. PEP 668 also blocks system-wide pip installs. - -**Fix:** Update systemd service `ExecStart` to use the venv Python: -``` -ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py --listen 0.0.0.0 --enable-manager -``` +The deployed configuration uses `HSA_OVERRIDE_GFX_VERSION=11.0.0` as a workaround, forcing gfx1100 kernel fallback. This sacrifices ~3.4× compute throughput (10 vs 33 TFLOPS) but avoids the deadlock. --- -### 2. CUDA torch in venv — no NVIDIA GPU +## Deployed Configuration -**Symptom:** After fixing the venv path, crashed with `RuntimeError: Found no NVIDIA driver on your system`. +### Venv -**Root cause:** The venv contained `torch 2.12.0+cu130` (CUDA/NVIDIA build). Someone had run `pip install` which pulled the default CUDA torch from PyPI. - -**Fix:** Recreate the venv with Python 3.14 (system Python, which has the pacman-installed ROCm pytorch), enable system site-packages, and install ROCm torchvision/torchaudio: +Uses system Python 3.14 with `include-system-site-packages = true` to access the pacman-installed ROCm torch: ```sh -rm -rf ~/git/ComfyUI/venv -python3 -m venv ~/git/ComfyUI/venv -sed -i 's/include-system-site-packages = false/include-system-site-packages = true/' ~/git/ComfyUI/venv/pyvenv.cfg +rm -rf ~/git/ComfyUI-bak/venv +/usr/bin/python3 -m venv ~/git/ComfyUI-bak/venv +sed -i 's/include-system-site-packages = false/include-system-site-packages = true/' \ + ~/git/ComfyUI-bak/venv/pyvenv.cfg -# Install ROCm torchvision/torchaudio (--no-deps to prevent pip pulling CUDA torch as dependency) -~/git/ComfyUI/venv/bin/pip install torchvision torchaudio \ - --index-url https://download.pytorch.org/whl/rocm7.2 --no-deps +# Install deps excluding torch (system torch used via site-packages) +grep -viE '^torch$|^torchvision$|^torchaudio$' \ + ~/git/ComfyUI-bak/requirements.txt > /tmp/old_reqs.txt +~/git/ComfyUI-bak/venv/bin/pip install -r /tmp/old_reqs.txt -# Install everything else except torch/torchvision/torchaudio/torchsde -grep -viE "^torch|^torchvision|^torchaudio" requirements.txt > /tmp/comfy_reqs.txt -~/git/ComfyUI/venv/bin/pip install -r /tmp/comfy_reqs.txt -~/git/ComfyUI/venv/bin/pip install torchsde -~/git/ComfyUI/venv/bin/pip install comfyui-frontend-package==1.45.15 +# Install ROCm-compatible torchvision/torchaudio +~/git/ComfyUI-bak/venv/bin/pip install --force-reinstall --no-deps torchvision torchaudio \ + --index-url https://download.pytorch.org/whl/rocm7.2 ``` -The system ROCm torch is provided by `python-pytorch-rocm` (pacman). Verify: -```sh -~/git/ComfyUI/venv/bin/python3 -c "import torch; print(torch.__version__, torch.version.rocm)" -# Expected: 2.12.0 7.2.3 -``` +### Aotriton Symlinks ---- - -### 3. ROCm has no precompiled kernels for gfx1151 - -**Symptom:** Service started, GPU detected, but `rocm-smi` showed 0% GPU utilization during inference. All compute fell through to CPU (200% CPU, generation took 30+ minutes). - -**Root cause:** The rocBLAS library only ships precompiled kernels up to gfx1103 (RDNA 3). gfx1151 (Strix Halo / RDNA 3.5) has no matching kernel files, so every GPU operation silently fell back to CPU. - -**Fix:** Set `HSA_OVERRIDE_GFX_VERSION=11.0.0` to tell the ROCm runtime to use gfx1100 (RDNA 3) kernels as a fallback. Add to `/etc/systemd/system/comfyui.service`: - -```ini -[Service] -Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0 -``` - -This makes `torch.cuda.get_device_properties(0).gcnArchName` report `gfx1100` instead of `gfx1151`. - ---- - -### 4. Sub-quadratic attention (CPU fallback) used instead of pytorch attention - -**Symptom:** Logs showed `Using sub quadratic optimization for attention` — the slowest CPU-only attention path. - -**Root cause:** ComfyUI's `model_management.py` enables pytorch attention for AMD only if `aotriton_supported(arch)` returns True. It looks for precompiled aotriton kernel images at `torch.__path__[0]/lib/aotriton.images/`. On Arch, `python-aotriton` installs these to `/usr/lib/aotriton.images/` instead — a different path. The directory lookup raised `FileNotFoundError`, caught silently by a bare `except: pass`, leaving `ENABLE_PYTORCH_ATTENTION = False`. - -Additionally, even if the path was found, there are no `gfx1151`-specific aotriton images. With `HSA_OVERRIDE_GFX_VERSION=11.0.0`, the arch reports as `gfx1100`, which matches the `gfx11xx` wildcard pattern in the aotriton package. - -**Fix:** Two symlinks (requires sudo): +Required for pytorch attention to find kernel images: ```sh -# Point torch to the system aotriton images sudo ln -s /usr/lib/aotriton.images \ /usr/lib/python3.14/site-packages/torch/lib/aotriton.images -# Add gfx1151 entry pointing to gfx120x kernels as fallback sudo ln -s /usr/lib/aotriton.images/amd-gfx120x \ /usr/lib/aotriton.images/amd-gfx1151 ``` -After restart, logs should show `Using pytorch attention` instead of `Using sub quadratic optimization`. - ---- - -### 5. AIMDO async weight offloading deadlocks on AMD - -**Symptom:** Even with pytorch attention fixed, ComfyUI hung indefinitely on "Load Checkpoint". RSS grew ~5MB/3s (model streaming from disk to CPU RAM) but never completed. GPU stayed at 0%. - -**Root cause:** ComfyUI's AIMDO (Async Inference Model Data Offloading) system uses CUDA streams for async weight prefetching from CPU to GPU. At startup it logs `Could not autodetect AIMDO implementation, assuming Nvidia` — it falls back to an Nvidia-specific CUDA stream implementation that deadlocks on AMD/ROCm. - -**Fix:** Add `--disable-smart-memory` to bypass AIMDO entirely: - -```ini -ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py \ - --listen 0.0.0.0 --enable-manager --disable-smart-memory -``` - ---- - -## Final Service File +### Systemd Service `/etc/systemd/system/comfyui.service`: @@ -124,20 +64,79 @@ Wants=network-online.target [Service] User=schmeeve -WorkingDirectory=/home/schmeeve/git/ComfyUI +WorkingDirectory=/home/schmeeve/git/ComfyUI-bak Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0 -ExecStart=/home/schmeeve/git/ComfyUI/venv/bin/python3 main.py --listen 0.0.0.0 --enable-manager --disable-smart-memory +ExecStart=/home/schmeeve/git/ComfyUI-bak/venv/bin/python3 main.py \ + --listen 0.0.0.0 --enable-manager --disable-smart-memory Restart=always [Install] WantedBy=multi-user.target ``` +Flags: +- `HSA_OVERRIDE_GFX_VERSION=11.0.0` — workaround for kernel fallback + deadlock avoidance +- `--disable-smart-memory` — avoids AIMDO "assuming Nvidia" deadlock path + --- -## Gotchas for Future Upgrades +## Startup Verification -- **Never run `pip install torch` or `pip install -r requirements.txt` in the venv** — pip will pull the CUDA build from PyPI, overwriting the ROCm setup. Always use `--no-deps` for torch-family packages or install from `--index-url https://download.pytorch.org/whl/rocm7.2`. -- **The aotriton symlinks live in `/usr/lib/python3.14/site-packages/torch/lib/`** — a pacman upgrade of `python-pytorch-rocm` will recreate that directory and wipe the symlink. Re-run the `ln -s` command after any pytorch upgrade. -- **`HSA_OVERRIDE_GFX_VERSION=11.0.0`** makes the GPU advertise as gfx1100. If ROCm ever ships proper gfx1151 support (rocBLAS kernels + aotriton images), remove this override. -- **`include-system-site-packages = true`** in `venv/pyvenv.cfg` is required. The ROCm torch lives at `/usr/lib/python3.14/site-packages/torch`, not inside the venv. +Expected log output on healthy start: + +``` +pytorch version: 2.12.0 +AMD arch: gfx1100 +Device: cuda:0 AMD Radeon 8060S Graphics : native +Using async weight offloading with 2 streams +Using pytorch attention +Starting server +``` + +--- + +## Performance + +| Mode | TFLOPS (fp16 matmul) | Status | +|---|---|---| +| Native gfx1151 (system ROCm 7.2.4) | 33 | Deadlocks in v0.24.0; works in v0.13.0 but old code lacks Chroma nodes | +| HSA_OVERRIDE=gfx1100 | 10 | Stable, deployed | +| Old AMD nightly torch 2.9.1 (bundled ROCm) | 33 | Venv lost; nightly wheel now requires rocm-sdk not available for Arch | + +--- + +## Known Issues + +### 1. ComfyUI v0.24.0 native gfx1151 deadlock + +**Symptom:** With HSA_OVERRIDE removed (native gfx1151), ComfyUI v0.24.0 hangs after logging "VAE load device". Process idle, GPU 0%, no model loading. Both system torch 2.12.0 and nightly torch affected. + +**Root cause:** Something in the v0.24.0 execution engine (`comfy_execution/` rewrite between v0.13.0 and v0.24.0) deadlocks when the GPU reports as gfx1151 natively. Not related to torch version, attention mode (`--use-split-cross-attention` doesn't fix), or async offloading (`--disable-async-offload` doesn't fix). The old v0.13.0 execution engine handles native gfx1151 fine. + +**Workaround:** `HSA_OVERRIDE_GFX_VERSION=11.0.0` forces gfx1100 which avoids the deadlock but costs ~3.4× kernel performance. + +**Fix needed:** Bisect ComfyUI commits between v0.13.0 and v0.24.0 to find the breaking change, or file upstream bug. + +### 2. AMD nightly torch wheels require rocm-sdk + +The pre-Feb-2026 AMD nightly torch builds (e.g., 2.9.1+rocm7.12.0a) bundled their own ROCm libraries (175MB `libtorch_hip.so`) and worked standalone. Current nightlies for the same version tags now require `rocm-sdk` as a separate package, which in turn requires system ROCm libraries newer than the Arch-packaged 7.2.4. The old standalone torch wheel is no longer available. + +### 3. No system ROCm >7.2.4 for Arch + +AMD's `amdgpu-install` only supports Ubuntu/RHEL/SUSE. Arch/CachyOS maintain their own ROCm packages, currently at 7.2.4. Upgrading to 7.12+ would require building from source or waiting for distro package updates. + +### 4. ComfyUI v0.24.0 Chroma nodes missing from v0.13.0 + +The Chroma workflow (SamplerCustomAdvanced, BasicScheduler, EmptySD3LatentImage, ModelSamplingAuraFlow, T5TokenizerOptions) is only available in ComfyUI v0.24.0+, which has the gfx1151 deadlock. The deployed v0.13.0 has these nodes in `comfy_extras/` (they were present in the Feb 2026 codebase) but the Chroma workflow submitted during testing appeared to stall — possibly due to the aimdo/package updates applied to the venv. + +--- + +## Gotchas + +- **Never run `pip install torch` or bare `pip install -r requirements.txt`** — pip pulls CUDA torch from PyPI, overwriting the ROCm setup. Always filter out torch/torchvision/torchaudio from requirements, or use `--no-deps` with `--index-url https://download.pytorch.org/whl/rocm7.2`. +- **Aotriton symlinks at `/usr/lib/python3.14/site-packages/torch/lib/`** — a pacman upgrade of `python-pytorch-rocm` wipes this directory. Re-run the `ln -s` after pytorch upgrades. +- **`include-system-site-packages = true`** in venv `pyvenv.cfg` is required — the ROCm torch lives at `/usr/lib/python3.14/site-packages/torch`. +- **Two ComfyUI directories exist:** + - `~/git/ComfyUI` — v0.24.0 (current master, has deadlock) + - `~/git/ComfyUI-bak` — v0.13.0 (deployed, stable) +- The old ComfyUI-bak venv was destroyed and recreated using system Python 3.14 + ROCm torch from pacman. If weird import errors appear, check that all pip deps from `requirements.txt` are installed.