143 lines
6.2 KiB
Markdown
143 lines
6.2 KiB
Markdown
# ComfyUI ROCm Fix — nimo-cachy (Strix Halo / gfx1151)
|
||
|
||
## Machine
|
||
|
||
- **Host:** nimo-cachy (nimo.loc)
|
||
- **GPU:** AMD Radeon 8060S (Strix Halo APU, gfx1151 / RDNA 3.5)
|
||
- **OS:** CachyOS (Arch-based), kernel 7.0.12-1-cachyos
|
||
- **ComfyUI:** `~/git/ComfyUI-bak` (v0.13.0), served via systemd on port 8188
|
||
- **System ROCm:** 7.2.4 (pacman: `rocm-hip-sdk`, `python-pytorch-rocm` 2.12.0-4)
|
||
|
||
---
|
||
|
||
## Current State (as of 2026-06-14)
|
||
|
||
ComfyUI v0.13.0 (old codebase) is deployed because v0.24.0 (current master) has a **native gfx1151 execution deadlock** — it hangs after "VAE load device" when the GPU reports as gfx1151 natively, regardless of torch version, attention mode, or async offload settings. See [Known Issues](#known-issues) below.
|
||
|
||
The deployed configuration uses `HSA_OVERRIDE_GFX_VERSION=11.0.0` as a workaround, forcing gfx1100 kernel fallback. This sacrifices ~3.4× compute throughput (10 vs 33 TFLOPS) but avoids the deadlock.
|
||
|
||
---
|
||
|
||
## Deployed Configuration
|
||
|
||
### Venv
|
||
|
||
Uses system Python 3.14 with `include-system-site-packages = true` to access the pacman-installed ROCm torch:
|
||
|
||
```sh
|
||
rm -rf ~/git/ComfyUI-bak/venv
|
||
/usr/bin/python3 -m venv ~/git/ComfyUI-bak/venv
|
||
sed -i 's/include-system-site-packages = false/include-system-site-packages = true/' \
|
||
~/git/ComfyUI-bak/venv/pyvenv.cfg
|
||
|
||
# Install deps excluding torch (system torch used via site-packages)
|
||
grep -viE '^torch$|^torchvision$|^torchaudio$' \
|
||
~/git/ComfyUI-bak/requirements.txt > /tmp/old_reqs.txt
|
||
~/git/ComfyUI-bak/venv/bin/pip install -r /tmp/old_reqs.txt
|
||
|
||
# Install ROCm-compatible torchvision/torchaudio
|
||
~/git/ComfyUI-bak/venv/bin/pip install --force-reinstall --no-deps torchvision torchaudio \
|
||
--index-url https://download.pytorch.org/whl/rocm7.2
|
||
```
|
||
|
||
### Aotriton Symlinks
|
||
|
||
Required for pytorch attention to find kernel images:
|
||
|
||
```sh
|
||
sudo ln -s /usr/lib/aotriton.images \
|
||
/usr/lib/python3.14/site-packages/torch/lib/aotriton.images
|
||
|
||
sudo ln -s /usr/lib/aotriton.images/amd-gfx120x \
|
||
/usr/lib/aotriton.images/amd-gfx1151
|
||
```
|
||
|
||
### Systemd Service
|
||
|
||
`/etc/systemd/system/comfyui.service`:
|
||
|
||
```ini
|
||
[Unit]
|
||
Description=ComfyUI Service
|
||
After=network-online.target
|
||
Wants=network-online.target
|
||
|
||
[Service]
|
||
User=schmeeve
|
||
WorkingDirectory=/home/schmeeve/git/ComfyUI-bak
|
||
Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0
|
||
ExecStart=/home/schmeeve/git/ComfyUI-bak/venv/bin/python3 main.py \
|
||
--listen 0.0.0.0 --enable-manager --disable-smart-memory
|
||
Restart=always
|
||
|
||
[Install]
|
||
WantedBy=multi-user.target
|
||
```
|
||
|
||
Flags:
|
||
- `HSA_OVERRIDE_GFX_VERSION=11.0.0` — workaround for kernel fallback + deadlock avoidance
|
||
- `--disable-smart-memory` — avoids AIMDO "assuming Nvidia" deadlock path
|
||
|
||
---
|
||
|
||
## Startup Verification
|
||
|
||
Expected log output on healthy start:
|
||
|
||
```
|
||
pytorch version: 2.12.0
|
||
AMD arch: gfx1100
|
||
Device: cuda:0 AMD Radeon 8060S Graphics : native
|
||
Using async weight offloading with 2 streams
|
||
Using pytorch attention
|
||
Starting server
|
||
```
|
||
|
||
---
|
||
|
||
## Performance
|
||
|
||
| Mode | TFLOPS (fp16 matmul) | Status |
|
||
|---|---|---|
|
||
| Native gfx1151 (system ROCm 7.2.4) | 33 | Deadlocks in v0.24.0; works in v0.13.0 but old code lacks Chroma nodes |
|
||
| HSA_OVERRIDE=gfx1100 | 10 | Stable, deployed |
|
||
| Old AMD nightly torch 2.9.1 (bundled ROCm) | 33 | Venv lost; nightly wheel now requires rocm-sdk not available for Arch |
|
||
|
||
---
|
||
|
||
## Known Issues
|
||
|
||
### 1. ComfyUI v0.24.0 native gfx1151 deadlock
|
||
|
||
**Symptom:** With HSA_OVERRIDE removed (native gfx1151), ComfyUI v0.24.0 hangs after logging "VAE load device". Process idle, GPU 0%, no model loading. Both system torch 2.12.0 and nightly torch affected.
|
||
|
||
**Root cause:** Something in the v0.24.0 execution engine (`comfy_execution/` rewrite between v0.13.0 and v0.24.0) deadlocks when the GPU reports as gfx1151 natively. Not related to torch version, attention mode (`--use-split-cross-attention` doesn't fix), or async offloading (`--disable-async-offload` doesn't fix). The old v0.13.0 execution engine handles native gfx1151 fine.
|
||
|
||
**Workaround:** `HSA_OVERRIDE_GFX_VERSION=11.0.0` forces gfx1100 which avoids the deadlock but costs ~3.4× kernel performance.
|
||
|
||
**Fix needed:** Bisect ComfyUI commits between v0.13.0 and v0.24.0 to find the breaking change, or file upstream bug.
|
||
|
||
### 2. AMD nightly torch wheels require rocm-sdk
|
||
|
||
The pre-Feb-2026 AMD nightly torch builds (e.g., 2.9.1+rocm7.12.0a) bundled their own ROCm libraries (175MB `libtorch_hip.so`) and worked standalone. Current nightlies for the same version tags now require `rocm-sdk` as a separate package, which in turn requires system ROCm libraries newer than the Arch-packaged 7.2.4. The old standalone torch wheel is no longer available.
|
||
|
||
### 3. No system ROCm >7.2.4 for Arch
|
||
|
||
AMD's `amdgpu-install` only supports Ubuntu/RHEL/SUSE. Arch/CachyOS maintain their own ROCm packages, currently at 7.2.4. Upgrading to 7.12+ would require building from source or waiting for distro package updates.
|
||
|
||
### 4. ComfyUI v0.24.0 Chroma nodes missing from v0.13.0
|
||
|
||
The Chroma workflow (SamplerCustomAdvanced, BasicScheduler, EmptySD3LatentImage, ModelSamplingAuraFlow, T5TokenizerOptions) is only available in ComfyUI v0.24.0+, which has the gfx1151 deadlock. The deployed v0.13.0 has these nodes in `comfy_extras/` (they were present in the Feb 2026 codebase) but the Chroma workflow submitted during testing appeared to stall — possibly due to the aimdo/package updates applied to the venv.
|
||
|
||
---
|
||
|
||
## Gotchas
|
||
|
||
- **Never run `pip install torch` or bare `pip install -r requirements.txt`** — pip pulls CUDA torch from PyPI, overwriting the ROCm setup. Always filter out torch/torchvision/torchaudio from requirements, or use `--no-deps` with `--index-url https://download.pytorch.org/whl/rocm7.2`.
|
||
- **Aotriton symlinks at `/usr/lib/python3.14/site-packages/torch/lib/`** — a pacman upgrade of `python-pytorch-rocm` wipes this directory. Re-run the `ln -s` after pytorch upgrades.
|
||
- **`include-system-site-packages = true`** in venv `pyvenv.cfg` is required — the ROCm torch lives at `/usr/lib/python3.14/site-packages/torch`.
|
||
- **Two ComfyUI directories exist:**
|
||
- `~/git/ComfyUI` — v0.24.0 (current master, has deadlock)
|
||
- `~/git/ComfyUI-bak` — v0.13.0 (deployed, stable)
|
||
- The old ComfyUI-bak venv was destroyed and recreated using system Python 3.14 + ROCm torch from pacman. If weird import errors appear, check that all pip deps from `requirements.txt` are installed.
|