schmeeve-toolz/docs/NIMO-COMFYUI-ROCM.md

ComfyUI ROCm Fix — nimo-cachy (Strix Halo / gfx1151)

Machine

• Host: nimo-cachy (nimo.loc)
• GPU: AMD Radeon 8060S (Strix Halo APU, gfx1151 / RDNA 3.5)
• OS: CachyOS (Arch-based), kernel 7.0.12-1-cachyos
• ComfyUI: ~/git/ComfyUI-bak (v0.13.0), served via systemd on port 8188
• System ROCm: 7.2.4 (pacman: rocm-hip-sdk, python-pytorch-rocm 2.12.0-4)

---

Current State (as of 2026-06-14)

ComfyUI v0.13.0 (old codebase) is deployed because v0.24.0 (current master) has a native gfx1151 execution deadlock — it hangs after "VAE load device" when the GPU reports as gfx1151 natively, regardless of torch version, attention mode, or async offload settings. See Known Issues below.

The deployed configuration uses HSA_OVERRIDE_GFX_VERSION=11.0.0 as a workaround, forcing gfx1100 kernel fallback. This sacrifices ~3.4× compute throughput (10 vs 33 TFLOPS) but avoids the deadlock.

---

Deployed Configuration

Venv

Uses system Python 3.14 with include-system-site-packages = true to access the pacman-installed ROCm torch:

rm -rf ~/git/ComfyUI-bak/venv
/usr/bin/python3 -m venv ~/git/ComfyUI-bak/venv
sed -i 's/include-system-site-packages = false/include-system-site-packages = true/' \
    ~/git/ComfyUI-bak/venv/pyvenv.cfg

# Install deps excluding torch (system torch used via site-packages)
grep -viE '^torch$|^torchvision$|^torchaudio$' \
    ~/git/ComfyUI-bak/requirements.txt > /tmp/old_reqs.txt
~/git/ComfyUI-bak/venv/bin/pip install -r /tmp/old_reqs.txt

# Install ROCm-compatible torchvision/torchaudio
~/git/ComfyUI-bak/venv/bin/pip install --force-reinstall --no-deps torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/rocm7.2

Aotriton Symlinks

Required for pytorch attention to find kernel images:

sudo ln -s /usr/lib/aotriton.images \
    /usr/lib/python3.14/site-packages/torch/lib/aotriton.images

sudo ln -s /usr/lib/aotriton.images/amd-gfx120x \
    /usr/lib/aotriton.images/amd-gfx1151

Systemd Service

/etc/systemd/system/comfyui.service:

[Unit]
Description=ComfyUI Service
After=network-online.target
Wants=network-online.target

[Service]
User=schmeeve
WorkingDirectory=/home/schmeeve/git/ComfyUI-bak
Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0
ExecStart=/home/schmeeve/git/ComfyUI-bak/venv/bin/python3 main.py \
    --listen 0.0.0.0 --enable-manager --disable-smart-memory
Restart=always

[Install]
WantedBy=multi-user.target

Flags:

• HSA_OVERRIDE_GFX_VERSION=11.0.0 — workaround for kernel fallback + deadlock avoidance
• --disable-smart-memory — avoids AIMDO "assuming Nvidia" deadlock path

---

Startup Verification

Expected log output on healthy start:

pytorch version: 2.12.0
AMD arch: gfx1100
Device: cuda:0 AMD Radeon 8060S Graphics : native
Using async weight offloading with 2 streams
Using pytorch attention
Starting server

---

Performance

Mode	TFLOPS (fp16 matmul)	Status
Native gfx1151 (system ROCm 7.2.4)	33	Deadlocks in v0.24.0; works in v0.13.0 but old code lacks Chroma nodes
HSA_OVERRIDE=gfx1100	10	Stable, deployed
Old AMD nightly torch 2.9.1 (bundled ROCm)	33	Venv lost; nightly wheel now requires rocm-sdk not available for Arch

---

Known Issues

1. ComfyUI v0.24.0 native gfx1151 deadlock

Symptom: With HSA_OVERRIDE removed (native gfx1151), ComfyUI v0.24.0 hangs after logging "VAE load device". Process idle, GPU 0%, no model loading. Both system torch 2.12.0 and nightly torch affected.

Root cause: Something in the v0.24.0 execution engine (comfy_execution/ rewrite between v0.13.0 and v0.24.0) deadlocks when the GPU reports as gfx1151 natively. Not related to torch version, attention mode (--use-split-cross-attention doesn't fix), or async offloading (--disable-async-offload doesn't fix). The old v0.13.0 execution engine handles native gfx1151 fine.

Workaround: HSA_OVERRIDE_GFX_VERSION=11.0.0 forces gfx1100 which avoids the deadlock but costs ~3.4× kernel performance.

Fix needed: Bisect ComfyUI commits between v0.13.0 and v0.24.0 to find the breaking change, or file upstream bug.

2. AMD nightly torch wheels require rocm-sdk

The pre-Feb-2026 AMD nightly torch builds (e.g., 2.9.1+rocm7.12.0a) bundled their own ROCm libraries (175MB libtorch_hip.so) and worked standalone. Current nightlies for the same version tags now require rocm-sdk as a separate package, which in turn requires system ROCm libraries newer than the Arch-packaged 7.2.4. The old standalone torch wheel is no longer available.

3. No system ROCm >7.2.4 for Arch

AMD's amdgpu-install only supports Ubuntu/RHEL/SUSE. Arch/CachyOS maintain their own ROCm packages, currently at 7.2.4. Upgrading to 7.12+ would require building from source or waiting for distro package updates.

4. ComfyUI v0.24.0 Chroma nodes missing from v0.13.0

The Chroma workflow (SamplerCustomAdvanced, BasicScheduler, EmptySD3LatentImage, ModelSamplingAuraFlow, T5TokenizerOptions) is only available in ComfyUI v0.24.0+, which has the gfx1151 deadlock. The deployed v0.13.0 has these nodes in comfy_extras/ (they were present in the Feb 2026 codebase) but the Chroma workflow submitted during testing appeared to stall — possibly due to the aimdo/package updates applied to the venv.

---

Gotchas

• Never run pip install torch or bare pip install -r requirements.txt — pip pulls CUDA torch from PyPI, overwriting the ROCm setup. Always filter out torch/torchvision/torchaudio from requirements, or use --no-deps with --index-url https://download.pytorch.org/whl/rocm7.2.
• Aotriton symlinks at /usr/lib/python3.14/site-packages/torch/lib/ — a pacman upgrade of python-pytorch-rocm wipes this directory. Re-run the ln -s after pytorch upgrades.
• include-system-site-packages = true in venv pyvenv.cfg is required — the ROCm torch lives at /usr/lib/python3.14/site-packages/torch.
• Two ComfyUI directories exist:
  • ~/git/ComfyUI — v0.24.0 (current master, has deadlock)
  • ~/git/ComfyUI-bak — v0.13.0 (deployed, stable)
• The old ComfyUI-bak venv was destroyed and recreated using system Python 3.14 + ROCm torch from pacman. If weird import errors appear, check that all pip deps from requirements.txt are installed.