# ComfyUI ROCm Fix — nimo-cachy (Strix Halo / gfx1151)

## Machine

- **Host:** nimo-cachy (nimo.loc)
- **GPU:** AMD Radeon 8060S (Strix Halo APU, gfx1151 / RDNA 3.5)
- **OS:** CachyOS (Arch-based), kernel 7.0.12-1-cachyos
- **ComfyUI:** `~/git/ComfyUI-bak` (v0.13.0), served via systemd on port 8188
- **System ROCm:** 7.2.4 (pacman: `rocm-hip-sdk`, `python-pytorch-rocm` 2.12.0-4)

---

## Current State (as of 2026-06-14)

ComfyUI v0.13.0 (old codebase) is deployed because v0.24.0 (current master) has a **native gfx1151 execution deadlock** — it hangs after "VAE load device" when the GPU reports as gfx1151 natively, regardless of torch version, attention mode, or async offload settings. See [Known Issues](#known-issues) below.

The deployed configuration uses `HSA_OVERRIDE_GFX_VERSION=11.0.0` as a workaround, forcing gfx1100 kernel fallback. This sacrifices ~3.4× compute throughput (10 vs 33 TFLOPS) but avoids the deadlock.

---

## Deployed Configuration

### Venv

Uses system Python 3.14 with `include-system-site-packages = true` to access the pacman-installed ROCm torch:

```sh
rm -rf ~/git/ComfyUI-bak/venv
/usr/bin/python3 -m venv ~/git/ComfyUI-bak/venv
sed -i 's/include-system-site-packages = false/include-system-site-packages = true/' \
    ~/git/ComfyUI-bak/venv/pyvenv.cfg

# Install deps excluding torch (system torch used via site-packages)
grep -viE '^torch$|^torchvision$|^torchaudio$' \
    ~/git/ComfyUI-bak/requirements.txt > /tmp/old_reqs.txt
~/git/ComfyUI-bak/venv/bin/pip install -r /tmp/old_reqs.txt

# Install ROCm-compatible torchvision/torchaudio
~/git/ComfyUI-bak/venv/bin/pip install --force-reinstall --no-deps torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/rocm7.2
```

### Aotriton Symlinks

Required for pytorch attention to find kernel images:

```sh
sudo ln -s /usr/lib/aotriton.images \
    /usr/lib/python3.14/site-packages/torch/lib/aotriton.images

sudo ln -s /usr/lib/aotriton.images/amd-gfx120x \
    /usr/lib/aotriton.images/amd-gfx1151
```

### Systemd Service

`/etc/systemd/system/comfyui.service`:

```ini
[Unit]
Description=ComfyUI Service
After=network-online.target
Wants=network-online.target

[Service]
User=schmeeve
WorkingDirectory=/home/schmeeve/git/ComfyUI-bak
Environment=HSA_OVERRIDE_GFX_VERSION=11.0.0
ExecStart=/home/schmeeve/git/ComfyUI-bak/venv/bin/python3 main.py \
    --listen 0.0.0.0 --enable-manager --disable-smart-memory
Restart=always

[Install]
WantedBy=multi-user.target
```

Flags:
- `HSA_OVERRIDE_GFX_VERSION=11.0.0` — workaround for kernel fallback + deadlock avoidance
- `--disable-smart-memory` — avoids AIMDO "assuming Nvidia" deadlock path

---

## Startup Verification

Expected log output on healthy start:

```
pytorch version: 2.12.0
AMD arch: gfx1100
Device: cuda:0 AMD Radeon 8060S Graphics : native
Using async weight offloading with 2 streams
Using pytorch attention
Starting server
```

---

## Performance

| Mode | TFLOPS (fp16 matmul) | Status |
|---|---|---|
| Native gfx1151 (system ROCm 7.2.4) | 33 | Deadlocks in v0.24.0; works in v0.13.0 but old code lacks Chroma nodes |
| HSA_OVERRIDE=gfx1100 | 10 | Stable, deployed |
| Old AMD nightly torch 2.9.1 (bundled ROCm) | 33 | Venv lost; nightly wheel now requires rocm-sdk not available for Arch |

---

## Known Issues

### 1. ComfyUI v0.24.0 native gfx1151 deadlock

**Symptom:** With HSA_OVERRIDE removed (native gfx1151), ComfyUI v0.24.0 hangs after logging "VAE load device". Process idle, GPU 0%, no model loading. Both system torch 2.12.0 and nightly torch affected.

**Root cause:** Something in the v0.24.0 execution engine (`comfy_execution/` rewrite between v0.13.0 and v0.24.0) deadlocks when the GPU reports as gfx1151 natively. Not related to torch version, attention mode (`--use-split-cross-attention` doesn't fix), or async offloading (`--disable-async-offload` doesn't fix). The old v0.13.0 execution engine handles native gfx1151 fine.

**Workaround:** `HSA_OVERRIDE_GFX_VERSION=11.0.0` forces gfx1100 which avoids the deadlock but costs ~3.4× kernel performance.

**Fix needed:** Bisect ComfyUI commits between v0.13.0 and v0.24.0 to find the breaking change, or file upstream bug.

### 2. AMD nightly torch wheels require rocm-sdk

The pre-Feb-2026 AMD nightly torch builds (e.g., 2.9.1+rocm7.12.0a) bundled their own ROCm libraries (175MB `libtorch_hip.so`) and worked standalone. Current nightlies for the same version tags now require `rocm-sdk` as a separate package, which in turn requires system ROCm libraries newer than the Arch-packaged 7.2.4. The old standalone torch wheel is no longer available.

### 3. No system ROCm >7.2.4 for Arch

AMD's `amdgpu-install` only supports Ubuntu/RHEL/SUSE. Arch/CachyOS maintain their own ROCm packages, currently at 7.2.4. Upgrading to 7.12+ would require building from source or waiting for distro package updates.

### 4. ComfyUI v0.24.0 Chroma nodes missing from v0.13.0

The Chroma workflow (SamplerCustomAdvanced, BasicScheduler, EmptySD3LatentImage, ModelSamplingAuraFlow, T5TokenizerOptions) is only available in ComfyUI v0.24.0+, which has the gfx1151 deadlock. The deployed v0.13.0 has these nodes in `comfy_extras/` (they were present in the Feb 2026 codebase) but the Chroma workflow submitted during testing appeared to stall — possibly due to the aimdo/package updates applied to the venv.

---

## Gotchas

- **Never run `pip install torch` or bare `pip install -r requirements.txt`** — pip pulls CUDA torch from PyPI, overwriting the ROCm setup. Always filter out torch/torchvision/torchaudio from requirements, or use `--no-deps` with `--index-url https://download.pytorch.org/whl/rocm7.2`.
- **Aotriton symlinks at `/usr/lib/python3.14/site-packages/torch/lib/`** — a pacman upgrade of `python-pytorch-rocm` wipes this directory. Re-run the `ln -s` after pytorch upgrades.
- **`include-system-site-packages = true`** in venv `pyvenv.cfg` is required — the ROCm torch lives at `/usr/lib/python3.14/site-packages/torch`.
- **Two ComfyUI directories exist:**
  - `~/git/ComfyUI` — v0.24.0 (current master, has deadlock)
  - `~/git/ComfyUI-bak` — v0.13.0 (deployed, stable)
- The old ComfyUI-bak venv was destroyed and recreated using system Python 3.14 + ROCm torch from pacman. If weird import errors appear, check that all pip deps from `requirements.txt` are installed.