From 32859a5df0c29e2f236dc9494bcb53ed865e678d Mon Sep 17 00:00:00 2001 From: schmeeve Date: Thu, 30 Jul 2026 21:04:37 -0700 Subject: [PATCH] re-separate rename/organize, canonical --resort for AI folders rename-ai-snaps is rename-only again (+--include-renamed, +--only-generic); extract-ai-meta is the shared recursive metadata engine emitting canonical model/lora dest paths; organize-images does all sorting, gains --resort for idempotent tree re-sorts (used to fix ~10K old-scheme folders on the NAS). --- AGENTS.md | 21 +++ FIX-AI-FOLDERS.md | 110 ++++++++++++++++ extract-ai-meta | 150 +++++++++++++++++----- organize-images | 308 +++++++++++++++++++++++++++++++++++--------- rename-ai-snaps | 309 +++++++++++++-------------------------------- sync-comfyui-snaps | 24 +++- 6 files changed, 611 insertions(+), 311 deletions(-) create mode 100644 FIX-AI-FOLDERS.md diff --git a/AGENTS.md b/AGENTS.md index e33c467..c1a4e5a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -50,8 +50,29 @@ sleep ──→ idlecheck_caffeinatestuck pull-all ──→ iterates repos in ~/git/ matching *schmeeve* remote checkin-all ──→ iterates repos in ~/git/ matching *schmeeve* remote + +comfyui-sync (systemd timer on nimo.loc, ~/git checkout) + ├── neatcli organize --by-type (drops → Pictures/Images/) + ├── rename-ai-snaps . -n (rename in place, top-level only) + └── organize-images -e (sort into AI/{model}/{lora}/) ``` +### Image pipeline (ComfyUI) + +Strict separation of concerns — do NOT re-merge them: + +| Script | Role | +|---|---| +| `rename-ai-snaps` | Rename ONLY, in place, to `schmeeve-AI-{keywords}.png`. Skips `schmeeve-AI-*` unless `--include-renamed`; `--only-generic` limits to `ComfyUI_*` names. PNG only. | +| `organize-images` | ALL sorting/moving. Default: top-level → `AI/ Photos/ Screenshots/ Unsorted/`. `--resort `: recursive canonical re-sort into `AI/{model}/{lora}/` (never renames files). | +| `extract-ai-meta` | Shared metadata engine. JSON manifest `relpath → {model, loras, dest}`; `dest` is the canonical sanitized folder (`+`-joined LoRAs, capped 200 chars). Presence in manifest = is an AI image. | + +Canonical folder sanitizer lives in `extract-ai-meta` (`sanitize_dir_name`): +path prefixes stripped (`\`→`/`, basename), spaces/specials → `_`. Anything else +creating `AI/` subfolders must match it exactly. + +`sync-comfyui-snaps` is the dormant cron predecessor of `comfyui-sync`. + Sub-scripts are invoked via `/bin/bash` or `/bin/sh`, never sourced. ### Trigger chain diff --git a/FIX-AI-FOLDERS.md b/FIX-AI-FOLDERS.md new file mode 100644 index 0000000..83296d7 --- /dev/null +++ b/FIX-AI-FOLDERS.md @@ -0,0 +1,110 @@ +# FIX-AI-FOLDERS — Re-sort plan + +## Problem + +~17K files under `/Volumes/miniShare1/Pictures/Images/AI/` were sorted by an older +scheme. Two scripts currently create `AI/{model}/{lora}/` folders with **different +sanitizers**, fragmenting the tree: + +| | `organize-images` (`clean()`) | `rename-ai-snaps` (`sanitize_dir_name`) | +|---|---|---| +| `Pony XL\trserkanWildpony_v20.safetensors` | `Pony XL\trserkanWildpony_v20/` or `Pony XL-trserkanWildpony_v20/` (its `\\`→`-` fix only catches *double* backslashes) | `trserkanWildpony_v20/` (strips path prefix, `_`-sanitizes) | +| spaces | kept (`FLUX Handsome Male Model_1.0/`) | → underscores | + +Legacy artifacts also exist: `None/` and `unknown/` folders (at Images root and inside +`AI/`), plus loose files at the `AI/` root. + +Additional blockers: + +- `rename-ai-snaps` **skips all `schmeeve-AI-*` files** — 17,178 of 17,255 files in + `AI/`. As-is it would touch almost nothing. +- `rename-ai-snaps` is PNG-only (fine: `AI/` is 99.97% PNG; 5 jpgs reported, not moved). +- All 30 sampled `AI/` files still carry full ComfyUI `prompt` metadata — a + metadata-driven re-sort is viable. +- `Unsorted/` sweep: 0 of 120 sampled files (jpg/png/webp) have ComfyUI metadata — + they are web downloads, not ComfyUI outputs. Only files with real `prompt` + metadata qualify for moving. +- `nas_mount_base()` does not know `/Volumes/miniShare1`. +- `~/.local/bin/organize-images` is stale (older than repo copy). + +## Requirements (user decisions) + +1. **Scope:** Everything — `AI/`, `None/`, `unknown/`, plus sweep `Unsorted/` and + `Screenshots/` for PNGs with real ComfyUI metadata. +2. **Filenames:** Keep existing names, **except** generic names (`ComfyUI_*` etc.) + which get a proper keyword-based rename. +3. **Architecture:** Re-separate concerns — `rename-ai-snaps` renames only (the + folder-moving added in `e85fce1 "model first"` was a mistake), `organize-images` + does all sorting. Fix its sanitizer; sync the stale `~/.local/bin` copy. + +## Plan + +### 1. `rename-ai-snaps` → rename-only + +- Strip the move/folder logic (`output_root`, `mkdir`, `shutil.move`, model/lora + folder computation). Keep metadata extraction, keyword scoring, and + interactive/batch renaming **in place** (filename changes only, never directory + changes). +- Default: skip `schmeeve-AI-*` as today. New flag `--include-renamed` for + corpus-wide renames later. + +### 2. `extract-ai-meta` → shared, recursive metadata engine + +- Add `-r` recursive walk; relative-path keys instead of bare basenames. +- Port the robust extraction from `rename-ai-snaps` (`_strip_model_name`, + `UNETLoader`/extra checkpoint classes, generic `*lora_name*` fields). It currently + only knows `CheckpointLoaderSimple` and keeps backslash path prefixes — the cause + of `Pony XL\…` folders. +- Keep its connected-LoRA check (excludes disconnected LoRA nodes). +- Add a `prompt_present` flag per file so `organize-images` can classify AI files + **without** `mdls`, dimension heuristics, or filename guesses. + +### 3. `organize-images` → all sorting, canonical scheme + +- **Canonical sanitizer** (single implementation; fixes the `\\` bug and space + handling): strip path prefix → spaces → `_` → non `[a-zA-Z0-9_.-]` → `_` → + collapse. `Pony XL\trserkanWildpony_v20.safetensors` → `trserkanWildpony_v20` + everywhere. +- **New `--resort` mode:** recursively walk `AI/` (including `schmeeve-AI-*` files), + compute canonical `AI/{model}/{lora}/` per file, move only what's misplaced. + **Filenames never touched.** Files already correct are skipped (idempotent). + Missing model → `unknown/`; no loras → `no-lora/`. +- **Sort `None/` and `unknown/`:** pass as targets; files with metadata route into + `AI/{model}/{lora}/`, files without go to `AI/unknown/`. +- **Sweep `Unsorted/` + `Screenshots/`:** only files with actual ComfyUI `prompt` + metadata move into `AI/{model}/{lora}/`. +- **Generic-name hook:** before moving files matching `ComfyUI_*`/hash-style names, + invoke `rename-ai-snaps -n` on them (its skip-rule already protects + `schmeeve-AI-*`). +- **Collision policy:** destination exists → append `_1`, `_2`… (never overwrite, + never delete). +- **Performance:** one metadata scan → JSON manifest in a temp file; move logic + reads the manifest (no per-file `mdls`/`sips` subprocess storms). +- Keep dry-run default, `-e` to execute, `--limit` for batching. + +### 4. Verification + +- Full dry-run first: summary counts per destination folder + sample moves. +- Spot-check that `Pony XL\…`, `Pony XL-…`, `None/`, loose-root files land + correctly; confirm already-correct files are untouched; re-run dry-run + post-execute to confirm ~0 remaining moves (idempotency proof). + +### 5. Deploy + docs + +- Copy updated scripts to `~/.local/bin/` (fixes stale `organize-images`) and + `~/Dropbox/bin/`. +- Add a short "Image pipeline" section to `AGENTS.md` documenting the restored + separation: `rename-ai-snaps` = rename, `organize-images` = sort, + `extract-ai-meta` = shared engine. + +## Untouched + +- `Unsorted/` web downloads (no metadata = no move), `Screenshots/` non-AI files, + non-PNGs in `AI/` (5 jpgs — reported, not moved). +- Nothing is ever deleted; worst case is a `_1` suffix. + +## Note + +With `schmeeve-AI-*` names kept as-is, a handful may not match what today's keyword +scorer would produce — but placement will be correct, and any folder can be +re-renamed later with `rename-ai-snaps --include-renamed`. diff --git a/extract-ai-meta b/extract-ai-meta index 0d1a374..cef2e9e 100755 --- a/extract-ai-meta +++ b/extract-ai-meta @@ -1,13 +1,22 @@ #!/usr/bin/env python3 """Extract model and LoRA metadata from ComfyUI-generated images. -Outputs a JSON mapping of filename → {"model": "...", "loras": ["...", ...]}. +Outputs a JSON mapping of path (relative to target) → + {"model": "...", "loras": ["...", ...], "dest": "{model_dir}/{lora_dir}"} + +Only files with parseable ComfyUI 'prompt' metadata are included, so +presence in the output doubles as "is an AI-generated image" for +organize-images. 'dest' is the canonical folder path (sans leading "AI/"): +model/LoRA names are path-stripped and sanitized (spaces/specials → _), +LoRAs joined with '+', missing model → "unknown", no LoRAs → "no-lora". + +Usage: extract-ai-meta [path] [-r|--recursive] """ import json import os +import re import sys -from pathlib import Path try: from PIL import Image @@ -17,6 +26,43 @@ except ImportError: IMAGE_EXTS = {".png", ".jpg", ".jpeg", ".webp"} +CHECKPOINT_CLASS_TYPES = { + "CheckpointLoaderSimple", + "Checkpoint Loader with Name (Image Saver)", + "CheckpointLoader", +} + +MODEL_CLASS_TYPES = { + "UNETLoader", +} + +LORA_CLASS_TYPES = { + "LoraLoader", + "Power Lora Loader (rgthree)", +} + +# Filesystem-safe cap for the joined LoRA combo directory name +MAX_LORA_DIR_LEN = 200 + + +def _strip_model_name(raw): + """Strip path prefix and .safetensors extension from a model/LoRA name.""" + name = raw.strip() + # Handle backslash/forward-slash paths: "Pony\model.safetensors" or "Pony/model.safetensors" + name = name.replace("\\", "/") + name = os.path.basename(name) + if name.endswith(".safetensors"): + name = name[:-len(".safetensors")] + return name + + +def sanitize_dir_name(name): + """Canonical directory-name sanitizer (spaces/specials → _).""" + name = name.strip().replace(" ", "_") + name = re.sub(r"[^a-zA-Z0-9_.-]", "_", name) + name = re.sub(r"_+", "_", name).strip("_") + return name + def is_lora_connected(node_id, data): """Check if a LoRA node's output is consumed by any downstream node.""" @@ -30,19 +76,23 @@ def is_lora_connected(node_id, data): def extract_ai_meta(filepath): - """Return (model, loras) or (None, []) if no ComfyUI metadata found.""" + """Return (model, loras, has_prompt). + + has_prompt is True whenever the file carries parseable ComfyUI 'prompt' + metadata, even if no model/LoRA nodes were found in it. + """ try: img = Image.open(filepath) except Exception: - return None, [] + return None, [], False if "prompt" not in img.info: - return None, [] + return None, [], False try: data = json.loads(img.info["prompt"]) except (json.JSONDecodeError, TypeError): - return None, [] + return None, [], False model = None loras = [] @@ -51,24 +101,59 @@ def extract_ai_meta(filepath): cls = node.get("class_type", "") inputs = node.get("inputs", {}) - if cls == "CheckpointLoaderSimple": - ckpt = inputs.get("ckpt_name", "") - if ckpt: - model = ckpt.rsplit(".", 1)[0] + if model is None: + if cls in CHECKPOINT_CLASS_TYPES and inputs.get("ckpt_name"): + model = _strip_model_name(inputs["ckpt_name"]) + elif cls in MODEL_CLASS_TYPES and inputs.get("unet_name"): + model = _strip_model_name(inputs["unet_name"]) - if "lora" in cls.lower(): - lora = inputs.get("lora_name", "") - if lora: - # Only include LoRAs whose output is actually consumed - if is_lora_connected(nid, data): - loras.append(lora.rsplit(".", 1)[0]) + if cls in LORA_CLASS_TYPES and inputs.get("lora_name"): + # Only include LoRAs whose output is actually consumed + if is_lora_connected(nid, data): + loras.append(_strip_model_name(inputs["lora_name"])) + else: + # Some custom nodes use fields ending in 'lora'/'lora_name' + for field, val in inputs.items(): + if isinstance(val, str) and val.endswith(".safetensors") and "lora" in field.lower(): + name = _strip_model_name(val) + if name: + loras.append(name) - loras.sort() - return model, loras + loras = sorted(set(loras)) + return model, loras, True + + +def dest_for(model, loras): + """Canonical relative destination dir (sans 'AI/' prefix).""" + model_dir = sanitize_dir_name(model) if model else "unknown" + if loras: + lora_dir = "+".join(sanitize_dir_name(l) for l in loras if l) + if len(lora_dir) > MAX_LORA_DIR_LEN: + lora_dir = lora_dir[:MAX_LORA_DIR_LEN].rstrip("+_") + if not lora_dir: + lora_dir = "no-lora" + else: + lora_dir = "no-lora" + return f"{model_dir}/{lora_dir}" + + +def iter_images(target, recursive): + if recursive: + for dirpath, _dirnames, filenames in os.walk(target): + for fn in sorted(filenames): + if os.path.splitext(fn)[1].lower() in IMAGE_EXTS: + yield os.path.join(dirpath, fn) + else: + for entry in sorted(os.listdir(target)): + fpath = os.path.join(target, entry) + if os.path.isfile(fpath) and os.path.splitext(entry)[1].lower() in IMAGE_EXTS: + yield fpath def main(): - target = sys.argv[1] if len(sys.argv) > 1 else os.path.expanduser("~/Pictures") + args = [a for a in sys.argv[1:] if not a.startswith("-")] + recursive = any(a in ("-r", "--recursive") for a in sys.argv[1:]) + target = args[0] if args else os.path.expanduser("~/Pictures") target = os.path.abspath(target) if not os.path.isdir(target): @@ -76,18 +161,25 @@ def main(): sys.exit(1) meta = {} - - for entry in sorted(os.listdir(target)): - fpath = os.path.join(target, entry) - if not os.path.isfile(fpath): - continue - ext = os.path.splitext(entry)[1].lower() - if ext not in IMAGE_EXTS: + scanned = 0 + for fpath in iter_images(target, recursive): + scanned += 1 + if scanned % 200 == 0: + print(f"\r …scanned {scanned} images ({len(meta)} with AI metadata)", + end="", file=sys.stderr, flush=True) + model, loras, has_prompt = extract_ai_meta(fpath) + if not has_prompt: continue + rel = os.path.relpath(fpath, target) + meta[rel] = { + "model": model, + "loras": loras, + "dest": dest_for(model, loras), + } - model, loras = extract_ai_meta(fpath) - if model or loras: - meta[entry] = {"model": model, "loras": loras} + if scanned: + print(f"\r Scanned {scanned} images, {len(meta)} with AI metadata. ", + file=sys.stderr, flush=True) print(json.dumps(meta, indent=2)) diff --git a/organize-images b/organize-images index 5382bd8..f38deeb 100755 --- a/organize-images +++ b/organize-images @@ -1,23 +1,34 @@ #!/bin/bash # organize-images -- Sort images into AI/, Photos/, Screenshots/, Unsorted/ -# Default: sub-categorizes AI by model + LoRA combo (uses extract-ai-meta). -# Use --flat for one flat AI/ folder. +# AI files route into AI/{model}/{lora}/ via extract-ai-meta (canonical scheme). +# +# Modes: +# (default) Sort top-level files of [path] into the 4 buckets. +# --resort Recursively re-sort [path] (the Images root containing AI/), +# moving any file with ComfyUI metadata to its canonical +# AI/{model}/{lora}/ location. Filenames are never changed. +# +# Generic ComfyUI_* filenames are renamed first via rename-ai-snaps +# (--only-generic) before sorting, when that script is available. TARGET="" DRY_RUN=true VERBOSE=false INSTALL=false FLAT=false +RESORT=false LIMIT=0 usage() { - echo "Usage: $(basename "$0") [path] [-e|--execute] [-v|--verbose] [-i|--install] [--flat]" + echo "Usage: $(basename "$0") [path] [-e|--execute] [-v|--verbose] [-i|--install] [--flat] [--resort] [--limit N]" echo " path Directory to organize (default: ~/Pictures)" echo " -e Execute moves (default: dry-run only)" - echo " -v Verbose: show per-file classification" + echo " -v Verbose: show per-file classification/moves" echo " -i Install this script to ~/.local/bin/" echo " --flat Don't sub-categorize by model/LoRA (flat AI/ folder)" - echo " --limit N Only process N files (for batching)" + echo " --resort Recursively re-sort [path] into canonical AI/{model}/{lora}/" + echo " ([path] must be the Images root containing AI/)" + echo " --limit N Only process N files (for batching; default mode only)" exit 0 } @@ -34,6 +45,7 @@ for arg in "$@"; do -v|--verbose) VERBOSE=true ;; -i|--install) INSTALL=true ;; --flat) FLAT=true ;; + --resort) RESORT=true ;; --limit) LIMIT_NEXT=true ;; --limit=*) LIMIT="${arg#*=}" ;; -h|--help) usage ;; @@ -63,34 +75,223 @@ if [ ! -d "$TARGET" ]; then exit 1 fi -echo "organize-images — Classifying images in: $TARGET" -[ "$DRY_RUN" = true ] && echo " DRY RUN (use -e to execute)" || echo " EXECUTING" -[ "$LIMIT" -gt 0 ] && echo " Limit: first $LIMIT files" +# --- Resolve helper scripts (same dir as this script, else PATH) --- +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" +find_tool() { + local n="$1" + if [ -x "$SCRIPT_DIR/$n" ]; then + echo "$SCRIPT_DIR/$n" + elif command -v "$n" >/dev/null 2>&1; then + command -v "$n" + else + return 1 + fi +} -# --- Pre-scan: extract AI metadata for model/LoRA routing --- +META_SCRIPT="$(find_tool extract-ai-meta)" || { + echo "Error: extract-ai-meta not found (looked next to $0 and in PATH)" >&2 + exit 1 +} +RENAME_SCRIPT="$(find_tool rename-ai-snaps || true)" + +echo "organize-images — $([ "$RESORT" = true ] && echo "Re-sorting" || echo "Classifying") images in: $TARGET" +[ "$DRY_RUN" = true ] && echo " DRY RUN (use -e to execute)" || echo " EXECUTING" +[ "$LIMIT" -gt 0 ] && [ "$RESORT" = false ] && echo " Limit: first $LIMIT files" + +# --- Resort mode: recursive canonical re-sort of an existing tree --- +if [ "$RESORT" = true ]; then + if [ ! -d "$TARGET/AI" ]; then + echo "Error: --resort expects the Images root (a directory containing AI/)" >&2 + exit 1 + fi + + # Generic-name hook: rename ComfyUI_* files in place before sorting + GENERIC_COUNT=$(find "$TARGET" -type f -name 'ComfyUI*.png' 2>/dev/null | wc -l | tr -d ' ') + if [ "$GENERIC_COUNT" -gt 0 ]; then + if [ -n "$RENAME_SCRIPT" ]; then + if [ "$DRY_RUN" = true ]; then + echo " $GENERIC_COUNT generic ComfyUI_*.png file(s) would be renamed first (rename-ai-snaps --only-generic)" + else + echo " Renaming $GENERIC_COUNT generic ComfyUI_*.png file(s) first..." + "$RENAME_SCRIPT" "$TARGET" -r -n --only-generic + fi + else + echo " Note: $GENERIC_COUNT generic ComfyUI_*.png file(s) found; rename-ai-snaps not available, keeping names" + fi + fi + + # Metadata manifest (recursive) + echo " Scanning AI metadata (recursive)..." + AI_META=$(mktemp) + if ! "$META_SCRIPT" "$TARGET" -r > "$AI_META"; then + echo "Error: AI metadata scan failed" >&2 + rm -f "$AI_META" + exit 1 + fi + + python3 - "$TARGET" "$AI_META" "$DRY_RUN" "$VERBOSE" <<'PYEOF' +import json, os, shutil, sys + +root = os.path.abspath(sys.argv[1]) +meta = json.load(open(sys.argv[2])) +dry = sys.argv[3] == "true" +verbose = sys.argv[4] == "true" + +IMAGE_EXTS = {".png", ".jpg", ".jpeg", ".webp"} +LEGACY_DIRS = {"None", "unknown"} + +moved = ok = nometa = collisions = errors = 0 +samples = [] +legacy_no_meta = [] +planned = set() + +for dirpath, dirnames, filenames in os.walk(root): + dirnames.sort() + for fn in sorted(filenames): + if os.path.splitext(fn)[1].lower() not in IMAGE_EXTS: + continue + src = os.path.join(dirpath, fn) + rel = os.path.relpath(src, root) + entry = meta.get(rel) + if not entry: + nometa += 1 + top = rel.split(os.sep)[0] + if top in LEGACY_DIRS: + legacy_no_meta.append(rel) + continue + + dest_dir = os.path.join(root, "AI", entry["dest"]) + if os.path.abspath(dest_dir) == os.path.abspath(dirpath): + ok += 1 + continue + + dest = os.path.join(dest_dir, fn) + taken = planned if dry else None + if os.path.exists(dest) or (taken is not None and dest in taken): + stem, ext = os.path.splitext(fn) + c = 1 + while os.path.exists(os.path.join(dest_dir, f"{stem}_{c}{ext}")) or \ + (taken is not None and os.path.join(dest_dir, f"{stem}_{c}{ext}") in taken): + c += 1 + dest = os.path.join(dest_dir, f"{stem}_{c}{ext}") + collisions += 1 + if taken is not None: + taken.add(dest) + + dest_rel = os.path.relpath(dest, root) + if dry: + moved += 1 + if len(samples) < 20: + samples.append((rel, dest_rel)) + if verbose: + print(f" WOULD MOVE {rel} → {dest_rel}") + else: + try: + os.makedirs(dest_dir, exist_ok=True) + shutil.move(src, dest) + moved += 1 + if verbose: + print(f" MOVED {rel} → {dest_rel}") + elif moved % 200 == 0: + print(f"\r …moved {moved}", end="", flush=True) + except OSError as e: + errors += 1 + print(f" SKIPPED ({e}): {rel}") + +if not verbose and not dry and moved: + print(f"\r Moved {moved} file(s). ") + +# Prune dirs left empty by the moves (keep top-level buckets) +pruned = 0 +if not dry and moved: + keep = {"AI", "Photos", "Screenshots", "Unsorted"} + for dirpath, dirnames, filenames in os.walk(root, topdown=False): + if dirpath == root: + continue + if os.path.relpath(dirpath, root).split(os.sep)[0] in keep and \ + os.path.dirname(dirpath) == root: + continue + try: + os.rmdir(dirpath) # only succeeds when empty + pruned += 1 + except OSError: + pass + +print() +print(f" {'DRY RUN — would move' if dry else 'Moved'}: {moved}") +print(f" Already correctly placed: {ok}") +print(f" Collisions (renamed with _N suffix): {collisions}") +if errors: + print(f" Errors: {errors}") +print(f" No AI metadata (left in place): {nometa}") +if pruned: + print(f" Empty folders pruned: {pruned}") + +if samples: + print() + print(" Sample moves:") + for s, d in samples: + print(f" {s}") + print(f" → {d}") + +if legacy_no_meta: + print() + print(f" Note: {len(legacy_no_meta)} file(s) in None//unknown/ have no AI metadata") + print(" and were left in place for manual triage (not all are AI images).") + for rel in legacy_no_meta[:10]: + print(f" {rel}") + if len(legacy_no_meta) > 10: + print(f" … and {len(legacy_no_meta) - 10} more") +print() +PYEOF + STATUS=$? + rm -f "$AI_META" + exit $STATUS +fi + +# --- Default mode: classify top-level files into AI/Photos/Screenshots/Unsorted --- + +# Generic-name hook (non-recursive) +GENERIC_COUNT=$(find "$TARGET" -maxdepth 1 -type f -name 'ComfyUI*.png' 2>/dev/null | wc -l | tr -d ' ') +if [ "$GENERIC_COUNT" -gt 0 ] && [ -n "$RENAME_SCRIPT" ]; then + if [ "$DRY_RUN" = true ]; then + echo " $GENERIC_COUNT generic ComfyUI_*.png file(s) would be renamed first (rename-ai-snaps --only-generic)" + else + echo " Renaming $GENERIC_COUNT generic ComfyUI_*.png file(s) first..." + "$RENAME_SCRIPT" "$TARGET" -n --only-generic + fi +fi + +# Pre-scan: extract AI metadata for model/LoRA routing AI_META="" if [ "$FLAT" = false ]; then - META_SCRIPT="$(cd "$(dirname "$0")" && pwd -P)/extract-ai-meta" - if [ -x "$META_SCRIPT" ]; then - echo " Scanning AI metadata for model/LoRA routing..." - AI_META=$(mktemp) - if ! "$META_SCRIPT" "$TARGET" > "$AI_META" 2>/dev/null; then - echo " Warning: AI metadata scan failed, falling back to flat AI/" - FLAT=true - rm -f "$AI_META" - elif [ ! -s "$AI_META" ]; then - FLAT=true - rm -f "$AI_META" - fi - else - echo " Warning: extract-ai-meta not found, falling back to flat AI/" + AI_META=$(mktemp) + if ! "$META_SCRIPT" "$TARGET" > "$AI_META" 2>/dev/null; then + echo " Warning: AI metadata scan failed, falling back to flat AI/" FLAT=true + rm -f "$AI_META" + AI_META="" + elif [ ! -s "$AI_META" ]; then + FLAT=true + rm -f "$AI_META" + AI_META="" fi fi [ "$FLAT" = false ] && echo " AI sub-categorization: by model/LoRA (use --flat to disable)" echo +# Basenames of files with AI metadata (manifest presence = is an AI image) +AI_KEYS="" +if [ -n "$AI_META" ]; then + AI_KEYS=$(mktemp) + python3 -c " +import json, sys +for k in json.load(open('$AI_META')): + print(k) +" > "$AI_KEYS" 2>/dev/null +fi + classify() { local f="$1" @@ -110,6 +311,11 @@ classify() { ComfyUI_*|*_DEFI_*|R.*.png|schmeeve-AI-*) echo "AI"; return ;; esac + # Files with ComfyUI metadata are AI regardless of name + if [ -n "$AI_KEYS" ] && grep -Fxq "$name" "$AI_KEYS" 2>/dev/null; then + echo "AI"; return + fi + if mdls -name kMDItemAcquisitionMake "$f" 2>/dev/null | grep -q '= "'; then echo "Photos"; return fi @@ -121,24 +327,10 @@ classify() { echo "Photos"; return ;; esac - if [ "$ext" = "png" ]; then - local dims w h area ratio - dims=$(sips --getProperty pixelWidth --getProperty pixelHeight "$f" 2>/dev/null | grep -E 'pixel(Width|Height)' | awk '{print $NF}') - w=$(echo "$dims" | head -1) - h=$(echo "$dims" | tail -1) - if [ -n "$w" ] && [ -n "$h" ] && [ "$w" -gt 0 ] && [ "$h" -gt 0 ]; then - area=$((w * h)) - ratio=$(echo "scale=4; $w / $h" | bc 2>/dev/null || echo 1) - if [ "$area" -gt 1048576 ] && [ "$(echo "$ratio > 0.75" | bc -l)" = 1 ] && [ "$(echo "$ratio < 1.33" | bc -l)" = 1 ]; then - echo "AI"; return - fi - fi - fi - echo "Unsorted" } -# --- Pre-scan: collect all image files --- +# --- Collect top-level image files --- FILES=() for f in "$TARGET"/*; do [ -f "$f" ] || continue @@ -155,6 +347,8 @@ if [ "$LIMIT" -gt 0 ] && [ "$LIMIT" -lt "$TOTAL" ]; then fi if [ "$TOTAL" -eq 0 ]; then echo " No image files found." + [ -n "$AI_META" ] && rm -f "$AI_META" + [ -n "$AI_KEYS" ] && rm -f "$AI_KEYS" exit 0 fi @@ -174,27 +368,16 @@ progress_bar() { printf "] %3d%% (%d/%d)" "$pct" "$current" "$total" } +# Destination for an AI file: canonical AI/{model}/{lora}/ from the manifest ai_dest() { - python3 -c " -import json, sys -m = json.load(open('$AI_META')) + AI_META_PATH="$AI_META" python3 -c " +import json, os, sys +m = json.load(open(os.environ['AI_META_PATH'])) entry = m.get(sys.argv[1]) -if not entry or not isinstance(entry, dict): +if not entry or not isinstance(entry, dict) or not entry.get('dest'): print('AI/unknown') - sys.exit(0) -def clean(s): - if not s: - return 'unknown' - return s.replace('/', '-').replace('\\\\', '-') - -model = clean(entry.get('model')) -loras = entry.get('loras') or [] -if loras: - combo = '+'.join(clean(l.strip()) for l in loras if l) - combo = (combo[:200]).rstrip('+') - print(f'AI/{model}/{combo}') else: - print(f'AI/{model}/no-lora') + print('AI/' + entry['dest']) " "$1" } @@ -224,7 +407,15 @@ for f in "${FILES[@]}"; do if [ "$DRY_RUN" = false ]; then mkdir -p "$dest" - mv -f "$f" "$dest" + base=$(basename "$f") + final="$dest/$base" + if [ -e "$final" ]; then + stem="${base%.*}"; fext="${base##*.}" + n=1 + while [ -e "$dest/${stem}_${n}.${fext}" ]; do n=$((n + 1)); done + final="$dest/${stem}_${n}.${fext}" + fi + mv "$f" "$final" fi case "$cat" in @@ -235,13 +426,16 @@ for f in "${FILES[@]}"; do esac done +[ -n "$AI_META" ] && rm -f "$AI_META" +[ -n "$AI_KEYS" ] && rm -f "$AI_KEYS" + if [ "$VERBOSE" = false ]; then printf "\r %-60s\n" "Done." fi echo echo " Results:" -if [ "$FLAT" = false ] && [ -n "$AI_META" ]; then +if [ "$FLAT" = false ] && [ "$cAI" -gt 0 ]; then echo " AI (by model/LoRA): $cAI" else echo " AI: $cAI" diff --git a/rename-ai-snaps b/rename-ai-snaps index 02c743d..87e80de 100755 --- a/rename-ai-snaps +++ b/rename-ai-snaps @@ -1,19 +1,20 @@ #!/usr/bin/env python3 """ -rename-ai-snaps — Scan PNGs for AI prompt metadata and reorg into model/lora folders. +rename-ai-snaps — Scan PNGs for AI prompt metadata and rename descriptively. Usage: ./rename-ai-snaps [path] [options] -Scans for ComfyUI PNGs, reads embedded prompt metadata, extracts the -checkpoint model and LoRA names, and moves files into: - {output}/{model}/{lora}/schmeeve-AI-{keywords}.png +Scans for ComfyUI PNGs, reads embedded prompt metadata, extracts keywords, +and renames files IN PLACE to: + schmeeve-AI-{keywords}.png + +Renaming only — use organize-images to sort files into model/lora folders. """ import json import os import re -import shutil import sys import time from pathlib import Path @@ -85,6 +86,9 @@ NEGATIVE_INDICATORS = { "body out of frame", "bad hands", "bad face", "blurry", } +# Generic ComfyUI output filenames (ComfyUI_00042_.png, ComfyUI_temp_….png) +GENERIC_RE = re.compile(r"^ComfyUI", re.IGNORECASE) + # ── Helpers ──────────────────────────────────────────────────────────────── def spinner(): @@ -115,82 +119,24 @@ def is_quality_only(text): # ── Metadata extraction from ComfyUI prompt JSON ─────────────────────────── -CHECKPOINT_CLASS_TYPES = { - "CheckpointLoaderSimple", - "Checkpoint Loader with Name (Image Saver)", - "CheckpointLoader", -} - -LORA_CLASS_TYPES = { - "LoraLoader", - "Power Lora Loader (rgthree)", -} - -MODEL_CLASS_TYPES = { - "UNETLoader", -} - - -def _strip_model_name(raw): - """Strip path prefix and .safetensors extension from a model/LoRA name.""" - name = raw.strip() - # Handle backslash/forward-slash paths: "Pony\\model.safetensors" or "Pony/model.safetensors" - name = name.replace("\\", "/") - name = os.path.basename(name) - # Strip .safetensors extension - if name.endswith(".safetensors"): - name = name[:-len(".safetensors")] - return name - - -def extract_model_name(data): - """Return the checkpoint/UNET model name from the prompt JSON, or None.""" - model_name = None - for node in data.values(): - inputs = node.get("inputs", {}) - class_type = node.get("class_type", "") - if class_type in CHECKPOINT_CLASS_TYPES and "ckpt_name" in inputs: - model_name = _strip_model_name(inputs["ckpt_name"]) - break - if class_type in MODEL_CLASS_TYPES and "unet_name" in inputs: - model_name = _strip_model_name(inputs["unet_name"]) - break - return model_name if model_name else None - - -def extract_lora_names(data): - """Return a sorted list of unique LoRA names from the prompt JSON.""" - names = [] - for node in data.values(): - inputs = node.get("inputs", {}) - class_type = node.get("class_type", "") - if class_type in LORA_CLASS_TYPES and "lora_name" in inputs: - names.append(_strip_model_name(inputs["lora_name"])) - # Some custom nodes use 'lora' or similar — look for any field ending - # in 'lora_name' or 'lora' that contains '.safetensors' - for field, val in inputs.items(): - if isinstance(val, str) and val.endswith(".safetensors") and "lora" in field.lower(): - name = _strip_model_name(val) - if name and name not in names: - names.append(name) - return sorted(set(names)) - - -def extract_prompts(filepath): - """Return (positive_prompt, negative_prompt) from a ComfyUI PNG.""" +def load_prompt_data(filepath): + """Return parsed ComfyUI prompt JSON from a PNG, or None.""" try: img = Image.open(filepath) except Exception: - return None, None + return None if "prompt" not in img.info: - return None, None + return None try: - data = json.loads(img.info["prompt"]) + return json.loads(img.info["prompt"]) except (json.JSONDecodeError, TypeError): - return None, None + return None + +def extract_prompts(data): + """Return (positive_prompt, negative_prompt) from parsed prompt JSON.""" # Gather all text fields from all nodes candidates = [] for node in data.values(): @@ -325,69 +271,22 @@ def keywords_to_name(keywords): return f"schmeeve-AI-{desc}.png" if desc and len(desc) > 3 else None -def extract_metadata(filepath): - """Return (model_name, lora_names, positive_prompt, negative_prompt).""" - try: - img = Image.open(filepath) - except Exception: - return None, None, None, None - - if "prompt" not in img.info: - return None, None, None, None - - try: - data = json.loads(img.info["prompt"]) - except (json.JSONDecodeError, TypeError): - return None, None, None, None - - model = extract_model_name(data) - loras = extract_lora_names(data) - pos, neg = extract_prompts(filepath) - return model, loras, pos, neg - - -def sanitize_dir_name(name): - """Sanitize a string for use as a directory name.""" - name = name.strip().replace(" ", "_") - name = re.sub(r"[^a-zA-Z0-9_.-]", "_", name) - name = re.sub(r"_+", "_", name).strip("_") - return name - - # ── main ─────────────────────────────────────────────────────────────────── -def nas_mount_base(): - """Return the NAS mount base path for this machine.""" - candidates = [ - "/mini.nas/miniShare1", - os.path.expanduser("~/mnt/mini.nas/miniShare1"), - os.path.expanduser("~/mnt/miniShare1"), - ] - for p in candidates: - if os.path.isdir(p): - return p - return candidates[-1] - - def main(): import argparse - DEFAULT_OUTPUT = os.path.join(nas_mount_base(), "Pictures", "Images", "AI") - parser = argparse.ArgumentParser( - description="Scan PNGs for AI prompt metadata and reorg into model/lora folders.", + description="Scan PNGs for AI prompt metadata and rename in place. " + "Does not move files — use organize-images for sorting.", ) parser.add_argument( "path", nargs="?", default=os.path.expanduser("~/Pictures"), help="Directory to scan for PNG files (default: ~/Pictures)", ) - parser.add_argument( - "-o", "--output", default=DEFAULT_OUTPUT, - help=f"Output root directory (default: {DEFAULT_OUTPUT})", - ) parser.add_argument( "-n", "--no-interactive", action="store_true", - help="Auto-reorg without prompting", + help="Auto-rename without prompting", ) parser.add_argument( "-r", "--recursive", action="store_true", @@ -396,7 +295,15 @@ def main(): parser.add_argument( "-p", "--preview", "--dry-run", action="store_true", dest="dry_run", - help="Show proposed changes without making any (combine with -n for full listing)", + help="Show proposed changes without making any", + ) + parser.add_argument( + "--include-renamed", action="store_true", + help="Also process files already named schmeeve-AI-* (default: skip them)", + ) + parser.add_argument( + "--only-generic", action="store_true", + help="Only rename generic ComfyUI_* filenames", ) args = parser.parse_args() @@ -405,8 +312,6 @@ def main(): print(f"Error: {scan_dir} is not a directory") sys.exit(1) - output_root = Path(args.output).expanduser().resolve() - # Gather PNGs glob_pattern = "**/*.png" if args.recursive else "*.png" pngs = sorted(scan_dir.glob(glob_pattern)) @@ -425,20 +330,20 @@ def main(): for p in pngs: sys.stdout.write(f"\r {next(spin)} Analyzing PNGs") sys.stdout.flush() - # Skip already-renamed files (schmeeve-AI-*) - if p.name.startswith("schmeeve-AI-"): + # Skip already-renamed files (schmeeve-AI-*) unless requested + if not args.include_renamed and p.name.startswith("schmeeve-AI-"): continue - model, loras, pos, neg = extract_metadata(str(p)) + if args.only_generic and not GENERIC_RE.match(p.name): + continue + data = load_prompt_data(str(p)) + if not data: + continue + pos, neg = extract_prompts(data) source = pos or neg if source: candidates = extract_all_candidates(source) if candidates: - image_data[p] = { - "candidates": candidates, - "model": model, - "loras": loras, - "source": source, - } + image_data[p] = candidates for kw, _ in candidates: if kw not in word_images: word_images[kw] = set() @@ -447,48 +352,35 @@ def main(): freq_map = {kw: len(images) for kw, images in word_images.items()} total_images = len(image_data) - # ── Phase 1b: Propose file paths ── + # ── Phase 1b: Propose new names (in place — same directory) ── raw_proposals = {} - for p, info in image_data.items(): - candidates = info["candidates"] - model = info["model"] - loras = info["loras"] - + for p, candidates in image_data.items(): if total_images > 1: keywords = select_best_keywords(candidates, freq_map, total_images) else: keywords = [kw for kw, _ in candidates[:5]] name = keywords_to_name(keywords) - if not name: + if not name or name == p.name: continue - - # Determine folder structure - model_dir = sanitize_dir_name(model) if model else "unknown" - if loras: - lora_dir = sanitize_dir_name("+".join(loras)) - else: - lora_dir = "no-lora" - - rel_path = Path(model_dir) / lora_dir / name - raw_proposals[p] = rel_path + raw_proposals[p] = name # Clear the spinner line sys.stdout.write("\r" + " " * 60 + "\r") sys.stdout.flush() - # Deduplicate file names within their target directories + # Deduplicate file names within their directories proposals = {} used_paths = {} - for p, rel_path in raw_proposals.items(): - new_path = rel_path - counter = 0 + for p, name in raw_proposals.items(): + new_path = p.with_name(name) key = str(new_path) + counter = 0 while key in used_paths: counter += 1 ts = time.strftime("%m%d-%H%M%S", time.localtime(p.stat().st_mtime)) - stem = rel_path.stem - ext = rel_path.suffix - new_path = rel_path.with_name(f"{stem}-{ts}{'_' + str(counter) if counter > 1 else ''}{ext}") + stem = new_path.stem + ext = new_path.suffix + new_path = new_path.with_name(f"{stem}-{ts}{'_' + str(counter) if counter > 1 else ''}{ext}") key = str(new_path) used_paths[key] = True proposals[p] = new_path @@ -499,9 +391,9 @@ def main(): return print(f"\n {'DRY RUN' if args.dry_run else 'PROPOSED'} — " - f"{len(proposals)} file(s) to organize into {output_root}/\n") + f"{len(proposals)} file(s) to rename in place\n") - for i, (old_path, rel_path) in enumerate(proposals.items(), 1): + for i, (old_path, new_path) in enumerate(proposals.items(), 1): try: src_display = str(old_path.relative_to(scan_dir)) except ValueError: @@ -509,7 +401,7 @@ def main(): if len(src_display) > 55: src_display = src_display[:25] + "…" + src_display[-27:] print(f" {i:>3}. {src_display}") - print(f" → {output_root / rel_path}") + print(f" → {new_path.name}") print() @@ -517,40 +409,46 @@ def main(): print(f" Dry-run complete. No files were changed.\n") return - # ── Phase 3: Rename/move (interactive or auto) ── + def do_rename(old_path, new_path): + """Rename with collision suffix. Returns final path or None on failure.""" + if old_path == new_path: + return new_path + if new_path.exists(): + stem = new_path.stem + ext = new_path.suffix + counter = 1 + while new_path.exists(): + new_path = new_path.with_name(f"{stem}_{counter}{ext}") + counter += 1 + print(f" (name existed, using {new_path.name})") + try: + old_path.rename(new_path) + return new_path + except (FileNotFoundError, OSError) as e: + print(f" SKIPPED ({e}): {old_path.name}") + return None + + # ── Phase 3: Rename (interactive or auto) ── if args.no_interactive: - moved = 0 - for old_path, rel_path in proposals.items(): - new_path = output_root / rel_path - new_path.parent.mkdir(parents=True, exist_ok=True) - if new_path.exists(): - stem = new_path.stem - counter = 1 - while new_path.exists(): - new_path = new_path.with_name(f"{stem}_{counter}{new_path.suffix}") - counter += 1 - try: - shutil.move(old_path, new_path) - moved += 1 - except (FileNotFoundError, OSError) as e: - print(f" SKIPPED ({e}): {old_path.name}") - print(f" Moved {moved} file(s)." if not args.dry_run else "") + renamed = 0 + for old_path, new_path in proposals.items(): + if do_rename(old_path, new_path): + renamed += 1 + print(f" Renamed {renamed} file(s).") else: - moved = 0 + renamed = 0 skipped = 0 items = list(proposals.items()) i = 0 while i < len(items): - old_path, rel_path = items[i] - new_path = output_root / rel_path - new_path_display = str(rel_path) + old_path, new_path = items[i] print(f"\n [{i+1}/{len(items)}]") print(f" From: {old_path.name}") - print(f" To: {output_root / rel_path}") + print(f" To: {new_path.name}") remaining = len(items) - i - 1 - rlabel = f"move all {remaining}" if remaining else "" - sys.stdout.write(f" [Enter]=move [e]=edit [s]=skip{' [a]=' + rlabel if rlabel else ''} [q]=quit: ") + rlabel = f"rename all {remaining}" if remaining else "" + sys.stdout.write(f" [Enter]=rename [e]=edit [s]=skip{' [a]=' + rlabel if rlabel else ''} [q]=quit: ") sys.stdout.flush() choice = sys.stdin.readline().strip().lower() @@ -564,15 +462,14 @@ def main(): i += 1 continue elif choice == "e": - sys.stdout.write(f" Edit name (will be 'schmeeve-AI-{rel_path.parent}/...'): ") + sys.stdout.write(f" Edit keywords (name becomes 'schmeeve-AI-...'): ") sys.stdout.flush() custom = sys.stdin.readline().strip() if custom: custom_desc = re.sub(r"[^a-z0-9-]", "-", custom.lower()) custom_desc = re.sub(r"-+", "-", custom_desc).strip("-") if custom_desc: - new_name = f"schmeeve-AI-{custom_desc}.png" - rel_path = rel_path.with_name(new_name) + new_path = old_path.with_name(f"schmeeve-AI-{custom_desc}.png") else: print(" Invalid name, skipping.") skipped += 1 @@ -586,20 +483,9 @@ def main(): pass elif choice == "a": for j in range(i, len(items)): - p, rp = items[j] - np = output_root / rp - np.parent.mkdir(parents=True, exist_ok=True) - if np.exists(): - stem = np.stem - counter = 1 - while np.exists(): - np = np.with_name(f"{stem}_{counter}{np.suffix}") - counter += 1 - try: - shutil.move(p, np) - except (FileNotFoundError, OSError) as e: - print(f" SKIPPED ({e}): {p.name}") - moved += len(items) - i + p, np = items[j] + if do_rename(p, np): + renamed += 1 break else: print(f" Unknown option '{choice}', skipping.") @@ -607,24 +493,11 @@ def main(): i += 1 continue - # Perform move - new_path = output_root / rel_path - new_path.parent.mkdir(parents=True, exist_ok=True) - if new_path.exists(): - stem = new_path.stem - counter = 1 - while new_path.exists(): - new_path = new_path.with_name(f"{stem}_{counter}{new_path.suffix}") - counter += 1 - print(f" (file existed, saved as {new_path.name})") - try: - shutil.move(old_path, new_path) - moved += 1 - except (FileNotFoundError, OSError) as e: - print(f" SKIPPED ({e}): {old_path.name}") + if do_rename(old_path, new_path): + renamed += 1 i += 1 - print(f"\n Moved: {moved} Skipped: {skipped}") + print(f"\n Renamed: {renamed} Skipped: {skipped}") # Check for JPEGs with AI metadata jpg_pattern = "**/*.jpg" if args.recursive else "*.jpg" diff --git a/sync-comfyui-snaps b/sync-comfyui-snaps index 066b403..c06de38 100755 --- a/sync-comfyui-snaps +++ b/sync-comfyui-snaps @@ -3,6 +3,9 @@ # sync-comfyui-snaps — Pull new images from nimo.loc ComfyUI output, sync to NAS, # then rename AI snaps and organize. # +# NOTE: superseded by comfyui-sync (systemd user timer on nimo.loc). Kept for +# reference/manual use; the cron entry below is NOT currently installed. +# # Crontab (every 2 hours): # 0 */2 * * * /home/schmeeve/.local/bin/sync-comfyui-snaps >> /home/schmeeve/.local/log/sync-comfyui-snaps.log 2>&1 # @@ -72,19 +75,26 @@ rsync -vu --checksum \ --progress --stats \ "${SRC}" "${DEST}/" -# 3. Rename AI snaps (non-interactive for automation) -AI_OUTPUT="${DEST}/Images/AI" +# 3. Route fresh drops into Images/ (neatcli equivalent), then rename in place. +# NOTE: rename-ai-snaps no longer takes --output; it only renames. +# organize-images does the model/lora sorting. +IMAGES_DIR="${DEST}/Images" +mkdir -p "${IMAGES_DIR}" +find "${DEST}" -maxdepth 1 -type f \ + \( -iname '*.png' -o -iname '*.jpg' -o -iname '*.jpeg' -o -iname '*.webp' \) \ + -exec mv -n {} "${IMAGES_DIR}/" \; + if [ -x "${RENAME_SCRIPT}" ]; then - echo "[sync-comfyui-snaps] Running rename-ai-snaps on ${DEST} → ${AI_OUTPUT}" - "${RENAME_SCRIPT}" "${DEST}" --output "${AI_OUTPUT}" --no-interactive + echo "[sync-comfyui-snaps] Running rename-ai-snaps on ${IMAGES_DIR}" + "${RENAME_SCRIPT}" "${IMAGES_DIR}" --no-interactive else echo "[sync-comfyui-snaps] WARNING: rename-ai-snaps not found at ${RENAME_SCRIPT}" fi -# 4. Organize images +# 4. Organize images into AI/{model}/{lora}/ if [ -x "${ORGANIZE_SCRIPT}" ]; then - echo "[sync-comfyui-snaps] Running organize-images on ${DEST}" - "${ORGANIZE_SCRIPT}" "${DEST}" -e + echo "[sync-comfyui-snaps] Running organize-images on ${IMAGES_DIR}" + "${ORGANIZE_SCRIPT}" "${IMAGES_DIR}" -e else echo "[sync-comfyui-snaps] WARNING: organize-images not found at ${ORGANIZE_SCRIPT}" fi