Fix ipfs-cluster wedge: blox-ai bind-mounts identity.json as a directory

[ehsan6sha]

Problem

On a blox, /uniondrive/ipfs-cluster/identity.json was an empty directory instead of a file. ipfs_cluster was stuck forever printing Waiting for /internal and /uniondrive to become available and writable... and the device could not earn. Neither fula-readiness-check nor the blox-ai "not earning" path detected or repaired it.

Root cause (proven on a lab device)

The blox-ai plugin bind-mounts the identity FILE directly (plugins/blox-ai/docker-compose.yml):

- /uniondrive/ipfs-cluster/identity.json:/etc/fula/host-ipfs-cluster-identity.json:ro

When the Docker daemon sets up a bind mount whose host source path is absent, it auto-creates the source as a root-owned directory. blox-ai runs as its own systemd service with no ordering dependency on the identity existing, so whenever identity.json is absent at blox-ai container-create time (fresh/reformatted /uniondrive, or before go-fula writes it), Docker poisons the path. That permanently wedges the cluster ([ -f identity.json ] fails forever) and makes go-fula's initipfscluster panic (WriteFile over a directory); go-fula.sh has no set -e, so it marks .ipfscluster_setup done anyway, masking it.

Reproduced byte-for-byte on the lab device: a docker run with that exact mount + absent source created identity.json as drwxr-xr-x root root. The regenerated cluster peerID is deterministic (derived from /internal/config.yaml), so recovery preserves the on-chain registration -> earnings resume without re-registration.

Why the safety nets missed it

• readiness-check only matched specific daemon-error log strings; the wait-loop matches none, and a directory identity.json was treated as "legitimately absent."
• not-earning tree only flagged the cluster if oom_killed or state != running; the wedged container is running, so it fell through to a chain check that couldn't read the peerID -> "indeterminate."

Changes (this PR, fula-ota)

• Fix 1 (prevent recurrence) -- plugins/blox-ai/docker-compose.yml: mount the parent dir /uniondrive/ipfs-cluster instead of the file, and point BLOX_AI_CLUSTER_IDENTITY_PATH at identity.json inside it. Docker can then only ever auto-create the harmless directory, never the file-as-dir. identity_health honors the env var, so no blox-ai image rebuild is needed.
• Fix 2 (recover already-wedged devices) -- readiness-check.py check_and_fix_ipfs_cluster(): detect os.path.isdir(identity.json) -> stop blox-ai -> stop fula -> rm -rf the dir -> restart fula (go-fula regenerates the deterministic identity) -> restart blox-ai once the file is back. All subprocess calls are timeout-bounded so a D-state container cannot hang the watchdog.
• Fix 3a (defense) -- docker/go-fula/go-fula.sh: remove a directory-shaped identity.json before initipfscluster.
• Fix 4a (detection) -- plugins/blox-ai/trees/not-earning.yaml: branch on the new cluster_identity_is_directory reason -> concrete "cluster identity is a folder" verdict + restart_fula recommendation instead of "indeterminate."

Companion PRs

• go-fula: initipfscluster self-heals a directory-shaped identity.json (Fix 3b).
• blox-ai: identity_health reports cluster_identity_is_directory (Fix 4b) -- Fix 4a's tree branch is inert until this ships.

Validation (lab device, fxblox-rk1)

• Reproduced the mechanism (Docker creates the dir on real mergerfs).
• Proved deterministic peerID regeneration (file == cluster API == regenerated).
• Full wedge->recover cycle: reproduced the exact wedge, recovery restored ** IPFS Cluster is READY **, rejoined the pool, same peerID.
• Applied Fix 1 on-device: blox-ai reads identity via the new path, healthy.
• Verified propagation: fxsupport image -> docker cp -> /usr/bin/fula -> install.sh refreshes the active compose on boot.

Notes

No fleet deploy happens on merge -- CI builds fxsupport:release only when a GitHub Release is published. Ship Fix 1 + Fix 2 together (self-heal alone could re-wedge without the mount fix).

🤖 Generated with Claude Code