v1.3.1 — defensive default-route reconcile

What's new

Defensive fix for the service-anchor's default-route reconcile loop.

Background

Before v1.3.1, the service-anchor's reconcile loop compared the resolver-supplied gateway IP against an in-memory cache of the last IP it installed. If the two matched, it returned without touching the kernel. That covered the happy case (network-anchor recreated → new IP → reinstall) but silently missed the case where the kernel route was flushed externally and the resolved IP stayed the same:

• ip route del default in the netns (an unrelated tool, an operator's hand)
• a kernel quirk on link renumber
• a sidecar bug

In all of these the resolver still returns the original IP, the in-memory cache matches it, and anchord believes everything is fine forever — but the application container has lost egress.

Fix

applyRoute now reads the kernel's current default gateway via RecordDefaultRoute each tick and only short-circuits when the kernel actually has our gateway installed. Mirrors the extroute package's assert pattern (issue #6) that's been doing this defensively for the network-anchor's external default route since v1.0.

Cost: one extra netlink list per family per ResolveInterval (default 5s). Negligible — two syscalls every 5 seconds per service-anchor.

Behavior:

• Kernel has our gateway → no-op, no log noise.
• Kernel has a different gateway → replace, log reason=drifted with prev=<old>.
• Kernel has no default route at all → replace, log reason=missing.

Tests

Two new regression tests in internal/serviceanchor/serviceanchor_test.go:

• TestReconcile_ReinstallsAfterExternalFlush — flushes the fake kernel's default and confirms the next reconcile re-installs it.
• TestReconcile_ReinstallsWhenKernelHasDifferentGateway — pre-seeds a stale gateway and confirms reconcile replaces it.

The existing fake stubRouter now persists ReplaceDefaultRoute into its internal "kernel state" map, matching real netlink semantics. The pre-existing TestReconcile_NoOpWhenUnchanged still passes — three reconciles in a row install exactly once.

Full module: 394/394 unit + 74/74 e2e green.

Migration

None. Drop-in replacement for v1.3.0 in all existing modes. No env-var changes, no SPEC changes (the fix tightens the F-26 contract without changing its surface).

docker pull ghcr.io/alexcherrypi/anchord:v1.3.1

Full changelog

v1.3.0...v1.3.1

v1.3.0 — sidecar rebinders for peer-stack recreates

What's new

Two new sidecar modes that close the "my peer stack ran compose down/up and silently broke me" failure class. Both modes share the same binary and image as the existing network-anchor / service-anchor; they're opt-in additions, not migrations.

F-48 — External-network follower auto-rebind (issue #12)

ANCHORD_MODE=external-rebinder. A sidecar in a follower's compose project that re-attaches a configured container to a target bridge network every time that network's Docker ID changes — typically after the target stack does compose down/up and Docker hands out a new ID for the same network name.

• Bootstrap-recheck on startup + docker events stream (type=network, event=create|destroy|connect|disconnect).
• Reconnect-only by default; ANCHORD_FOLLOW_RESTART=true opt-in for apps with long-lived persistent client pools (HikariCP, http.Client with large IdleConnTimeout, etc.).
• Spec: SPEC-EXTERNAL-REBINDER-DRAFT.md.

F-49 — Wrap-anchor netns-target auto-rebind (issue #13)

ANCHORD_MODE=wrap-rebinder. A sidecar in a wrap-stack that auto-discovers every wrap-anchor in its own compose project via HostConfig.NetworkMode and recreates each one whenever its target is recreated under the same name.

• Zero required env vars beyond the mode selector — pool is discovered, not configured.
• Bootstrap-recheck + docker events stream (type=container, event=start) filtered in-process against the dynamic target pool.
• Own-project sibling-start events trigger pool re-enumeration, so the sidecar survives compose up --force-recreate of its own wrap-stack.
• Spec: SPEC-WRAP-REBINDER-DRAFT.md.

Implementation note worth flagging

F-49's original design assumed docker restart would re-resolve network_mode: container:<X>. Empirically it does not — Docker resolves the reference at container creation time and stores the resolved long-ID; a restart against a dead target ID fails with joining network namespace of container: No such container. The only working recovery is remove + create + start with a fresh NetworkMode string, which is the same RecreateWithNetworkMode primitive internal/autostart uses for F-45 wrap-dep rebinding. The SPEC has a > Critical implementation note block documenting this for future readers.

What didn't change

• network-anchor, service-anchor, doctor modes are byte-for-byte the same. Existing deployments don't need any changes; the new sidecars are purely additive.
• No new required env vars in existing modes. The new ANCHORD_FOLLOW_* / ANCHORD_WRAP_* env vars are scoped to the new modes only.

Deployment

Pull the new image:

docker pull ghcr.io/alexcherrypi/anchord:v1.3.0

External-rebinder sidecar (follower's compose):

services:
  rebinder:
    image: ghcr.io/alexcherrypi/anchord:v1.3.0
    environment:
      ANCHORD_MODE: external-rebinder
      ANCHORD_FOLLOW_NETWORK: <target-bridge-network-name>
      ANCHORD_FOLLOW_TARGET: <service-or-container-name>
      # ANCHORD_FOLLOW_RESTART: "true"   # opt-in
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped

Wrap-rebinder sidecar (wrap-stack's compose):

services:
  # ...existing wrap-anchors...
  rebinder:
    image: ghcr.io/alexcherrypi/anchord:v1.3.0
    environment:
      ANCHORD_MODE: wrap-rebinder
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped

The docker socket is mounted read-write (the rebinders need POST for network/{connect,disconnect} / containers/<id>/restart / containers/create+remove). A POST=1 docker-socket-proxy works just as well.

Test surface

• Go unit tests: 392/392 — race-clean across all 18 packages, including 24 new rebinder and 23 new wraprebinder tests.
• End-to-end harness: 74/74 across 5 DHCP scenarios.
• Integration tests (build tag integration, against real Docker): 6 scenarios for rebinder, 4 for wraprebinder including the headline 2026-06-07 Mailcow-incident reproduction. Not gating CI; run locally with go test -tags=integration ./internal/{rebinder,wraprebinder}/....

Acknowledgements

Both features triggered by the 2026-06-07 production incident chain. Design refinement from the issue-comment round-trip caught two important annotations (bootstrap-recheck race, sibling-recreate re-enumeration) and one design correction (restart vs recreate) before they reached production.

Full changelog

v1.2.1...v1.3.0

v1.2.0 — F-45 auto-rebind of dead-netns wrap dependents

Closes #10

Closes the dead-netns cascade that anchord itself triggers as a side effect of every F-45 SA recreate (image-drift in v1.0.2, destroy-event respawn in v1.0.2, stale-netns in #5). v1.1.0 added detection-only; v1.2.0 closes the loop for the case anchord caused.

What's new

When anchord removes its F-45-managed SA (stale-netns or image-drift path), it now:

1. Enumerates dependents whose network_mode: container:<X> matches the SA's IDs/names before the Remove.
2. Removes the old SA, creates + starts the new one.
3. Re-creates each enumerated dependent via the docker SDK (inspect → patch HostConfig.NetworkMode → remove → create → start), bound to the new SA's container ID.

Failures per-dep are warn-logged and the rebind loop continues — one broken dep doesn't justify leaving the others stranded.

Config flag

ANCHORD_AUTOFIX_DEAD_NETNS — default true. Set to false to revert to v1.1.0 detection-only behaviour. Same opt-out shape as ANCHORD_AUTOSTART_SIBLINGS.

Why the narrow scope

The auto-fix only fires for orphans anchord knows it created — i.e., it's about to call Remove(saID) and walks the container list for things pinned to saID. This avoids the hard problems the broader v1.1.0 detector kept WARN-only:

• No race with operator-driven SA recreates (anchord caused this one, by definition)
• No scope-discovery ambiguity (we know the exact ID, no project-scan needed)
• No multi-SA confusion (one anchord, one managed SA)
• No state-tracking needed across restarts

v1.1.0's dependents watcher still WARNs about the other orphan sources: operator docker rm, OOM-killed SAs, external orchestrators recreating something. Those still need anchord doctor stale-netns or manual recovery.

Operator notes

• Same docker-socket-proxy permission surface F-45 already uses for SA recreate (DELETE + container create/start), just applied to wrap-dep containers in our project.
• No breaking changes vs v1.1.0.
• After upgrading: docker compose pull && up for the network-anchor stack just works — parent restart triggers image-drift, image-drift triggers SA recreate, SA recreate triggers dep rebind. No more manual sweep needed.

Commits

• 2bfc25c — feat(autostart): rebind wrap dependents on F-45 SA recreate

Full changelog: v1.1.0...v1.2.0

v1.1.0 — dead-netns dependent detection + anchord doctor CLI

Closes #9

Cluster-rolling-deploy of v1.0.1 hit the anchord-v2 wrap pattern's silent failure mode: recreating a service-anchor pins every dependent (per-stack Traefik, acme renewers, the wrapped service itself) to the now-dead container ID. The dependent stays running to Docker but has no interface, no routes, no DNAT.

Production incident scope (2026-05-23): 21 dependents orphaned across 9 service-anchors during the rollout, including authentik_server itself — chain-failing every OIDC-protected service. The fix was to walk every stack and docker compose up -d --no-deps --force-recreate <svc> each victim.

What's new

Daemon-side detection

Network-anchor mode now runs a periodic dead-netns watcher scoped to its compose project. New victims fire a structured WARN per first detection:

WARN dependent in dead netns container=ix-authentik-traefik-frigate-1 \
     container_id=abcdef0123 stale_target=158f0cc2 \
     hint="docker compose -p ix-authentik up -d --no-deps --force-recreate traefik-frigate"

Deduped by (container_id, stale_target) so a sticky victim doesn't spam every poll. Seen-set rebuilt every tick, so a fix-then-rebreak fires a fresh warn.

Disabled (no-op) in F-42 label-selector mode — without a compose-project boundary, every anchord instance on the host would broadcast the same victims.

anchord doctor stale-netns CLI

One-shot diagnostic for post-mortem use or hosts without a running anchord. Scans every container on the host, groups victims by dead target, prints the exact compose recovery command per victim:

$ anchord doctor stale-netns
Found 11 dependent(s) in dead netns across 9 target(s):

  dead target: 158f0cc2  (1 victim(s))
    ix-authentik-traefik-frigate-1
      → docker compose -p ix-authentik up -d --no-deps --force-recreate traefik-frigate

  dead target: a7c53426  (3 victim(s))
    acme-init-xibo
    acme-renewer-xibo
    ix-xibo-traefik-1
      → docker compose -p ix-xibo up -d --no-deps --force-recreate traefik
  ...

No daemon config needed — speaks docker.sock, exits when done.

Docs

README gains an Operator tooling section with example output and a new caveats entry on the wrap-recreate pitfall pointing at both the WARN log and the doctor command.

Operator notes

• Detection-only by design. The watcher tells you what to recreate; it never recreates anything on its own. Auto-fix is a separate policy change with its own failure modes (loops, fighting operator removes) and deserves its own opt-in flag — tracked separately.
• No breaking changes vs v1.0.2. The watcher logs new WARN lines but never affects data-plane behaviour.
• Coverage: 301 unit tests (was 280; +21 for the new package + CLI), 74 e2e tests still green, race detector clean.

Commits

• f5c7f37 — feat(dependents): detect dead-netns wrap dependents + anchord doctor CLI
• 9877a82 — Refresh TEST-REPORT for v1.1.0 cut

Full changelog: v1.0.2...v1.1.0

v1.0.2 — F-45: SA destroy-respawn + parent image-drift cascade

Closes #8

Discovered during the 2026-05-23 production binary rollout when standalone F-45-managed SAs (ldap-service-anchor, nextcloud-aio-talk-anchor) silently kept running on the old image after operators tried to upgrade them.

Two related F-45 gaps closed

1. SA destroy event was ignored

The watcher's docker events filter was hard-wired to event=start only. An SA removed at runtime (docker rm -f, accidental prune, OOM-killer, operator-driven upgrade by recreate) was never respawned — the wrapped service kept anchord's DNAT rule but lost its service-anchor in the netns, silently losing default-route enforcement until the parent network-anchor itself restarted.

Fix: filter now includes destroy; new handleSAGone dispatcher takes the create-fresh path when the destroyed name matches the recipe and the target is still running. destroy rather than die so we don't fight docker's restart-policy on plain crashes and so the SA is actually gone from the daemon list by the time we react.

Log: managed service-anchor gone; respawning reason=absent

2. Parent image upgrade did not cascade to the SA

With the natural rolling-deploy workflow (docker compose pull && up), parents restarted on the new image digest but standalone F-45-managed SAs kept running on the old one — maybeManage's "SA already running + netns not stale" branch returned early.

Fix: backfill compares the parent's image digest (insp.Image) against each managed SA's (Summary.ImageID). On mismatch the SA is force-removed and the existing create-fresh path picks up the parent's current image.

Design choices:

• By digest, not tag string — tag-floats like :main trigger correctly across rolling deploys where the tag stays the same and only the underlying digest moves
• Backfill-only — fires once per process lifetime at parent startup. Subsequent target-start / SA-gone events skip the check, so a noisy event source can't trigger churning recreates
• Operator pin wins — ANCHORD_MANAGED_SA_IMAGE keeps its pin; the cascade only fires when the recipe uses the default "match self" behaviour

Log: managed service-anchor running on stale image; recreating reason=image_drift

Operator notes

Behaviour additions are purely additive — pre-existing deployments that explicitly pin SA images see no change. No breaking changes vs v1.0.1.

After upgrading to v1.0.2, the rolling-deploy workflow now Just Works:

docker compose pull        # parent image updated to new digest
docker compose up -d       # parents restart
                           # backfill detects image drift → SA force-recreated

For ad-hoc SA upgrades by docker rm: now respawns automatically within ~event-latency, no parent restart needed.

Commits in this release

• 05c37a1 — fix(autostart): respawn managed SA on destroy + cascade on parent image drift
• c339008 — Refresh TEST-REPORT for v1.0.2 cut

Full changelog: v1.0.1...v1.0.2

v1.0.1 — patch: docker event-stream connection leak

Critical production fix

v1.0.0 deployments running anchord against a busy docker daemon (e.g. Frigate cycling many ffmpeg subprocesses) accumulate hundreds of ESTAB sockets to docker(-proxy) per anchord instance, eventually exhausting the kernel's tcp_mem.

Observed in production at ~490 ESTAB per anchord × 32 anchord containers, ~6.4 GB of kernel TCP memory in use (above tcp_mem max of 5.9 GB), TCP: out of memory in dmesg, second-order slowness across every service sharing the kernel TCP pool.

Diagnosis

eventLoop's inner select sat inside an outer for, so every <-msgs receive fell out of the select and re-called cli.Events, opening a fresh long-poll HTTP request to docker per event while the prior request's goroutine stayed parked on its still-valid ctx. The bug lived in the discovery loop since the initial skeleton.

Fix

• 47b445d — split the consume-loop out (consumeEventStream). It stays on the same (msgs, errs) pair until ctx cancels or the stream actually ends. Reconnect only on real stream-end.
• 161397a — extracted runEventLoop(ctx, source, consume) with injectable source/consume so a regression test counts source-calls directly: 5 events produce exactly 1 source call, then stream-close produces a 2nd. Backoff is now a parameter for deterministic tests.

Upgrade

Strongly recommended for any deployment touching docker event sources with non-trivial event rates. No breaking changes vs v1.0.0.

Full changelog: v1.0.0...v1.0.1

v1.0.0 — production-ready

The v0.9.0 release was beta with one outstanding gap before v1.0: real-host validation on a Linux box with a physical VLAN. That gap is closed by weeks of uptime on a small self-hosted fleet under real workloads (SMTP, IMAP IDLE, LDAP binds, OIDC, Nextcloud Talk).

What v2.x added on top of v2

Each feature lives in its own SPEC-*-DRAFT.md:

• F-39 / F-40 — Wrap pattern. Service-anchors that join an existing app container's netns via network_mode: container:<X>. Wrap third-party Compose stacks (Mailcow, Nextcloud-AIO) without forking them.
• F-41 — Default-route authority on the macvlan. Priority DHCP Option 3 > ANCHORD_EXT_GATEWAY_IP > IPAM.Config.Gateway. Fixes the asymmetric-reply failure mode that killed long-lived TCP at ~17 min via TCP RTO.
• F-42 — Label-selector discovery (ANCHORD_LABEL_SELECTOR). Multiple anchord stacks per Compose project, or runtime-spawned targets that carry no project label.
• F-43 — Sibling auto-start. Network-anchor watches docker events and bootstraps Created-state siblings whose network_mode: container:<X> matches a now-running target.
• F-44 — Co-attachment shared-network picker. When anchord is on multiple non-EXT networks, pick the one with the most observed backends. Settles deterministically, never flaps.
• F-45 — Managed service-anchors. Network-anchor creates and rebinds the service-anchor on demand. Auto-recovery when an outside orchestrator (Authentik outpost controller, K8s-style operators) recreates the target mid-flight.
• F-46 — Port-translating DNAT. anchord.expose=tcp/636:6636 rewrites the destination port at DNAT time. Lets DMZ-side reservation (LDAPS on 636) meet app-side reality (Authentik outpost listening on non-privileged 6636).

Production-driven fixes shipped between v0.9.0 and v1.0.0

• #1 F-46 reconcile fails with 'nft map flush: numerical result out of range' on the v6 path — register-numbering bug, fixed by using legacy reg 3 for both families.
• #2 F-45 stamped com.docker.compose.project without .service — broke TrueNAS app.stop. Compose labels dropped from the spawn.
• #3 F-45 had no way to inject anchord.identity / anchord.expose labels on the spawn, and discovery couldn't follow network_mode: container:<X> for IP resolution. ANCHORD_MANAGED_SA_LABELS added; discovery now follows wrap references.
• #4 F-37 false alarm — turned out to be a Docker mac_address quirk under macvlan, not anchord. Documented in the close comment.
• #5 F-45 managed SA not respawned when target container is recreated. Stale-netns detect + force-recreate on target-start event.
• #6 F-41 itself.

New operator-facing env vars (all optional)

• ANCHORD_LABEL_SELECTOR — F-42 discovery scope override.
• ANCHORD_MANAGED_SA_TARGET, ANCHORD_MANAGED_SA_NAME, ANCHORD_MANAGED_SA_IMAGE, ANCHORD_MANAGED_SA_GATEWAY_IP, ANCHORD_MANAGED_SA_EXTRA_ENV, ANCHORD_MANAGED_SA_LABELS — F-45 recipe.
• ANCHORD_AUTOSTART_SIBLINGS — F-43 opt-out (default on).
• ANCHORD_SHARED_NETWORK — F-44 pin override.
• ANCHORD_EXT_GATEWAY_IP — F-41 default-route pin.

No breaking changes since v0.9.0 — all new vars are optional, all defaults preserve prior behaviour.

Tests

343/343 green (source hash sha256:5c3f9bad3f33eb4df324026d8c62a8c2667390ebe3ea02b6f317c17373643f3b):

• Go unit + integration: 269/269
• E2E (5 scenarios — v4-only, v6-only, both, none, dhcpv6-stateful): 74/74

See the auto-generated test report in README.md.

Compatibility

• Kernel ≥ 4.18 (atomic nftables map replaces).
• Docker ≥ 20.10 (macvlan + named maps).
• Tested on Linux 6.6 LTS (TrueNAS SCALE host) and Docker Desktop on Windows / macOS for development.

v0.9.0 — beta, feature-complete

v0.9.0 — beta, feature-complete

This is the first beta of anchord. Every functional requirement in
SPEC.md
is implemented and every acceptance scenario covered by the e2e
harness. One outstanding item before v1.0: real-host validation
on a Linux box with a physical VLAN sub-interface, to confirm the
Docker Desktop bridge-flood workaround isn't needed in production.

Added since v0.1.0-alpha

• Prometheus metrics + health endpoints (SPEC §2.7 + §2.8).
  Both modes serve /metrics, /healthz (always-200 liveness)
  and /readyz (mode-specific readiness — network-anchor needs
  nft tables installed AND first reconcile complete; service-anchor
  needs at least one default route installed) on a single
  loopback-only listener (ANCHORD_METRICS_ADDR=127.0.0.1:9090 by
  default, so the macvlan never sees them). 12 metrics total,
  bounded label cardinality, custom collector recomputes
  dhcp_lease_remaining_seconds at scrape time so the gauge decays
  between renewals.
• Architecture diagram redrawn in Mermaid with shape-based
  semantics — hexagons for Docker bridges, rectangles for full
  containers, rounded rectangles for shared-netns processes,
  parallelograms for boundaries (LAN, DBs).
• Cross-doc linking cleaned up — README points at
  ARCHITECTURE / SPEC / CONTEXT in the order CLAUDE.md prescribes
  for new contributors.
• GitHub community standards completed — Code of Conduct,
  bug-report and proposal issue templates, PR template that
  mirrors the CONTRIBUTING.md "Definition of done" checklist.

What was already there at v0.1.0-alpha

For context — these landed before this release line started:
service-anchor mode (SPEC §2.6), pure-Go DHCP client
(github.com/insomniacslk/dhcp, no more dhclient subprocess),
DHCPv6 feature parity, multi-arch images for 8 platforms,
atomic-replace verification for nftables maps, the Phase-2 e2e
harness with real listeners, code-hash test-report system, and
release-gate workflow.

Tests

97 unit + 70 e2e = 167/167 green. The auto-generated
TEST-REPORT block at the bottom of README.md is the canonical
release-readiness signal — the recorded code hash matches the
source this release was cut against.

Image

ghcr.io/alexcherrypi/anchord:v0.9.0 (multi-arch — linux/amd64,
linux/arm64, linux/arm/v7, linux/386, linux/ppc64le, linux/s390x,
linux/riscv64, linux/loong64).

Outstanding before v1.0

• Real-host validation: run scripts/update-test-report.sh on a
  Linux host with a physical VLAN sub-interface, confirm 70/70
  without E2E_BRIDGE_FLOOD_FIX=1. Closes the env-quirk caveat
  for good and unblocks the v1.0 tag.

v0.1.0-alpha — first tagged release

First tagged release of anchord. Pre-alpha but functional — use as a starting point, not in production.

What works

• Network-anchor + service-anchor modes (single binary, dispatched on ANCHORD_MODE).
• Pure-Go DHCP client (insomniacslk/dhcp) — no dhclient subprocess, DHCPRELEASE sent on shutdown, supports v4 + stateful DHCPv6.
• nftables NAT with atomic map updates — DNAT rules update without ever falling through, masquerade for outbound source-IP consistency.
• Conntrack flush when backend IPs change.
• Service-anchor default-route management via Docker DNS.
• 152/152 tests pass across unit + e2e (5 scenarios: v4-only, v6-only, both, none, dhcpv6-stateful). The README's auto-generated test report records the source-tree hash this tag was cut from. The release gate validates that the recorded hash matches the tagged source.

Multi-arch images

Available on ghcr.io for eight platforms:

```
linux/amd64 linux/arm64 linux/arm/v7 linux/arm/v6
linux/386 linux/ppc64le linux/s390x linux/riscv64
```

```bash
docker pull ghcr.io/alexcherrypi/anchord:v0.1.0-alpha
```

Open before v1.0

• Real-host validation on a Linux box with a physical VLAN sub-interface (e2e currently runs on Docker Desktop with a one-shot bridge-nf-call-iptables=0 workaround flag).
• SPEC decisions on the metrics surface and the health-endpoint shape.

See SPEC.md, ARCHITECTURE.md, CONTEXT.md, and CLAUDE.md/AGENTS.md for AI agents.

🤖 Generated with Claude Code