Follow-up: post-merge verification for hosted CI gating from #4036

[justin808]

Why this follow-up exists

PR #4036 changed the repository's CI control plane: required PR gate, hosted CI labels, comment commands, local CI defaults, selector logic, merge-queue base handling, and agent/human process docs. The pre-merge PR was validated heavily, but several GitHub behaviors only run from the default branch after merge. This issue is the post-merge audit plan and the durable handoff for humans and agents.

Merged PR: #4036
Merge commit: 6b4efdf11e53f58fa61b429dede6fc0322221de9
Merged at: 2026-06-16T11:29:16Z
Pre-merge final head: 522955b72ba1e3a6dba8a9a32ae689b3a11dfadb
Release tracker at merge time: #3823, accelerated-rc

Audit goals

• Prove the new default-branch issue_comment CI commands work from real PR comments.
• Prove optimized hosted CI and force-full hosted CI are separate and auditable decisions.
• Prove labels persist intended future behavior without duplicating workflow runs on command-triggered label writes.
• Prove bin/ci-local and bin/ci-rerun-failures auto-discover the PR base and do not hard-code unrelated refs.
• Prove merge queue and release-target branches use the right diff base.
• Preserve a process that future humans and agents can follow without reading the whole PR history.

Already verified before merge

• PR Clarify hosted CI gating and required PR gate #4036 merged with current-head GitHub checks complete: 50 passed, 0 failed, 0 pending, 5 expected skips.
• Current-head Claude Code Review passed: https://github.com/shakacode/react_on_rails/actions/runs/27612336985
• CodeRabbit approved the current head.
• GraphQL unresolved review-thread count was 0 before merge.
• Local final-head validation passed:
  • Ruby safe-load check for all .github/workflows/*.{yml,yaml} with YAML.safe_load_file(..., permitted_classes: [], aliases: false)
  • bash -n bin/ci-local bin/ci-rerun-failures bin/request-hosted-ci script/ci-changes-detector script/ci-changes-detector-test.bash script/lib/git-diff-base-test.bash
  • bash script/ci-changes-detector-test.bash -> 49 run, 0 failed
  • bash script/lib/git-diff-base-test.bash -> 41 run, 0 failed
  • actionlint
  • git diff --check
  • pre-commit hook
  • pre-push hook

Phase 0: main-branch baseline

Owner: first agent/human picking up this issue.

1. Confirm main contains merge commit 6b4efdf11e53f58fa61b429dede6fc0322221de9.
2. Inspect the first main-branch workflow batch triggered by the merge.
3. Record each failure, skip, or cancellation with workflow URL and whether it is related to Clarify hosted CI gating and required PR gate #4036.
4. Run the local static baseline from a clean origin/main worktree:

git fetch origin main
ruby -e 'require "yaml"; Dir[".github/workflows/*.{yml,yaml}"].each { |f| YAML.safe_load_file(f, permitted_classes: [], aliases: false) }'
actionlint
bash script/ci-changes-detector-test.bash
bash script/lib/git-diff-base-test.bash
bash -n bin/ci-local bin/ci-rerun-failures bin/request-hosted-ci script/ci-changes-detector script/ci-changes-detector-test.bash script/lib/git-diff-base-test.bash

Expected result: all static checks pass. Any main-branch CI failure gets either a fix PR or an explicit unrelated-failure note.

Phase 1: sacrificial optimized-hosted PR

Create a tiny draft PR from current main that changes only a harmless internal planning/test file. Keep it small enough that reviewing the CI behavior is the point.

Required checks:

• Opening the draft PR creates the required PR gate.
• Draft status is not treated as the hosted-CI trigger.
• Review tools that are configured for draft PRs run; if any review tool skips drafts, record the exact check/comment behavior.
• +ci-status reports the current labels, selector mode, and available commands.
• +ci-run-hosted dispatches optimized hosted workflows for the current head and adds ready-for-hosted-ci.
• The command does not create duplicate hosted workflow batches when it adds the label. Note: workflow-token label writes should not trigger new workflow runs; only the explicit workflow_dispatch calls should be the current-head trigger.
• +ci-stop-hosted removes hosted readiness and future pushes return to the required gate only.
• +ci-skip-hosted [reason] creates a SHA-bound waiver comment and does not accidentally leave hosted labels behind.
• Invalid legacy commands (+ci-run-full, +ci-skip-full, /run-skipped-ci, /run-skipped-tests, /stop-run-skipped-ci) get the new guidance, not silent success.

Expected result: optimized hosted CI is requestable by comment, auditable by comment, and separate from draft/readiness status.

Phase 2: force-full hosted PR path

On the same sacrificial PR or a second tiny PR, test the broad-matrix path.

Required checks:

• +ci-force-full dispatches hosted workflows with force_full_hosted: true.
• The PR receives both ready-for-hosted-ci and force-full-hosted-ci.
• Hosted workflows bypass optimized selection only under force-full mode.
• +ci-stop-full removes only force-full-hosted-ci; if ready-for-hosted-ci remains, later pushes continue optimized hosted CI.
• hosted-ci-no-benchmarks suppresses benchmark labels when present.
• Benchmark labels remain opt-in and are not implied by hosted CI or force-full hosted CI.

Expected result: force-full is an explicit escalation, not a synonym for ordinary hosted readiness.

Phase 3: release-target branch behavior

Create or use a safe temporary release-target branch and a tiny PR targeting it. Do not disturb real release coordination.

Required checks:

• bin/ci-local discovers the PR base automatically.
• Hosted workflow selectors treat release target branches as release targets without agents passing arbitrary <base-ref> values.
• script/ci-changes-detector compares against the actual PR base, not unrelated origin/main.
• Agent docs remain accurate: agents should not ask maintainers to choose base refs for normal PR validation.

Expected result: release-target PRs get the same process with the correct base.

Phase 4: merge queue and branch protection

This requires maintainer/admin access.

Required checks:

• Branch protection requires ci-required / required-pr-gate.
• Merge queue, if enabled, triggers the merge_group paths.
• Workflows that run script/ci-changes-detector use github.event.merge_group.base_sha for merge-group diffing.
• A tiny PR can enter merge queue without bypassing the required gate.
• If merge queue is not enabled yet, record the exact missing repository setting rather than treating it as code failure.

Expected result: merge queue does not compare against the wrong base and does not skip required validation.

Phase 5: fork, bot, and Dependabot paths

Required checks:

• A fork PR receives a clear message if hosted CI cannot be command-dispatched from the fork branch.
• Maintainers have a documented manual path for fork PR hosted validation.
• Dependabot PRs follow the required gate and the new label vocabulary.
• Old labels on existing PRs are treated as stale vocabulary and mapped in comments rather than silently relied on.

Expected result: non-human branches are clear, auditable, and do not rely on unavailable token permissions.

Phase 6: local script behavior

Use at least one PR targeting main and one PR targeting a release-like base.

Required checks:

bin/ci-local --help
bin/ci-local
bin/ci-local --changed
bin/ci-local --all
bin/ci-rerun-failures --help

Also verify bin/ci-rerun-failures uses the PR's actual base ref when invoking script/ci-changes-detector.

Expected result: optimized local CI is the default, --changed is an alias for the default, and --all is an explicit broad local run.

Phase 7: open PR rollout

After #4036 is merged, every open PR should receive a rollout comment that tells authors/agents to rebase onto current main and use the new CI vocabulary.

Required checks:

• Each open PR receives exactly one comment with marker <!-- hosted-ci-rollout-4036-v1 -->.
• The comment explains replacement labels:
  • full-ci -> ready-for-hosted-ci only when optimized hosted CI should persist.
  • full-ci-no-benchmarks -> hosted-ci-no-benchmarks when suppressing benchmarks.
  • force-full intent -> force-full-hosted-ci or +ci-force-full.
• Draft and do not merge PRs are told no action is needed until resumed, but they still get the process notice.

Expected result: no open PR continues under the old CI vocabulary without a visible warning.

Agent process instructions

Agents picking this up should:

• Start from this issue, not from memory of the PR discussion.
• Use a fresh branch/worktree for any sacrificial PRs.
• Keep sacrificial PR changes tiny and easy to close.
• Prefer comment commands over direct label changes when validating user-facing CI flows.
• Record exact workflow URLs, PR URLs, command comments, labels, and head SHAs.
• Do not close this issue until every phase has either passed or has a linked follow-up PR/issue with an owner.
• If a behavior differs from this plan, update this issue with the observed behavior and decide whether docs or code are wrong.

Human process instructions

Humans should:

• Use bin/ci-local before pushing; it auto-detects the PR base.
• Use +ci-status before deciding whether hosted CI is needed.
• Use +ci-run-hosted for optimized hosted confirmation after the final push.
• Use +ci-force-full only when intentionally bypassing optimized selection.
• Use +ci-stop-hosted or +ci-stop-full to remove persistent CI intent while iterating.
• Keep PR descriptions self-contained: include a short why even when an issue is linked.

Known risk to watch

The generator shards are slow: recent successful runs for the generator matrix took roughly 33 to 35 minutes for the long-tail jobs, and one earlier ready-event run appeared stuck before being canceled/rerun. Treat new generator hangs as real audit findings and capture job URLs plus elapsed time before rerunning.

Definition of done

• Main-branch post-merge checks are reviewed and any Clarify hosted CI gating and required PR gate #4036-related failures are fixed.
• At least one optimized-hosted command path and one force-full command path are proven on default-branch workflow code.
• Branch protection and merge-queue expectations are verified or explicitly recorded as admin work.
• Open PR rollout comments are posted.
• Any code/doc corrections discovered during the audit are linked here.
• This issue has a final comment summarizing evidence, residual risk, and whether the new CI process is ready for routine use.

[justin808]

Phase 0 update: immediate post-merge audit

Timestamp: 2026-06-16T11:38Z
Merge commit audited: 6b4efdf11e53f58fa61b429dede6fc0322221de9

Local/default-branch static checks

The PR branch tree matches origin/main after the squash merge (git diff --quiet HEAD origin/main exited 0). These checks were rerun against that merged tree and passed:

ruby -e 'require "yaml"; Dir[".github/workflows/*.{yml,yaml}"].each { |f| YAML.safe_load_file(f, permitted_classes: [], aliases: false) }'
actionlint
bash script/ci-changes-detector-test.bash              # 49 run, 0 failed
bash script/lib/git-diff-base-test.bash                # 41 run, 0 failed
bash -n bin/ci-local bin/ci-rerun-failures bin/request-hosted-ci script/ci-changes-detector script/ci-changes-detector-test.bash script/lib/git-diff-base-test.bash

Main push workflow status

Current status for the merge-commit push batch:

• Passed: Git Diff Base and CI Detector Tests, Lint GitHub Actions, Check Markdown Links, JS unit tests for Renderer package, Assets Precompile Check, CodeQL, Lint JS and Ruby, Playwright E2E Tests, React on Rails Pro - Integration Tests, Integration Tests, React on Rails Pro - Package Tests, Generator tests.
• Still running: Benchmark Workflow, Rspec test for gem.
• Failed: none observed at this checkpoint.

Default-branch issue-comment signal from rollout comments

Posting the rollout comments to existing PRs exercised the newly merged default-branch issue_comment workflows. The CI Commands and Detect Invalid CI Commands workflows completed successfully for those non-command rollout comments; Claude Code skipped the comment events.

This is useful evidence that the default-branch comment workflows are installed and parsing ordinary comments safely. It does not replace the explicit Phase 1/2 tests for +ci-status, +ci-run-hosted, and +ci-force-full.

Open PR rollout

Rollout comments with marker <!-- hosted-ci-rollout-4036-v1 --> were posted to every other open PR.

Verification count:

• Other open PRs: 12
• PRs with rollout marker: 12
• Missing marker: none

Commented PRs: #4054, #4052, #4048, #4037, #4035, #4023, #4022, #4018, #4006, #3985, #3693, #3245.

Next audit work remains open in the phase checklist above, especially real command comments, force-full hosted behavior, release-target base detection, and merge queue/branch protection verification.