Adapter conformance canary¶

Upstream agent CLIs release on their own schedule, and a release can break an adapter contract (rename a flag, drop a subcommand, change output shape) without any change on our side. Version-floor advisories (bernstein doctor, see src/bernstein/adapters/advisories.py) warn about known-old versions, but they cannot catch a fresh upstream release breaking a contract. The canary closes that gap: adapter breakage becomes our finding, not a user's broken unattended run.

What runs¶

A nightly workflow (.github/workflows/adapter-conformance-canary.yml) drives scripts/adapter_canary.py, which runs the canary matrix defined in src/bernstein/adapters/canary.py:

• every primary adapter (agy, aider, claude, codex, copilot, gemini, opencode, qwen) against whatever upstream version the runner installed that night;
• one pinned tiny goal on the cheapest usable model per adapter, so daily spend stays bounded and run-to-run diffs isolate upstream drift;
• the same in-process conformance check operators get from bernstein adapters check: the installed binary's --help surface is matched against the adapter's declared contract (tests/contract/contracts/<adapter>.yaml).

Every probe is a receipt¶

Each probe seals a content-addressed receipt: canonical JSON whose SHA-256 is its identity. Two runs that observed the same upstream surface at the same timestamp produce byte-identical receipts; a mutated receipt fails verification (bernstein.adapters.canary.verify_canary_receipt) exactly like a tampered chain entry. Receipt hashes are mirrored into the HMAC audit chain (adapter.canary_receipt events), so a canary finding is reconstructable offline rather than living only in a CI log.

Regression handling¶

• A regression must repeat: two consecutive failures with the same failure fingerprint are required before the canary proposes an issue, so one upstream flake never pages anyone.
• A --help that advertises none of a contract's required tokens (or prints nothing) is classified as an inconclusive skip, not a fail -- an installed CLI cannot legitimately drop its entire required surface in one release, so this signals a broken, paginated, or wholesale-redesigned --help (or a shim binary on PATH) that an operator must investigate, rather than genuine per-flag drift. A partial miss (at least one required token still present) remains a real drift fail. The skip is independent of the process exit code.
• Issues are deduped on the failure fingerprint (adapter + version + failure lines): the same regression never opens two issues, while a new upstream version failing fresh reports again.
• The opened issue carries the failing conformance transcript and the receipt hash, so the finding is reproducible from the issue alone.

Last-green table¶

The table below is regenerated by the canary from passing receipts; it is a projection, never a hand-maintained list. Primary adapters without a passing receipt (currently aider) have no last-green row, so the primary-adapter list above and this table diverge until the next green canary run for that adapter. Each row names the receipt hash prefix that attested it. bernstein doctor reads the same projection (shipped as src/bernstein/adapters/last_green.json) and warns when a locally installed agent version is ahead of last-green: the canary has not verified that release yet, so unattended runs are one upstream regression away from failing without warning.

Operator knobs¶

• Pin an adapter for unattended runs to its last-green version when doctor warns.
• Run one probe locally: uv run python scripts/adapter_canary.py --adapter agy.
• Regenerate this table from a local run: uv run python scripts/adapter_canary.py --update-docs.