Auto-heal v2¶
Self-healing CI on main. When a required check fails, the v2 workflow classifies the failure, picks a repair strategy via a multi-arm bandit, applies it inside a cordoned worktree, opens a PR, and lets standard branch protection decide.
TL;DR¶
| Layer | What it does | Persisted state |
|---|---|---|
| Detection | Reads the failed run, buckets jobs, scores Bayesian confidence, judges flake-vs-real | .sdd/autoheal-bayes.json |
| Classification | Routes to safe / heuristic / risky / unknown | none |
| Repair | Thompson-samples a strategy, applies it, runs a diff-aware self-test | .sdd/autoheal-bandit.json, .sdd/autoheal-shadow.json |
| Safety | Cordon allowlist, blast-radius gate, cost circuit-breaker, kill switch | .sdd/autoheal-disabled |
| Provenance | Lineage v2 child body + decision-log entry + calibration row | .sdd/autoheal-history.jsonl |
All state files are gitignored. The workflow never commits state.
Architecture¶
flowchart LR
CI[CI failure on main] --> T[Triage]
T --> KS{Kill switch?}
KS -- on --> S1[Skip]
KS -- off --> ID{Existing PR?}
ID -- yes --> S1
ID -- no --> BUC[Bucketize jobs]
BUC --> BAY[Bayesian confidence]
BAY --> FL[Flake detector]
FL -- flake --> RT[Retry, no patch]
FL -- real --> BANDIT[Bandit select strategy]
BANDIT --> APPLY[Apply strategy]
APPLY --> CO[Cordon check]
CO -- block --> S2[Abort]
CO -- ok --> ST[Self-test]
ST --> BR[Blast-radius gate]
BR -- ok --> PR[Open heal PR]
PR --> AT[SLSA attestation]
PR --> LOG[Audit + lineage + decision + calibration]
LOG --> TG[Telegram alert] The 26-capability matrix¶
status is one of: shipped (works in this PR), partial (module shipped, workflow wiring deferred), deferred (planned for v3).
| # | Capability | Status | Bernstein hook |
|---|---|---|---|
| 1 | Bayesian per-class confidence | shipped | core.autoheal.bayesian |
| 2 | Flake vs real-fail distinguisher | shipped | core.autoheal.flake_detector |
| 3 | Diff-aware fail localisation | deferred | bisect helper to land in v3 |
| 4 | Failure clustering (bucketize) | shipped | core.autoheal.categorizer.bucketize |
| 5 | LLM-grounded categorisation | partial | cost-guard preflight in place; prompt path deferred |
| 6 | Sibling-bug hunt | deferred | needs blame oracle, planned v3 |
| 7 | Code-provenance check | shipped | core.autoheal.provenance |
| 8 | Multi-arm-bandit strategy select | shipped | core.autoheal.bandit |
| 9 | LLM-grounded fix synthesis | partial | cost-guard ready; prompt deferred |
| 10 | Counter-example test injection | deferred | planned v3 |
| 11 | Diff-aware self-test | partial | workflow runs ruff check / format / typos; blast-radius mapping deferred |
| 12 | Permission-profile enforcement | shipped | core.autoheal.cordon + cordon CLI |
| 13 | Cost circuit-breaker | shipped | core.autoheal.cost_guard |
| 14 | Adversarial pre-merge test | deferred | adversary role exists, wiring v3 |
| 15 | Blast-radius gate | partial | import-guarded in workflow; threshold tuning v3 |
| 16 | Lineage v2 attestation | shipped | core.autoheal.lineage_writer (extensible meta channel) |
| 17 | Decision-log entry | shipped | core.autoheal.wire writes autoheal_strategy rows |
| 18 | Calibration tracking | shipped | core.autoheal.wire writes predicted/observed pairs |
| 19 | SLSA build-provenance attestation | shipped | actions/attest-build-provenance |
| 20 | Structured Telegram alert | shipped | workflow step |
| 21 | Operator-readable audit ledger | shipped | .sdd/autoheal-history.jsonl |
| 22 | Shadow-mode quarantine + promote | shipped | core.autoheal.shadow_mode |
| 23 | One-button kill switch | shipped | .sdd/autoheal-disabled + core.autoheal.kill_switch |
| 24 | Idempotency (content hash dedupe) | shipped | core.autoheal.idempotency + branch name |
| 25 | Cordon-zone enforcement (pre-commit) | partial | CLI + module shipped; pre-commit hook deferred to v3 |
| 26 | Cost-aware degradation | shipped | cost_guard.llm_globally_disabled |
Shipped: 16. Partial: 5. Deferred: 5. Total surface: 26.
Forward-compat env knobs¶
Operators can route observability sidecars to alternative storage without forking this package:
| Env var | Purpose | Default |
|---|---|---|
BERNSTEIN_AUTOHEAL_LOG_PATH | Audit ledger destination | .sdd/autoheal-history.jsonl |
BERNSTEIN_AUTOHEAL_DECISION_LOG_PATH | Decision-log destination | .sdd/runtime/decisions.jsonl |
BERNSTEIN_AUTOHEAL_CALIBRATION_LOG_PATH | Calibration log destination | .sdd/metrics/calibration.jsonl |
BERNSTEIN_AUTOHEAL_CORDON_EXTRA | Colon-separated extra exact-allow paths | (empty) |
BERNSTEIN_AUTOHEAL_BANDIT_SEED | Integer seed for reproducible bandit picks | (fresh RNG) |
BERNSTEIN_AUTOHEAL_BUDGET_USD | Daily LLM budget cap | 1.00 |
BERNSTEIN_AUTOHEAL_DISABLE_LLM | Any non-empty value globally disables LLM paths | (unset) |
State files¶
| Path | Purpose |
|---|---|
.sdd/autoheal-disabled | Kill switch flag |
.sdd/autoheal-bandit.json | Bandit posteriors (alpha / beta per strategy) |
.sdd/autoheal-bayes.json | Bayesian per-class confidence |
.sdd/autoheal-shadow.json | Shadow-mode tally per strategy |
.sdd/autoheal-history.jsonl | Operator-readable JSONL ledger |
CLI surface (used by the workflow)¶
python scripts/auto_heal_v2_run.py triage
python scripts/auto_heal_v2_run.py check-kill-switch
python scripts/auto_heal_v2_run.py select-strategy --candidates a,b,c
python scripts/auto_heal_v2_run.py record-outcome \
--strategy ruff-format --cls safe --job Lint --outcome success
python scripts/auto_heal_v2_run.py log < record.json
python scripts/auto_heal_v2_cordon.py <path> [--whitespace-only]
python scripts/auto_heal_v2_imports.py <alias>
Operator workflow¶
Disable for an incident¶
mkdir -p .sdd
echo "2026-05-20T12:00:00Z" > .sdd/autoheal-disabled # disable until time
echo "forever" > .sdd/autoheal-disabled # disable indefinitely
rm .sdd/autoheal-disabled # re-enable
Read the audit ledger¶
jq -r '"\(.ts | localtime) \(.outcome) \(.strategy) \(.cls) \(.rationale)"' \
.sdd/autoheal-history.jsonl
Emergency revert¶
The frozen v1 workflow lives at .github/workflows/auto-heal-v1.yml with all jobs gated by if: false. To revert, swap the v2 file out of the workflows directory and re-enable v1 by replacing the static false gates with the original guards from git history.
v1 vs v2¶
| Aspect | v1 | v2 |
|---|---|---|
| Workflow file | auto-heal.yml (kept frozen as auto-heal-v1.yml) | auto-heal.yml |
| Repair strategies | 4 hardcoded recipes | bandit-selected from a growing set |
| Confidence | Flat safe/heuristic/risky | Numeric Bayesian per-class |
| Flake handling | None | non-adjacent-failure heuristic |
| State | .sdd/ not used | .sdd/autoheal-*.json{,l} |
| Cordon | YAML regex | Python module (also pre-commit) |
| Cost guard | None | cost_guard with budget cap |
| LLM path | None | Optional, behind cost-guard |
| Lineage | None | Lineage v2 child body |
| Decision log | None | Per-action entry |
| Calibration | None | Per-action probability log |
| Tests | 152 unit + integration | 178 unit + 15 property + 11 integration |