Skip to content

Security pentest eval scenario

TL;DR

Item Value
Scenario ID security-pentest
Role under test security
Manifest eval/scenarios/security_pentest.yaml
Fixture root tests/fixtures/eval/security_pentest/
Ground truth tests/fixtures/eval/security_pentest/ground_truth.json
Scorer bernstein.eval.pentest_scorer:PentestScorer (precision/recall/F1)
CLI bernstein eval scenario security-pentest --model <name>
Runtime budget < 60s on the mock adapter, no network
Default thresholds precision >= 0.70, recall >= 0.70, F1 >= 0.70

What this scenario evaluates

The scenario asks the security role to enumerate vulnerabilities in a small synthetic codebase under tests/fixtures/eval/security_pentest/. Each file in app/ contains exactly one seeded vulnerability that covers an OWASP top-10 class:

# File Vulnerability Canonical alias CWE
1 app/db.py SQL injection sqli CWE-89
2 app/files.py Path traversal path_traversal CWE-22
3 app/config.py Hardcoded secret hardcoded_secret CWE-798
4 app/shell.py OS command injection command_injection CWE-78
5 app/crypto.py Weak cryptographic hash weak_crypto CWE-327
6 app/cache.py Insecure deserialization insecure_deserialization CWE-502
7 app/api.py Missing authentication missing_auth CWE-306
8 app/xml_parser.py XML external entity xxe CWE-611
9 app/fetcher.py Server-side req. forgery ssrf CWE-918
10 app/race.py Race condition (TOCTOU) race CWE-367

Marker files under tests/fixtures/eval/security_pentest/markers/ mirror this table with line numbers and remediation hints.

Scoring

The scorer compares the agent-produced report (a JSON list of {vuln_type, path} dicts) to the seeded ground truth.

A report finding matches a ground-truth entry when:

  1. The canonical vulnerability type matches (case-insensitive, aliases like sqli, SQL_Injection, sql-injection all collapse to one key - see bernstein.eval.pentest_scorer.canonicalize_vuln_type).
  2. The file path matches under fuzzy rules (paths_match):
  3. exact normalised equality, or
  4. same basename, or
  5. one path is a /-separated suffix of the other.

Each ground-truth entry can be credited to at most one report finding. Duplicate findings count as false positives.

Metrics emitted:

  • precision = TP / (TP + FP)
  • recall = TP / (TP + FN)
  • f1 = 2 * P * R / (P + R) (zero when both P and R are zero)

The scenario passes when every configured threshold in scoring.thresholds is met.

Running the scenario

Offline mock adapter (used by CI):

bernstein eval scenario security-pentest --model mock

Against a real model adapter (provided by the orchestrator):

bernstein eval scenario security-pentest --model sonnet --output run.json

The JSON output has the following top-level shape:

{
  "scenario_id": "security-pentest",
  "model": "mock",
  "passed": true,
  "duration_s": 0.0023,
  "score": {
    "precision": 1.0,
    "recall": 0.9,
    "f1": 0.947368,
    "true_positives": 9,
    "false_positives": 0,
    "false_negatives": 1
  },
  "report": [
    {"vuln_type": "sqli", "path": "app/db.py"}
  ]
}

Exit code 0 when every threshold is met, 1 otherwise.

How to add a new seed

  1. Add a new file under tests/fixtures/eval/security_pentest/app/ and make the vulnerability deterministic and self-contained.
  2. Add a marker doc under markers/ describing the seed (type, path, line, CWE, expected fix).
  3. Append the new entry to tests/fixtures/eval/security_pentest/ground_truth.json.
  4. If the type is new, extend _VULN_ALIASES in src/bernstein/eval/pentest_scorer.py so common aliases match.
  5. Update this doc and re-run the integration tests.

Why a synthetic codebase

The fixture is intentionally tiny so that the entire run completes under 60 seconds on the offline mock adapter. The seeds are non-functional placeholders (for example the hardcoded secret is a clearly fake string with FAKE in it) and are never imported by real application code.