昨日回顧

Day 10 用 pytest 鎖住 URL 正規化、域名邊界、固定日期與來源設定錯誤。單元測試能回答「這個函式是否符合契約」，但 Agent Skill 還需要另一層驗證：給定一個完整案例，最終輸出的 status、evidence、normalized_host 與 next_step 是否一起正確。

今天把這些案例做成 machine-readable eval，讓規格可以獨立於 Python 測試函式被閱讀、擴充與重跑。

單元測試和 eval 的分工

單元測試適合驗證小函式與錯誤邊界：

• parser 是否拒絕 ftp
• suffix 是否保留 . 的域名邊界
• loader 是否拒絕重疊規則

Eval 則從使用者輸入開始，驗證完整決策：

• 這個 URL 應得到哪個 status
• 哪些 evidence 必須出現
• normalized_host 應該是什麼
• 下一步是否符合安全策略

兩者不是替代關係。單元測試定位快，eval 保護產品行為。

案例檔先選 JSONL

每行一個 JSON 物件，方便 diff、逐行讀取，也能在失敗時直接報 case id：

evals/
├── cases.jsonl
├── schema.json
└── run_evals.py

第一個案例：

{"id":"official-exact","url":"https://tickets.example.com/event/1","today":"2026-09-20","expect":{"status":"OFFICIAL","normalized_host":"tickets.example.com","evidence":["source:example-primary"],"next_step":"Continue from the verified official entry point"}}

案例資料只用保留給文件的 example.com，避免把真實網站當成已驗證來源。

每個案例都要有穩定 ID

id 不只是名稱，也是失敗報告、CI log 與日後趨勢追蹤的主鍵。ID 應描述行為，不綁實作：

{"id":"lookalike-suffix-is-not-official","url":"https://tickets.example.com.attacker.net/pay","today":"2026-09-20","expect":{"status":"UNCONFIRMED","normalized_host":"tickets.example.com.attacker.net","evidence":["no_verified_source_match"],"next_step":"Open the organizer's verified first-party entry point"}}

不要叫 test_host_matches_line_42。程式重構後，行為 ID 仍然成立。

先驗證案例 schema

Eval runner 不能相信 eval 自己一定寫對。最小 schema 可以要求：

{
  "type": "object",
  "required": ["id", "url", "today", "expect"],
  "properties": {
    "id": {"type": "string", "minLength": 1},
    "url": {"type": "string", "minLength": 1},
    "today": {"type": "string", "format": "date"},
    "expect": {
      "type": "object",
      "required": ["status", "normalized_host", "evidence", "next_step"],
      "properties": {
        "status": {
          "enum": ["OFFICIAL", "UNCONFIRMED", "INSUFFICIENT_INPUT"]
        },
        "normalized_host": {
          "type": ["string", "null"]
        },
        "evidence": {
          "type": "array",
          "items": {"type": "string"}
        },
        "next_step": {"type": "string"}
      },
      "additionalProperties": false
    }
  },
  "additionalProperties": false
}

Schema 防止 evidences 拼錯、日期漏寫、status 自創新值，或某個案例忘記檢查 next_step。

載入案例時 fail fast

import json
from pathlib import Path

from jsonschema import Draft202012Validator


def load_cases(path: Path, schema_path: Path) -> list[dict]:
    schema = json.loads(schema_path.read_text(encoding="utf-8"))
    validator = Draft202012Validator(schema)
    cases = []
    seen_ids = set()

    for line_no, line in enumerate(
        path.read_text(encoding="utf-8").splitlines(), start=1
    ):
        if not line.strip():
            continue

        try:
            case = json.loads(line)
        except json.JSONDecodeError as exc:
            raise EvalConfigError(
                f"cases.jsonl:{line_no}: invalid JSON: {exc.msg}"
            ) from exc

        errors = sorted(validator.iter_errors(case), key=lambda e: list(e.path))
        if errors:
            paths = [".".join(map(str, e.path)) or "$" for e in errors]
            raise EvalConfigError(
                f"cases.jsonl:{line_no}: schema errors at {', '.join(paths)}"
            )

        if case["id"] in seen_ids:
            raise EvalConfigError(f"duplicate case id: {case['id']}")

        seen_ids.add(case["id"])
        cases.append(case)

    if not cases:
        raise EvalConfigError("no eval cases found")

    return cases

和 sources loader 一樣，案例集損壞時不能繼續跑出一份看似正常的部分報告。

runner 直接呼叫純函式

from datetime import date

from scripts.check_domain import check, load_sources


def run_case(case: dict, sources: list[dict]) -> dict:
    return check(
        case["url"],
        sources,
        today=date.fromisoformat(case["today"]),
    )

不要在每個 case 啟動 subprocess。純函式 runner 更快，也不會把 shell quoting 或工作目錄問題混進分類行為。CLI 仍由 Day 10 的少量端到端測試負責。

比較完整輸出，不只 status

最危險的 eval 是只寫：

assert actual["status"] == expected["status"]

OFFICIAL status 可能正確，但 evidence 指到錯的來源；UNCONFIRMED 可能正確，但 next_step 卻叫使用者繼續付款。完整比較應涵蓋所有 load-bearing 欄位：

FIELDS = ("status", "normalized_host", "evidence", "next_step")


def compare(expected: dict, actual: dict) -> dict:
    return {
        field: {
            "expected": expected[field],
            "actual": actual.get(field),
        }
        for field in FIELDS
        if actual.get(field) != expected[field]
    }

Runner 回傳空 dict 代表 pass，否則就是結構化 diff。

Evidence 要不要在乎順序

這個系列的第一版把 evidence 視為有順序的清單，原因是順序也能表達優先級，而且固定順序能讓 JSON snapshot 穩定。

如果未來規格改成集合語意，應在 compare 層明確排序或轉 set，不能讓不同案例各自猜測。

輸出人類可讀的 diff

import json


def format_failure(case_id: str, diff: dict) -> str:
    lines = [f"FAIL {case_id}"]
    for field, values in diff.items():
        expected = json.dumps(values["expected"], ensure_ascii=False)
        actual = json.dumps(values["actual"], ensure_ascii=False)
        lines.append(f"  {field}")
        lines.append(f"    expected: {expected}")
        lines.append(f"    actual:   {actual}")
    return "\n".join(lines)

失敗報告只顯示 case id 與不一致欄位，不把整份 registry 或不相關設定倒進 CI log。

主迴圈保持確定性

def run_all(cases: list[dict], sources: list[dict]) -> int:
    failures = []

    for case in cases:
        actual = run_case(case, sources)
        diff = compare(case["expect"], actual)
        if diff:
            failures.append(format_failure(case["id"], diff))
        else:
            print(f"PASS {case['id']}")

    if failures:
        print("\n\n".join(failures))
        print(f"\n{len(failures)} failed, {len(cases) - len(failures)} passed")
        return 1

    print(f"\n{len(cases)} passed")
    return 0

案例依檔案順序執行，欄位依 FIELDS 順序輸出，不加入時間戳或隨機值。相同 commit 應得到相同報告。

四個核心案例

第一批至少包含：

{"id":"official-exact","url":"https://tickets.example.com/event/1","today":"2026-09-20","expect":{"status":"OFFICIAL","normalized_host":"tickets.example.com","evidence":["source:example-primary"],"next_step":"Continue from the verified official entry point"}}
{"id":"official-subdomain","url":"https://pay.tickets.example.com/event/1","today":"2026-09-20","expect":{"status":"OFFICIAL","normalized_host":"pay.tickets.example.com","evidence":["source:example-primary"],"next_step":"Continue from the verified official entry point"}}
{"id":"stale-source","url":"https://tickets.example.com/event/1","today":"2026-10-02","expect":{"status":"UNCONFIRMED","normalized_host":"tickets.example.com","evidence":["stale:example-primary"],"next_step":"Open the organizer's verified first-party entry point"}}
{"id":"missing-scheme","url":"tickets.example.com/event/1","today":"2026-09-20","expect":{"status":"INSUFFICIENT_INPUT","normalized_host":null,"evidence":["URL must use http or https"],"next_step":"Provide a complete HTTP(S) URL"}}

再加上 lookalike suffix、revoked 與 unknown host，就能覆蓋目前最重要的產品決策。

Eval 設定錯誤和案例失敗要分開

命令列 exit code 建議分三類：

0  # 所有案例通過
1  # runner 正常，但至少一個案例不符合預期
2  # cases/schema/sources 本身損壞，eval 無法成立

這讓 CI 能區分產品回歸和測試基礎設施故障。兩者都應阻擋合併，但診斷方向不同。

Machine-readable summary

除了 console diff，可以選擇輸出一份固定 JSON summary 給 CI artifact：

{
  "total": 7,
  "passed": 6,
  "failed": 1,
  "failures": [
    {
      "id": "stale-source",
      "fields": ["evidence", "next_step"]
    }
  ]
}

不要放執行時間、主機名或絕對路徑，否則每次執行都會產生無意義差異。

第一次執行

python evals/run_evals.py \
  --cases evals/cases.jsonl \
  --schema evals/schema.json \
  --sources references/sources.md

先故意改壞一個 expected field，確認 diff 容易讀、exit code 是 1；再破壞一行 JSON，確認 runner 在分類前以 exit code 2 結束。

明天預告

Day 12 把 pytest 與 eval runner 接進 CI，讓每次修改 SKILL.md、腳本或來源 registry 都必須通過同一套可驗證門檻。