# Agent Seek public_v1 — pairwise LLM-judge suite

Private / public-ready **evidence harness** for Progressive Disclosure Search Ranking.

**Hypothesis:** on the **same You.com discover pool**, Agent Seek’s cascade top-10 is preferred over raw You.com order top-10 by a blinded LLM judge.

This directory is the frozen suite + contract. The runner is [`scripts/eval_llm_judge.py`](../../scripts/eval_llm_judge.py). The first published run is rendered at [`/eval`](/eval) ([`/eval.md`](/eval.md)).

The legacy substring harness (`evals/queries.json` + `scripts/eval_compare.py`) is unchanged.

## Methodology

| Side | Definition |
| --- | --- |
| **Baseline** `you_com_order` | Top-`k` of the discover list in You.com order |
| **Treatment** `agent_seek_cascade` | Cascade rank with `mode=deep` (eval treatment; website demo also uses deep) on **that same** candidate list, then top-`k` |

Fairness rules:

1. **One discover per query.** Do not call search twice with different modes if that rediscovers.
2. In-process path: `YouComDiscover.search` once → `raw_as_ranked(..., k)` → `cascade_rank(...)` on the same objects.
3. HTTP path (optional): a **single** `POST /v1/search` and read `raw_results` + `results` from that response.
4. **Blind labels.** Each query shuffles A/B. The mapping is stored only in run output (`per_query.jsonl`), never sent to the judge.
5. The judge sees titles, URLs, and snippets only — no scores, flags, or system names.

Suite uses `k=10`, `max_candidates=50`, `mode=deep` (eval treatment). API/MCP default is `snip`.

## Judge (locked)

| Field | Value |
| --- | --- |
| API | OpenAI **Responses** (`POST /v1/responses`) |
| Model | `gpt-5.6-sol` (alias `gpt-5.6` also routes to Sol; this suite pins the explicit id) |
| Reasoning | `{ "effort": "medium", "mode": "standard" }` |
| Temperature | not set (reasoning-model default) |

Pro mode (`reasoning.mode=pro`) is **off** unless the runner is given an explicit override. Do not change the default pin.

Published prompt: [`judge/PROMPT_v1.md`](./judge/PROMPT_v1.md).  
JSON Schema: [`judge/schema.json`](./judge/schema.json).  
Shape example (not a live verdict): [`judge/example.json`](./judge/example.json).

The judge must output JSON only:

```json
{
  "winner": "A",
  "confidence": 0.72,
  "rationale": "List A leads with the canonical specification; list B opens on a mirror.",
  "a_first_relevant_rank": 1,
  "b_first_relevant_rank": 4
}
```

`winner` is remapped after the call to `baseline` | `agent_seek` | `tie`.

## Queries

[`queries.json`](./queries.json) — **50** items, ids `p1-001` … `p1-050`.

| Tier | Count | Role |
| --- | ---: | --- |
| `L1` | 15 | Easy — clear official docs |
| `L2` | 20 | Medium — fact-in-body / SEO fog / product clarity |
| `L3` | 15 | Hard — entity, republisher, lookalike, source quality |

Each row: `{ "id", "q", "tier", "intent" }`. No Agent Seek / TypeSafe / Jev / personal-brand queries.

## How to run

Keys stay in `.env` (never commit them):

| Variable | Used for |
| --- | --- |
| `YDC_API_KEY` | You.com discover (in-process) |
| `TYPESAFE_API_KEY` | Jev cascade (in-process) |
| `OPENAI_API_KEY` | Judge only — not used by product search |
| `AGENT_SEEK_API_KEY` + `AGENT_SEEK_BASE` | Optional HTTP path (one `/v1/search` per query) |

```bash
# Load suite, print plan, call no APIs
python scripts/eval_llm_judge.py --dry-run

# Two-query smoke (needs the keys above)
python scripts/eval_llm_judge.py --limit 2

# Full 50
python scripts/eval_llm_judge.py
```

Missing keys soft-skip (exit 0) unless `AGENT_SEEK_EVAL_STRICT=1`.

## Output

Each live run writes `evals/public_v1/results/run_<UTC>/`. Those directories are gitignored. This repo does **not** ship a filled `results.json`.

`summary.json` shape after a real run (zeros shown only as a schema — not a measured result):

```json
{
  "suite_id": "agent-seek-public-v1",
  "n": 50,
  "agent_seek_wins": null,
  "baseline_wins": null,
  "ties": null,
  "errors": null,
  "by_tier": {
    "L1": {"n": 15, "agent_seek_wins": null, "baseline_wins": null, "ties": null, "errors": null},
    "L2": {"n": 20, "agent_seek_wins": null, "baseline_wins": null, "ties": null, "errors": null},
    "L3": {"n": 15, "agent_seek_wins": null, "baseline_wins": null, "ties": null, "errors": null}
  }
}
```

Live summaries fill the counts from that run. Do not invent win rates in docs or commits.

## Offline tests

```bash
pytest -q tests/test_eval_public_v1.py
```

No OpenAI / You.com / TypeSafe calls.

## Published runs

Public page: [`/eval`](/eval) · [`/eval.md`](/eval.md). Scratch `results/run_*` stays gitignored.

| Run | Score (AS / tie / baseline) | Judge |
| --- | --- | --- |
| [v1.0.0-first-run](./published/v1.0.0-first-run/) | 32 / 2 / 16 | gpt-5.6-sol · medium |