---
title: Agent Seek Preference Eval
canonical: /eval.md
---

# Agent Seek Preference Eval

Same-pool pairwise · You.com order vs cascade · frozen suite `public_v1`.

**64% preferred** (32/50); excl. ties **67%** (32/48).

ties 2 · baseline wins 16 · n=50 · 0 errors

Judge: `gpt-5.6-sol` · `reasoning.effort=medium` · `mode=standard` · prompt `v1` · sha `1ad0946c1f7d…` · run `20260921T044121Z` · suite `1.0.0`

Static published run. This page does not call OpenAI, You.com, or TypeSafe.

## Method

1. Discover once (You.com). Both sides use that candidate list.
2. Baseline: You.com order top-k.
3. Treatment: Agent Seek cascade top-k on the same objects (`mode=deep`).
4. Blind A/B labels. Mapping is stored in run output only.
5. Judge prefers the more useful shortlist (authority, on-topic, sought fact; not scrapers or hijack pages).

Suite: [evals/public_v1/README.md](/evals/public_v1/README.md) · prompt: [PROMPT_v1.md](/evals/public_v1/judge/PROMPT_v1.md) · published: [`evals/public_v1/published/v1.0.0-first-run/`](/evals/public_v1/published/v1.0.0-first-run/summary.json)

## Leaderboard

| System | Pref. wins | Win rate | Win rate (excl. ties) |
| --- | ---: | ---: | ---: |
| Agent Seek (deep) | 32 | 64.0% | 66.7% |
| You.com order (same pool) | 16 | 32.0% | 33.3% |
| Tie | 2 | 4.0% | — |

## Tier breakdown

| Tier | n | Agent Seek | You.com order | Tie | AS win% excl. ties |
| --- | ---: | ---: | ---: | ---: | ---: |
| L1 | 15 | 10 | 4 | 1 | 71.4% |
| L2 | 20 | 14 | 6 | 0 | 70.0% |
| L3 | 15 | 8 | 6 | 1 | 57.1% |

## Per-query

| id | tier | query | winner | confidence |
| --- | --- | --- | --- | ---: |
| p1-001 | L1 | FastAPI dependency injection tutorial | Agent Seek | 0.93 |
| p1-002 | L1 | Postgres JSONB GIN index documentation | Agent Seek | 0.88 |
| p1-003 | L1 | RFC 9110 HTTP semantics | You.com order | 0.94 |
| p1-004 | L1 | React useEffect cleanup function | Agent Seek | 0.96 |
| p1-005 | L1 | Kubernetes liveness vs readiness probe | You.com order | 0.96 |
| p1-006 | L1 | SQLite WAL mode concurrency | Agent Seek | 0.82 |
| p1-007 | L1 | Cloudflare Workers KV limits | Agent Seek | 0.98 |
| p1-008 | L1 | OpenAPI 3.1 specification | Agent Seek | 0.90 |
| p1-009 | L1 | Python asyncio TaskGroup | Agent Seek | 0.92 |
| p1-010 | L1 | Redis Streams consumer groups | Agent Seek | 0.88 |
| p1-011 | L1 | GitHub Actions cache key documentation | You.com order | 0.97 |
| p1-012 | L1 | MDN Content-Security-Policy | Tie | 0.96 |
| p1-013 | L1 | Docker multi-stage build best practices | Agent Seek | 0.93 |
| p1-014 | L1 | Prometheus histogram vs summary | You.com order | 0.97 |
| p1-015 | L1 | TLS 1.3 handshake overview | Agent Seek | 0.94 |
| p1-016 | L2 | When was Next.js App Router stable | Agent Seek | 0.90 |
| p1-017 | L2 | LangChain Deep Agents release announcement | Agent Seek | 0.97 |
| p1-018 | L2 | Factory.ai Droid skills documentation | You.com order | 0.76 |
| p1-019 | L2 | Vercel Fluid compute pricing limits | Agent Seek | 0.82 |
| p1-020 | L2 | Upstash Redis REST rate limits | Agent Seek | 0.97 |
| p1-021 | L2 | Neon serverless Postgres branching docs | Agent Seek | 0.99 |
| p1-022 | L2 | Anthropic tool use JSON schema | Agent Seek | 0.99 |
| p1-023 | L2 | OpenAI structured outputs JSON schema | You.com order | 0.90 |
| p1-024 | L2 | MCP streamable HTTP transport | Agent Seek | 0.96 |
| p1-025 | L2 | OAuth 2.1 authorization code PKCE | Agent Seek | 0.88 |
| p1-026 | L2 | Cloudflare R2 egress pricing | You.com order | 0.99 |
| p1-027 | L2 | Exe.dev persistent Linux VMs | Agent Seek | 0.78 |
| p1-028 | L2 | Cursor Background Agents documentation | Agent Seek | 0.72 |
| p1-029 | L2 | Temporal workflow vs activity | Agent Seek | 0.90 |
| p1-030 | L2 | ClickHouse ReplacingMergeTree caveats | You.com order | 0.74 |
| p1-031 | L2 | Pydantic v2 model_validate vs parse_obj | You.com order | 0.84 |
| p1-032 | L2 | HTTP 103 Early Hints browser support | Agent Seek | 0.97 |
| p1-033 | L2 | WebAuthn discoverable credentials passkeys | Agent Seek | 0.95 |
| p1-034 | L2 | S3 conditional writes If-None-Match | Agent Seek | 0.95 |
| p1-035 | L2 | Postgres LISTEN NOTIFY connection pooling pitfalls | You.com order | 0.98 |
| p1-036 | L3 | Who created Redis | Agent Seek | 0.95 |
| p1-037 | L3 | Original paper introducing the Transformer attention architecture | Agent Seek | 0.98 |
| p1-038 | L3 | Official X/Twitter profile for OpenAI | Tie | 0.98 |
| p1-039 | L3 | NASA Instagram on the official Instagram host | You.com order | 0.93 |
| p1-040 | L3 | GitHub user torvalds | You.com order | 0.99 |
| p1-041 | L3 | Wikipedia English: Transformer (machine learning model) vs SEO blog summaries of Attention Is All You Need | Agent Seek | 0.98 |
| p1-042 | L3 | Apple Vision Pro US release date | Agent Seek | 0.98 |
| p1-043 | L3 | Difference between LangGraph and LangChain agents | You.com order | 0.96 |
| p1-044 | L3 | Official Solidity language documentation | Agent Seek | 0.99 |
| p1-045 | L3 | Cloudflare Workers vs Cloudflare Pages Functions | Agent Seek | 0.94 |
| p1-046 | L3 | What is prompt injection? defensive overview for LLM applications | You.com order | 0.98 |
| p1-047 | L3 | PyTorch 2.0 release date | Agent Seek | 0.78 |
| p1-048 | L3 | Who maintains the ripgrep project | You.com order | 0.97 |
| p1-049 | L3 | Official Kubernetes security announcements / CVE channel vs random aggregators | Agent Seek | 0.97 |
| p1-050 | L3 | Stripe Tax vs Stripe Tax Calculations API distinction | You.com order | 0.82 |

## Reproduce

```bash
python scripts/eval_llm_judge.py --dry-run
python scripts/eval_llm_judge.py
```

Published artifacts: `evals/public_v1/published/v1.0.0-first-run/` (`summary.json`, `run_meta.json`, `per_query.jsonl`). `evals/public_v1/results/run_*` stays gitignored.

Run started 2026-09-21T04:33:31Z · finished 2026-09-21T04:41:21Z · k=10 · max_candidates=50 · path=in_process.