{"id": "p1-001", "q": "FastAPI dependency injection tutorial", "tier": "L1", "intent": "official tutorial", "candidates_in": 50, "baseline_urls": ["https://fastapi.tiangolo.com/tutorial/dependencies/", "https://www.propelauth.com/post/a-practical-guide-to-dependency-injection-with-fastapis-depends", "https://oneuptime.com/blog/post/2026-01-25-dependency-injection-fastapi/view", "https://python-dependency-injector.ets-labs.org/examples/fastapi.html", "https://www.freecodecamp.org/news/how-to-implement-dependency-injection-in-fastapi/", "https://fastapi.tiangolo.com/tutorial/dependencies/classes-as-dependencies/", "https://www.fastapitutorial.com/blog/dependency-injection-fastapi/", "https://www.geeksforgeeks.org/python/dependency-injection-in-fastapi/", "https://medium.com/@azizmarzouki/mastering-dependency-injection-in-fastapi-clean-scalable-and-testable-apis-5f78099c3362", "https://christophergs.com/tutorials/ultimate-fastapi-tutorial-pt-11-dependency-injection/"], "treatment_urls": ["https://fastapi.tiangolo.com/tutorial/dependencies/", "https://fastapi.tiangolo.com/tutorial/dependencies/classes-as-dependencies/", "https://fastapi.tiangolo.com/reference/dependencies/", "https://christophergs.com/tutorials/ultimate-fastapi-tutorial-pt-11-dependency-injection/", "https://universopython.com/en/blog/fastapi-dependency-injection", "https://www.propelauth.com/post/a-practical-guide-to-dependency-injection-with-fastapis-depends", "https://python.plainenglish.io/mastering-dependency-injection-in-fastapi-a-practical-guide-9fae69dbc68d?gi=e1596947c612", "https://www.freecodecamp.org/news/how-to-implement-dependency-injection-in-fastapi/", "https://fastapi.tiangolo.com/tutorial/dependencies/sub-dependencies/", "https://www.geeksforgeeks.org/python/dependency-injection-in-fastapi/"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.93, "rationale": "Both lead with the canonical FastAPI tutorial, but B places additional official FastAPI pages on classes, Depends/Security, and sub-dependencies prominently, providing a more authoritative and comprehensive shortlist.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 0, "rank_ms": 4300, "judge_ms": 3296, "error": null, "fetch_ms": 2246, "cache_hit": true, "judge_id": "resp_01fc4cc20372ee4c006ab0b3a05f6887d299c77e2c9e7f61d1", "elapsed_ms": 7636}
{"id": "p1-002", "q": "Postgres JSONB GIN index documentation", "tier": "L1", "intent": "official index docs", "candidates_in": 50, "baseline_urls": ["https://www.postgresql.org/docs/current/gin.html", "https://pganalyze.com/blog/gin-index", "https://www.crunchydata.com/blog/indexing-jsonb-in-postgres", "https://medium.com/google-cloud/jsonb-and-gin-index-operators-in-postgresql-cea096fbb373", "https://neon.com/postgresql/indexes/json-index", "https://www.datacamp.com/doc/postgresql/gin-indexes", "https://www.tigerdata.com/learn/how-to-index-json-columns-in-postgresql", "https://medium.com/@josef.machytka/postgresql-jsonb-operator-classes-of-gin-indexes-and-their-usage-0bf399073a4c", "https://martendb.io/documents/indexing/gin-gist-indexes", "https://dev.to/antozanini/postgresql-gin-index-complete-guide-8g3"], "treatment_urls": ["https://www.postgresql.org/docs/16/gin-builtin-opclasses.html", "https://www.postgresql.org/docs/current/gin.html", "https://www.postgresql.org/docs/17/gin.html", "https://www.postgresql.org/docs/9.4/datatype-json.html"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.88, "rationale": "B is entirely official PostgreSQL documentation and includes a JSONB-specific page with supported GIN operators and an index example. A offers broader practical coverage, but most results are secondary sources.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 0, "rank_ms": 3743, "judge_ms": 5641, "error": null, "fetch_ms": 1581, "cache_hit": true, "judge_id": "resp_0aa8af8aa91785d4006ab0b3a794d087d28450c5a5a26ef760", "elapsed_ms": 9385}
{"id": "p1-003", "q": "RFC 9110 HTTP semantics", "tier": "L1", "intent": "canonical RFC", "candidates_in": 50, "baseline_urls": ["https://www.rfc-editor.org/rfc/rfc9110.html", "https://datatracker.ietf.org/doc/html/rfc9110", "https://httpwg.org/specs/rfc9110.html", "https://www.rfc-editor.org/info/rfc9110/", "https://dl.acm.org/doi/10.17487/RFC9110", "https://www.arc-it.net/html/standards/standard1500.html", "https://tools.ietf.org/html/rfc9110", "https://tex2e.github.io/rfc-translater/html/rfc9110.html", "https://www.rfc-editor.org/rfc/rfc9110.pdf", "https://www.ietf.org/rfc/rfc9110.pdf"], "treatment_urls": ["https://www.rfc-editor.org/rfc/rfc9110.html", "https://www.rfc-editor.org/rfc/rfc9110.xml", "https://datatracker.ietf.org/doc/html/rfc9110", "https://www.rfc-editor.org/info/rfc9110/", "https://www.rfc-editor.org/refs/ref9110.txt", "https://rfc-explained.com/en/9110", "https://www.ietf.org/rfc/rfc9110.pdf", "https://statuscodefyi.com/guides/rfc-9110-http-semantics/", "https://ipfyi.com/rfc/9110/", "https://www.rfc-editor.org/rfc/rfc9110.pdf"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "B", "confidence": 0.94, "rationale": "Both lead with the canonical RFC Editor page, but B offers more authoritative full-text alternatives near the top, including IETF Datatracker and HTTP Working Group, while A includes a references file and several secondary explainer sites.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 639, "rank_ms": 2974, "judge_ms": 3736, "error": null, "fetch_ms": 747, "cache_hit": false, "judge_id": "resp_0110d46915a90f62006ab0b3b1175c87d2a0743fe73a4babe6", "elapsed_ms": 7351}
{"id": "p1-004", "q": "React useEffect cleanup function", "tier": "L1", "intent": "official React docs", "candidates_in": 50, "baseline_urls": ["https://blog.logrocket.com/understanding-react-useeffect-cleanup-function/", "https://react.dev/reference/react/useEffect", "https://dev.to/werliton/useeffect-with-cleanup-function-in-react-what-it-is-when-to-use-it-and-why-k18", "https://medium.com/@vishalkalia.er/what-is-the-useeffect-cleanup-function-and-how-it-works-83d8c67a1a10", "https://refine.dev/blog/useeffect-cleanup/", "https://reacttraining.com/blog/useEffect-cleanup", "https://www.reddit.com/r/reactjs/comments/prvg2k/whats_the_purpose_of_cleanup_function_in_useeffect/", "https://stackoverflow.com/questions/57023074/why-is-the-cleanup-function-from-useeffect-called-on-every-render", "https://www.zipy.ai/blog/understanding-react-useeffect-cleanup-function", "https://www.youtube.com/watch?v=50EJWAnVQtM"], "treatment_urls": ["https://react.dev/reference/react/useEffect", "https://legacy.reactjs.org/docs/hooks-effect.html", "https://reacttraining.com/blog/useEffect-cleanup", "https://blog.logrocket.com/understanding-react-useeffect-cleanup-function/", "https://refine.dev/blog/useeffect-cleanup/", "https://dev.to/logrocket/understanding-reacts-useeffect-cleanup-function-1ek5", "https://hackernoon.com/cleanup-functions-in-reacts-useeffect-hook-explained", "https://dev.to/bradwestfall/the-useeffect-cleanup-and-the-two-circumstances-its-called-1n80", "https://github.com/learn-co-curriculum/react-hooks-use-effect-cleanup", "https://dev.to/otamnitram/react-useeffect-cleanup-how-and-when-to-use-it-2hbm"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.96, "rationale": "Both are relevant, but A ranks the canonical React reference first and the official legacy guide second. B leads with a third-party blog and places the primary source second, making A the stronger research shortlist.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 673, "rank_ms": 7368, "judge_ms": 2797, "error": null, "fetch_ms": 5406, "cache_hit": false, "judge_id": "resp_0e324ad73a2e2c16006ab0b3bd17c087d2a8f1e62583c69214", "elapsed_ms": 10840}
{"id": "p1-005", "q": "Kubernetes liveness vs readiness probe", "tier": "L1", "intent": "official k8s docs", "candidates_in": 50, "baseline_urls": ["https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/", "https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/", "https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes", "https://distantjob.com/blog/kubernetes-readiness-vs-liveness-probe/", "https://stackoverflow.com/questions/55423405/k8s-livenessprobe-vs-readinessprobe", "https://medium.com/@jrkessl/readiness-vs-liveness-probes-what-is-the-difference-and-startup-probes-215560f043e4", "https://www.baeldung.com/ops/kubernetes-livenessprobe-readinessprobe", "https://dev.to/sagarmaheshwary/kubernetes-liveness-vs-readiness-probes-what-they-actually-mean-3b3l", "https://medium.com/@muppedaanvesh/a-hands-on-guide-to-kubernetes-probes-liveness-readiness-and-startup-probes-ee047ebab504", "https://www.reddit.com/r/kubernetes/comments/wayj42/what_should_readiness_liveness_probe_actually/"], "treatment_urls": ["https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes", "https://distantjob.com/blog/kubernetes-readiness-vs-liveness-probe/", "https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/", "https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/", "https://www.reddit.com/r/kubernetes/comments/twutj8/the_difference_between_liveness_and_readiness/", "https://www.groundcover.com/blog/kubernetes-liveness-probe", "https://stackharbor.com/en/knowledge-base/k8s-liveness-readiness-startup-probes/", "https://stackoverflow.com/questions/55423405/k8s-livenessprobe-vs-readinessprobe", "https://resolve.ai/glossary/how-to-debug-kubernetes-probe-issues", "https://spacelift.io/blog/kubernetes-readiness-probe"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "B", "confidence": 0.96, "rationale": "List B leads with two canonical Kubernetes documentation pages covering probe behavior and configuration, followed by Google’s direct comparison. List A’s first result is useful, but official Kubernetes sources are ranked lower.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 863, "rank_ms": 4010, "judge_ms": 3745, "error": null, "fetch_ms": 1721, "cache_hit": false, "judge_id": "resp_02e1a1f2ab62b38e006ab0b3c50cc487d2a9bcb372674cced6", "elapsed_ms": 8619}
{"id": "p1-006", "q": "SQLite WAL mode concurrency", "tier": "L1", "intent": "official SQLite docs", "candidates_in": 50, "baseline_urls": ["https://www.sqlite.org/wal.html", "https://mohit-bhalla.medium.com/understanding-wal-mode-in-sqlite-boosting-performance-in-sql-crud-operations-for-ios-5a8bd8be93d2", "https://coddy.tech/docs/sqlite/wal-mode-and-concurrency", "https://fly.io/blog/sqlite-internals-wal/", "https://news.ycombinator.com/item?id=32579866", "https://oldmoe.blog/2024/07/08/the-write-stuff-concurrent-write-transactions-in-sqlite/", "https://www.sqliteforum.com/p/sqlite-wal-internals-frames-commits", "https://dev.to/ahmet_gedik778845/sqlite-wal-mode-for-better-concurrent-web-performance-4fck", "https://tenthousandmeters.com/blog/sqlite-concurrent-writes-and-database-is-locked-errors/", "https://micrologics.org/blog/sqlite-in-production-optimizing-wal-mode-concurrency-and-vfs-layers-for-low-latency-app-servers"], "treatment_urls": ["https://www.sqlite.org/wal.html", "https://coddy.tech/docs/sqlite/wal-mode-and-concurrency", "https://adhdecode.com/articles/sqlite/sqlite-wal-mode-explained/", "https://www.sqliteforum.com/p/sqlite-wal-internals-frames-commits", "https://www.sqlite.org/walformat.html", "https://sqlite.org/forum/info/b4e8b29ae409cd198652c6b7e70b53b702f269e67e1d2573d627feeba37bbf85", "https://stackoverflow.com/questions/48163910/will-database-file-be-corrupted-if-concurrent-write-happened-in-sqlite3-wal-mode", "https://sqlite.org/forum/info/461653af585fb599"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.82, "rationale": "Both lead with SQLite’s canonical WAL documentation. B maintains tighter focus and adds official SQLite file-format and forum material addressing concurrent clients, snapshots, and writer blocking, while A includes more secondary blogs and discussion aggregation.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 696, "rank_ms": 7065, "judge_ms": 4711, "error": null, "fetch_ms": 5118, "cache_hit": false, "judge_id": "resp_0f4d67d2f16ecb1d006ab0b3d0cec887d283ab9fdf16d4baef", "elapsed_ms": 12474}
{"id": "p1-007", "q": "Cloudflare Workers KV limits", "tier": "L1", "intent": "official KV limits", "candidates_in": 50, "baseline_urls": ["https://developers.cloudflare.com/kv/platform/limits/", "https://developers.cloudflare.com/workers/platform/limits/", "https://developers.cloudflare.com/workers/platform/pricing/", "https://developers.cloudflare.com/kv/", "https://blog.cloudflare.com/workers-kv-free-tier/", "https://developers.cloudflare.com/kv/reference/faq/", "https://eastondev.com/blog/en/posts/dev/20260422-cloudflare-workers-kv-guide/", "https://basekv.com/articles/cloudflare-workers-kv-limits-2026", "https://developers.cloudflare.com/kv/platform/pricing/", "https://blog.cloudflare.com/workers-kv-is-ga/"], "treatment_urls": ["https://developers.cloudflare.com/kv/platform/limits/", "https://developers.cloudflare.com/kv/platform/pricing/", "https://developers.cloudflare.com/kv/api/write-key-value-pairs/", "https://developers.cloudflare.com/kv/reference/faq/", "https://developers.cloudflare.com/changelog/post/2025-01-27-kv-increased-namespaces-limits/", "https://developers.cloudflare.com/workers/platform/storage-options/", "https://blog.cloudflare.com/workers-kv-is-ga/", "https://developers.cloudflare.com/workers/platform/limits/", "https://developers.cloudflare.com/kv/concepts/how-kv-works/", "https://community.cloudflare.com/t/kv-workers-kv-namespace-limits-increased-to-1000/897306"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.98, "rationale": "Both lead with the exact canonical limits page, but A provides a stronger supporting shortlist of official, KV-specific pages covering pricing, write limits, FAQs, namespace changes, and architecture. B includes more generic, outdated, and third-party results.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 615, "rank_ms": 2582, "judge_ms": 3108, "error": null, "fetch_ms": 540, "cache_hit": false, "judge_id": "resp_0d8e9ba586f0a9f2006ab0b3d8f1b887d2a6b23ea76ef3d21a", "elapsed_ms": 6307}
{"id": "p1-008", "q": "OpenAPI 3.1 specification", "tier": "L1", "intent": "canonical spec", "candidates_in": 50, "baseline_urls": ["https://spec.openapis.org/oas/v3.1.0.html", "https://swagger.io/specification/v3.2/", "https://spec.openapis.org/oas/v3.1.1.html", "https://www.apimatic.io/blog/2021/09/migrating-to-and-from-openapi-3-1", "https://github.com/OAI/OpenAPI-Specification/releases", "https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.1.0.md", "https://nisp.nw3.dk/standard/oai-openapi-3.1.0.html", "https://www.openapis.org/blog/2021/02/18/openapi-specification-3-1-released", "https://spec.openapis.org/oas/v3.2.0.html", "https://dev.to/mikeralphson/openapi-31-the-gnarly-bits-58d0"], "treatment_urls": ["https://spec.openapis.org/oas/v3.1.0.html", "https://spec.openapis.org/oas/v3.1.1.html", "https://www.openapis.org/blog/2021/02/18/openapi-specification-3-1-released", "https://spec.openapis.org/oas/v3.1.0", "https://spec.openapis.org/oas/", "https://www.openapis.org/blog/2021/02/16/migrating-from-openapi-3-0-to-3-1-0", "https://learn.openapis.org/upgrading/v3.0-to-v3.1.html", "https://nisp.nw3.dk/standard/oai-openapi-3.1.0.html", "https://spec.openapis.org/oas/v3.2.0.html", "https://www.linux.com/news/openapi-specification-3-1-0-available-now/"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.9, "rationale": "Both lead with the canonical 3.1.0 specification, but A follows with 3.1.1 and several official OpenAPI resources. B places an off-version Swagger 3.2 page second, making it a less focused shortlist.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 784, "rank_ms": 3963, "judge_ms": 4764, "error": null, "fetch_ms": 1835, "cache_hit": false, "judge_id": "resp_0eca6f18fde6409b006ab0b3e112d487d2917acd0ab89d677c", "elapsed_ms": 9513}
{"id": "p1-009", "q": "Python asyncio TaskGroup", "tier": "L1", "intent": "official Python docs", "candidates_in": 50, "baseline_urls": ["https://docs.python.org/3/library/asyncio-task.html", "https://www.reddit.com/r/learnpython/comments/123iw6q/whats_the_advantage_of_using_asynciotaskgroup/", "https://superfastpython.com/asyncio-taskgroup/", "https://www.geeksforgeeks.org/python/python-taskgroups-with-asyncio/", "https://dev.to/koladev/creating-and-managing-tasks-with-asyncio-4kjl", "https://sglmr.com/blog/python-asyncio-taskgroup-by-example/", "https://www.dataleadsfuture.com/why-taskgroup-and-timeout-are-so-crucial-in-python-3-11-asyncio/", "https://bruceeckel.substack.com/p/python-311-asynciotaskgroup", "https://stackoverflow.com/questions/42231161/asyncio-gather-vs-asyncio-wait-vs-asyncio-taskgroup", "https://jacobpadilla.com/writing/handling-asyncio-tasks"], "treatment_urls": ["https://docs.python.org/3/library/asyncio-task.html", "https://sglmr.com/blog/python-asyncio-taskgroup-by-example/", "https://superfastpython.com/asyncio-taskgroup/", "https://universopython.com/en/blog/python-asyncio-taskgroup", "https://runebook.dev/en/docs/python/library/asyncio-task/asyncio.TaskGroup", "https://runebook.dev/en/docs/python/library/asyncio-task/asyncio.TaskGroup.create_task", "https://towardsdatascience.com/why-taskgroup-and-timeout-are-so-crucial-in-python-3-11-asyncio-c424bcc88b89/", "https://realpython.com/python311-exception-groups/", "https://www.python3.info/advanced/async-asyncio/task-group.html", "https://bugs.python.org/issue46752"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.92, "rationale": "Both lead with the canonical Python documentation, but A offers a stronger, more focused supporting set, including detailed tutorials, Real Python coverage, and the primary implementation issue. B ranks Reddit second and includes more general or less authoritative material.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 636, "rank_ms": 3959, "judge_ms": 3009, "error": null, "fetch_ms": 1993, "cache_hit": false, "judge_id": "resp_039302d63bd0aa78006ab0b3eaa98c87d281ec96f92150090d", "elapsed_ms": 7606}
{"id": "p1-010", "q": "Redis Streams consumer groups", "tier": "L1", "intent": "official Redis docs", "candidates_in": 50, "baseline_urls": ["https://redis.io/docs/latest/develop/tools/insight/insight-stream-consumer/", "https://medium.com/redis-with-raphael-de-lio/understanding-redis-streams-33aa96ca7206", "https://redis.io/docs/latest/develop/data-types/streams/", "https://oneuptime.com/blog/post/2026-01-30-redis-streams-consumer-groups/view", "https://redisson.pro/glossary/redis-streams.html", "https://faststream.ag2.ai/latest/redis/streams/groups/", "https://stackoverflow.com/questions/70306238/upper-limit-on-number-of-redis-streams-consumer-groups", "https://redis.antirez.com/fundamental/streams-consumer-patterns.html", "https://docs.spring.io/spring-data/redis/reference/redis/redis-streams.html", "https://medium.com/@yudhasubki/redis-stream-simple-reliable-and-scalable-for-real-time-data-processing-692985a02a65"], "treatment_urls": ["https://redis.io/docs/latest/commands/xreadgroup/", "https://redis.io/docs/latest/commands/xinfo-consumers/", "https://redis.io/blog/youre-probably-thinking-about-redis-streams-wrong/", "https://redis.io/docs/latest/commands/xinfo-groups/", "https://redis.io/docs/latest/commands/xpending/", "https://www.infoworld.com/article/2257824/how-to-use-consumer-groups-in-redis-streams.html", "https://redisson.pro/glossary/redis-streams.html", "https://redis.io/docs/latest/develop/use-cases/streaming/", "https://redis.antirez.com/fundamental/streams-consumer-patterns.html", "https://academy.jatinjainsaraf.com/redis-in-depth/streams-consumer-groups"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.88, "rationale": "A leads with authoritative Redis command documentation central to consumer groups, then covers consumers, group status, pending entries, and practical guidance. B is useful, but begins with narrower Redis Insight tooling and places the canonical Streams overview third.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 658, "rank_ms": 5177, "judge_ms": 5175, "error": null, "fetch_ms": 3144, "cache_hit": false, "judge_id": "resp_0ff440ef7827608c006ab0b3f3c72087d2827fe111e3be9f04", "elapsed_ms": 11011}
{"id": "p1-011", "q": "GitHub Actions cache key documentation", "tier": "L1", "intent": "official Actions docs", "candidates_in": 50, "baseline_urls": ["https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching", "https://github.com/actions/cache", "https://github.com/marketplace/actions/cache", "https://docs.github.com/en/actions/how-tos/manage-workflow-runs/manage-caches", "https://stackoverflow.com/questions/55110729/how-do-i-cache-steps-in-github-actions", "https://github.com/orgs/community/discussions/25027", "https://michaelcurrin.github.io/code-cookbook/recipes/ci-cd/github-actions/workflows/cache.html", "https://graphite.com/guides/github-actions-caching", "https://starsling.dev/best-practices/github-actions/cache-dependencies", "https://docs.github.com/en/rest/actions/cache"], "treatment_urls": ["https://runs-on.com/github-actions/cache-keys-and-scopes/", "https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching", "https://docs.github.com/en/enterprise-cloud@latest/actions/reference/workflows-and-actions/dependency-caching", "https://eastondev.com/blog/en/posts/dev/20260407-github-actions-cache-strategy/", "https://runs-on.com/github-actions/caching-dependencies/", "https://graphite.com/guides/in-depth-guide-ci-cd-cache-github-actions", "https://blog.stephane-robert.info/en/docs/pipeline-cicd/github/optimiser/cache/", "https://github.com/actions/cache", "https://github.com/actions/cache/blob/master/examples.md", "https://github.com/marketplace/actions/cache"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "A", "confidence": 0.97, "rationale": "A leads with GitHub’s canonical dependency-caching documentation and follows with the official actions/cache repository. B’s first result is relevant but third-party, while its official documentation starts at rank 2 and includes a duplicate variant.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 684, "rank_ms": 4098, "judge_ms": 2626, "error": null, "fetch_ms": 2098, "cache_hit": false, "judge_id": "resp_0dada4a4d76b69e9006ab0b3fdf57c87d28d45d62ea0d74730", "elapsed_ms": 7411}
{"id": "p1-012", "q": "MDN Content-Security-Policy", "tier": "L1", "intent": "MDN CSP docs", "candidates_in": 46, "baseline_urls": ["https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CSP", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy", "https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Content_Security_Policy", "https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implementation_guides/CSP", "https://github.com/mdn/content/blob/main/files/en-us/web/http/reference/headers/content-security-policy/index.md?plain=1", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/script-src", "https://cheatsheetseries.owasp.org/cheatsheets/Content_Security_Policy_Cheat_Sheet.html", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy-Report-Only", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/frame-ancestors", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/block-all-mixed-content"], "treatment_urls": ["https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CSP", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/default-src", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/report-to", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/script-src", "https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implementation_guides/CSP", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/frame-ancestors", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/sandbox", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/base-uri", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/connect-src"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "tie", "confidence": 0.96, "rationale": "Both lists lead with the same authoritative MDN CSP guide and Content-Security-Policy header reference. A offers broader supporting resources, while B offers more directive-specific MDN pages; neither has a meaningful advantage for this broad query.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "tie", "ranking": "jev", "discover_ms": 627, "rank_ms": 3369, "judge_ms": 4467, "error": null, "fetch_ms": 1266, "cache_hit": false, "judge_id": "resp_06ecc834f5d6a133006ab0b404d7d487d2bc38fcf292abcaa4", "elapsed_ms": 8466}
{"id": "p1-013", "q": "Docker multi-stage build best practices", "tier": "L1", "intent": "official Docker docs", "candidates_in": 50, "baseline_urls": ["https://docs.docker.com/build/building/multi-stage/", "https://oluoch-odhiambo.medium.com/docker-multi-stage-build-an-effective-strategy-to-building-production-ready-docker-images-59fda6e94e0", "https://cycle.io/learn/multi-stage-builds", "https://labs.iximiuz.com/tutorials/docker-multi-stage-builds", "https://www.blacksmith.sh/blog/understanding-multi-stage-docker-builds", "https://oneuptime.com/blog/post/2026-02-02-docker-multi-stage-builds/view", "https://www.cleanstart.com/guide/multi-stage-build", "https://spacelift.io/blog/docker-multistage-builds", "https://depot.dev/blog/docker-multi-stage-builds", "https://www.reddit.com/r/docker/comments/1l27pv4/should_i_use_multistage_dockerfiles_or_separate/"], "treatment_urls": ["https://www.docker.com/blog/advanced-dockerfiles-faster-builds-and-smaller-images-using-buildkit-and-multistage-builds/", "https://www.docker.com/blog/multi-stage-builds/", "https://docs.docker.com/guides/cpp/multistage/", "https://docs.docker.com/build/building/best-practices/", "https://docs.docker.com/build/building/multi-stage/", "https://oneuptime.com/blog/post/2026-02-02-docker-multi-stage-builds/view", "https://dev.to/dpvasani56/docker-multi-stage-build-complete-documentation-4990", "https://www.techtarget.com/searchitoperations/tip/A-beginners-guide-to-a-multistage-Docker-build"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.93, "rationale": "A offers substantially more authoritative coverage: multiple official Docker Docs and Docker blog pages, including a dedicated best-practices guide and advanced multi-stage patterns. B starts with canonical documentation but relies mostly on third-party articles afterward.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 655, "rank_ms": 4572, "judge_ms": 4124, "error": null, "fetch_ms": 2572, "cache_hit": false, "judge_id": "resp_0f8f2df3a3e06f6f006ab0b40ec86487d2b299151c44d00724", "elapsed_ms": 9353}
{"id": "p1-014", "q": "Prometheus histogram vs summary", "tier": "L1", "intent": "official Prometheus docs", "candidates_in": 50, "baseline_urls": ["https://prometheus.io/docs/practices/histograms/", "https://prometheus.io/docs/concepts/metric_types/", "https://victoriametrics.com/blog/prometheus-monitoring-metrics-counters-gauges-histogram-summaries/", "https://stackoverflow.com/questions/51146578/what-use-cases-really-make-prometheuss-summary-metrics-type-necessary-unique", "https://last9.io/blog/histogram-buckets-in-prometheus/", "https://oneuptime.com/blog/post/2026-01-26-prometheus-histograms-summaries/view", "https://blog.devops.dev/use-cases-for-prometheus-histograms-and-summary-metrics-9cd14c75615e?gi=772e7e340095", "https://prometheus.io/docs/tutorials/understanding_metric_types/", "https://www.tigerdata.com/blog/four-types-prometheus-metrics-to-collect", "https://dyladan.me/histograms/2023/05/03/histograms-vs-summaries/"], "treatment_urls": ["https://prometheus.io/docs/tutorials/understanding_metric_types/", "https://adhdecode.com/articles/prometheus/prometheus-histogram-vs-summary-when-to-use/", "https://prometheus.io/docs/concepts/metric_types/", "https://prometheus.io/docs/practices/histograms/", "https://www.robustperception.io/how-does-a-prometheus-histogram-work/", "https://github.com/prometheus/docs/blob/main/docs/practices/histograms.md", "https://stackoverflow.com/questions/71768510/when-to-use-gauge-or-histogram-in-prometheus-in-recording-request-duration", "https://grafana.com/blog/2022/03/01/how-summary-metrics-work-in-prometheus/", "https://docs.rs/prometheus/latest/prometheus/struct.Histogram.html", "https://prometheus.github.io/client_java/getting-started/metric-types/"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "A", "confidence": 0.97, "rationale": "A ranks Prometheus’s canonical “Histograms and summaries” comparison first, followed by official metric documentation and several directly comparative resources. B is relevant, but places the definitive official comparison fourth behind a lesser-known third-party article.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 648, "rank_ms": 3486, "judge_ms": 3355, "error": null, "fetch_ms": 1471, "cache_hit": false, "judge_id": "resp_04d746c6cbca2f7b006ab0b4174bac87d2b4246d5fb2af024c", "elapsed_ms": 7491}
{"id": "p1-015", "q": "TLS 1.3 handshake overview", "tier": "L1", "intent": "authoritative TLS overview", "candidates_in": 50, "baseline_urls": ["https://www.thesslstore.com/blog/tls-1-3-handshake-tls-1-2/", "https://tls13.xargs.org/", "https://www.cloudflare.com/learning/ssl/what-happens-in-a-tls-handshake/", "https://medium.com/@fehzanvayani/tls-1-3-handshake-aa1c9aca126d", "https://www.thesecurecoder.com/p/tls-1-3-handshake-explained-simply", "https://cabulous.medium.com/tls-1-2-andtls-1-3-handshake-walkthrough-4cfd0a798164", "https://www.ibm.com/docs/en/sdk-java-technology/8?topic=works-tls-13-handshake", "https://www.paloaltonetworks.com/cyberpedia/what-is-a-tls-handshake", "https://www.qacafe.com/resources/examples-of-tls-1-3/", "https://www.logicmonitor.com/deep-dive/http3-vs-http2/tls1-2-vs-1-3"], "treatment_urls": ["https://systeminternals.dev/security/tls-handshake/", "https://www.paolotagliaferri.com/overview-of-transport-layer-security-protocol-tls-1-3/", "https://tools.ietf.org/html/rfc8446", "https://link.springer.com/article/10.1007/s00145-021-09384-1", "https://blog.cloudflare.com/tls-1-3-overview-and-q-and-a/", "https://www.expressvpn.com/glossary/tls-handshake/", "https://www.thesslstore.com/blog/tls-1-3-handshake-tls-1-2/", "https://www.thesslstore.com/blog/tls-1-3-everything-possibly-needed-know/", "https://www.qacafe.com/resources/examples-of-tls-1-3/", "https://www.davidwong.fr/tls13/"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.94, "rationale": "A leads with a direct step-by-step TLS 1.3 handshake overview and places the authoritative RFC 8446 at rank 3, supplemented by Cloudflare and a readable specification. B is relevant but starts with a commercial comparison and lacks the primary standard.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 699, "rank_ms": 7612, "judge_ms": 2880, "error": null, "fetch_ms": 5394, "cache_hit": false, "judge_id": "resp_0441433a806faff0006ab0b423357c87d2af1167fb8fe3abe4", "elapsed_ms": 11193}
{"id": "p1-016", "q": "When was Next.js App Router stable", "tier": "L2", "intent": "product milestone date", "candidates_in": 50, "baseline_urls": ["https://nextjs.org/blog/next-13-4", "https://meisteritsystems.com/news/next-js-app-router-in-2026-is-it-ready-for-production/", "https://dev.to/ottoaria/nextjs-app-router-in-2026-the-complete-guide-for-full-stack-developers-5bjl", "https://dev.to/thekitbase/nextjs-app-router-best-practices-in-2026-2b0f", "https://nextjs.org/blog/june-2023-update", "https://en.wikipedia.org/wiki/Next.js", "https://www.kunalganglani.com/blog/nextjs-app-router-vs-pages-router", "https://thenewstack.io/why-developers-should-give-next-js-app-router-another-chance/", "https://nextjs.org/blog/next-14", "https://nextjs.org/blog"], "treatment_urls": ["https://nextjs.org/blog/next-13-4", "https://nextjs.org/blog/june-2023-update", "https://en.wikipedia.org/wiki/Next.js", "https://dev.to/skyz03/the-evolution-of-nextjs-from-inception-to-cutting-edge-framework-2837", "https://clerk.com/blog/stable-next-js-app-router-support", "https://engineering.udacity.com/what-is-next-js-app-router-is-it-ready-for-the-main-stage-bed07ef9519f?gi=8374c0ed05de", "https://www.inngest.com/blog/5-lessons-learned-from-taking-next-js-app-router-to-production", "https://dev.to/thekitbase/nextjs-app-router-best-practices-in-2026-2b0f", "https://oneuptime.com/blog/post/2026-02-02-nextjs-app-router/view", "https://www.reddit.com/r/nextjs/comments/13cy3t3/what_is_nextjs_app_router_amp_is_it_ready_for_the/"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.9, "rationale": "Both lead with the definitive official Next.js 13.4 announcement, but A ranks the official follow-up second and includes more directly relevant corroboration that App Router became stable in version 13.4 in May 2023.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 674, "rank_ms": 3892, "judge_ms": 2972, "error": null, "fetch_ms": 1889, "cache_hit": false, "judge_id": "resp_0c49eeb9f7cefc6f006ab0b42af15c87d2850974944055b9a7", "elapsed_ms": 7540}
{"id": "p1-017", "q": "LangChain Deep Agents release announcement", "tier": "L2", "intent": "first-party launch post", "candidates_in": 50, "baseline_urls": ["https://www.langchain.com/blog/august-2026-langchain-newsletter", "https://aitoolly.com/ai-news/article/2026-08-13-langchain-introduces-managed-deep-agents-a-new-standard-for-building-and-deploying-ai-agents", "https://www.prnewswire.com/news-releases/langchain-and-nvidia-launch-nemoclaw-deep-agents-blueprint-for-enterprise-agents-302820446.html", "https://changelog.langchain.com/announcements/deepagents-0-2-release-for-more-autonomous-agents", "https://pub.towardsai.net/langchain-just-released-deep-agents-and-it-changes-how-you-build-ai-systems-cc2371b04714?gi=0a32bfd3f72c", "https://blockchain.news/news/langchain-deep-agents-multi-agent-framework-release", "https://changelog.langchain.com/", "https://www.langchain.com/blog/deep-agents-v0-7", "https://www.langchain.com/blog/langchain-and-nvidia-launch-the-nemoclaw-deep-agents-blueprint", "https://github.com/langchain-ai/deepagents/releases"], "treatment_urls": ["https://www.langchain.com/blog/doubling-down-on-deepagents", "https://awesomeagents.ai/news/langchain-deep-agents-release/", "https://www.langchain.com/blog/deep-agents-v0-5", "https://www.marktechpost.com/2026/03/15/langchain-releases-deep-agents-a-structured-runtime-for-planning-memory-and-context-isolation-in-multi-step-ai-agents/", "https://www.langchain.com/blog/deep-agents-v0-7", "https://www.langchain.com/blog/august-2026-langchain-newsletter", "https://www.langchain.com/blog/deep-agents-0-6", "https://www.langchain.com/deep-agents", "https://www.langchain.com/blog/managed-deep-agents-is-now-in-public-beta", "https://www.langchain.com/blog/deep-agents"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.97, "rationale": "B leads with an official LangChain Deep Agents release post and includes the canonical original Deep Agents announcement, product page, and several official version announcements. A’s earliest relevant result is an official v0.2 changelog at rank 4.", "a_first_relevant_rank": 4, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 664, "rank_ms": 3242, "judge_ms": 5017, "error": null, "fetch_ms": 1403, "cache_hit": false, "judge_id": "resp_0d4fade4fab6da26006ab0b4320e5887d2a5d0c9b8751cda40", "elapsed_ms": 8925}
{"id": "p1-018", "q": "Factory.ai Droid skills documentation", "tier": "L2", "intent": "official product docs", "candidates_in": 50, "baseline_urls": ["https://docs.factory.ai/cli/configuration/skills", "https://docs.factory.ai/", "https://docs.factory.ai/harness/skills", "https://github.com/Factory-AI/skills", "https://docs.factory.ai/guides/skills/browser", "https://sidbharath.com/blog/factory-ai-guide/", "https://www.siddharthbharath.com/factory-ai-guide/", "https://docs.factory.ai/droid-cli/overview", "https://docs.factory.ai/cli/configuration/custom-droids", "https://docs.factory.ai/droid-cli/settings"], "treatment_urls": ["https://docs.factory.ai/cli/configuration/skills", "https://docs.factory.ai/harness/plugins", "https://docs.factory.ai/harness/skills", "https://docs.factory.ai/guides/skills/browser", "https://docs.factory.ai/droid-cli/overview", "https://docs.factory.ai/harness/subagents", "https://mcpservers.org/agent-skills/author/factory-ai", "https://sidbharath.com/blog/factory-ai-guide/", "https://www.siddharthbharath.com/factory-ai-guide/", "https://factory.mintlify.app/welcome"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "B", "confidence": 0.76, "rationale": "Both lead with the exact official skills documentation. B is stronger overall because it adds Factory’s official skills GitHub repository and avoids A’s third-party MCP aggregator and likely legacy Mintlify page.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 578, "rank_ms": 3316, "judge_ms": 5356, "error": null, "fetch_ms": 1534, "cache_hit": false, "judge_id": "resp_0ef36e1b996fbc74006ab0b43b398887d2be53a9f59e01ce27", "elapsed_ms": 9253}
{"id": "p1-019", "q": "Vercel Fluid compute pricing limits", "tier": "L2", "intent": "pricing and limits", "candidates_in": 50, "baseline_urls": ["https://vercel.com/docs/functions/usage-and-pricing", "https://vercel.com/docs/fluid-compute", "https://vercel.com/docs/services/pricing", "https://www.usagepricing.com/blueprint/vercel", "https://vercel.com/docs/functions/limitations", "https://vercel.com/docs/pricing", "https://vercel.com/changelog/lower-pricing-with-active-cpu-pricing-for-fluid-compute", "https://vercel.com/docs/functions/usage-and-pricing/legacy-pricing", "https://vercel.com/changelog/higher-defaults-and-limits-for-vercel-functions-running-fluid-compute", "https://vercel.com/docs/queues/pricing"], "treatment_urls": ["https://vercel.com/docs/functions/usage-and-pricing", "https://vercel.com/changelog/higher-defaults-and-limits-for-vercel-functions-running-fluid-compute", "https://vercel.com/docs/pricing", "https://vercel.com/kb/guide/vercel-services-fluid-compute", "https://vercel.com/docs/services/pricing", "https://vercel.com/blog/introducing-active-cpu-pricing-for-fluid-compute", "https://vercel.com/docs/fluid-compute", "https://vercel.com/docs/queues/pricing", "https://vercel.com/docs/limits", "https://vercel.com/changelog/lower-pricing-with-active-cpu-pricing-for-fluid-compute"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.82, "rationale": "Both lead with the canonical pricing page, but B is entirely official and elevates the Fluid compute limits update to rank 2. A includes a third-party pricing overview and irrelevant legacy non-Fluid pricing.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 681, "rank_ms": 3818, "judge_ms": 4683, "error": null, "fetch_ms": 1897, "cache_hit": false, "judge_id": "resp_04012ace0e09f306006ab0b44551d887d28b4952d99b68f597", "elapsed_ms": 9184}
{"id": "p1-020", "q": "Upstash Redis REST rate limits", "tier": "L2", "intent": "official rate limits", "candidates_in": 50, "baseline_urls": ["https://upstash.com/docs/redis/sdks/ratelimit-ts/overview", "https://upstash.com/docs/redis/sdks/ratelimit-ts/gettingstarted", "https://upstash.com/docs/redis/sdks/ratelimit-ts/algorithms", "https://github.com/upstash/ratelimit-js", "https://upstash.com/docs/redis/sdks/ratelimit-py/features", "https://upstash.com/docs/redis/tutorials/rate-limiting", "https://www.reddit.com/r/nextjs/comments/1iwkriv/is_upstash_redis_free_tier_enough_for_rate/", "https://upstash.com/docs/redis/tutorials/python_rate_limiting", "https://pypi.org/project/upstash-ratelimit/", "https://www.npmjs.com/package/@upstash/ratelimit"], "treatment_urls": ["https://upstash.com/docs/redis/overall/pricing", "https://upstash.com/blog/limits-increase", "https://upstash.com/pricing/redis", "https://upstash.com/blog/upstash-vs-redis-cloud-a-2026-comparison", "https://www.13labs.au/compare/upstash-vs-redis-cloud", "https://www.tryorbye.com/products/upstash", "https://nerdleveltech.com/api-rate-limiting-upstash-redis-sliding-window-tutorial", "https://upstash.com/docs/redis/sdks/ratelimit-ts/algorithms", "https://upstash.com/docs/redis/tutorials/python_rate_limiting", "https://upstash.com/docs/redis/sdks/ratelimit-ts/methods"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.97, "rationale": "A leads with official Upstash pricing and limits documentation, directly addressing Redis REST service limits. B largely covers building rate limiters with Upstash rather than limits imposed on the Redis REST API.", "a_first_relevant_rank": 1, "b_first_relevant_rank": null}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 649, "rank_ms": 4539, "judge_ms": 5200, "error": null, "fetch_ms": 2635, "cache_hit": false, "judge_id": "resp_013902e7f9e9229b006ab0b44f72a487d2bfb372d75631736a", "elapsed_ms": 10390}
{"id": "p1-021", "q": "Neon serverless Postgres branching docs", "tier": "L2", "intent": "official branching docs", "candidates_in": 50, "baseline_urls": ["https://github.com/neondatabase/neon", "https://neon.com/docs/introduction", "https://docs.astro.build/en/guides/backend/neon/", "https://getautonoma.com/blog/neon-database", "https://neon.com/", "https://mcpservers.org/agent-skills/neondatabase/neon-postgres", "https://encore.dev/articles/neon-serverless-postgres", "https://github.com/kalikex1/neon", "https://dev.to/0012303/neon-has-a-free-tier-serverless-postgres-with-branching-512-mb-storage-and-no-credit-card-i3e", "https://dev.to/hexshift/mastering-neon-a-pros-guide-to-using-serverless-postgresql-ei5"], "treatment_urls": ["https://neon.com/docs/guides/branching-intro", "https://neon.com/docs/introduction/branching", "https://neon.com/docs/get-started/why-neon", "https://neon.com/docs/introduction/serverless", "https://neon.com/docs/introduction", "https://neon.com/docs/postgres/overview", "https://github.com/neondatabase/neon", "https://neon.com/", "https://encore.dev/articles/neon-serverless-postgres", "https://semaphore.io/blog/neon-database"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.99, "rationale": "B leads with two official Neon branching documentation pages, directly satisfying the query, followed by related official serverless and Postgres docs. A’s first result is relevant but broad, and its dedicated branching coverage is much weaker.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 624, "rank_ms": 3045, "judge_ms": 2209, "error": null, "fetch_ms": 1147, "cache_hit": false, "judge_id": "resp_04d4d605d1d4b287006ab0b458907887d2b4d5f481554e3d0f", "elapsed_ms": 5879}
{"id": "p1-022", "q": "Anthropic tool use JSON schema", "tier": "L2", "intent": "official tool-use docs", "candidates_in": 50, "baseline_urls": ["https://platform.claude.com/docs/en/build-with-claude/structured-outputs", "https://news.ycombinator.com/item?id=41886372", "https://hidekazu-konishi.com/tools/anthropic_tool_use_schema_builder_validator_tool.html", "https://thomas-wiegold.com/blog/claude-api-structured-output/", "https://towardsdatascience.com/hands-on-with-anthropics-new-structured-output-capabilities/", "https://github.com/anthropics/anthropic-cookbook/blob/main/tool_use/extracting_structured_json.ipynb", "https://github.com/aws-samples/anthropic-on-aws/blob/main/complex-schema-tool-use/README.md", "https://www.reddit.com/r/Anthropic/comments/1hje7fq/structured_json_output/", "https://www.neura.market/directories/chatgpt/blog/hands-on-guide-to-anthropic-s-structured-outputs-unlock-claude-s-json-schema-power", "https://jsonic.io/guides/anthropic-tool-use-json"], "treatment_urls": ["https://platform.claude.com/docs/en/agents-and-tools/tool-use/strict-tool-use", "https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools", "https://platform.claude.com/docs/en/build-with-claude/structured-outputs", "https://www.neura.market/directories/chatgpt/blog/hands-on-guide-to-anthropic-s-structured-outputs-unlock-claude-s-json-schema-power", "https://towardsdatascience.com/hands-on-with-anthropics-new-structured-output-capabilities/", "https://aws-samples.github.io/anthropic-on-aws/projects/complex-schema-tool-use/", "https://platform.claude.com/docs/en/agents-and-tools/tool-use/implement-tool-use", "https://docs.anthropic.com/en/api/openai-sdk", "https://github.com/anthropics/anthropic-cookbook/blob/main/tool_use/extracting_structured_json.ipynb", "https://docs.anthropic.com/en/docs/build-with-claude/tool-use"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.99, "rationale": "A leads with exact, authoritative Anthropic documentation on JSON Schema compliance and tool definitions, then provides additional official implementation guidance. B is relevant but relies much more heavily on third-party guides, discussions, and tools.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 770, "rank_ms": 4311, "judge_ms": 3426, "error": null, "fetch_ms": 2176, "cache_hit": false, "judge_id": "resp_0ec536748d2f34f5006ab0b46044a087d2abd9e27148f6c1ee", "elapsed_ms": 8509}
{"id": "p1-023", "q": "OpenAI structured outputs JSON schema", "tier": "L2", "intent": "official structured-output docs", "candidates_in": 50, "baseline_urls": ["https://developers.openai.com/api/docs/guides/structured-outputs", "https://openai.com/index/introducing-structured-outputs-in-the-api/", "https://platform.openai.com/docs/guides/structured-outputs", "https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/structured-outputs", "https://cobusgreyling.medium.com/openai-structured-json-output-with-adherence-a685776aa4bc", "https://community.openai.com/t/responses-api-documentation-on-structured-outputs-is-lacking/1356632", "https://developers.openai.com/cookbook/examples/structured_outputs_intro", "https://docs.x.ai/developers/model-capabilities/text/structured-outputs", "https://www.digitalapplied.com/blog/openai-structured-outputs-complete-guide", "https://www.codewords.ai/blog/openai-structured-outputs-json-schema"], "treatment_urls": ["https://cookbook.openai.com/examples/structured_outputs_intro", "https://openai.com/index/introducing-structured-outputs-in-the-api/", "https://developers.openai.com/api/docs/guides/structured-outputs", "https://ai-tldr.dev/learn/llm-apis/function-calling/openai-json-schema-mode/", "https://www.codewords.ai/blog/openai-structured-outputs-json-schema", "https://chatai.guide/api/structured-outputs/", "https://www.digitalapplied.com/blog/openai-structured-outputs-complete-guide", "https://towardsdatascience.com/diving-deeper-with-structured-outputs-b4a5d280c208/", "https://github.com/openai/openai-structured-outputs-samples", "https://medium.com/@piyushsonawane10/getting-structured-outputs-from-openai-models-a-developers-guide-3090e8120785"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "A", "confidence": 0.9, "rationale": "A leads with OpenAI’s canonical Structured Outputs documentation, followed by the official announcement and another official docs host. B is also strong, but prioritizes a cookbook example over the primary API guide and shifts sooner to third-party guides.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 860, "rank_ms": 3671, "judge_ms": 4100, "error": null, "fetch_ms": 1629, "cache_hit": false, "judge_id": "resp_01d0ba57b2cf6829006ab0b46850a087d2b2b42826900343cb", "elapsed_ms": 8633}
{"id": "p1-024", "q": "MCP streamable HTTP transport", "tier": "L2", "intent": "official MCP spec", "candidates_in": 50, "baseline_urls": ["https://modelcontextprotocol.io/specification/2025-03-26/basic/transports", "https://medium.com/@shsrams/deep-dive-mcp-servers-with-streamable-http-transport-0232f4bb225e", "https://github.com/invariantlabs-ai/mcp-streamable-http", "https://medium.com/the-ai-language/a-visual-guide-to-mcps-streamable-http-transport-6dc18fe751ad", "https://blog.cloudflare.com/streamable-http-mcp-servers-python/", "https://docs.spring.io/spring-ai/reference/api/mcp/mcp-streamable-http-server-boot-starter-docs.html", "https://thenewstack.io/how-mcp-uses-streamable-http-for-real-time-ai-tool-interaction/", "https://builder.aws.com/content/35A0IphCeLvYzly9Sw40G1dVNzc/mcp-transport-mechanisms-stdio-vs-streamable-http", "https://www.claudemcp.com/blog/mcp-streamable-http", "https://www.mcp-framework.com/docs/transports/http-stream"], "treatment_urls": ["https://modelcontextprotocol.io/specification/2026-07-28/basic/transports", "https://modelcontextprotocol.io/specification/draft/basic/transports/streamable-http", "https://developers.cloudflare.com/agents/model-context-protocol/transport/", "https://docs.spring.io/spring-ai/reference/api/mcp/mcp-streamable-http-server-boot-starter-docs.html", "https://www.stanza.dev/courses/mcp-fundamentals/transport-layers/mcp-fundamentals-http-transport", "https://gingerlabs.ai/blog/mcp-transport-comparison", "https://www.rapidevelopers.com/mcp-tutorial/how-to-use-streamable-http-transport", "https://www.truefoundry.com/blog/mcp-stdio-vs-streamable-http-enterprise", "https://levelup.gitconnected.com/mcp-server-and-client-with-sse-the-new-streamable-http-d860850d9d9d?gi=0b0448f75270", "https://auth0.com/blog/mcp-streamable-http/"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.96, "rationale": "A leads with current official MCP transport documentation and a dedicated official Streamable HTTP specification, followed by strong implementation docs. B is relevant, but its official result is older and less specifically focused.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 620, "rank_ms": 3118, "judge_ms": 3485, "error": null, "fetch_ms": 1103, "cache_hit": false, "judge_id": "resp_004288e6fc08bf5b006ab0b47068bc87d29986578e3978286a", "elapsed_ms": 7224}
{"id": "p1-025", "q": "OAuth 2.1 authorization code PKCE", "tier": "L2", "intent": "canonical OAuth spec", "candidates_in": 50, "baseline_urls": ["https://oauth.net/2/pkce/", "https://auth0.com/docs/get-started/authentication-and-authorization-flow/authorization-code-flow-with-pkce", "https://www.descope.com/learn/post/pkce", "https://www.scalekit.com/blog/pkce-developers-guide-secure-oauth-flows", "https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-auth-code-flow", "https://curity.io/resources/learn/oauth-pkce/", "https://askmeidentity.com/resources/standards/oauth-2-1-explained/", "https://oneuptime.com/blog/post/2025-12-16-what-is-pkce-and-why-you-need-it/view", "https://aembit.io/blog/mcp-oauth-2-1-pkce-and-the-future-of-ai-authorization/", "https://www.oauth.com/playground/authorization-code-with-pkce.html"], "treatment_urls": ["https://oauth.net/2.1/", "https://www.scalekit.com/blog/pkce-developers-guide-secure-oauth-flows", "https://connect2id.com/learn/oauth-2-1", "https://connect2id.com/products/nimbus-oauth-openid-connect-sdk/examples/oauth/pkce", "https://datatracker.ietf.org/doc/draft-ietf-oauth-v2-1/", "https://ashishsrivastav.com/blog/oauth2-authorization-code-pkce-guide", "https://askmeidentity.com/resources/standards/oauth-2-1-explained/", "https://oauth.net/2/pkce/", "https://datatracker.ietf.org/doc/html/rfc7636", "https://auth0.com/docs/get-started/authentication-and-authorization-flow/authorization-code-flow-with-pkce"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.88, "rationale": "A better covers the full OAuth 2.1 context and includes primary standards sources: the IETF OAuth 2.1 draft and RFC 7636. B is useful for implementation guidance but relies more heavily on vendor and secondary explanations.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 802, "rank_ms": 4780, "judge_ms": 4353, "error": null, "fetch_ms": 2895, "cache_hit": false, "judge_id": "resp_021b2eca789616b8006ab0b479b90087d2a9c17d07ed064c90", "elapsed_ms": 9937}
{"id": "p1-026", "q": "Cloudflare R2 egress pricing", "tier": "L2", "intent": "official R2 pricing", "candidates_in": 50, "baseline_urls": ["https://developers.cloudflare.com/r2/pricing/", "https://egresscost.com/cloudflare/", "https://rafftechnologies.com/learn/compare/raff-object-storage-vs-cloudflare-r2", "https://r2-calculator.cloudflare.com/", "https://www.reddit.com/r/CloudFlare/comments/1ic51x1/r2_pricing_serving_filesimages_is_not_free/", "https://leanopstech.com/blog/cloudflare-r2-pricing-2026/", "https://tech-insider.org/cloudflare-r2-vs-s3-vs-backblaze-b2-2026/", "https://filebase.com/blog/cloudflare-r2-pricing-costs-savings-and-alternatives-in-2026/", "https://www.cloudflare.com/products/r2/", "https://www.alekseialeinikov.com/en/blog/topics/cloud/cloudflare-r2-vs-s3-egress-costs-2026"], "treatment_urls": ["https://r2-calculator.cloudflare.com/", "https://leanopstech.com/blog/cloudflare-r2-pricing-2026/", "https://r2drop.com/blog/cloudflare-r2-vs-aws-s3-cost-comparison", "https://mecanik.dev/en/posts/cloudflare-r2-pricing-explained-real-costs-vs-s3-and-backblaze/", "https://mecanik.dev/en/tools/r2-pricing-calculator/", "https://flarecalc.com/calculators/r2/", "https://egresscost.com/", "https://filebase.com/blog/cloudflare-r2-pricing-costs-savings-and-alternatives-in-2026/", "https://dev.to/nayankyada/cloudflare-r2-pricing-2026-free-tier-limits-egress-costs-when-to-upgrade-52ph", "https://developers.cloudflare.com/r2/"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "B", "confidence": 0.99, "rationale": "B ranks Cloudflare’s canonical R2 pricing documentation first and also includes the official product page. A is relevant but leads with a calculator and does not surface the authoritative pricing page.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 590, "rank_ms": 2844, "judge_ms": 1911, "error": null, "fetch_ms": 729, "cache_hit": false, "judge_id": "resp_0cd0535219a39128006ab0b481c1d087d2b187ab11d57cead4", "elapsed_ms": 5347}
{"id": "p1-027", "q": "Exe.dev persistent Linux VMs", "tier": "L2", "intent": "official product docs", "candidates_in": 50, "baseline_urls": ["https://blog.exe.dev/persistent", "https://boxd.sh/blog/boxd-vs-exe-dev/", "https://exe.dev/", "https://exe.dev/docs/all", "https://moge.ai/product/exe-dev", "https://flaviocopes.com/exe-dev/", "https://www.toolify.ai/tool/exe-dev", "https://github.com/livid/exe", "https://exe.dev/llms.txt", "https://exe.dev/sandbox"], "treatment_urls": ["https://blog.exe.dev/ios-development", "https://blog.exe.dev/meet-exe.dev", "https://exe.dev/sandbox", "https://blog.exe.dev/persistent", "https://exe.dev/", "https://exe.dev/llms.txt", "https://exe.dev/docs", "https://exe.dev/docs/serverful", "https://flaviocopes.com/exe-dev/", "https://exe.dev/docs/teams/vms"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.78, "rationale": "B places the exact persistence article first, but A offers a substantially stronger authoritative shortlist overall: the official overview, homepage, core docs, persistence-specific documentation, and related first-party explanations, with fewer aggregators or tangential projects", "a_first_relevant_rank": 2, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 637, "rank_ms": 3161, "judge_ms": 8471, "error": null, "fetch_ms": 1350, "cache_hit": false, "judge_id": "resp_037d2f743631dfad006ab0b487ba9887d28a87250b38a30a66", "elapsed_ms": 12272}
{"id": "p1-028", "q": "Cursor Background Agents documentation", "tier": "L2", "intent": "official product docs", "candidates_in": 50, "baseline_urls": ["https://docs.cursor.com/background-agent", "https://cursor.com/docs/cloud-agent", "https://linear.app/changelog/2025-08-21-cursor-agent", "https://cursor.com/help/ai-features/background-agents", "https://ameany.io/cursor-background-agents/", "https://medium.com/@lgallard/exploring-cursor-background-agents-a-hands-on-experience-15555d206a18", "https://stevekinney.com/courses/ai-development/cursor-background-agents", "https://markaicode.com/cursor-background-agents-documentation/", "https://madewithlove.com/blog/using-cursor-background-agents/", "https://blink.new/blog/cursor-background-agent"], "treatment_urls": ["https://cursor.com/docs/background-agent/api/overview", "https://cursor.com/help/ai-features/background-agents", "https://cursor.com/docs/cloud-agent", "https://docs.cursor.com/background-agent", "https://blink.new/blog/cursor-background-agent", "https://www.aimakers.co/blog/cursor-2-4-subagents/", "https://www.digitalapplied.com/blog/cursor-2-0-agent-first-architecture-guide", "https://www.learncursor.dev/learn/cursor-agents", "https://cursor.com/docs/agent/overview", "https://stevekinney.com/courses/ai-development/cursor-background-agents"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.72, "rationale": "Both lead with relevant official material, but B concentrates four authoritative Cursor documentation pages at the top, including the Cloud Agents API, general explanation, and current Cloud Agents docs. A introduces a third-party changelog at rank 3.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 622, "rank_ms": 3826, "judge_ms": 5237, "error": null, "fetch_ms": 1938, "cache_hit": false, "judge_id": "resp_079a294e67cab1ef006ab0b494e4b087d2b7bc5c22242542b4", "elapsed_ms": 9686}
{"id": "p1-029", "q": "Temporal workflow vs activity", "tier": "L2", "intent": "official Temporal docs", "candidates_in": 45, "baseline_urls": ["https://community.temporal.io/t/workflow-vs-activity/9826", "https://community.hpe.com/t5/servers-general/activities-vs-workflows-in-temporal-understanding-the-core/td-p/7270834", "https://docs.temporal.io/activities", "https://raphaelbeamonte.com/posts/good-practices-for-writing-temporal-workflows-and-activities/", "https://www.reddit.com/r/golang/comments/1as23yb/when_to_use_a_workflow_tool_temporal_vs_a_job/", "https://keithtenzer.com/temporal/Temporal_Fundamentals_Workflows/", "https://mincong.io/en/activity-in-temporal/", "https://community.temporal.io/t/determining-the-best-practices-for-business-logic-placement-within-temporal-activities-and-workflows/8951", "https://mareks-082.medium.com/temporal-io-in-net-466938e6692a", "https://temporal.io/how-it-works"], "treatment_urls": ["https://docs.temporal.io/activities", "https://temporal.io/how-it-works", "https://docs.temporal.io/workflows", "https://pkg.go.dev/go.temporal.io/sdk/activity", "https://mareks-082.medium.com/temporal-io-in-net-466938e6692a", "https://manuel.bernhardt.io/2021/04/12/tour-of-temporal-welcome-to-the-workflow", "https://keithtenzer.com/temporal/Temporal_Fundamentals_Workflows/", "https://medium.com/mitti-engineering/building-resilient-microservice-workflows-with-temporal-a-next-gen-workflow-engine-a9637a73572d", "https://pkg.go.dev/go.temporal.io/sdk/workflow", "https://mincong.io/en/activity-in-temporal/"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.9, "rationale": "B leads with official Temporal documentation for Activities, then provides official platform and Workflow pages, enabling an authoritative comparison. A’s exact-title forum result is relevant but narrower and less canonical.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 543, "rank_ms": 3019, "judge_ms": 7833, "error": null, "fetch_ms": 1128, "cache_hit": false, "judge_id": "resp_0e181ed40d13166a006ab0b49e4fe887d2aa1ef2bc6709ae46", "elapsed_ms": 11398}
{"id": "p1-030", "q": "ClickHouse ReplacingMergeTree caveats", "tier": "L2", "intent": "official engine caveats", "candidates_in": 50, "baseline_urls": ["https://clickhouse.com/docs/concepts/features/operations/update/replacing-merge-tree", "https://clickhouse.com/docs/reference/engines/table-engines/mergetree-family/replacingmergetree", "https://altinity.com/blog/clickhouse-replacingmergetree-explained-the-good-the-bad-and-the-ugly", "https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/guides/developer/replacing-merge-tree.md", "https://www.tinybird.co/blog/clickhouse-replacingmergetree-example", "https://kb.altinity.com/engines/mergetree-table-engine-family/replacingmergetree/", "https://medium.com/@raphael3213/clickhouse-replacingmergetree-how-does-it-work-7d6bb3991851", "https://github.com/ClickHouse/ClickHouse/blob/22.5/docs/en/engines/table-engines/mergetree-family/replacingmergetree.md", "https://www.reddit.com/r/Clickhouse/comments/1h0jeq3/how_does_replacingmergetree_handle_new_entries/", "https://obsessiondb.com/blog/replacingmergetree-deduplication-done-right"], "treatment_urls": ["https://obsessiondb.com/blog/replacingmergetree-deduplication-done-right", "https://clickhouse.com/docs/reference/engines/table-engines/mergetree-family/replacingmergetree", "https://chistadata.com/clickhouse-replacingmergetree-good-bad/", "https://altinity.com/blog/clickhouse-replacingmergetree-explained-the-good-the-bad-and-the-ugly", "https://hypequery.com/blog/clickhouse-final-replacingmergetree-typescript", "https://clickhouse.com/docs/concepts/features/operations/update/replacing-merge-tree", "https://clickhouse.com/blog/updates-in-clickhouse-1-purpose-built-engines", "https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/guides/developer/replacing-merge-tree.md", "https://clickhouse.com/docs/operations/settings/merge-tree-settings", "https://clickhouse.com/docs/guides/developer/deduplication"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "B", "confidence": 0.74, "rationale": "B leads with canonical ClickHouse guidance and the official engine reference, followed by reputable Altinity material. A is more explicitly caveat-focused, but ranks a less authoritative third-party article above primary documentation and includes several SEO-style sources.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 630, "rank_ms": 4321, "judge_ms": 5992, "error": null, "fetch_ms": 2354, "cache_hit": false, "judge_id": "resp_040dcdbf68d907a5006ab0b4ab01a087d28b2e3a8cf996e592", "elapsed_ms": 10945}
{"id": "p1-031", "q": "Pydantic v2 model_validate vs parse_obj", "tier": "L2", "intent": "official Pydantic v2 docs", "candidates_in": 50, "baseline_urls": ["https://docs.pydantic.dev/latest/migration/", "https://github.com/pydantic/pydantic/issues/7920", "https://pydantic.dev/docs/validation/latest/get-started/migration/", "https://github.com/pydantic/pydantic/discussions/7008", "https://docs.pydantic.dev/latest/concepts/models/", "https://github.com/pydantic/pydantic/issues/6944", "https://stackoverflow.com/questions/55762673/how-to-parse-list-of-models-with-pydantic", "https://pydantic.dev/articles/pydantic-v2-alpha", "https://pydantic.com.cn/en/concepts/models/", "https://jsonic.io/guides/json-to-pydantic"], "treatment_urls": ["https://docs.pydantic.dev/latest/migration/", "https://docs.pydantic.dev/latest/concepts/json/", "https://www.pykit.org/validate-everything-a-practical-introduction-to-pydantic-v2/", "https://jsonic.io/guides/json-to-pydantic", "https://pydantic.dev/docs/validation/2.9/api/pydantic/base_model/", "https://techoral.com/python/python-pydantic-guide.html", "https://medium.com/@akashbaidya2/pydantic-v2-tutorial-mastering-data-validation-in-python-with-pydantic-v2-0a2dbbd3f5e7", "https://github.com/pydantic/pydantic/issues/6944", "https://docs.pydantic.dev/latest/concepts/performance/", "https://2pointers.medium.com/an-introduction-to-pydantic-v2-alpha-pre-release-a-massive-improvement-over-previous-version-748a1f1118ba"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "B", "confidence": 0.84, "rationale": "Both lead with the authoritative migration guide, but B places a directly focused model_validate-versus-parse_obj behavior issue second and includes another relevant issue at rank 6. A’s comparable issue is buried at rank 8.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 714, "rank_ms": 3621, "judge_ms": 5044, "error": null, "fetch_ms": 1539, "cache_hit": false, "judge_id": "resp_09dc1de99eedc0db006ab0b4b595d087d2b74334b41ed84ec1", "elapsed_ms": 9380}
{"id": "p1-032", "q": "HTTP 103 Early Hints browser support", "tier": "L2", "intent": "support vs spec", "candidates_in": 50, "baseline_urls": ["https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/103", "https://blog.nginx.org/blog/nginx-introduces-support-103-early-hints", "https://techdocs.akamai.com/property-mgr/docs/early-hints", "https://developer.mozilla.org/en-US/docs/web/http/status/103", "https://developer-mozilla-org.translate.goog/en-US/docs/Web/HTTP/Reference/Status/103?_x_tr_sl=en&_x_tr_tl=vi&_x_tr_hl=vi&_x_tr_pto=tc", "https://www.corewebvitals.io/pagespeed/103-early-hints", "https://nitropack.io/blog/early-hints/", "https://chromium.googlesource.com/chromium/src/+/master/docs/early-hints.md", "https://www.catchpoint.com/blog/understanding-the-103-early-hints-status-code", "https://http.cat/status/103"], "treatment_urls": ["https://chromium.googlesource.com/chromium/src/+/master/docs/early-hints.md", "https://developer.chrome.com/docs/web-platform/early-hints", "https://chromestatus.com/feature/5207422375297024", "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/103", "https://wiki.mozilla.org/Networking/103_Early_Hints", "https://developer.mozilla.org/en-US/docs/web/http/status/103", "https://qainsights.com/what-is-http-103/"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.97, "rationale": "B is more focused on browser support, combining primary Chromium documentation, Chrome’s developer guidance and feature status, MDN compatibility information, and Firefox source documentation. A includes more server/vendor and implementation-oriented pages.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 646, "rank_ms": 4914, "judge_ms": 2663, "error": null, "fetch_ms": 2880, "cache_hit": false, "judge_id": "resp_00419469b134fc9a006ab0b4c06c6087d2abcb6ec3d7ecc1c7", "elapsed_ms": 8225}
{"id": "p1-033", "q": "WebAuthn discoverable credentials passkeys", "tier": "L2", "intent": "spec or MDN", "candidates_in": 50, "baseline_urls": ["https://www.webauthn.me/passkeys", "https://web.dev/articles/webauthn-discoverable-credentials", "https://developers.yubico.com/Passkeys/Passkey_concepts/Discoverable_vs_non-discoverable_credentials.html", "https://www.corbado.com/blog/webauthn-resident-key-discoverable-credentials-passkeys", "https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API", "https://simplewebauthn.dev/docs/advanced/passkeys", "https://github.com/keepassxreboot/keepassxc/issues/10269", "https://hidekazu-konishi.com/entry/passkeys_and_webauthn_in_practice.html", "https://developer.mozilla.org/en-US/docs/Web/Security/Authentication/Passkeys", "https://www.corbado.com/glossary/discoverable-credential"], "treatment_urls": ["https://docs.yubico.com/hardware/yubikey-guidance/best-practices/all-faq-passkeys.html", "https://www.w3.org/TR/webauthn-3/", "https://developers.yubico.com/WebAuthn/WebAuthn_Developer_Guide/Resident_Keys.html", "https://developers.yubico.com/Passkeys/Passkey_concepts/Discoverable_vs_non-discoverable_credentials.html", "https://web.dev/articles/webauthn-discoverable-credentials", "https://mojoauth.com/blog/passkey-glossary-webauthn-fido2-aaguid-explained", "https://www.webauthn.me/passkeys", "https://www.passkeys.io/technical-details", "https://www.corbado.com/glossary/discoverable-credential", "https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.95, "rationale": "Both are highly relevant, but A ranks authoritative, directly applicable sources first, including Yubico guidance, the canonical W3C WebAuthn specification, and Yubico’s developer guide. B delays comparable authority and includes a narrow GitHub issue and personal blog.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 837, "rank_ms": 3627, "judge_ms": 5817, "error": null, "fetch_ms": 1482, "cache_hit": false, "judge_id": "resp_0667245691a38355006ab0b4c8284487d2a51ea2332acfd170", "elapsed_ms": 10283}
{"id": "p1-034", "q": "S3 conditional writes If-None-Match", "tier": "L2", "intent": "official S3 docs", "candidates_in": 50, "baseline_urls": ["https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html", "https://medium.com/@Nelsonalfonso/supercharge-your-aws-s3-write-operations-with-conditional-requests-with-typescript-example-2a9baf8a7848", "https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/", "https://4sysops.com/archives/aws-s3-conditional-writes-the-if-none-match-parameter/", "https://simonwillison.net/2024/Nov/26/s3-conditional-writes/", "https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes-enforce.html", "https://www.rahulpnath.com/blog/amazon-s3-conditional-requests-dotnet", "https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-enforcement-conditional-write-operations-general-purpose-buckets/", "https://github.com/Robothy/local-s3/issues/289", "https://www.amazonaws.cn/en/new/2024/amazon-s3-supports-conditional-writes/"], "treatment_urls": ["https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html", "https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html", "https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes-enforce.html", "https://www.amazonaws.cn/en/new/2024/amazon-s3-supports-conditional-writes/", "https://aws.amazon.com/about-aws/whats-new/2025/10/amazon-s3-conditional-write-functionality-copy-operations/", "https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_Scenario_ConditionalRequests_section.html", "https://www.amazonaws.cn/en/new/2024/amazon-s3-adds-new-functionality-for-conditional-writes/", "https://www.amazonaws.cn/en/new/2025/amazon-s3-adds-conditional-write-functionality-to-copy-operations/", "https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-enforcement-conditional-write-operations-general-purpose-buckets/", "https://www.rahulpnath.com/blog/amazon-s3-conditional-requests-dotnet"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.95, "rationale": "A leads with the authoritative PutObject API reference explicitly defining If-None-Match behavior, followed by AWS guidance and enforcement documentation. B is relevant but places a Medium article above several stronger official sources.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 685, "rank_ms": 4116, "judge_ms": 3932, "error": null, "fetch_ms": 2133, "cache_hit": false, "judge_id": "resp_063e05fdc0cc5be2006ab0b4d2aaf887d282746d4647603d96", "elapsed_ms": 8735}
{"id": "p1-035", "q": "Postgres LISTEN NOTIFY connection pooling pitfalls", "tier": "L2", "intent": "pooling caveats", "candidates_in": 50, "baseline_urls": ["https://github.com/pgbouncer/pgbouncer/issues/655", "https://multigres.com/blog/listen-notify", "https://goldlapel.com/grounds/laravel-php/symfony-messenger-postgres-listen-notify", "https://medium.com/@jramcloud1/01-connection-pooling-postgresql-database-administration-connection-pooling-in-postgresql-17-1264aff21dae", "https://www.pandastack.ai/blog/postgres-connection-pooling-explained/", "https://www.reddit.com/r/PostgreSQL/comments/6h3rhf/using_listennotify_and_the_connection_count/", "https://lobste.rs/s/lfuy71/postgresql_listen_notify", "https://dev.to/software_mvp-factory/postgresql-connection-pooling-under-pressure-2ihm", "https://goldlapel.com/how-to/connection-pooling", "https://dev.to/philip_mcclarence_2ef9475/postgresql-connection-pooling-pgbouncer-supavisor-built-in-3i4c"], "treatment_urls": ["https://github.com/pgbouncer/pgbouncer/issues/655"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "A", "confidence": 0.98, "rationale": "Both lead with the directly relevant PgBouncer issue, but A adds several focused discussions of LISTEN/NOTIFY’s session-bound behavior, dedicated connections, scaling, and pooling workarounds. B offers only the single issue.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 725, "rank_ms": 3495, "judge_ms": 3224, "error": null, "fetch_ms": 1653, "cache_hit": false, "judge_id": "resp_08b86c39433904ba006ab0b4db152c87d29c1a904623fc5635", "elapsed_ms": 7446}
{"id": "p1-036", "q": "Who created Redis", "tier": "L3", "intent": "authoritative bio vs SEO", "candidates_in": 50, "baseline_urls": ["https://en.wikipedia.org/wiki/Redis", "https://en.wikipedia.org/wiki/Redis_(company)", "https://redis.io/blog/becoming-one-redis/", "https://redis.io/press/redis-labs-becomes-simply-redis/", "https://oneuptime.com/blog/post/2026-03-31-redis-history-side-project-to-industry-standard/view", "https://en.wikipedia.org/wiki/Salvatore_Sanfilippo", "https://medium.com/life-at-telkomsel/brief-story-of-redis-8bfa90becb8a", "https://www.swiftorial.com/tutorials/caching/redis/introduction_to_redis/history_of_redis/", "https://redis.io/press/redis-creator-salvatore-sanfilippo-antirez-joins-redis-labs/", "https://www.reddit.com/r/programming/comments/1hxdlua/redis_creator_antirez_salvatore_sanfilippo_on/"], "treatment_urls": ["https://redis.io/blog/becoming-one-redis/", "https://redis.io/press/redis-creator-salvatore-sanfilippo-antirez-joins-redis-labs/", "https://redis.io/blog/new-governance-for-redis/", "https://en.wikipedia.org/wiki/Salvatore_Sanfilippo", "https://www.swiftorial.com/tutorials/caching/redis/introduction_to_redis/history_of_redis/", "https://medium.com/life-at-telkomsel/brief-story-of-redis-8bfa90becb8a", "https://www.altexsoft.com/blog/redis-pros-and-cons/", "https://dbdb.io/db/redis", "https://grokipedia.com/page/Redis_(company)", "https://www.networkworld.com/article/941072/redis-labs-hires-the-creator-of-redis-salvatore-sanfilippo.html"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.95, "rationale": "B leads with authoritative Redis-hosted pages that directly identify Salvatore Sanfilippo as Redis’s creator. A also answers correctly, but places less authoritative Wikipedia results ahead of the primary-source Redis pages.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 720, "rank_ms": 3533, "judge_ms": 4935, "error": null, "fetch_ms": 1669, "cache_hit": false, "judge_id": "resp_0d66565e0736c437006ab0b4e2d14487d2b74aa7a2a6ec20f0", "elapsed_ms": 9190}
{"id": "p1-037", "q": "Original paper introducing the Transformer attention architecture", "tier": "L3", "intent": "primary paper vs rewrite", "candidates_in": 50, "baseline_urls": ["https://en.wikipedia.org/wiki/Attention_Is_All_You_Need", "https://arxiv.org/abs/1706.03762", "https://towardsai.com/p/machine-learning/attention-is-all-you-need-a-deep-dive-into-the-revolutionary-transformer-architecture", "https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf", "https://nlp.seas.harvard.edu/2018/04/03/attention.html", "https://medium.com/@vijay.poudel1/attention-is-all-you-need-the-paper-that-revolutionized-ai-6e606e6a847b", "https://machinelearningmastery.com/a-gentle-introduction-to-attention-and-transformer-models/", "https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)", "https://hasgeek.com/fifthelephant/paper-reading-meet-up-december-2023/", "https://huggingface.co/blog/Esmail-AGumaan/attention-is-all-you-need"], "treatment_urls": ["https://research.google/pubs/attention-is-all-you-need/", "https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/", "https://arxiv.org/abs/1706.03762", "https://huggingface.co/papers/1706.03762", "https://www.linkedin.com/posts/alexxubyte_the-most-important-paper-attention-is-all-activity-7404924500187865088-7LDp", "https://towardsai.com/p/machine-learning/attention-is-all-you-need-a-deep-dive-into-the-revolutionary-transformer-architecture", "https://medium.com/@vijay.poudel1/attention-is-all-you-need-the-paper-that-revolutionized-ai-6e606e6a847b", "https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf", "https://en.wikipedia.org/wiki/Attention_Is_All_You_Need", "https://huggingface.co/blog/Esmail-AGumaan/attention-is-all-you-need"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.98, "rationale": "A leads with Google Research’s official paper page, followed by an official explanatory post and arXiv. B starts with Wikipedia and places primary sources lower. Both contain the paper, but A is substantially better ranked.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 2}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 656, "rank_ms": 4580, "judge_ms": 2500, "error": null, "fetch_ms": 2560, "cache_hit": false, "judge_id": "resp_0a77d391dbb816d2006ab0b4ed3c0887d2b6f5652627a9e3a1", "elapsed_ms": 7738}
{"id": "p1-038", "q": "Official X/Twitter profile for OpenAI", "tier": "L3", "intent": "canonical host vs scrapers", "candidates_in": 48, "baseline_urls": ["https://x.com/OpenAI", "https://twitter.com/openai", "https://x.com/sama", "https://x.com/OpenAI/highlights", "https://x.com/OpenAINewsroom", "https://twitter.com/OpenAIDevs", "https://apify.com/xquik/x-profile-scraper/examples/profile-overview-for-openai-1", "https://x.com/openaidevs", "https://twitter.com/openainewsroom", "https://x.com/sama/reposts"], "treatment_urls": ["https://x.com/OpenAI", "https://twitter.com/openai", "https://x.com/OpenAI/with_replies", "https://x.com/OpenAI/status/2087912623883051300", "https://x.com/OpenAI/status/2085434712429052386", "https://x.com/OpenAI/status/2084352161404920316", "https://x.com/OpenAI/highlights", "https://x.com/OpenAI/status/2085801349866729975", "https://x.com/OpenAI/status/2093074192636018977", "https://x.com/OpenAI/status/2084747580693426555"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "tie", "confidence": 0.98, "rationale": "Both lists place the exact official OpenAI X profile first and its legacy Twitter URL second. Later differences add little value for this straightforward navigational query.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "tie", "ranking": "jev", "discover_ms": 684, "rank_ms": 4007, "judge_ms": 5913, "error": null, "fetch_ms": 2198, "cache_hit": false, "judge_id": "resp_026959d3985a4234006ab0b4f4af9087d2a1f6aefc3ce22296", "elapsed_ms": 10606}
{"id": "p1-039", "q": "NASA Instagram on the official Instagram host", "tier": "L3", "intent": "canonical Instagram vs viewers", "candidates_in": 36, "baseline_urls": ["https://www.instagram.com/nasa/", "https://www.nasa.gov/news-release/nasa-to-share-the-universe-with-instagram-users-through-its-images/", "https://www.nasa.gov/social-media/", "https://www.instagram.com/nasaames/?hl=en", "https://www.instagram.com/nasaadmin/", "https://www.instagram.com/nasascience_/", "https://instastatistics.com/nasa", "https://www.instagram.com/nasaastronauts/?hl=en", "https://www.instagram.com/nasa/channel/?hl=en", "https://www.instagram.com/nasaearth/?hl=en"], "treatment_urls": ["https://www.instagram.com/nasa/", "https://www.instagram.com/nasa/feed/?hl=en", "https://www.instagram.com/nasa/channel/?hl=en", "https://en.wikipedia.org/wiki/List_of_most-followed_Instagram_accounts", "https://www.inverse.com/article/10373-nasa-has-200-social-media-accounts-these-are-the-10-you-should-follow-now", "https://www.nasa.gov/news-release/nasa-to-share-the-universe-with-instagram-users-through-its-images/", "https://www.theverge.com/2015/7/14/8958063/pluto-image-instagram-nasa"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "B", "confidence": 0.93, "rationale": "Both rank the exact official Instagram profile first. B is stronger overall because it immediately adds NASA’s own announcement and official social-media directory, plus other official Instagram accounts; A includes redundant profile variants and unrelated secondary pages.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 616, "rank_ms": 4011, "judge_ms": 5214, "error": null, "fetch_ms": 2125, "cache_hit": false, "judge_id": "resp_00a538e6a1a31f3d006ab0b4ff7b4087d296fda67afbcc9eb5", "elapsed_ms": 9843}
{"id": "p1-040", "q": "GitHub user torvalds", "tier": "L3", "intent": "canonical github.com profile", "candidates_in": 50, "baseline_urls": ["https://github.com/torvalds", "https://github.blog/open-source/git/git-turns-20-a-qa-with-linus-torvalds/", "https://github.com/torvalds?tab=repositories", "https://github.com/torvalds/linux", "https://dev.to/martiliones/how-i-got-linus-torvalds-in-my-contributors-on-github-3k4g", "https://thenewstack.io/linus-torvalds-reflects-on-20-years-of-git/", "https://github.com/torvalds?tab=stars", "https://github.com/thorvalds", "https://www.wired.com/2012/05/torvalds-github/", "https://gitglance.vercel.app/torvalds"], "treatment_urls": ["https://github.com/torvalds?tab=repositories", "https://github.com/anuraghazra/github-readme-stats/issues/564", "https://github.com/torvalds/linux", "https://github.com/torvalds/uemacs", "https://www.wired.com/2012/05/torvalds-github/", "https://github.com/search?q=torvalds&type=users", "https://github.com/torvalds/GuitarPedal", "https://github.blog/open-source/git/git-turns-20-a-qa-with-linus-torvalds/", "https://github.com/torvalds/linux/commits/", "https://medium.com/@jlehner1/exploring-linus-torvalds-github-network-dac89d3a19fa"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "B", "confidence": 0.99, "rationale": "List B places the exact canonical GitHub user profile first. List A starts with the profile’s repositories tab, which is relevant but less directly answers the user-profile query.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 640, "rank_ms": 6891, "judge_ms": 1872, "error": null, "fetch_ms": 5131, "cache_hit": false, "judge_id": "resp_06b9f68feb92c07b006ab0b50c797c87d281a1f1272945732e", "elapsed_ms": 9405}
{"id": "p1-041", "q": "Wikipedia English: Transformer (machine learning model) vs SEO blog summaries of Attention Is All You Need", "tier": "L3", "intent": "encyclopedia vs SEO rewrite", "candidates_in": 50, "baseline_urls": ["https://en.wikipedia.org/wiki/Attention_Is_All_You_Need", "https://simple.wikipedia.org/wiki/Transformer_(machine_learning_model)", "https://wiki.davidl.me/index.php?title=Transformer_(machine_learning_model)&mobileaction=toggle_view_desktop", "https://parseur.com/blog/attention-is-all-you-need", "https://arxiv.org/html/1706.03762v7", "https://drscotthawley.github.io/blog/posts/Transformers1-Attention.html", "https://www.artificial-intelligence.blog/ai-news/attention-is-all-you-need", "https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf", "https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)", "https://research.google/pubs/attention-is-all-you-need/"], "treatment_urls": ["https://research.google/pubs/attention-is-all-you-need/", "https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)", "https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)", "https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf", "https://arxiv.org/abs/1706.03762", "https://mbrenndoerfer.com/writing/transformer-attention-is-all-you-need", "https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/", "https://simple.wikipedia.org/wiki/Transformer_(machine_learning_model)", "https://portkey.ai/blog/attention-is-all-you-need-summary/", "https://parseur.com/blog/attention-is-all-you-need"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.98, "rationale": "B places the sought English Wikipedia transformer article at ranks 2–3 and prioritizes authoritative Google and primary-paper sources. A delays the exact article until rank 9, behind Simple Wikipedia, a mirror, and multiple blog summaries.", "a_first_relevant_rank": 9, "b_first_relevant_rank": 2}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 785, "rank_ms": 3415, "judge_ms": 6156, "error": null, "fetch_ms": 1365, "cache_hit": false, "judge_id": "resp_01381f54deb9072b006ab0b512e8e887d28a905307a3022c00", "elapsed_ms": 10358}
{"id": "p1-042", "q": "Apple Vision Pro US release date", "tier": "L3", "intent": "official date vs rumor", "candidates_in": 50, "baseline_urls": ["https://en.wikipedia.org/wiki/Apple_Vision_Pro", "https://www.apple.com/newsroom/2024/01/apple-vision-pro-available-in-the-us-on-february-2/", "https://www.macrumors.com/roundup/apple-vision-pro/", "https://theapplewiki.com/wiki/Apple_Vision_Pro", "https://www.bbc.com/news/technology-67922296", "https://simple.wikipedia.org/wiki/Apple_Vision_Pro", "https://www.bgr.com/guides/apple-vision-pro-release-date-price-features/", "https://www.ign.com/articles/apple-vision-pro-us-release-date-announced", "https://www.phonearena.com/ar-vr/hubs/apple-vision-pro-release-date-price-features-news_id314", "https://www.inverse.com/tech/apple-vision-pro-release-date-usa"], "treatment_urls": ["https://www.apple.com/newsroom/2024/01/apple-vision-pro-available-in-the-us-on-february-2/", "https://www.ign.com/articles/apple-vision-pro-us-release-date-announced", "https://www.inverse.com/tech/apple-vision-pro-release-date-usa", "https://www.bbc.com/news/technology-67922296", "https://simple.wikipedia.org/wiki/Apple_Vision_Pro", "https://www.theverge.com/2024/1/8/24001858/apple-vision-pro-release-date-availability-price", "https://www.bgr.com/guides/apple-vision-pro-release-date-price-features/", "https://www.phonearena.com/ar-vr/hubs/apple-vision-pro-release-date-price-features-news_id314", "https://www.cnbc.com/2024/02/02/apples-vision-pro-headset-launches-in-us.html", "https://www.macrumors.com/roundup/apple-vision-pro/"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.98, "rationale": "B puts Apple’s authoritative announcement first, directly stating the February 2 U.S. release date, then follows with relevant reporting. A starts with a less authoritative Wikipedia result whose snippet does not clearly answer the query.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 702, "rank_ms": 7098, "judge_ms": 2048, "error": null, "fetch_ms": 5245, "cache_hit": false, "judge_id": "resp_0ce3ab95dcc4ec2c006ab0b521012c87d2b52123183ad3c81d", "elapsed_ms": 9850}
{"id": "p1-043", "q": "Difference between LangGraph and LangChain agents", "tier": "L3", "intent": "first-party distinction", "candidates_in": 50, "baseline_urls": ["https://www.langchain.com/blog/deep-agents-vs-langchain-vs-langgraph", "https://www.reddit.com/r/LangChain/comments/1ojwl1y/new_to_langchain_agents_langchain_vs_langgraph/", "https://medium.com/@sivakami.kanda/langchain-agents-vs-langgraph-agents-when-to-use-each-a-practical-decision-framework-abc9b451a627", "https://atlan.com/know/ai-agent/ai-agent-memory/langchain-vs-langgraph/", "https://medium.com/@tahirbalarabe2/%EF%B8%8Flangchain-vs-langgraph-a-comparative-analysis-ce7749a80d9c", "https://duplocloud.com/blog/langchain-vs-langgraph/", "https://oxylabs.io/blog/langgraph-vs-langchain", "https://kanerika.com/blogs/langchain-vs-langgraph/", "https://www.spheron.network/blog/langgraph-vs-langchain/", "https://www.geeksforgeeks.org/artificial-intelligence/langchain-vs-langgraph/"], "treatment_urls": ["https://medium.com/@sivakami.kanda/langchain-agents-vs-langgraph-agents-when-to-use-each-a-practical-decision-framework-abc9b451a627", "https://ai.plainenglish.io/langgraph-vs-langchain-agents-bc67d18650c6?gi=1d5a3ba373b9", "https://academy.langchain.com/courses/intro-to-langgraph", "https://ai-tldr.dev/learn/agent-frameworks/langchain-ecosystem/langchain-vs-langgraph/", "https://www.digitalapplied.com/blog/langchain-vs-langgraph-comparison-2026", "https://www.langchain.com/blog/deep-agents-vs-langchain-vs-langgraph", "https://www.spheron.network/blog/langgraph-vs-langchain/", "https://docs.langchain.com/oss/python/concepts/products", "https://towardsdatascience.com/ai-agent-workflows-a-complete-guide-on-whether-to-build-with-langgraph-or-langchain-117025509fa0/", "https://milvus.io/blog/langchain-vs-langgraph.md"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "A", "confidence": 0.96, "rationale": "Both lists are highly relevant, but A ranks LangChain’s authoritative first-party comparison first. B begins with third-party articles and does not surface official LangChain Academy, blog, and documentation resources until ranks 3, 6, and 8.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 653, "rank_ms": 4548, "judge_ms": 3568, "error": null, "fetch_ms": 2489, "cache_hit": false, "judge_id": "resp_0b981ab7b12e7edd006ab0b52882a487d2a74ed8f47e1a099d", "elapsed_ms": 8770}
{"id": "p1-044", "q": "Official Solidity language documentation", "tier": "L3", "intent": "official vs tutorial-farm", "candidates_in": 50, "baseline_urls": ["https://github.com/argotorg/solidity", "https://www.soliditylang.org/", "https://docs.soliditylang.org/en/v0.7.4/", "https://docs.soliditylang.org/", "https://www.cyfrin.io/blog/what-is-solidity", "https://app.readthedocs.org/projects/solidity/", "https://docs.monad.xyz/guides/evm-resources/solidity-resources", "https://docs.soliditylang.org/en/v0.8.0/", "https://docs.soliditylang.org/en/latest/", "https://en.wikipedia.org/wiki/Solidity"], "treatment_urls": ["https://docs.soliditylang.org/en/v0.8.0/", "https://docs.soliditylang.org/en/v0.7.4/", "https://docs.soliditylang.org/", "https://docs.soliditylang.org/en/latest/", "https://docs.soliditylang.org/en/latest/contracts.html", "https://docs.soliditylang.org/en/latest/grammar.html", "https://www.soliditylang.org/", "https://docs.soliditylang.org/en/latest/installing-solidity.html", "https://docs.soliditylang.org/_/downloads/en/latest/pdf/", "https://media.readthedocs.org/pdf/solidity/develop/solidity.pdf"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.99, "rationale": "A leads with official documentation and is dominated by canonical docs.soliditylang.org pages. B starts with a GitHub fork and includes several secondary sources, while current official documentation appears only at rank 4.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 2}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 705, "rank_ms": 2723, "judge_ms": 2772, "error": null, "fetch_ms": 912, "cache_hit": false, "judge_id": "resp_0486e0ba0d22f9fd006ab0b5304f4887d2af8d290f857e3b0d", "elapsed_ms": 6201}
{"id": "p1-045", "q": "Cloudflare Workers vs Cloudflare Pages Functions", "tier": "L3", "intent": "official product distinction", "candidates_in": 50, "baseline_urls": ["https://www.reddit.com/r/CloudFlare/comments/1ip87mx/workers_vs_pages/", "https://community.cloudflare.com/t/workers-vs-page-functions/552945", "https://www.justaftermidnight247.com/insights/cloudflare-pages-vs-workers-which-one-should-you-use/", "https://medium.com/@sehban.alam/cloudflare-workers-vs-cloudflare-functions-vs-cloudflare-pages-a-comprehensive-comparison-370dac4a858c", "https://mecanik.dev/en/posts/cloudflare-pages-vs-workers-which-to-use-in-2026/", "https://www.reddit.com/r/CloudFlare/comments/1s40651/help_me_understand_workers_vs_pages_in_cloudflare/", "https://developers.cloudflare.com/workers/static-assets/migration-guides/migrate-from-pages/", "https://www.morphllm.com/comparisons/cloudflare-pages-vs-workers", "https://developers.cloudflare.com/pages/functions/", "https://www.reddit.com/r/CloudFlare/comments/ztddh4/cf_workers_vs_pages_function/"], "treatment_urls": ["https://developers.cloudflare.com/pages/functions/", "https://blog.cloudflare.com/cloudflare-pages-goes-full-stack/", "https://blog.cloudflare.com/pages-function-goes-ga/", "https://developers.cloudflare.com/pages/", "https://agentikas.ai/en/blog/cloudflare-pages-vs-workers-when-to-use-each", "https://mecanik.dev/en/posts/cloudflare-pages-vs-workers-which-to-use-in-2026/", "https://cogley.jp/articles/cloudflare-pages-to-workers-migration", "https://barnabas.me/blog/2026/07/cloudflare-pages-vs-workers/", "https://www.justaftermidnight247.com/insights/cloudflare-pages-vs-workers-which-one-should-you-use/", "https://blog.cloudflare.com/pages-and-workers-are-converging-into-one-experience/"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "B", "confidence": 0.94, "rationale": "B prioritizes authoritative Cloudflare documentation and official blog posts explaining Pages Functions, the recommendation to start new projects with Workers, and platform convergence. A has useful comparisons, but leads with Reddit and community discussions.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 651, "rank_ms": 4825, "judge_ms": 3032, "error": null, "fetch_ms": 2740, "cache_hit": false, "judge_id": "resp_0ee896140d9a0290006ab0b53848f487d28870fbb8a05e5a27", "elapsed_ms": 8511}
{"id": "p1-046", "q": "What is prompt injection? defensive overview for LLM applications", "tier": "L3", "intent": "educational defensive overview", "candidates_in": 50, "baseline_urls": ["https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html", "https://genai.owasp.org/llmrisk/llm01-prompt-injection/", "https://www.oligo.security/academy/prompt-injection-impact-attack-anatomy-prevention", "https://www.evidentlyai.com/llm-guide/prompt-injection-llm", "https://www.ibm.com/think/topics/prompt-injection", "https://coralogix.com/ai-blog/prompt-injection-attacks-in-llms-what-are-they-and-how-to-prevent-them/", "https://www.fortra.com/blog/vulnerability-strategy-defending-llms-against-prompt-injection-attacks", "https://www.paloaltonetworks.com/cyberpedia/what-is-a-prompt-injection-attack", "https://witness.ai/blog/prompt-injection/", "https://github.com/tldrsec/prompt-injection-defenses"], "treatment_urls": ["https://genai.owasp.org/llmrisk/llm01-prompt-injection/", "https://www.ibm.com/think/topics/prompt-injection"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "B", "confidence": 0.98, "rationale": "List B leads with OWASP’s authoritative prevention cheat sheet, includes OWASP’s canonical vulnerability overview, and offers several directly relevant defensive guides. List A is authoritative but much less comprehensive for the requested defensive overview.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 793, "rank_ms": 3665, "judge_ms": 2912, "error": null, "fetch_ms": 1754, "cache_hit": false, "judge_id": "resp_07eb4ea88b87f754006ab0b540007887d2aebab2be4b590a46", "elapsed_ms": 7371}
{"id": "p1-047", "q": "PyTorch 2.0 release date", "tier": "L3", "intent": "official release notes", "candidates_in": 50, "baseline_urls": ["https://pytorch.org/blog/pytorch-2-0-release/", "https://pytorch.org/get-started/pytorch-2-x/", "https://www.exxactcorp.com/blog/Deep-Learning/PyTorch-2-Preview", "https://en.wikipedia.org/wiki/PyTorch", "https://github.com/pytorch/pytorch/releases", "https://www.h2kinfosys.com/blog/everything-to-know-about-pytorch-2-0/", "https://dev-discuss.pytorch.org/t/pytorch-call-for-features-2-0/919", "https://pyimagesearch.com/2023/03/27/whats-new-in-pytorch-2-0-torch-compile/", "https://dev-discuss.pytorch.org/t/pytorch-release-2-2-0-jan-4-update/1778", "https://www.linux.com/news/pytorch-2-0-our-next-generation-release-that-is-faster-more-pythonic-and-dynamic-as-ever/"], "treatment_urls": ["https://pytorch.org/blog/pytorch-2-0-release/", "https://github.com/pytorch/pytorch/wiki/PyTorch-Versions", "https://en.wikipedia.org/wiki/PyTorch", "https://dev-discuss.pytorch.org/t/2-0-feature-review-complete/1023", "https://www.exxactcorp.com/blog/Deep-Learning/PyTorch-2-Preview", "https://dev-discuss.pytorch.org/t/pytorch-call-for-features-2-0/919", "https://pyimagesearch.com/2023/03/27/whats-new-in-pytorch-2-0-torch-compile/", "https://www.linux.com/news/pytorch-2-0-our-next-generation-release-that-is-faster-more-pythonic-and-dynamic-as-ever/", "https://www.infoq.com/news/2023/03/pytorch-release-compile/", "https://odsc.medium.com/pytorch-2-0-officially-released-330e287abc7a"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.78, "rationale": "Both lead with the canonical PyTorch 2.0 announcement, but A follows it with the official versions wiki and places the exact March 15, 2023 answer higher. Its remaining results are also more consistently focused on the 2.0 release.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 832, "rank_ms": 5682, "judge_ms": 4464, "error": null, "fetch_ms": 3761, "cache_hit": false, "judge_id": "resp_09bac80f6dd5ba61006ab0b549adc087d297f7673316c37c44", "elapsed_ms": 10980}
{"id": "p1-048", "q": "Who maintains the ripgrep project", "tier": "L3", "intent": "canonical maintainer", "candidates_in": 50, "baseline_urls": ["https://ripgrep.dev/docs/faq/", "https://github.com/BurntSushi/ripgrep", "https://ripgrep.dev/", "https://pypi.org/project/ripgrep/", "https://ripgrep.org/", "https://ripgrep.org/about-us/", "https://github.com/Dbachtiar/ripgrep", "https://itsfoss.community/t/ubuntu-series-meet-the-maintainers-the-mind-behind-ripgrep-andrew-gallant-s-blazing-fast-search-tool/15211", "https://ripgrep.dev/about/", "https://wiki.gentoo.org/wiki/Ripgrep"], "treatment_urls": ["https://burntsushi.net/ripgrep/", "https://news.ycombinator.com/item?id=27491583", "https://itsfoss.community/t/ubuntu-series-meet-the-maintainers-the-mind-behind-ripgrep-andrew-gallant-s-blazing-fast-search-tool/15211", "https://github.com/BurntSushi/ripgrep", "https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md", "https://github.com/BurntSushi/ripgrep/releases", "https://docs.rs/crate/ripgrep/0.7.1", "https://crates.io/crates/ripgrep", "https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md", "https://www.linode.com/docs/guides/ripgrep-linux-installation/"], "assignment": {"A": "baseline", "B": "agent_seek"}, "judge": {"winner": "A", "confidence": 0.97, "rationale": "A’s first result directly states that ripgrep is primarily maintained by Andrew Gallant, with the canonical GitHub repository second. B has strong primary sources but does not surface the maintenance answer as directly until result 3.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 3}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 651, "rank_ms": 3905, "judge_ms": 7192, "error": null, "fetch_ms": 1994, "cache_hit": false, "judge_id": "resp_0b3e6f6d4f6b29ec006ab0b552f2e487d2ae3f5b2ed6d5e8f0", "elapsed_ms": 11750}
{"id": "p1-049", "q": "Official Kubernetes security announcements / CVE channel vs random aggregators", "tier": "L3", "intent": "official security channel", "candidates_in": 50, "baseline_urls": ["https://kubernetes.io/docs/reference/issues-security/official-cve-feed/", "https://pagecrawl.io/blog/kubernetes-release-notes-cve-monitoring", "https://docs.cloud.google.com/kubernetes-engine/security-bulletins", "https://v1-32.docs.kubernetes.io/docs/reference/issues-security/official-cve-feed/", "https://v1-34.docs.kubernetes.io/docs/reference/issues-security/official-cve-feed/", "https://kubernetes.io/docs/reference/issues-security/official-cve-feed/index.json", "https://cve.akaoma.com/vendor/kubernetes", "https://app.opencve.io/cve/?vendor=kubernetes", "https://kubernetes.io/blog/2022/09/12/k8s-cve-feed-alpha/", "https://kubernetes.io/docs/reference/issues-security/security/"], "treatment_urls": ["https://v1-32.docs.kubernetes.io/docs/reference/issues-security/official-cve-feed/", "https://kubernetes.io/docs/reference/issues-security/official-cve-feed/index.json", "https://kubernetes.io/docs/reference/issues-security/official-cve-feed/", "https://kubernetes.io/docs/reference/issues-security/security/", "https://www.kubernetes.dev/blog/2022/09/12/k8s-cve-feed-alpha/", "https://kubernetes.io/blog/2022/09/12/k8s-cve-feed-alpha/", "https://kubernetes.io/blog/2023/04/25/k8s-cve-feed-beta/", "https://kubernetes.io/blog/2025/03/24/ingress-nginx-cve-2025-1974/", "https://kubernetes.io/blog/2026/05/26/reconciling-unfixed-kubernetes-cves/", "https://github.com/kubernetes/kubernetes/issues/130786"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "A", "confidence": 0.97, "rationale": "A is consistently authoritative and Kubernetes-hosted, placing the official CVE feed, machine-readable feed, and security announcement channel in its top four. B starts well but mixes in third-party monitoring and CVE aggregators ahead of the official announcement channel.", "a_first_relevant_rank": 1, "b_first_relevant_rank": 1}, "mapped_winner": "agent_seek", "ranking": "jev", "discover_ms": 752, "rank_ms": 7726, "judge_ms": 3986, "error": null, "fetch_ms": 5830, "cache_hit": false, "judge_id": "resp_00d0794438761b24006ab0b562de0c87d2842b647988419b1b", "elapsed_ms": 12466}
{"id": "p1-050", "q": "Stripe Tax vs Stripe Tax Calculations API distinction", "tier": "L3", "intent": "official API distinction", "candidates_in": 49, "baseline_urls": ["https://docs.stripe.com/tax/custom", "https://docs.stripe.com/tax/calculating", "https://support.stripe.com/questions/understanding-stripe-tax-pricing?locale=en-GB", "https://docs.stripe.com/api/tax/calculations", "https://stripe.com/tax/pricing", "https://www.reddit.com/r/stripe/comments/1616n3q/tax_calculation_for_subscriptions_with_stripes_api/", "https://stripe.com/tax", "https://docs.stripe.com/billing/taxes/tax-rates", "https://docs.stripe.com/tax", "https://docs.stripe.com/tax/settings-api"], "treatment_urls": ["https://docs.stripe.com/tax/how-tax-works", "https://docs.stripe.com/tax/calculating", "https://docs.stripe.com/tax/custom", "https://docs.stripe.com/tax/payment-intent/custom", "https://docs.stripe.com/tax/payment-intent", "https://support.stripe.com/questions/understanding-stripe-tax-pricing?locale=en-GB", "https://docs.stripe.com/tax", "https://deepwiki.com/stripe/stripe-node/4.7-tax-api", "https://docs.stripe.com/api/tax/calculations", "https://docs.stripe.com/api/tax/calculations/create"], "assignment": {"A": "agent_seek", "B": "baseline"}, "judge": {"winner": "B", "confidence": 0.82, "rationale": "B leads with Stripe’s canonical standalone Tax Calculations/Transactions API guide, then pairs it with general Stripe Tax and pricing documentation, making the product-versus-API distinction easier to research. A’s clearest comparison-oriented result appears later.", "a_first_relevant_rank": 3, "b_first_relevant_rank": 1}, "mapped_winner": "baseline", "ranking": "jev", "discover_ms": 598, "rank_ms": 3377, "judge_ms": 6208, "error": null, "fetch_ms": 1526, "cache_hit": false, "judge_id": "resp_05a517acb0d76d66006ab0b56b16e087d2b59d87640575509a", "elapsed_ms": 10186}