Methodolgy
The question this benchmark answers
Is agronomic advice from large language models and AI-based agricultural advisory tools accurate? Rather than testing models on generic benchmarks, AI AgriBench evaluates them on realistic agronomic questions grounded in extension knowledge.
Source corpus
The benchmark is built from extension publications from US land-grant universities — the documents farmers and advisors actually use — together with peer-reviewed research. The v0.5 source corpus comprises more than 40,000 extension PDFs and research papers.
The Smallholders track draws instead on more than 3,100 real farmer–expert question-and-answer pairs contributed by Digital Green, GAIA/IFPRI and Precision Development.
Question generation and expert review
Synthetic question–answer pairs are generated from the source corpus using the YourBench method, then filtered and reviewed.
- 45,172 synthetic question–answer pairs generated
- 951 pairs put through expert review
- 416 pairs retained in the final v0.5 benchmark
Each pair was reviewed by one to four experts. Many questions were edited by hand during review. Twenty-three agronomists conducted the review.
Questions focus on real-world decisions: diagnosing nutrient deficiencies, timing fungicides, setting seeding rates, managing weeds, responding to weather events. Answers are multi-paragraph, practical, and written in plain language.
Contamination-aware evaluation
To avoid favouring models that may have seen the source documents during pretraining, a subset of question–answer pairs was extracted from documents published after 30 September 2024. Results on this subset are marked with an asterisk in the leaderboard.
Scoring
Every response is scored from 0 to 100 on four dimensions rather than reduced to a single number.
Scoring uses a calibrated LLM-as-judge framework with human expert validation. Judge models in v0.5 include Claude Opus 4.5, Gemini 3 Pro, Kimi-K2 and GPT-5.1. No model is ever used to judge itself.
Topic categories
Results are filterable across nine categories: pests, diseases, weeds, nutrition, soils, seed hybrids, horticulture, water and weather.
Research foundations
- YourBench — synthetic question–answer generation. arXiv:2504.01853.
- MIRAGE — multimodal interactive question answering. NeurIPS 2025 Datasets and Benchmarks track.
The benchmarking methodology originates in research conducted within the CropWizard project of the AIFARMS National AI Institute.
What v0.5 does not cover
Version 0.5 evaluates technical agronomic understanding only. It does not cover field-level decisions such as seed or hybrid selection, input product choices, or recommendations conditioned on farm-specific data. Image-based question answering, data-driven recommendations using weather or price forecasts, and open-ended reasoning are planned, not present.
Stating this scope is required whenever a v0.5 result is cited. AI AgriBench v0.5 is a starting point, not a verdict on AI in agriculture.
Known limitations and planned improvements
The current accuracy metric bundles several properties into one yardstick and does not always capture fine-grained factual correctness within longer answers. Planned work includes decomposing accuracy into sub-components (species identification, diagnostic correctness, recommendation appropriateness, terminology precision), fact-level verification of individual claims within a response, and additional metrics for safety and temporal relevance.