Findings
The capability gap
As of AI AgriBench v0.5, frontier models score between 94% and 98% accuracy on technical agronomy questions in the main track. The same models score between 65% and 81% on real smallholder farmer questions in the Smallholders track.
The two tracks use different datasets and different sources, so the numbers are not a like-for-like comparison of the same task made harder. What they show is that performance on textbook-style technical agronomy does not predict performance on the practical, contextual questions farmers actually ask.
Why the gap matters
More than 500 million farmers worldwide could come to rely on AI advisory tools. Inaccurate advice carries real consequences: crop failures from a wrong pest identification, financial losses from a wrong input recommendation, and erosion of trust in digital agriculture generally.
Buyers — farmers, agronomists, retailers, governments — have had no standardised way to compare these tools, and vendors have an incentive to self-report favourably. General AI benchmarks such as MMLU and HELM do not measure agronomic accuracy, regional appropriateness or smallholder relevance.
Scoring is multi-dimensional for a reason
A single score hides the thing a buyer needs to know. Reporting accuracy, completeness, relevance and conciseness separately, and filtering by topic category, lets an extension professional check performance in their own specialty and lets a vendor's engineering team see where to invest.
What these findings do not say
- They do not say AI advisory tools are unsafe, or that they are ready. They say how a defined set of systems performed on a defined set of questions.
- They do not generalise beyond text question answering in English, mostly in a US context.
- They are not a verdict. AI AgriBench v0.5 is a starting point, and the leaderboard changes as models change.
Cite these findings
See how to cite AI AgriBench for the citation formats and the permanent URL for each release.