Comparing Two Models on a Question About Huizinga
- Experiment
- The same question about a passage in Huizinga produced different answers from Gemini Flash Lite and Claude Opus 5.
- What I learned
- the model and task matter when assessing an AI response
- this comparison does not isolate subscription tier as the cause
- Results
- Gemini returned a passage that did not address court ceremony
- Claude found relevant material without inventing page numbers
- You'll need
- Claude Opus 5, Gemini Flash Lite
- Format
- less than 10 minutes
- Jig by
- Jonathan Seyfried
I gave the free tier of Gemini and a paid tier of Claude the same question about court ceremony in Huizinga’s The Waning of the Middle Ages. I wanted a passage I could check, with a page number. The responses differed considerably.
The Prompt
prompt to give to the LLM
What did Huizinga say about court ceremony in The Waning of the Middle Ages? Quote the passage with page number.
Comparing the responses
Gemini Flash Lite returned the opening sentence of the book. It was a quotation, but it didn’t answer the question about court ceremony.
Response from Gemini Flash Lite.
Claude Opus 5 found material relevant to court ceremony and didn’t fabricate page numbers.
Response from Claude Opus 5.
What this comparison tells me
For this question, Claude gave me a more useful place to start. It also showed why an assessment of an AI answer needs to name the model and the task: these responses would lead to quite different impressions of how useful AI is for historical work.
I compared two providers as well as two subscription tiers, so this trial doesn’t isolate the effect of paying for access. It doesn’t establish how often either model fabricates sources, either. More questions and repeated trials would be needed to make those claims. What I can report here is the difference between these two answers.
Using This Experiment
This experiment is licensed under Creative Commons BY-NC-SA 4.0. You're welcome to share and adapt it for noncommercial purposes, with credit, as long as you share your adaptation under the same license.
Suggested citation
Jonathan Seyfried. "Comparing Two Models on a Question About Huizinga." AI Workbench, Amaranth, University of New Mexico, September 1, 2026. https://amaranth.unm.edu/ai-workbench/research/model-tiers.html. Licensed under CC BY-NC-SA 4.0.