You’ve seen the headline claim: one method got AI models to quote Scripture with 100% accuracy. Claims like that are easy to write and hard to trust without seeing the methodology behind them.
The short version: 33,000 observations, five methods, rigorous controls, and an open-sourced test harness anyone can run themselves to check the result. Here’s exactly how it was built.
What was Actually Being Tested?
Given a Bible passage already correctly selected, how reliably can different AI architectures reproduce its exact wording? The study explicitly didn’t evaluate whether a model can choose the right verse for a question, or interpret Scripture soundly, those are separate problems tied to a system’s broader retrieval pipeline. This isolates one question: once you know what to quote, does the output match the source exactly.
How Big was the Test, and Why?
33,000 observations, from multiplying every variable against every other: 5 methods, 10 translations, 10 passages, roughly 2.2 effective temperature settings, 3 iterations each, plus a second run against 10 non-English translations. Three iterations per combination matters specifically because these are probabilistic systems; a single run can look better or worse than a method’s true reliability by chance alone.
The Five Methods Compared
- Unassisted model: quotes purely from training, no external lookup.
- Unbounded web search: searches the open internet in real time.
- Simulated RAG: handed the exact verse directly, an unrealistic best-case ceiling, not a claim about real-world retrieval.
- Agentic tool calls: has access to Bible API integrations, decides for itself when to use them.
- Output transform: outputs only a placeholder reference; a deterministic process swaps in the real, licensed text.
Only the last method removes the model entirely from generating the verse text, which is the distinction the whole study turns on.
What Kept this a Fair Test?
Ten models spanning a wide capability range, ten translations across four Bible API providers, ten passages varying in length and obscurity, three realistic temperature settings. Two scoring standards were applied throughout: “exact” (every character, including punctuation) and “normalized” (discarding punctuation and case), so a method that reformats punctuation without changing the actual words isn’t unfairly penalized on the headline number.
What Did the Results Show?
| Method | Exact match (English) | Exact match (non-English) |
|---|---|---|
| Output transform | 100.0% | 99.4% |
| Agentic tool calls | 96.2% | 85.4% |
| Simulated RAG (ceiling) | 93.0% | 92.2% |
| Unbounded web search | 48.5% | 22.0% |
| Unassisted model | 32.6% | 7.3% |
Two findings beyond the headline number: the gap between methods widens sharply outside English (tool calls dropped nearly 11 points, output transform dropped only 0.6), and even the simulated RAG ceiling, handed the literally correct text, still misquoted it 7% of the time. Removing the retrieval problem entirely doesn’t remove the generation problem.
Can this Be Verified Independently?
Yes, the full test harness and complete result set are open-sourced, specifically so the numbers can be checked or reproduced rather than taken on faith. If you’re building something that touches Scripture quotation, the harness is built to be run against your own models or translations.
FAQ
How many AI models were tested? Ten, spanning smaller efficiency-focused models to top-tier flagship models, commercial and open source.
Why test non-English translations separately? Because accuracy patterns don’t transfer: methods relying more on the model’s own generation dropped sharply outside English, while output transform barely moved.
What’s the difference between “exact” and “normalized” scoring? Exact counts every character including punctuation; normalized discards punctuation and case, so reformatting doesn’t look like a misquotation.
Is the test harness public? Yes, open-sourced alongside the full result set for independent verification.
Does this study rank which AI model is “best” for faith applications? No, it isolates quotation accuracy from verse selection or interpretation, and its own findings show model choice barely affects quotation accuracy once a proper harness is in place.
Conclusion
The 100% headline number holds up because the methodology was built to be checked: a large sample, repeated trials, real controls, and an open-sourced harness anyone can run. Explore the open-sourced test harness to reproduce these results, or read the full study for the complete write-up.