Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA

Human and AI strengths depend on the question

Claims that an AI system “beats humans” at question answering can hide an important detail: which kinds of questions are being asked? A model may excel when the answer is a specific fact to retrieve, while a skilled person does better when clues are indirect and require connecting context to a missing idea. CAIMIRA, a framework for analyzing question difficulty and respondent skill, makes these differences visible across thousands of Quiz Bowl questions.

The researchers analyzed 3,042 question items and more than 300,000 response records from about 70 AI systems and 155 human players. Quiz Bowl questions reveal clues gradually, so the dataset also shows how accuracy changes as more information becomes available. The humans were experienced players, not a representative sample of the general public. CAIMIRA uses multidimensional item response theory, a statistical approach that estimates both an agent’s skills and question characteristics. It learns question dimensions from text, allowing it to model new questions without prior response records.

Five kinds of question-solving

The model identifies five latent dimensions: abductive recall, history and events, scientific facts, cultural records and complex semantics. “Abductive recall” means inferring what information is missing from indirect clues, then retrieving a specific fact to fill that gap. Humans tended to have an advantage on questions in this area. Large models performed strongly on historical and fact-based questions with clearer information gaps. Retrieval systems were effective when answers appeared in supporting documents, while complex sentence structure could still challenge systems that needed to extract the final answer.

These dimensions expose a weakness in a single aggregate score. Two agents may have similar overall accuracy while relying on distinct strengths. A benchmark dominated by direct, entity-rich questions can favor broad retrieval and memory, while a set with indirect narrative clues can reward flexible interpretation. The authors describe these patterns through their learned skill profiles and question clusters (Figures 5–9, pages 21538–21541).

The study has meaningful boundaries. Quiz Bowl is a specialized trivia format, and the human sample consists of experienced, fluent U.S. English players. The authors group individual players’ responses to address sparse data, which creates synthetic human agents rather than measuring intact teams. Public availability of questions and unknown model training data also make it difficult to know whether high model scores reflect genuine reasoning or exposure. The analysis does not establish that either humans or AI systems are categorically better at question answering.

Complementarity as a design question

For organizations using AI to retrieve and synthesize information, the paper suggests that task design should test distinct reasoning demands instead of relying on a single accuracy figure. The evidence points toward a possible division of strengths: systems can retrieve explicit facts efficiently, while people may contribute when clues are indirect or conceptually grounded. That interpretation needs validation in workplace tasks, but CAIMIRA offers a way to ask more precise questions about where human oversight adds value—and where an AI system needs better support.

This kind of decomposition can reveal where two agents disagree even when their overall scores look similar. It is a diagnostic view of the evaluated question-answering setup, not evidence that human-AI collaboration will always outperform either participant alone. The choice of questions and evaluation conditions remains important.

Bibliography & sources

1. Maharshi Gor, Hal Daumé III, Tianyi Zhou, Jordan Boyd-Graber. 2024. “Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA.” In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, 21533–21564. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.1201.