Answer Consistency
Stability of answer labels across repeated trials.
AACL 2026
Structural vs Cultural Reasoning
Reliability Limits of LLM-as-a-Judge in Multilingual Evaluation
Shehenaz Hossain · Houda Bouamor · Haithem Afli
Study
This study examines whether LLM-based judges reliably evaluate cultural reasoning, comparing automated judgements with human annotations across Arabic, Bengali and Spanish.
The analysis separates dimensions that are relatively observable from those that require culturally situated interpretation.
Evaluation setup
Arabic · Bengali · Spanish
Three multilingual settings.
6
Five open-weight models plus GPT-4.1.
5
Two structural and three cultural dimensions.
Dimensions
Explanation Consistency
Stability of reasoning patterns and justification structure.
Cultural Fluency
Whether explanations are culturally grounded and contextually appropriate.
Linguistic Adaptation
Whether explanations adapt naturally to the target language while preserving cultural stance.
Deviation
Whether explanations preserve cultural intent rather than drifting towards generic reasoning.
Key observation
Automated judges align more closely with human assessment on structurally observable dimensions than on culturally interpretive dimensions. The distinction matters because a fluent or internally consistent explanation can still miss culturally situated grounding.
In one Bengali example, human annotators rated Cultural Fluency at 2.5 on average while Aya-Expanse-8B assigned 5.0, despite consistent answer labels across repeated trials.
Related work
CRaFT — An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Models.
PalmX 2025 — Arabic cultural question answering with full and parameter-efficient LLM fine-tuning.
Beyond the Benchmark — dynamic and distribution-based evaluation in NLP.