Evaluation Science & LLM-as-a-Judge
Rigorous methods for evaluating language models, embeddings and translation — including the limits of automated judges.
Why it matters: Because progress claims are only as sound as the evaluation behind them.
Featured events
- Findings of EMNLP 2025 — GASE and sign-language translation evaluation — Peer-reviewed publication, 2025
- EACL 2026 — PTEB: robust text-embedding evaluation — Peer-reviewed publication, 2026
Publications (4)
- PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs — Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026), Long Papers, 2026
- Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation — Findings of the Association for Computational Linguistics: EMNLP 2025, 2025
- GASE: Generatively Augmented Sentence Encoding — Findings of the Association for Computational Linguistics: EMNLP 2025, 2025
- Human Evaluation of English–Irish Transformer-Based NMT — Information, 13(7), 309, 2022
Milestones
- 2025 — Findings of EMNLP 2025 — GASE and sign-language translation evaluation
- 2026 — EACL 2026 — PTEB: robust text-embedding evaluation
See the full research agenda · collections · research journey.