Construct
Specify the capability and the claim the evaluation is intended to support.
Research programme · Evaluation science
Beyond the Benchmark
Dynamic and distribution-based evaluation in NLP — from stochastic paraphrasing, to multidimensional robustness, to Measurement-as-Distribution.
This programme asks what we learn when evaluation is treated as a measurement process rather than a single fixed score. The work makes variation, uncertainty and robustness visible through controlled generation, perturbation and repeated evaluation.
Research arc
PTEB
Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs.
Introduces stochastic paraphrasing at evaluation time and aggregates results across repeated runs.
ACL Anthology ↗HTEB
The Harder Text Embedding Benchmark: Beyond One-dimensional Static Robustness.
Tests robustness across Lexical/Stylistic, Length and Language axes using stochastic transformations.
arXiv ↗Measurement-as-Distribution
The Benchmark Is Not the Construct: Toward Distribution-Based Evaluation in NLP.
Generalises the earlier work into a model- and task-agnostic evaluation protocol.
Method ↗Selected publications
- 2026The Benchmark Is Not the Construct: Toward Distribution-Based Evaluation in NLP
Haithem Afli · Manuel Frank · EMNLP 2026 · Findings
Research summary ↗ - 2026The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness
Manuel Frank · Haithem Afli · EMNLP 2026 · Main Conference
Preprint ↗ - 2026PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs
Manuel Frank · Haithem Afli · EACL 2026 · Long Paper · pp. 2832–2851
Paper ↗
Measurement-as-Distribution
The proposed protocol represents an evaluation as P = (C, G, Φ, E, R): construct specification, data generation, validated perturbation, repeated evaluation, and reporting.
Generate
Create evaluation evidence through a principled and reproducible process.
Perturb
Introduce validated variation while preserving the intended construct.
Evaluate
Repeat evaluation so uncertainty and variability become observable.
Report
Report distributions and stability alongside point estimates.
What the work examines
Score versus rank
A model can retain its relative ordering while its absolute score changes under valid perturbation.
Multiple robustness axes
Lexical/stylistic, length and language variation can produce partly decoupled robustness profiles.
Evaluation as evidence
When inputs and conditions vary, the distribution of measurements becomes part of the evidence.
Related research
Cultural Adaptation in NLP — culturally grounded multilingual reasoning and evaluation.
Dr Haithem Afli — Academic Website — research, publications, projects and academic profile.