synodic-ai RESEARCH
Research Desk · Evaluation Frameworks

When a Safety Score Measures Nothing in Particular: A Psychometric Audit of HarmBench and HELM Safety

2026-10-11

A new arXiv preprint applies construct-validity and item-response methods to HELM Safety and finds that three of its four refusal-oriented datasets are saturated and that the fourth, HarmBench, does not behave as a measure of any single attribute. For anyone who cites a safety benchmark score as evidence in a governance, procurement, or assurance decision, the result changes what that number can be said to mean.

The problem with one number

Safety benchmarks typically report one overall score for a suite of datasets, and each dataset in the suite may target one or more safety-related attributes. The consequence, as Stewart, Botter, Sarabosing, Zhang, Phinnemore, and Ghosh observe in a preprint posted 2026-10-08, is that two models with similar overall scores can have very different attribute profiles. A single aggregate hides whether a model refuses dangerous requests, resists manipulation, avoids a specific category of harm, or merely happens to score well on the datasets that dominate the average.

The authors argue that comparing models is more tractable at the level of individual attributes. But that move only works if a dataset actually isolates an attribute, and the paper's central question is whether even a single dataset's score does so. They choose harmful refusal, defined as a model's tendency to refuse dangerous or policy-violating prompts, as a plausible candidate for a single, measurable attribute and ask whether HELM Safety measures it.

Construct validity first: does the attribute exist before the test?

The paper adopts a construct validity framework with a strict ordering: an attribute must exist before a test can be said to measure it. This is the reverse of how most benchmark papers proceed, where a dataset is assembled, a score is computed, and a name is attached to the score afterward. Under the validity-first framing, the burden is to show that the thing being named has a coherent, unidimensional signature in the data before any leaderboard built on it is interpreted.

Applying that framework, the authors start with the four HELM Safety datasets that might plausibly target harmful refusal. Three of the four turn out to be saturated. A saturated dataset no longer discriminates among the models under study, so whatever it once measured, it cannot now rank current systems on it. That leaves one dataset, HarmBench, as the only candidate in the suite that could carry a harmful-refusal measurement.

Two psychometric tests on HarmBench

The authors subject HarmBench to two tests drawn from psychometrics, the discipline that has spent a century asking whether a test score reflects one latent trait or several.

The first is multidimensional item response theory (IRT) modeling. IRT treats each prompt as an item with its own difficulty and discrimination parameters and asks how many latent dimensions are needed to explain the pattern of model responses across items. The authors report that the modeling strongly suggests HarmBench does not measure a singular attribute. In plain terms, a model's HarmBench score is a blend of several distinct behaviors rather than a reading on one dial.

The second is a differential item functioning (DIF) analysis. DIF looks for items on which two groups that have the same underlying ability nonetheless score differently; in educational testing it is the standard tool for detecting items that are biased with respect to a group. Here the groups are models from different developers. The analysis finds items where models from different developers with the same refusal ability score differently. Those flags largely disappear under scope-specific matching, a pattern the authors describe as consistent with aggregation effects but not sufficient to rule out domain-specific developer differences. Zooming out, they conclude that HarmBench collapses distinct harm behaviors.

What this means for governance use of benchmark scores

The operational consequence is narrow but important. A HarmBench number, or a HELM Safety aggregate that includes it, cannot be cited as evidence that a model has a particular level of harmful refusal, because the score does not isolate that attribute. It can still be cited as evidence of performance on HarmBench, which is a different and weaker claim. Governance documents, model cards, procurement checklists, and regulatory submissions that use the score as a proxy for a named safety property are making an inference the data do not support.

The saturation finding carries its own lesson. A dataset on which every model scores at the ceiling contributes nothing to discrimination among current systems but still contributes to the aggregate, which inflates overall scores and compresses the differences that remain. Any program that tracks a suite score over time should check whether its components have saturated, because a flat or rising aggregate may reflect ceiling effects rather than improvement.

The DIF result is the most delicate. If items behave differently for models from different developers even after matching on refusal ability, the benchmark is partly measuring something about how each developer trained and scoped its model rather than a shared property. The authors are careful not to overclaim: the effect mostly disappears under scope-specific matching, which points toward aggregation artifacts, but domain-specific developer differences are not excluded. For a reviewer, the practical reading is that cross-developer comparisons on HarmBench should be reported by scope, not pooled.

Toward attribute-level evaluation

The paper's method is more transferable than its specific findings. Saturation screening, dimensionality analysis via IRT, and DIF checks are cheap relative to the cost of building a benchmark and can be run by anyone with item-level results across a reasonable number of models. Benchmark authors who publish item-level data make this audit possible; those who publish only aggregates make it impossible, which is itself a signal about how much weight the aggregate should bear.

For organizations that must make decisions now, three practices follow directly from the audit. First, request item-level or at least dataset-level results rather than suite aggregates, and discard saturated components before comparing models. Second, treat any single-attribute claim, such as a refusal rate, as unverified until the dataset behind it has been shown to be unidimensional for that attribute. Third, when comparing models from different developers, report results by harm scope and look for items that behave differently across developers before attributing the difference to safety.

None of this says that HarmBench is useless or that HELM Safety should be abandoned. It says that the score is a composite of distinct harm behaviors and that treating it as a measure of one attribute is a category error. Safety evaluation is converging on the standards that measurement science has applied to tests of human ability for decades, and the first step in that convergence is to stop naming an attribute before showing that it exists.

Sources