When an Empty Evaluation Set Looks Like a Logging Crash
A merged DSPy fix turns an empty-devset division error into an explicit validation error at the evaluation boundary.
DSPyMerged 2026-08-26Hermes Labs · Roli Bosch
The failure and the consequence
An empty evaluation set reached the metric-summary calculation, where its length became a zero divisor. The resulting ZeroDivisionError pointed at a logging operation instead of telling the caller which input was invalid. Filtering a dataset down to zero examples could therefore produce a misleading traceback.
ZeroDivisionError during metric-summary loggingValueError: devset must contain at least one example, got an empty devset.The change
The patch checks the resolved devset at the start of Evaluate.__call__, before evaluation and metric aggregation. It raises ValueError with a message naming the empty devset. It preserves the existing requirement that evaluation needs examples; it does not invent a zero score for an evaluation that did not happen.
Regression evidence
The added regression calls Evaluate with an empty devset and asserts ValueError with a message containing devset. The public PR records the new test failing on the old source and passing with the guard, and reports 11 passing tests with 25 pre-existing skips in the evaluation module. This case study inspects that public record; it does not present those historical runs as a new execution.
What the fix delivers
An invalid evaluation input now fails where the contract is known. The error identifies the missing examples, and no empty evaluation is represented as a valid measured result.
Attribution and upstream outcome
The public PR follows a maintainer request on an earlier closed proposal (#1193): keep raising for empty data, but explain the failure. Roli Bosch submitted the guard and regression under roli-lpci. DSPy merged PR #9978 on 26 August 2026. This is an upstream framework bug fix; merge evidence alone does not identify an installed package version.
Inspect the public record
Pull request #9978 · Code and regression tests · Merge commit · Earlier upstream report or discussion
Complete contribution ledger · All case studies
Hermes Labs applies this diagnostic work to agent frameworks, retrieval, memory and evaluation systems. Read about diagnostic engineering or bring us a system and a symptom.