When an Empty Evaluation Set Looks Like a Logging Crash
A merged DSPy fix turns an empty-devset division error into an explicit validation error at the evaluation boundary.
DSPyMerged 2026-08-26Hermes Labs · Roli Bosch
The failure and the consequence
An empty evaluation set reached the metric-summary calculation, where its length became a zero divisor. The resulting ZeroDivisionError pointed at a logging operation instead of telling the caller which input was invalid. Filtering a dataset down to zero examples could therefore produce a misleading traceback.
ZeroDivisionError during metric-summary loggingValueError: devset must contain at least one example, got an empty devset.The change
The patch checks the resolved devset at the start of Evaluate.__call__, before evaluation and metric aggregation. It raises ValueError with a message naming the empty devset. It preserves the existing requirement that evaluation needs examples; it does not invent a zero score for an evaluation that did not happen.
Regression evidence
The added regression calls Evaluate with an empty devset and asserts ValueError with a message containing devset. The public PR records the new test failing on the old source and passing with the guard, and reports 11 passing tests with 25 pre-existing skips in the evaluation module. This case study inspects that public record; it does not present those historical runs as a new execution.
What the fix delivers
An invalid evaluation input now fails where the contract is known. The error identifies the missing examples, and no empty evaluation is represented as a valid measured result.
Attribution and upstream outcome
The public PR follows a maintainer request on an earlier closed proposal (#1193): keep raising for empty data, but explain the failure. Roli Bosch submitted the guard and regression under roli-lpci. DSPy merged PR #9978 on 26 August 2026. This is an upstream framework bug fix; merge evidence alone does not identify an installed package version.
Receipt
MethodTraced an empty evaluation set to the zero divisor in metric-summary logging, then added a check at the start of Evaluate.__call__ and a regression test, following a maintainer request on the earlier proposal #1193.
ResultDSPy merged PR #9978 on 2026-08-26. An empty devset now raises a ValueError that names the devset, instead of a ZeroDivisionError inside logging, and no zero score is invented for an evaluation that did not run.
Could not detectThe PR's recorded test runs were read from the public record, not re-executed for this study, and merge evidence alone does not identify an installed DSPy version.
ArtifactsPR #9978 · Merge commit · Earlier proposal #1193 · Contribution ledger
Seeing this in your own system? A paid diagnosis applies the same method to your system and symptom. The tools, papers and receipts stay free.
Inspect the public record
Pull request #9978 · Code and regression tests · Merge commit · Earlier upstream report or discussion
Complete contribution ledger · All case studies
Hermes Labs applies this diagnostic work to agent frameworks, retrieval, memory and evaluation systems. Read about diagnostic engineering or bring us a system and a symptom.