When the System Prompt Vanishes: A Silent Failure in Semantic Kernel

The same binary-gated discipline Hermes Labs offers as a diagnostic pipeline for private stacks also governs how it produces a public upstream fix: pass a gate, or stop. No gate is skippable. No gate is judgment-graded. The intent is that an opinionated process produces a falsifiable artifact at each step, so the diff at the end is one a maintainer can adopt on its own merits.

Each gate emits a falsifiable artifact, so the work can be re-checked rather than trusted. The proprietary tooling that runs the pipeline stays in-house.

The defect was the truncation reducer routing through a summarization-only helper that strips system and developer messages. The fix stops that routing and preserves the system prompt explicitly. The target behavior did not have to be invented: Microsoft had already corrected the same defect on the .NET side in PR #10344, which served as the verified reference for what correct looks like. The truncation reducer now locates and retains the first system or developer message:

# Preserve the first system/developer message so it is not
# lost during truncation (matches the .NET SDK behavior).
system_message_index = next(
    (i for i, msg in enumerate(history)
     if msg.role in (AuthorRole.SYSTEM, AuthorRole.DEVELOPER)),
    -1,
)
system_message = history[system_message_index] if system_message_index >= 0 else None

The reduction index calculation takes a new has_system_message flag so the preserved prompt is accounted for in the target count, with a guard added so a small target_count cannot produce an IndexError:

if has_system_message:
    target_count -= 1
    if target_count <= 0:
        logger.warning(
            "target_count after accounting for system message is %d; "
            "reduction will keep only the system message.",
            target_count,
        )

The patch touched the truncation reducer and the shared reducer utilities. The merged PR updated the existing truncation tests and added new ones covering system messages, developer messages, the no-system case, and the target_count=1 edge. The .NET fix gave the reference behavior, and the Microsoft maintainers reviewed and merged the Python port. The final merged patch also repairs the sibling summarization reducer: it identifies the first system or developer message that is not a generated summary, and prepends it if reduction removed it. Its regressions cover system and developer messages and exclude generated summary messages from that preservation rule. Inspect the merged reducer and its tests.

For a team that runs a history reducer to keep long conversations inside the context window, the cost of this failure is every long conversation in which the model ignored its system prompt: the safety framing, the tool-use rules, the persona, the task constraints. There is no error to catch, no metric that moves, no log to grep. A short test conversation never grows long enough to trigger truncation, so the deletion is invisible at runtime and surfaces only in long-running production sessions that are hardest to inspect. The failure is reproducible and testable once you know to look, which is precisely why a targeted regression test pins it and why the diagnosis is the work: the bug hides not because it cannot be tested, but because nothing prompts you to write that test.

This case is public, so the credit is shared. The symptom was reported by a community user, and the correct behavior was already established on the .NET side. The Python port carried that behavior over, held up in a Microsoft maintainer's review, and merged. When Hermes Labs audits a private stack, the same discipline turns to discovery, and the deliverable is the receipts: the reproduction script, the falsification log, the failing-then-passing regression test, and the verified contract behind the fix. A buyer can read the report cold and decide whether the failure applies to their stack and whether the audit checked the right things.

This is the same pipeline Hermes Labs runs whether the target is a chat history reducer, a retrieval layer, a prompt template, an agent harness, or a tool definition. The systematic failure categories vary; the gate structure does not. Our upstream contributions are listed at /open-source/contributions.

Seeing this in your own system? A paid diagnosis applies the same method to your system and symptom. The tools, papers and receipts stay free.

If you ship an LLM-based product where the model is supposed to follow a system prompt, ask what happens to that prompt when the conversation gets long. A silent instruction relaxation does not throw, does not log, and does not move a dashboard. A paid diagnosis can apply this method to your stack under agreed terms.

References: SK #13610 · SK #12612 · .NET #10344 · PR filed under our founder's GitHub handle, roli-lpci · Upstream contributions: /open-source/contributions