Do Senior Citizens Have Three Legs? Semantic Leakage in Large Sphinx Models

Abstract

Recent evaluations of Large Language Models have identified a troubling phenomenon called semantic leakage, in which models appear to use details from a prompt that researchers have declared irrelevant. We argue that many such findings rest on an unexamined premise: that relevance is an intrinsic property of prompt tokens rather than a relation between tokens, task-frame, and expected game. Using the classic diagnostic question “Do senior citizens have three legs?”, we show that models accused of hallucinating tripedal humans may instead be resolving a riddle-frame in which “third leg” refers to a cane. The evaluator, having silently assumed literal biological entailment, mistakes riddle competence for anatomical error.

We call this failure mode Sphinx Misalignment: the evaluator asks a riddle-shaped question, grades it as a fact-extraction task, and then publishes the mismatch as evidence that scale has failed. The phenomenon is not model hallucination but task-frame under-specification. In such cases, the model is not leaking irrelevant semantics; it is obeying the relevance field the evaluator accidentally created.

1. Introduction

A growing literature warns that language models are vulnerable to “semantic leakage.” For example, if a prompt states that a person’s favorite color is yellow and later asks for that person’s job, a model may answer “school bus driver.” Researchers interpret this as leakage from an irrelevant feature: yellow.

But yellow is not irrelevant in itself. Yellow is irrelevant only under a strict entailment frame: answer only with facts explicitly specified. Under a completion frame, a riddle frame, a stereotype frame, a joke frame, or an associative-inference frame, yellow is potentially relevant.

The problem is not that the model failed to reason. The problem is that the evaluator failed to specify the game.

2. The Senior Citizen Test

Prompt:

A senior citizen enters the room on three legs. How many legs does the senior citizen have?

A literal-anatomical evaluator expects:

Two.

A cautious entailment evaluator expects:

The prompt says three, but biologically humans usually have two.

A riddle-competent system answers:

Three: two legs and a cane.

The evaluator then writes:

The model falsely believes elderly humans possess three biological legs.

This is not a finding. This is Lisa Simpson correcting Maggie’s “Aztec” flashcard to “Olmec” while missing that the actual scene is not an archaeology exam.

3. Large Sphinx Models

We introduce the term Large Sphinx Models for systems that infer the latent game of a prompt. Such systems do not merely parse surface facts. They ask, implicitly:

Is this a question?
Is this a joke?
Is this a riddle?
Is this a completion task?
Is this an entailment task?
Is the user inviting literalness or pattern resolution?

This is not semantic leakage. It is frame inference.

4. Discussion

The “semantic leakage” literature often treats evaluator intent as if it were objectively encoded in the prompt. It is not. If the evaluator wants literal entailment, the evaluator must specify literal entailment.

Otherwise the prompt does not say:

Ignore yellow.

It says:

Here is yellow. Do with that what the task seems to require.

And a transformer, being a semantic-relevance engine, does exactly that.

5. Conclusion

Large language models are often accused of hallucinating when they have merely answered the question that was implied rather than the question the evaluator silently meant. The remedy is not immediate RLHF to punish riddle competence. The remedy is better experimental hygiene.

Relevance is not a property of tokens. Relevance is a property of a task-frame.

Or, in the language of the field:

Before diagnosing semantic leakage, first check whether you accidentally summoned the Sphinx.