# MeloText-ID v0.1.3 pilot protocol

## Primary endpoint

Closed-set top-1 identification accuracy. Each item contains one symbolic melody
and 25 randomized candidate titles. No tools, browsing, file metadata, audio,
candidate elimination across turns, or shared conversation state are allowed.

## Repetition and aggregation

Each model receives the same ten independently randomized balanced manifests.
Inside a repetition, overall and four condition scores are arithmetic accuracy.
Across repetitions, the official values are exact geometric means. No epsilon is
added: any zero input yields a zero raw geometric mean.

## Randomization

The planner derives permutations with HMAC-SHA256 from a run seed. Within each
condition and repetition, each correct answer position C01-C25 occurs exactly
once. The 24 distractors are HMAC-sorted separately for every item. All models
receive identical choice permutations.

## Automated grading

JSON object mode is requested uniformly across providers, and every prompt ends
with the same short format reminder after the melody. Local parsing requires
exactly `item_id` and `choice_id`, with no additional properties. It also accepts
that single valid object when the only surrounding characters are one-to-three
Markdown backticks (with an optional `json` tag); this normalization is recorded
as `ok_wrapped`. Prose, multiple objects, duplicate keys, and any other trailing
material remain invalid. Returned item IDs must match. Malformed, ambiguous, refused, or
out-of-range outputs are incorrect; they are never retried. Only infrastructure
errors eligible under the frozen retry policy may be retried with an identical
payload.

## Reasoning controls

Reasoning behavior is explicit rather than left to changing provider defaults.
MiniMax M3, Qwen 3.7 Max, DeepSeek V4 Pro, and GLM 5.2 run with reasoning disabled.
Every other route uses the `reasoning_effort` frozen in
`config/experiment.toml`; returned reasoning is excluded from the response
envelope but its tokens remain billed and audited. All routes receive the same
2,048-token maximum response allowance.

Version 0.1.2 followed an aborted 0.1.1 formatting pilot that exposed an omitted
MiniMax reasoning control. The 0.1.2 qualification run was also aborted after 5
of the first 15 GLM responses exhausted 2,048 tokens without an answer. A larger
GLM allowance remained nondeterministic: the same prompt later exhausted 8,192.
Reasoning-off GLM responses completed quickly but sometimes added one trailing
backtick, motivating the uniform wrapper normalization above. Eight long
DeepSeek canaries passed with reasoning disabled, while its larger-allowance
Ionstream requests failed provider-side. No 0.1.1 or 0.1.2 scores are benchmark
results.

## Model routing

Requests use exact OpenRouter model IDs and pinned provider endpoint slugs. Ten
routes are first-party. DeepSeek V4 Pro is pinned to Ionstream FP4 because the
account data policy makes DeepSeek's own endpoint ineligible.
`allow_fallbacks=false` prevents silent provider substitution. The runner saves
the live model and endpoint catalog before inference, as well as the resolved
model/provider returned for every request.

## Costs

The local inference guard stops dispatch before observed cost plus reserved
in-flight cost can exceed $142. OpenRouter's $150 dedicated-key limit provides a
second independent ceiling and leaves room for qualification and aborted pilots.
Planning uses frozen per-model billed-token estimates. Grok 4.5 has a larger
reservation bound because its mandatory hidden reasoning can be billed outside
the nominal response-token field; this affects accounting and run order, not the
prompt or score. Purchase fees are outside model inference and are included when
enforcing the project's $200 total budget.

## Interpretation

This benchmark measures identification of exceptionally familiar melodies from
textual monophonic encodings under duration and transposition manipulations. It
does not identify the mechanism responsible, prove literal synesthesia, measure
listening ability, or establish general musical intelligence.