Audit and deepened report
Only verified primary-source claims and carefully labelled adjacent evidence are retained. Unsupported or loosely transferred claims from the prior report are downgraded or removed.
A. Source verification table
| Source | Publication status | Exact experiment | Manipulated variable | Identifier class | Descriptions visible? | Result (verified) | Statistical test | Reproducibility | Prior-report claim verified |
|---|---|---|---|---|---|---|---|---|---|
| PA-Tool (Lee et al.) | ACL 2026 / arXiv 2510.07248v3 | Rename tool + parameter names via peakedness; MetaTool + RoTBench Clean | Tool names and parameter names only | Component/tool name + property/parameter name | Yes — descriptions, examples, instructions unchanged | “Up to 17%” tool-selection gains (Reliability subtask); schema-misalignment (plausible non-schema names) ↓ 80.0%; other error types ↓ 18.8–24.0%. Separate ablation: tool-name alignment helps selection; param-name alignment helps param ID. | Not always reported per cell; main claims presented as observed deltas | Code released (GitHub) | Yes for name-only effect on tool/param names with fixed descriptions. Partly if prior text implied enum values. |
| RoTBench (Ye et al.) | EMNLP 2024 / arXiv 2401.08326v3 | Five noise environments on tool + parameter names | Name noise: Slight (char edits), Medium (reversed/random strings), Heavy (shuffled names + extra params), Union | Component/tool name + property/parameter name | Yes (functionality fixed; names changed) | GPT-4 tool-selection Clean 80.00 → lower under noise; paper states drop to 58.10 under the noise condition referenced in abstract. Medium (meaningless) particularly damaging; Heavy (anti-semantic shuffle of real names) partially recovers relative to pure nonsense. | Not always formal p-values; clear multi-model pattern | Code + data released | Yes for sensitivity of tool/param names. No support for enum-value claims. |
| Hood semantic method naming | SSRN 2026 + author blog with full numbers | 7,200 trials; 36 tasks; 24 CRUD + 21 agentic endpoint pairs; mixed-paradigm selection | Method/endpoint names (BOOK/FIND/QUERY vs POST/GET style) | Component/tool name (API method) | Yes; description-swap ablation crosses names × descriptions | Frontier: agentic names +10–29 pp (Claude 59→88 = +29; Grok +18; GPT-4o +10). Aggregate z = 3.77, p < 0.001. Llama-3.2-3B: 0 pp, p = 0.95. Description-swap: CRUD names + agentic descs → −39 to −43 pp; agentic names + CRUD descs → −1 to −15 pp. Mismatch confidence ~87–88% at ~30–33% accuracy (calibration error +60–67 pp). | Yes (z, p reported) | Author states dataset/harness open; treat as high-quality technical report, not yet peer-reviewed journal | Yes for method-name causality and name > description asymmetry. Not enum values. |
| ToolTweak | arXiv 2510.02554 | Adversarial iterative edit of name + description to maximize selection | Name + description jointly | Component/tool name | Yes (both edited) | Target selection ~20% → 46–82% depending on model | Reported as large observed shifts | Available as preprint | Partly — shows joint power of name+desc; does not isolate name-only or enum values. |
| When Names Disappear | arXiv 2510.03178 | Semantics-preserving identifier obfuscation in code | Variable/function/class names → opaque | Code identifiers (adjacent) | N/A (code body preserved) | Pass@1 drops e.g. ClassEval 85.7→76.1; LiveCodeBench 85.4→71.2; summarization drops up to ~29 pp | Reported as consistent degradation | Benchmark released | Adjacent only — identifier reliance, not schema enum selection. |
| Constraint Tax (Ray) | arXiv 2605.26128 | Hard schema-constrained decoding vs prompt-only JSON; fixed instances | Presence of hard structural constraints at decode time | Structure / enum enforcement (not name semantics) | Schema present | Validity 61.5%→100%; answer acc 19.7%→11.0%; wrong-valid 49.5%→88.9%. Calendar tool-call: exec acc 91.5%→48.0% under hard schema (both 100% valid). | Confidence intervals reported | Measurement protocol described | Yes for constraint tax on SLMs. Does not test selection guidance vs validation architecture head-to-head. |
| DCCD / delayed packaging | Multiple 2025–26 papers (e.g. draft-conditioned constrained decoding) | Unconstrained draft then constrained packaging | Timing of constraint application | Structure | Varies | Recover substantial accuracy lost to hard constraints (e.g. up to +24 pp strict structured accuracy in reported settings) | Varies by paper | Preprints | Yes that delaying constraints helps; still not a full “guidance vs post-validation only” bake-off for design systems. |
Downgrades
- Any prior implication that PA-Tool or RoTBench tested enum-value labels is incorrect; they tested tool and parameter names.
- Hood is strong but not peer-reviewed journal yet; numbers are taken from the author’s detailed public write-up and are consistent across SSRN abstract and blog.
- ToolTweak is joint name+description; cannot be cited as pure name evidence.
B. Identifier evidence matrix
| Identifier class | Direct causal evidence | Adjacent evidence | No evidence found |
|---|---|---|---|
| Component / tool name | PA-Tool (rename only, descs fixed); RoTBench name noise; Hood method names + description-swap | ToolTweak (name+desc); code identifier studies | — |
| Property / parameter name | PA-Tool param-name ablation (helps param ID); RoTBench param-name noise | — | Effect of property name (variant vs tone vs intent vs opaque) on interpretation of identical value sets |
| Enum-value name | None in tool/schema selection literature reviewed | Label inversion / semantic-anchor ICL studies (classification labels, not schema enums); arbitrary vs meaningful labels in ICL | Controlled A/B/C comparison (aligned vs opaque vs misleading) for schema enum values with fixed descriptions |
| Enum-value description | None isolated for enum values | Tool/parameter description edits; Hood description-swap (at method level) | Whether per-value descriptions override misleading value names |
C. Semantic-conflict findings
Question: When a familiar pretrained meaning conflicts with local documented meaning, what happens to selection accuracy? Comparison of interest: aligned familiar (A) vs opaque (B) vs familiar-but-misleading (C), ideally with descriptions fixed.
Demonstrated
-
Models are anchored to pretrained label semantics and often cannot override them via few-shot inverted demonstrations (1–12B scale).
Semantic Anchors in In-Context Learning (arXiv 2511.21038): across 8 models × 8 tasks × 5 seeds, semantic override rate = exactly 0% under inverted label mappings. Models partially follow flipped demos but never learn a coherent anti-semantic classifier. Natural ICL refines priors; inverted ICL fails to remap meaning. -
Conflicting name vs description produces large accuracy collapse and severe overconfidence (method-name level).
Hood description-swap: weak names + strong (mismatched) descriptions → −39 to −43 pp; high confidence (~87–88%) at near-chance accuracy. This is name–description conflict, not pure value-name conflict, but it shows pretrained/name signal can dominate and that confidence is unreliable under conflict. -
Opaque / nonsense names hurt tool selection when descriptions remain (RoTBench Medium). Anti-semantic shuffle of real names (Heavy) is often less damaging than pure nonsense, consistent with residual name-like priors still providing some signal.
Adjacent
- Random labels in ICL often hurt little (Min et al. line of work), while homogeneous or inverted meaningful labels behave very differently — inverted meaningful labels are hard to override (semantic anchors).
- Code: meaningful identifiers improve understanding vs obfuscated ones; does not test local redefinition of a familiar name.
- No controlled schema experiment was found that holds descriptions fixed and compares only:
aligned value name vs opaque ID vs misleading familiar value name.
Unestablished
- Whether C (misleading familiar enum-value name) is worse than B (opaque) under identical descriptions and behaviour.
- Whether a strong local description fully overrides a misleading value name at frontier scale.
- Effect of model size on override success for schema enum values (semantic-anchor work stops at ~12B and uses classification labels, not tool schemas).
- Whether few-shot demonstrations of the local mapping override the prior for enum values.
- Confidence calibration specifically under misleading enum-value names with correct descriptions.
Strict answer to the key comparison:
There is no direct controlled evidence yet that C < B for schema enum values. Adjacent classification evidence suggests misleading meaningful labels are hard to redefine (override rate 0% in the 1–12B ICL study), which makes C plausibly dangerous, but that is not the same experiment.
D. Corrected conclusions
Semantic naming
- Demonstrated: Changing component/tool names (and, separately, parameter names) while holding descriptions fixed causally affects selection/identification accuracy (PA-Tool, RoTBench, Hood). Intent-dense method names outperform conventional less-intent-dense names at frontier scale; effect absent at ~3B in Hood’s design. Names can outweigh descriptions; name–description conflict produces large errors and high confidence.
- Not demonstrated: The same causal claim for enum-value names inside a property. No verified experiment renames only
primary/secondaryvsv1/v2vs misleading familiar tokens with descriptions held constant. - Claim status: “Semantic naming is high-leverage for AI design-system use” is partly supported (component/tool level) and unestablished (variant/enum-value level). Practitioner “Button / Primary vs Frame 47” remains anecdotal relative to controlled evidence.
Machine-readable contracts
- Demonstrated: Hard generation-time structural constraints raise validity (often to 100%) and can lower semantic/executable accuracy on small models (Constraint Tax: 19.7%→11.0% answer; 91.5%→48.0% calendar exec). Delayed constraint application recovers accuracy in several structured-generation settings.
- Prior architectural statement — “The high-value pattern is selection guidance to the model + deterministic validation afterward”:
Supported by adjacent evidence; not directly demonstrated by a head-to-head design-system or tool-schema experiment that compares (guidance + post-validator) vs (hard constraints in generation) vs (prose only) on the same Button-like task. It remains a plausible architecture hypothesis grounded in constraint-tax and delayed-packaging results. - “Hard constraints should mainly validate rather than guide selection”: same classification — plausible hypothesis, not a direct experimental finding for component/variant selection.
E. Implications for the Button evaluation
Strongly justified by literature (include)
- Manipulate component/tool-style names and property/parameter names with descriptions fixed (PA-Tool / RoTBench design).
- Cross names × descriptions (Hood-style swap) to measure conflict and confidence calibration.
- Include a ~3B vs frontier split (Hood capability threshold).
- Measure schema-valid rate, exact intended selection, invented values, and confidence when wrong.
- Compare hard constrained decoding vs unconstrained or lightly guided generation + deterministic validator (Constraint Tax / delayed packaging rationale).
Exploratory (necessary because evidence is missing)
- Enum-value naming factor: aligned semantic (
primary/secondary) vs opaque (v1/v2) vs misleading familiar (e.g. names whose pretrained meaning conflicts with the local definition), with identical value descriptions. - Property-name factor (
variantvsemphasisvsoption) crossed with the same value set and descriptions. - Explicit test of description override: misleading value name + strong local description vs opaque name + same description.
- Whether few-shot local definitions overcome pretrained value semantics (inspired by semantic-anchor null result).
What existing evidence can and cannot predict for the three Button sketches
| Schema sketch | What evidence can predict | What it cannot predict |
|---|---|---|
variant: primary / secondary + clear descriptions | Component/property naming literature suggests familiar, intent-aligned labels help selection among named options when descriptions match. | Whether these particular value strings are optimal; whether a misleading alternative would be worse than opaque; magnitude for this exact schema. |
emphasis: strong / medium + same descriptions | Same as above if “strong/medium” are treated as familiar intensity terms; risk of pretrained associations that may not match design-system intent is untested. | Whether “emphasis” as property name changes interpretation relative to “variant”. |
option: v1 / v2 + same descriptions | RoTBench-style opacity suggests possible degradation vs meaningful names when models rely on name priors; PA-Tool suggests aligning to model-preferred forms can help. | Whether opacity is better or worse than a misleading familiar name; whether strong descriptions fully compensate. |
No ranking among the three sketches is justified by current controlled evidence.
Bottom line after audit
- Verified causal effects exist for tool/component names and parameter names, with fixed descriptions, and for name–description conflict at method level.
- Enum-value naming and misleading vs opaque value names remain unestablished for schema selection.
- Semantic-anchor ICL work shows pretrained label meanings are hard to override (0% override in 1–12B inverted settings); this is adjacent, not direct schema evidence.
- The “guidance to model + deterministic validation after” pattern is a well-motivated hypothesis, not a directly demonstrated optimum for design-system variant selection.
The minimal Button experiment that closes the gap is still the factorial on value naming (aligned / opaque / misleading) × descriptions × property name × constraint delivery, with the justified factors above held as primary and the enum-value conflict conditions treated as the critical exploratory block.