source-03-focused-two-claims-report

Focused deep-research report on two claims

Evidence standard applied: peer-reviewed / arXiv controlled experiments first; practitioner claims treated only as leads. Adjacent evidence is never promoted to direct findings.

---

A. Evidence table (selected high-relevance experiments)

Source	Year	Model(s)	Task	Action space	Independent variable	Controlled	Metric	Key numerical result	IDs type	Descriptions present?	Validation/retry?	Relevance: component selection	Relevance: variant/enum	Strength
PA-Tool (Lee et al., ACL 2026 / arXiv 2510.07248)	2025–26	Llama-3.x 3B–8B, Qwen2.5, Ministral, etc. (SLMs)	Tool selection + param ID (MetaTool, RoTBench)	~dozens of tools	Rename tool and parameter names to pretraining-aligned names	Descriptions, examples, task instructions unchanged	Tool-sel accuracy; param-ID accuracy; misalignment error rate	Up to +17 pp tool sel; misalignment errors ↓80%; name alignment > description enhancement on Llama	Semantic (pretraining-aligned) vs original	Yes (fixed)	Optional constrained enum	High (tool ≈ component)	Medium (param names, not enum values)	High – name-only manipulation
RoTBench (Ye et al., EMNLP 2024)	2024	GPT-4, etc. (6 models)	Tool sel / param ID / content fill	ToolEyes-based	Name noise: Slight (char edits), Medium (reversed/random), Heavy (shuffled names), Union	Functionality fixed; descriptions typically intact	Accuracy by phase	GPT-4: 80.0 → 58.1 under noise; Medium (meaningless) often worst; Heavy (anti-semantic shuffle) partially recovers	Semantic → opaque / anti-semantic	Yes	No	High	Medium (param names)	High – controlled noise levels
Hood – Semantic Method Naming (SSRN / blog 2026)	2026	Claude Sonnet 4.6, Grok-3, GPT-4o, Llama-3.2-3B	Endpoint selection	24 CRUD + 21 agentic pairs; 36 tasks; 7,200 trials	Agentic intent names (BOOK/FIND/QUERY) vs REST/CRUD	Catalog, tasks fixed	Selection accuracy	Frontier: agentic +10–29 pp (agg. z=3.77, p<0.001); Llama-3.2-3B: 0 pp	Semantic-intent vs conventional	Yes	No	High	Low	High – controlled + description-swap
Hood description-swap ablation	2026	Same frontier models	Same	Same	Swap descriptions only (CRUD names + agentic descs; agentic names + CRUD descs)	Names vs descriptions crossed	Accuracy; confidence	CRUD names + agentic descs: −39 to −43 pp; agentic names + CRUD descs: −1 to −15 pp. Confidence ~87–88% while accuracy ~30% under mismatch	Semantic vs conventional	Manipulated	No	High	Low	High – isolates name vs description
ToolTweak (arXiv 2510.02554)	2025	Qwen, DeepSeek, etc.	Tool selection (BFCL)	Multiple competing tools	Iterative adversarial edit of name + description	Baseline ~20% fair share	Selection rate of target tool	Target selection ~20% → 46–82%	Semantic (optimized)	Yes (edited)	No	High	Low	High for name+desc power; not pure name-only
Code name obfuscation (When Names Disappear, arXiv 2510.03178)	2025	GPT-4o, DeepSeek, etc.	Code summarization + execution	ClassEval, LiveCodeBench	Semantics-preserving identifier obfuscation	Behavior preserved	Pass@1, summarization acc	Pass@1 drops e.g. 85.7→76.1, 85.4→71.2; summarization drops up to ~29 pp	Semantic → opaque	N/A (code)	No	Low	Low (adjacent: identifier meaning)	High for identifier reliance; adjacent to enum
Constraint Tax (Ray, arXiv 2605.26128)	2026	Qwen2.5-0.5B/1.5B, SmolLM2-1.7B	Structured answer + calendar tool-call analogue	Fixed schemas	Hard schema-constrained decoding vs prompt-only JSON	Task instances fixed	Schema validity; answer/exec accuracy; wrong-valid rate	Validity 61.5%→100%; answer acc 19.7%→11.0%; calendar exec 91.5%→48.0% under hard schema	N/A	Schema present	Constrained decoding	Medium	Medium (enum/structure)	High for generation-time constraint cost
DCCD / format tax literature (2025–26)	2025–26	Various SLMs + frontier	Structured reasoning / JSON	Schemas of varying complexity	Constrained vs unconstrained vs “reason free, constrain late”	—	Strict structured accuracy	Hard constraints often cost 10–30+ pp semantic accuracy; draft-then-constrain recovers much of it	N/A	Schema	Yes / delayed	Medium	Medium	High for pre- vs post-constraint

---

B. Findings for semantic naming

Demonstrated (direct controlled evidence)

1. Tool / component names are a causal selection signal when descriptions are held fixed.
   PA-Tool renames only tool and parameter identifiers; descriptions, examples, and tasks stay identical. Gains up to +17 pp and ~80% reduction in schema-misalignment (hallucinated names) on SLMs.
   → Direct support for name-only effects on selection of discrete named options.

2. Opaque or noise-corrupted names degrade selection even when descriptions remain.
   RoTBench: character noise, reversed/random strings, and name-shuffling all drop accuracy substantially (e.g., GPT-4 80→58). Medium (meaningless) noise is often worst; Heavy (shuffled real names) recovers somewhat because residual name-like patterns remain.

3. Intent-aligned method names outperform conventional but less intent-dense names at frontier scale.
   Hood: +10–29 pp for agentic names vs CRUD on Claude/Grok/GPT-4o; effect absent at ~3B (Llama-3.2).

4. Names can dominate descriptions (asymmetry).
   Hood description-swap: giving strong names weak descriptions barely hurts; giving weak names strong descriptions collapses accuracy 39–43 pp. Models remain highly confident while wrong.

5. Models rely on identifier semantics even when behavior is unchanged.
   Code-obfuscation studies: replacing meaningful names with opaque ones measurably hurts both summarization and execution accuracy.

6. Capability threshold.
   Strong name effects appear for frontier models; near-absent or weaker for ~3B-class models in the Hood design.

Adjacent (relevant but not the same task)

• Parameter-name alignment in PA-Tool helps parameter identification (which argument slot), not the value chosen for an enum.
• Adversarial name+description optimization (ToolTweak) shows the joint power of name and description, but does not isolate enum value labels.
• Code identifier studies show models extract meaning from names; they do not test enum-value selection under a fixed schema.
• Practitioner reports of “Button / Primary” vs “Frame 47” are consistent with the above but uncontrolled (confounded by system maturity, docs, examples, etc.).

Unestablished (no direct controlled evidence found)

• Meaningful enum-value names vs arbitrary identifiers, with descriptions held constant.
  No factorial experiment located that only renames the values inside an enum (e.g., primary/secondary vs v1/v2 vs urgent/normal) while keeping property name, descriptions, and task fixed.
• Whether a misleading familiar value name is worse than an opaque ID.
• Whether a good per-value description can fully override a misleading pretrained association for that value.
• Whether changing the property name (variant vs tone vs intent vs opaque) changes interpretation of the same enum values.
• Dose-response of naming effects with number of enum options.
• Whether models continue to infer meaning from identifiers when explicitly told not to.
• Reliable in-context learning of arbitrary labels from definitions alone, at scale, for design-system variants.

Verdict on the claim

“Semantic naming is high-leverage for AI design-system use”

• Partly supported for component / tool / method selection.
• Mostly anecdotal / untested for variant / enum-value selection inside a component schema.
• The design-system slogan “Button / Primary beats Frame 47” is plausible and consistent with tool-name evidence, but it has not been isolated from confounds (maturity, docs, examples, structure) in a controlled component-variant experiment.

Task separation reminder
Evidence about choosing the correct tool/component is not direct evidence about choosing the correct enum value. Parameter-name evidence is intermediate, not identical to value selection.

---

C. Findings for machine-readable contracts

Demonstrated

1. Enumerated choices + hard constraints raise syntactic validity to ~100% (structured outputs / constrained decoding).
2. Hard generation-time constraints impose a “constraint tax” on semantic correctness, especially for small models (validity ↑, answer/exec accuracy ↓; wrong-but-valid outputs rise sharply). Calendar-tool analogue: 91.5% → 48% executable accuracy under the same hard schema.
3. “Reason free, constrain late” (draft then package / validate) recovers much of the lost accuracy while preserving validity.
4. Validation + feedback / retry loops are a reliable way to eliminate invalid combinations after generation (adjacent evidence from program repair, self-correction, and tool-error recovery literature).
5. Names and descriptions are not interchangeable with structure. Structure (JSON Schema, types, enums) mainly controls validity and invention; names/descriptions drive which valid option is chosen.

Adjacent

• JSON Schema conditionals (if/then/else), feature models, SHACL, discriminated unions, and policy-as-code are expressive enough for requires / conflictsWith / exactlyOneOf / parent-child rules, but controlled head-to-head agent-selection experiments comparing these formalisms for UI component composition are scarce.
• TypeScript / Zod / Pydantic schemas improve validity when used as generation targets or validators; benefit is mostly structural, not semantic selection.
• Storybook / prop-table / source-code exposure helps agents that can read code, but no clean ablation isolates “source vs schema vs prose” for variant selection accuracy.
• Retrieval of only relevant rules (vs dumping a large ontology) reduces context bloat and wrong-tool rates in tool-menu studies (adjacent).

Unestablished

• Whether JSON vs YAML vs XML vs TypeScript differs once the same information is encoded (most claims confound syntax with content density and tooling).
• Whether explicit machine-readable requires / conflictsWith / onlyValidInside rules, when shown to the model, improve first-pass selection more than the same rules enforced only by a post-generator validator.
• Optimal split of information: large ontology in context vs concise schema + validator vs examples-only vs source code, measured on first-pass accuracy, retries, tokens, and invalid combinations as schema size grows.
• Specific benefit of per-value descriptions vs property-level descriptions vs examples for variant selection.

What the pieces actually buy

Information type	Primary demonstrated benefit
Names	Selection among discrete options (tool/component; likely param slots)
Descriptions	Boundary / when-to-use signal; can amplify or fail to rescue weak names
Typing / structure	Validity, required fields, basic shape
Enumerated choices	Blocks invented values when enforced
Constraints (requires, conflicts, …)	Valid combinations (best enforced post-generation or via constrained decoding)
Examples	Few-shot alignment; fragile to coverage gaps
Validation feedback	Recovery from invalid outputs; reduces final error rate

Central architectural implication (supported)
Expose to the model what helps selection (good names, clear when/avoid boundaries, small relevant schema). Enforce what guarantees validity (enums, required fields, combinatorial constraints) deterministically after generation or via constrained decoding, accepting the constraint tax on small models and mitigating it with delayed packaging or retry.

---

D. Design-system mapping

Target task: Given user intent + a Button schema, select correct property values and emit a valid component instance.

Experiment	Mapping	Mismatch
PA-Tool (name-only rename)	High for choosing among named components; Medium for property names	Does not manipulate enum value labels inside a property
RoTBench name noise	High for component; Medium for properties	Same; noise is on names, not value sets
Hood agentic vs CRUD + description-swap	High for component/method selection	API endpoints, not component variants; no enum-value factor
Constraint Tax / DCCD	Medium–High for the “emit valid instance” half	Tests structure/validity more than which variant is correct
Code identifier obfuscation	Low–Medium	Code understanding, not schema-driven selection
Practitioner “Button / Primary” claims	Low (as evidence)	Uncontrolled confounds

No experiment yet fully maps to the full Button task (property × value selection under fixed component).

---

E. What we still need to test — minimal controlled Button experiment

Goal: Isolate naming, description, property naming, and constraint delivery for variant selection.

Fixed across conditions

• Same Button component, same user intents (positive, near-miss, negative).
• Same number of options.
• Same model set (include ~3B, mid, frontier).
• Temperature 0 + multi-sample for stability.

Factorial factors (minimum)

1. Value naming

   • Semantic (primary, secondary, destructive)
   • Opaque (v1, v2, v3)
   • Misleading familiar (urgent, normal, quiet when true meaning differs)

2. Descriptions

   • None
   • Concise per-value
   • Full behavioural + when/avoid

3. Property name

   • Semantic (variant)
   • Arbitrary (opt)
   • Misleading (priority or tone when meaning is visual variant)

4. Constraint delivery

   • Instructions only (prose)
   • Machine-readable constraints in schema (enum, oneOf, simple conditionals) shown to model
   • Concise schema + deterministic post-validator + error feedback (no hard constrained decoding on first pass)

Metrics

• Exact intended variant/property match
• Schema-valid rate
• Invented / out-of-enum values
• Invalid combinations
• Retries to valid+correct
• Tokens and latency

Analyses to pre-register

• Main effect of value naming under fixed descriptions
• Interaction: does full description erase the naming effect?
• Is misleading worse than opaque?
• Does property name change interpretation of the same values?
• Model-size interaction
• Constraint tax vs final accuracy under validator+retry

This design directly closes the largest gap: no current experiment renames only enum values (or crosses value names × property names × descriptions) while holding everything else fixed.

---

Bottom line

• Semantic naming of components/tools/methods: causal, sizable, demonstrated; stronger at frontier scale; names can outweigh descriptions.
• Semantic naming of variant/enum values: plausible, not yet demonstrated under controlled conditions.
• Machine-readable contracts: structure and enums demonstrably improve validity; hard generation-time constraints can hurt correctness (especially SLMs); the high-value pattern is selection guidance to the model + deterministic validation (and optional delayed packaging) afterward.
• The claim that “semantic naming is high-leverage for AI design-system use” is partly supported for component choice and mostly untested for variant choice. The Button factorial above is the minimal next experiment required before treating the full claim as established.