Skill Injection Attribution

Attributing Structured-Output Gains in Function Calling: Interface Alignment versus Procedural Transfer

Accepted to Findings of EMNLP 2026. This is the compact public artifact for the accepted paper.

中文文档见 README_CN.md。

What this artifact tests

Function-calling scores combine two properties:

1. selecting the correct tool and arguments; and
2. expressing the call in the wrapper format expected by the evaluator.

This artifact provides the scorer and hand-written controls used to separate these effects. The paper's central diagnostic is simple: if a format-only instruction reproduces a gain, the gain should not be described as evidence of portable procedural transfer without additional controls.

What is included

src/
  evaluator.py          # JSON parsing and BFCL-style AST checks
  rescore_canonical.py  # standard vs. canonicalized rescoring
controls/
  format_only/          # explicit output-contract controls
  alias_only/           # wrapper-key controls
  wrong_alias/          # negative alias control
  schema_only/          # schema-only control
  schema_example/       # schema-plus-example control
  neutral_matched/      # length-matched neutral control

The public package intentionally contains method code and hand-written control templates only. It does not include model-generated skills, raw model outputs, benchmark manifests, per-item result dumps, or intermediate analysis files.

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

To rescore a directory of BFCL-style JSON files:

python src/rescore_canonical.py \
  --results-dir /path/to/bfcl_v2_results \
  --output /path/to/rescoring_analysis.json

Each input JSON file should contain records with model, condition, prediction, and ground_truth fields. The script reports standard scoring, canonicalized scoring, wrapper-key frequencies, and paired bootstrap summaries. Canonicalization is intentionally narrow: it normalizes wrapper-key aliases that preserve the same function-call structure; it is not a complete semantic executor or an official leaderboard rescore.

Paper and scope

• Venue: Findings of EMNLP 2026 (accepted)
• Authors: Wanyi Chen, Daoyuan Chen, Fang Kong
• Paper: arXiv:2607.02595
• English paper: README_EN.md

The paper evaluates BFCL-style structured output primarily, with API-Bank as a cross-contract diagnostic and MATH-500/MultiHop-RAG as scope checks. Reported numbers and the interpretation of each diagnostic are in the paper; this repository is the compact method release.

Citation

@misc{chen2026attributingstructuredoutputgainsfunction,
  title         = {Attributing Structured-Output Gains in Function Calling: Interface Alignment versus Procedural Transfer},
  author        = {Wanyi Chen and Daoyuan Chen and Fang Kong},
  year          = {2026},
  eprint        = {2607.02595},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SE},
  url           = {https://arxiv.org/abs/2607.02595}
}

License

The code and hand-written controls are released under the MIT License. The benchmarks and models referenced by the paper remain subject to their original licenses and terms of use.