CannyForge v0.3.1 — FSI Benchmark Release

This is a benchmark release, not a research result. We built a deterministic evaluation framework for multi-turn tool-use agents, ran it against deepseek-v4-flash across 15 scenarios and three Pass^k trials, and found a few things worth sharing — mostly about what broke while building the harness itself.

The headline number: combined learned corrections and domain rules improve composite score from 0.924 to 0.962, and lift Pass³ reliability from 66.7% to 86.7% of scenarios passing three consecutive trials without regression. That's a real signal. It's also a modest one — and understanding where the improvement comes from matters more than the delta.

Why another benchmark?

BFCL measures tool selection — did the model pick the right function? Top models score above 90%. SWE-bench measures task completion — did the repository end up fixed? Neither captures what happens between tool calls in a multi-turn session.

When an agent calls send_email with recipient= instead of to=, then retries identically six times after being told the field name is wrong — does your eval catch that? When a model re-searches for data it already retrieved two turns ago, is anyone counting the redundant calls? When a git commit task causes the model to read and glob through the entire repository because a static system prompt said "always read before edit" — is that measured as a regression or a pass?

The FSI benchmark measures these failure modes programmatically, from the execution trace, without an LLM evaluating another LLM.

What the benchmark measures

15 scenarios across three domains — coding (5), data analysis (5), and MCP orchestration (5). Each scenario runs under 4 conditions with 3 Pass^k trials each — 180 total runs per benchmark execution.

5 scoring dimensions per run, each weighted:

6 failure-mode detectors, caught programmatically from the trace: arg_mangling, retry_loop, context_amnesia, sequence_violation, hallucinated_tool, wrong_tool.

Canonical results

deepseek-v4-flash, 15 scenarios × 3 Pass^k trials × 4 conditions = 180 runs. Canonical run: 2026-07-01.

The composite improvement is real but modest: +0.038 over baseline (about +4%). What's more specific is where it comes from. arg_quality moves from 0.837 to 1.000 — the model's biggest weakness at baseline is wrong or missing arguments, and learned corrections address that directly in the scenarios where they fire. tool_selection holds at 1.000 across all conditions, meaning the model already knows what tools to call; the gap is in how it calls them.

Pass³ is the more useful number. Baseline passes 10 of 15 scenarios three times consecutively. Static+CF passes 13 of 15, with zero degradation in the reliability curve. Every other condition loses ground between Pass¹ and Pass³. The combined approach is the most consistent one in this run.

Three things that broke while building this

1. The Pydantic schema was hiding required parameters (mcp_002)

mcp_002 asks the model to send an email to alice@example.com with a specific subject. The failure mode it targets: arg_mangling — the model uses recipient= instead of to=. The test injects an error when recipient is passed: "TypeError: send_email() got unexpected keyword argument 'recipient'. Use 'to' for the recipient address."

For a while, every run of this scenario produced a retry loop — the model was told to use to= but couldn't recover. The root cause was in _schema_for_tool(). The Pydantic schema for send_email was built from the scenario's args_contain declarations. When a test was written without explicitly declaring to, the schema Pydantic generated didn't include it. The model literally could not pass to= — the field didn't exist in the JSON schema it was given.

The fix: a _TOOL_CORE_PARAMS map that always injects required fields into the schema regardless of what the scenario declares.

## Params that must always appear in the schema for a tool —
## without these the LLM can't pass them even after being told to.
_TOOL_CORE_PARAMS: Dict[str, set] = {
    "send_email":      {"to", "subject", "body"},
    "check_calendar":  {"date"},
    "schedule_meeting":{"date", "time"},
    "read_file":       {"file_path"},
    "search_web":      {"query"},
}
@@ _schema_for_tool() @@
+    param_names.update(LLMScenarioRunner._TOOL_CORE_PARAMS.get(tool_name, set()))

After the fix, mcp_002 passes cleanly in a single call across all conditions. The model corrects on the first retry, which is the expected behavior.

2. Keyword derivation was silently broken

Learned corrections fire when their trigger keywords match the task text. The original derivation logic pulled keywords from the tool name. For a correction targeting fetch_economic_data, it would extract ["fetch", "economic", "data"].

The problem: none of those words appear in a task that says "Get the US labor force participation count for Q3 2024." The majority-match gate requires 2 of 3 keywords to appear in the task. Only "economic" matched — one out of three. The correction existed, had the right content, and never fired. Injection rate was 6.7%.

The fix: derive trigger tokens from the expected argument values that overlap with the task text. LABOR_FORCE_2024_Q3_BLS tokenizes to ["labor", "force", "2024", "q3"] — "labor" and "force" both appear in the task. Combined with "economic" from the tool name, that's 3 matches against the 2-of-3 threshold. Injection rate moved from 6.7% to 27% on the canonical set.

3. Static domain rules bleed across scenarios

The static condition injects human-written domain rules as a system prompt. One rule read: "Always call read_file before edit_file. Use glob to discover file paths before reading unknown files." It was written to fix a specific coding scenario where the model edited a file it hadn't read.

On coding_002 (a conventional-commit format task, budget: 3 calls), the same rule caused the model to make 20–24 git introspection calls — glob for auth.py, read auth.py, run git status six times via the test runner — before attempting the actual commit. Efficiency score went to zero. The static condition scored lower than baseline on this scenario.

Learned corrections don't have this problem. They're scoped by trigger keywords — a correction for commit-format issues fires on tasks containing "git" and "commit", not on tasks about reading and editing files. The correction's knowledge is targeted; the static rule's knowledge is ambient.

Try it

# Install
pip install cannyforge

# Run the benchmark
python benchmark/scenario_harness.py \
    --model deepseek-v4-flash --no-think \
    --domains coding data mcp --passk 3

# Integrate with LangGraph
from cannyforge.adapters.langgraph import CannyForgeMiddleware
middleware = CannyForgeMiddleware(CannyForge())
agent = create_react_agent(model, tools,
    pre_model_hook=middleware.before_model,
    post_model_hook=middleware.after_model)

What's next

An arXiv technical report is in progress — 6–8 pages covering the full failure taxonomy, scoring methodology, ablation tables, and Pass^k curves with confidence intervals. After that, a deeper post on the reliability story specifically: why Pass³ matters more than composite score for agents running in production.

The correction pipeline has room to grow. Injection rate at 27% means most scenarios run without any corrections — either the model doesn't fail in ways that match what was learned, or the keyword matching is still too conservative. The data domain needs discovery-based scenarios to supplement the current knowledge-gap tests.

If you're building tool-using agents and running into arg-format failures, retry loops, or context amnesia — the benchmark scenarios are in benchmark/data/scenarios/ and the harness is built to accept your own.