← Spring of Winds Labs

Validation evidence

100% synthetic, no PII, no scraped content · 28,922 rows validated, 100% pass · SHA-256 provenance manifests · Apache-2.0 free samples

Measured validation evidence

Every number below is produced by running validate_jsonl.py (stdlib-only, included in the repository) against the shipped dataset files. No sampling — 100% of rows are validated. Last measured: 2026-11-19.

MetricResultMethodology
Schema validation pass rate28,922 / 28,922 (100.00%)Deterministic structural validation of every row across all lots and editions
Invalid JSON rows0Strict JSONL parsing, UTF-8, one sample per row
Phantom tool calls / undeclared arguments0 (fails the row)Every call checked against declared tool + argument schema
Near-duplicate rate0 at threshold3-gram Jaccard ≥ 0.85 exclusion vs. all prior rows incl. shipped lots
Validator self-test23 mutation classesGold rows must pass; deliberately corrupted variants must be caught

Full per-lot breakdown: evaluation/EVALS.md · Reproduce locally:

python validate_jsonl.py --track mcp_tool_use path/to/dataset.jsonl

Free 50-row samples (Apache-2.0)

The samples use the same schema as the full datasets, so you can wire them into your harness (Axolotl, Unsloth, custom) first. The MCP and CoT repositories use Hugging Face's information-sharing gate (access is automatic); the two Function Calling repositories are not gated.

One row, verbatim, from a shipped lot

Row ecom_research_pricing_retry_001 from the MCP Agent Trajectory 1K commercial lot — unedited, exactly as delivered. It shows the core training signal: a runtime failure, a strategy revision, and a verified recovery across chained tool calls.

TurnContent (verbatim)What this trains
userI'm looking at a competitor, 'ShopFlow Pro'. Can you find their monthly subscription pricing for the 'Enterprise' plan?Open-ended task requiring multi-step tool use
assistant
{"name": "web_search", "arguments":
 {"query": "ShopFlow Pro Enterprise plan pricing"}}
Strict tool-call JSON — declared tool, declared argument keys only
tool
{"error": {"code": "429", "message":
 "Rate limit exceeded. Please wait before retrying."}}
Injected failure. Most datasets stop at happy paths — this is where ours begins.
assistant
{"name": "web_search", "arguments":
 {"query": "ShopFlow Pro pricing page"}}
Recovery, not verbatim retry. The query is revised, not blindly re-sent — verbatim retries fail validation.
tool
{"results": [{"title": "ShopFlow Pro - Pricing Plans",
 "link": "https://shopflowpro.example.com/pricing"}]}
Partial progress — a page, not yet the answer
assistant
{"name": "scrape_page", "arguments":
 {"url": "https://shopflowpro.example.com/pricing"}}
Chained second tool, arguments grounded in the prior result
tool
{"content": "Basic: $49/mo. Pro: $99/mo.
 Enterprise: $299/mo. ..."}
Ground truth arrives
assistantThe ShopFlow Pro Enterprise plan is priced at $299 per month.Grounded final answer — matches tool output exactly; ungrounded numbers fail validation

All domains are synthetic (example.com) — no scraped content, no PII.

Free sample vs. commercial lots

Free sample (Hugging Face)1K lotExtended
Rows50 per trackMCP 1,003 · others 1,0001K lot + 2,300 non-overlapping rows (MCP 3,303 · others 3,300)
1K lot priceFreeMCP $7,300 · FC-EN $6,400 · FC-JA $8,500 · CoT $12,500
LicenceApache-2.0Perpetual, organization-wide, no redistribution
Schema validation reportPublic (EVALS.md)Included per delivery (stats.json)
SHA-256 integrity manifest—Included (SHA256SUMS.txt)
Overlap guarantee—Hash + ID + 3-gram Jaccard audited against all prior lots

How quality is verified

Generation-time schema gating → deterministic structural validation (declared tools/arguments only, strict role alternation) → 3-gram Jaccard ≥ 0.85 de-duplication against all shipped lots → independent LLM scoring → final review. The deterministic stage is public and reproducible (evaluation/).

New lots are audited (hash, ID, 3-gram Jaccard) against previously shipped lots, so no budget is spent twice on the same row.

Release notes

Dates are actual shipment and measurement dates.

DateRelease
2026-11-19Validator re-run on all lots: 28,922 / 28,922 rows, 100.00% schema pass, 0 invalid JSON. Self-pilot Run 1 report published (pilot-report.html).
2026-10-02Evidence hub launched. Published a verbatim failure→recovery row from a shipped commercial lot (ecom_research_pricing_retry_001).
2026-09-27Full-corpus deterministic validation measured: 28,922 / 28,922 rows, 100.00% schema pass. Validator published.
2026-09-16 / 17Commercial 1K v1.1 and Extended lots finalized for all four tracks, with cross-lot 3-gram Jaccard dedup audit.