100% synthetic, no PII, no scraped content · 28,922 rows validated, 100% pass · SHA-256 provenance manifests · Apache-2.0 free samples
Every number below is produced by running validate_jsonl.py (stdlib-only, included in the repository) against the shipped dataset files. No sampling — 100% of rows are validated. Last measured: 2026-11-19.
| Metric | Result | Methodology |
|---|---|---|
| Schema validation pass rate | 28,922 / 28,922 (100.00%) | Deterministic structural validation of every row across all lots and editions |
| Invalid JSON rows | 0 | Strict JSONL parsing, UTF-8, one sample per row |
| Phantom tool calls / undeclared arguments | 0 (fails the row) | Every call checked against declared tool + argument schema |
| Near-duplicate rate | 0 at threshold | 3-gram Jaccard ≥ 0.85 exclusion vs. all prior rows incl. shipped lots |
| Validator self-test | 23 mutation classes | Gold rows must pass; deliberately corrupted variants must be caught |
Full per-lot breakdown: evaluation/EVALS.md · Reproduce locally:
python validate_jsonl.py --track mcp_tool_use path/to/dataset.jsonl
The samples use the same schema as the full datasets, so you can wire them into your harness (Axolotl, Unsloth, custom) first. The MCP and CoT repositories use Hugging Face's information-sharing gate (access is automatic); the two Function Calling repositories are not gated.
Row ecom_research_pricing_retry_001 from the MCP Agent Trajectory 1K commercial lot — unedited, exactly as delivered. It shows the core training signal: a runtime failure, a strategy revision, and a verified recovery across chained tool calls.
| Turn | Content (verbatim) | What this trains |
|---|---|---|
user | I'm looking at a competitor, 'ShopFlow Pro'. Can you find their monthly subscription pricing for the 'Enterprise' plan? | Open-ended task requiring multi-step tool use |
assistant | {"name": "web_search", "arguments":
{"query": "ShopFlow Pro Enterprise plan pricing"}} | Strict tool-call JSON — declared tool, declared argument keys only |
tool | {"error": {"code": "429", "message":
"Rate limit exceeded. Please wait before retrying."}} | Injected failure. Most datasets stop at happy paths — this is where ours begins. |
assistant | {"name": "web_search", "arguments":
{"query": "ShopFlow Pro pricing page"}} | Recovery, not verbatim retry. The query is revised, not blindly re-sent — verbatim retries fail validation. |
tool | {"results": [{"title": "ShopFlow Pro - Pricing Plans",
"link": "https://shopflowpro.example.com/pricing"}]} | Partial progress — a page, not yet the answer |
assistant | {"name": "scrape_page", "arguments":
{"url": "https://shopflowpro.example.com/pricing"}} | Chained second tool, arguments grounded in the prior result |
tool | {"content": "Basic: $49/mo. Pro: $99/mo.
Enterprise: $299/mo. ..."} | Ground truth arrives |
assistant | The ShopFlow Pro Enterprise plan is priced at $299 per month. | Grounded final answer — matches tool output exactly; ungrounded numbers fail validation |
All domains are synthetic (example.com) — no scraped content, no PII.
| Free sample (Hugging Face) | 1K lot | Extended | |
|---|---|---|---|
| Rows | 50 per track | MCP 1,003 · others 1,000 | 1K lot + 2,300 non-overlapping rows (MCP 3,303 · others 3,300) |
| 1K lot price | Free | MCP $7,300 · FC-EN $6,400 · FC-JA $8,500 · CoT $12,500 | |
| Licence | Apache-2.0 | Perpetual, organization-wide, no redistribution | |
| Schema validation report | Public (EVALS.md) | Included per delivery (stats.json) | |
| SHA-256 integrity manifest | — | Included (SHA256SUMS.txt) | |
| Overlap guarantee | — | Hash + ID + 3-gram Jaccard audited against all prior lots | |
Generation-time schema gating → deterministic structural validation (declared tools/arguments only, strict role alternation) → 3-gram Jaccard ≥ 0.85 de-duplication against all shipped lots → independent LLM scoring → final review. The deterministic stage is public and reproducible (evaluation/).
New lots are audited (hash, ID, 3-gram Jaccard) against previously shipped lots, so no budget is spent twice on the same row.
Dates are actual shipment and measurement dates.
| Date | Release |
|---|---|
| 2026-11-19 | Validator re-run on all lots: 28,922 / 28,922 rows, 100.00% schema pass, 0 invalid JSON. Self-pilot Run 1 report published (pilot-report.html). |
| 2026-10-02 | Evidence hub launched. Published a verbatim failure→recovery row from a shipped commercial lot (ecom_research_pricing_retry_001). |
| 2026-09-27 | Full-corpus deterministic validation measured: 28,922 / 28,922 rows, 100.00% schema pass. Validator published. |
| 2026-09-16 / 17 | Commercial 1K v1.1 and Extended lots finalized for all four tracks, with cross-lot 3-gram Jaccard dedup audit. |