Self-Pilot Evaluation Report — MCP Agent Trajectory (Run 1)

Measured: 2026-11-19 · Track: MCP Agent Trajectory · Tasks: 60 · Base model: Qwen/Qwen2.5-1.5B-Instruct

1. Summary

This report measures whether a LoRA adapter trained on our MCP Agent Trajectory corpus improves a small base model's ability to recover from failures in MCP tool-calling tasks. The evaluation uses a fixed set of 60 failure-recovery tasks and compares two conditions on the same tasks.

This is a rehearsal run by the seller on its own fixed task set. It is not a customer pilot and does not use any buyer data or buyer-defined tasks.

On these 60 tasks, condition B (base + LoRA) recovered 34/60 (56.7%) versus 1/60 (1.7%) for condition A (base only). Limitations are listed in Section 6.

2. Conditions

ConditionDescription
ABase model only (Qwen/Qwen2.5-1.5B-Instruct)
BBase model + LoRA adapter trained on the corpus

3. Results

3.1 Overall (A vs. B)

MetricA: baseB: base + LoRA
Recovery success rate1/60 = 1.7%34/60 = 56.7%
Schema/validation error rate0/60 = 0.0%1/60 = 1.7%
Task completion rate1/60 = 1.7%31/60 = 51.7%
Mean seconds per task8.819.0
Crashed tasks00

3.2 Recovery success by injected failure type (recovered / tasks)

Failure typeA: baseB: base + LoRA
rate_limit0/103/10
conflict_lock0/105/10
auth_permission0/108/10
not_found0/106/10
invalid_argument0/107/10
other1/105/10

4. Method notes

5. Run status

Training completed 250/250 steps and the adapter was saved. No task crashed in either condition (A = 0, B = 0).

6. Limitations