Self-Pilot Evaluation Report — MCP Agent Trajectory (Run 1)
Measured: 2026-11-19 · Track: MCP Agent Trajectory · Tasks: 60 · Base model: Qwen/Qwen2.5-1.5B-Instruct
1. Summary
This report measures whether a LoRA adapter trained on our MCP Agent Trajectory corpus improves a small base model's ability to recover from failures in MCP tool-calling tasks. The evaluation uses a fixed set of 60 failure-recovery tasks and compares two conditions on the same tasks.
This is a rehearsal run by the seller on its own fixed task set. It is not a customer pilot and does not use any buyer data or buyer-defined tasks.
On these 60 tasks, condition B (base + LoRA) recovered 34/60 (56.7%) versus 1/60 (1.7%) for condition A (base only). Limitations are listed in Section 6.
2. Conditions
| Condition | Description |
| A | Base model only (Qwen/Qwen2.5-1.5B-Instruct) |
| B | Base model + LoRA adapter trained on the corpus |
3. Results
3.1 Overall (A vs. B)
| Metric | A: base | B: base + LoRA |
| Recovery success rate | 1/60 = 1.7% | 34/60 = 56.7% |
| Schema/validation error rate | 0/60 = 0.0% | 1/60 = 1.7% |
| Task completion rate | 1/60 = 1.7% | 31/60 = 51.7% |
| Mean seconds per task | 8.8 | 19.0 |
| Crashed tasks | 0 | 0 |
3.2 Recovery success by injected failure type (recovered / tasks)
| Failure type | A: base | B: base + LoRA |
| rate_limit | 0/10 | 3/10 |
| conflict_lock | 0/10 | 5/10 |
| auth_permission | 0/10 | 8/10 |
| not_found | 0/10 | 6/10 |
| invalid_argument | 0/10 | 7/10 |
| other | 1/10 | 5/10 |
4. Method notes
- Tasks: 60 MCP failure-recovery tasks with the seller's fixed task set (seed 42), 10 per injected failure type.
- Injected failure types: rate_limit, conflict_lock, auth_permission, not_found, invalid_argument, other.
- Decoding: greedy.
- Scoring: judged with the metric definitions frozen before the run; aggregates were computed mechanically from the raw per-task results.
- Training (condition B): LoRA on the base model; 2,000 examples drawn from the existing clean corpus, rank 16, learning rate 1e-4, 1 epoch (250/250 steps, loss 0.33 → 0.26, 3,949 seconds).
5. Run status
Training completed 250/250 steps and the adapter was saved. No task crashed in either condition (A = 0, B = 0).
6. Limitations
- Small base model. All measurements use a 1.5B-parameter base model (Qwen/Qwen2.5-1.5B-Instruct). Results may not carry over to other models.
- Schema errors in B. Condition B produced 1/60 schema/validation errors, whereas A produced 0/60.
- Longer runtime in B. B's mean time per task was 19.0 s versus 8.8 s for A.
- Weakest failure type. Recovery on rate_limit tasks was the weakest for B at 3/10.
- Seller-run rehearsal. A single run on a seller-defined task set of 60 tasks (10 per failure type); it is not a customer pilot.