Human-written agent skills often remain in the reasoning context as external manuals. SkillGym turns these workflows into executable task environments with checkable outcomes, then trains models on long trajectories that pass those checks. In the paper, under Claude Code, the same Qwen3.5-35B-A3B backbone is compared: on SkillsBench v1.1, the success rate with a skill condition rises from 23.34% to 51.47%, a same-backbone gain of 28.13 percentage points.
[Figure 2: Overview of the SkillGym construction framework] Human-written skills are first classified and converted into task templates; the templates are instantiated as Docker environments with inputs, constraints, and code checkers, then pass feasibility, structure, execution, and skill-dependency checks, after which multiple execution frameworks and models sample successful trajectories.
Figure 2
2,756 Verifiable Environments Turn Skill Descriptions into Execution Supervision
SkillGym does not directly have the model restate skill text. It designs task templates for different subcategories and assembles manuals, files, dependencies, and acceptance conditions into environments; an execution is successful only if it passes the code checker. The paper uses the contrast “passes with the target skill, fails when the skill is removed” to screen for tasks that genuinely depend on the skill, while also keeping tasks that pass verification but do not meet this contrast.
The final resource contains 2,756 environments, 12 major categories, and 63 subcategories. The research team collected 8,364 successful trajectories from multiple models and execution frameworks. On average, each has 49 tool calls, 63.4k text tokens recorded in logs, and 35.2 interaction steps, with the longest reaching 318 steps. The paper uses these trajectories to perform long-context supervised fine-tuning on Qwen3.5-35B-A3B; the environments and checkers can also be used for outcome-reward reinforcement learning, but the main results in this paper come from supervised fine-tuning.
[Table 4: General agent benchmark performance compared with the base model] Colored values in parentheses indicate the absolute improvement relative to the Qwen3.5-35B-A3B base model under the same execution framework; GDPval-AA v2 reports Elo, while the other three report success rates.
Table 4
All Four Same-Backbone Comparisons Under Claude Code Improve
Under Claude Code, GDPval-AA v2 rises from 974 to 1,173 Elo, an increase of 199 Elo; Terminal-Bench 2.1 rises from 39.33% to 58.43%, an increase of 19.10 percentage points. On SkillsBench v1.1, with a skill it rises from 23.34% to 51.47%, and without a skill it rises from 12.13% to 26.81%, increases of 28.13 and 14.68 percentage points, respectively.
Under Codex there are also improvements: Terminal-Bench 2.1 rises from 10.11% to 46.07%, an increase of 35.96 percentage points; on SkillsBench, with and without a skill, the scores rise from 5.33% to 33.02% and from 0.69% to 21.08%, respectively. These two sets of results mean different things: the Claude Code base model and fine-tuned model use the same standard system prompt; the Codex fine-tuned model switches to a no-applypatch system prompt, and the paper explicitly states that this difference cannot be attributed solely to fine-tuning. Separating with-skill and without-skill cases, the Claude Code fine-tuned model scores 26.81% without a skill, higher than the base model’s 23.34% with a skill; the corresponding Codex numbers are 21.08% versus 5.33%. This supports a narrower interpretation: verified trajectories taught the model some reusable workflows. When skills continue to be provided, scores reach 51.47% and 33.02%, showing that external instructions, constraints, and procedural experience in the weights can stack in this evaluation suite, but the model still needs these external conditions.
[Figure 1: Performance on general agent benchmarks] The figure also shows same-scale baselines and public reference models. On the left, the Qwen3.5-35B-A3B base model and SkillGym-Agent are same-backbone results; on the right, public reference scores come from each model’s original evaluation setup at release.
Figure 1
Public reference scores can serve only as positional references. The paper’s appendix notes that GDPval, Terminal-Bench, and SkillsBench results for different models may use different execution frameworks, inference budgets, runtime environments, and system prompts; for example, many public SkillsBench results use OpenHands, while Gemini 3.1 Pro uses Gemini CLI. The 51.47% is higher than the 47.2% listed for Claude Sonnet 4.6, 41.4% for GPT-5.4 Mini, and 50.1% for DeepSeek V4 Pro Preview, but these rows do not constitute a strictly controlled head-to-head win or loss.
This set of data supports that skill-derived supervision can transfer in specific evaluations. Under the paper’s Qwen3.5-35B-A3B, Claude Code, and SkillsBench v1.1 setup, SkillGym’s verified trajectories raise the with-skill success rate from 23.34% to 51.47%, and also raise the without-skill success rate from 12.13% to 26.81%. Combined with the same-backbone results on Terminal-Bench and GDPval-AA, the paper supports the conclusion that “successful trajectories from executable skills can provide cross-task procedural supervision.”
When deploying or reproducing this, three things must be checked separately: whether the task truly depends on the target skill, whether the checker covers key business risks, and whether the execution framework and system prompt match the control. Among SkillGym’s 2,756 environments, 1,081 satisfy the skill-dependency contrast, while the remaining 1,675 are verifiable fallback tasks; the overall success rate of trajectory sampling is 17.4%. These conditions show that data quality and evaluation criteria directly affect results, and 51.47% cannot be extrapolated into a stable gain for all agent workflows.
📄 Original paper title
SkillGym: Internalizing Large-Scale Human Skills into LLMs for Real-World Problem Solving
🔗 Original paper link