Homepage: http://qingkeai.online/

Core Thesis: The problem of SFT failing to learn cannot be solved by "adding epochs." For some issues, adding epochs only yields 1% improvement, but switching strategies can deliver 12.5%. This article focuses on "which strategy treats which disease" — a complete SFT failure repair toolkit.

Paper: Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models
Conference: ACL 2026 | Affiliation: Tencent Hunyuan × UNSW
arXiv: https://arxiv.org/abs/2604.10079

One: From "One Cage for All Diseases" to "Five Surgical Scalpels"

When practitioners face models that won't learn during SFT, the instinctive reaction is: add epochs.

The paper proves through experiments that this "blindly adding epochs" approach is extremely inefficient. For knowledge-deficiency-type unlearned samples (Root Cause I), adding epochs only improves performance by 1-2%. But switching to CPT (Continual Pre-Training) knowledge augmentation delivers +8-17% — an order of magnitude difference.

The authors' core argument: Incomplete Learning Phenomenon (ILP) isn't a single disease, but a syndrome of five root causes. Each cause needs a different "surgical scalpel." Across 10 standard SFT datasets, 15.3%±2.1% of training samples remain unlearned on average, a figure that holds steady across Qwen, LLaMA, and OLMo2.

Before diving into strategy breakdown, let's understand the basic framework. The authors propose a three-stage diagnostic pipeline:

The key technology enabling detection is MC Conversion (Multiple-Choice Conversion) — converting SFT supervised responses into multiple-choice format, using pass@5 < 0.2 to judge "unlearned samples."

Two: Scalpel 1 — CPT Knowledge Augmentation — When SFT Meets a Knowledge Vacuum

The Enemy

Root Cause I: Pre-training Knowledge Deficiency. The base model never encountered the relevant knowledge in pre-training corpus — diagnostic criteria for rare diseases, specific statute numbers, low-frequency financial product names. SFT cannot "create something from nothing."

Why Adding Epochs Fails

SFT's optimization signal strength is limited — a few epochs of gradient updates cannot fabricate knowledge representations that never existed in model parameters. You can make someone who's never seen a zebra answer ten thousand "zebra vs. horse" multiple-choice questions, they still won't have the concept of "stripes."

Treatment Pipeline

Step 1: Extract knowledge triples (h,r,t) from unlearned samples using OpenIE
Step 2: Base model BoN-10 probe, pass@10 < 0.2 → mark as "blind knowledge"
Step 3: Multi-source retrieval → WikiData + Google Search + OpenAI-o1 API
Step 4: Build augmented corpus (domain:general = 0.8:0.2 mix)
Step 5: Base model CPT (Continual Pre-Training)
Step 6: Standard SFT

Treatment Results

Key finding: The larger the model (14B vs 7B), the greater the CPT gain (+14.1% vs +9.4%) — larger models have stronger knowledge absorption capacity.

Three: Scalpel 2 — CPT Calibration — Correcting the Model's "False Beliefs"

The Enemy

Root Cause II: SFT Supervision Conflicts with Pre-training Knowledge. The base model formed strong false beliefs during pre-training. For instance, the knowledge point "French President" — the model saw massive amounts of "President Hollande" in pre-training data. SFT annotates the correct answer "Macron," but the pre-training prior is too strong.

Key Difference from Scalpel 1

Treatment Pipeline

1. Detect base model high-confidence errors (probability on wrong option > 0.9)

2. Retrieve authoritative external knowledge as correction data

3. CPT to realign internal representations

4. Re-run SFT again

Treatment Results

ARC +2.8%, CommonQA +2.4%. Conflict rate (proportion of base model high-confidence errors) drops significantly.

Four: Scalpel 3 — Dynamic Bucketing — Making Contradictory Data "Stay in Their Lanes"

The Enemy

Root Cause III: Internal Contradictions in SFT Data. The same training set contains semantically similar but label-contradictory sample pairs. For example, two samples both ask about a disease's incubation period — one labeled "3-7 days," another "1-14 days." When they co-occur in the same batch, gradient directions oppose each other, net gradient approaches zero — neither gets learned.

Why Not Just Delete

Treatment Pipeline

1. Sentence-BERT encode all training samples
2. Build conflict graph: Sim(i,j) > 0.85 AND labels contradict → add edge
3. Graph coloring to assign buckets: adjacent nodes (conflicting samples) different colors (buckets)
4. Training: each mini-batch samples from only one bucket
5. Re-evaluate every K steps, dynamically update buckets

The core idea is preserve total information, only change how information is presented — ensuring any two potentially contradictory knowledge points never appear in the same batch's gradient computation.

Five: Scalpel 4 — Global Shuffling + Dynamic Resampling — Combating Training Forgetting

The Enemy

Root Cause IV: Left-Side Forgetting. When SFT data is ordered by source (all MedQA first, then all LegalBench), later training "overwrites" earlier learning. The first 10% of data's ROUGE-L drops up to 29% at worst.

Global Shuffling

# Bad (sequential ordering → forgetting)
dataloader = [medqa_all, legal_all, finance_all]
# Good (global random mixing → anti-forgetting)
dataloader = RandomSampler(medqa + legal + finance, shuffle=True)

Dynamic Resampling

Monitor validation accuracy of each data subset every K steps. If a subset drops beyond threshold, oversample from that subset into current batch — "whoever gets forgotten gets more training opportunities."

Treatment Results

First 10% data improves 29%, while last 10% only drops 1.6%. Tiny cost, massive gain.

Six: Scalpel 5 — Progressive Epochs — Not Unwilling to Learn, Just Not Trained Enough

The Enemy

Root Cause V: Insufficient Optimization. Under fixed epochs, simple samples dominate (~80% quantity). Though individual gradients are small, their accumulation dominates average gradient direction. Hard samples (5%) have large gradients but get averaged out — model parameter updates are defined by simple samples.

Treatment Pipeline

# Traditional (fixed epochs)
for epoch in range(fixed_epochs):
    train()
# Progressive Epochs (adaptive stopping)
while val_performance_improving():
    train_one_epoch()

Treatment Results

Seven: Treatment Selection Logic

Decision Flow

Unlearned sample detected (pass@5 < 0.2)
    ↓
Base model zero-shot accuracy?
    ├── < 25% (near random) → Root Cause I → Scalpel 1 (CPT Knowledge Augmentation)
    ├── Low but high-confidence errors → Root Cause II → Scalpel 2 (CPT Calibration)
    └── Decent or normal → 
        ├── Semantically similar samples with contradictory labels? → Root Cause III → Scalpel 3 (Dynamic Bucketing)
        ├── Data in early training sequence? → Root Cause IV → Scalpel 4 (Shuffling + Resampling)
        └── Hard sample loss still decreasing? → Root Cause V → Scalpel 5 (Progressive Epochs)

Implementation Priority Recommendations

1. Zero-cost strategies first: Global shuffling (Scalpel 4) + Progressive Epochs (Scalpel 5) — modify dataloader and stopping criteria, zero extra compute.

2. Low-cost strategies next: Dynamic Bucketing (Scalpel 3) — only needs Sentence-BERT encoding and graph coloring, negligible compute overhead.

3. High-leverage strategies last: CPT (Scalpels 1 and 2) — requires external data retrieval and extra pre-training, high cost but highest returns.

This article uses five solution categories as the main thread, fully explaining each "scalpel's" applicable root cause, operational flow, key data, and implementation cost.