1
This paper tries to explain why mixing sampled teacher tokens with full teacher distributions can beat pure soft-label distillation in LLM compression. The authors argue the gain comes from lower exposure bias, with exact-token bridge steps and flexible garden steps tested across Qwen, Llama, Gemma, and DeepSeek teacher-student pairs.
discussion (0)