MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation
Published in arXiv preprint, 2026
MOPD-Router routes supervision from a pool of specialized teachers at each token, without prompt-level domain labels or a separately trained routing model. Its ExpertAlign metric weights each teacher’s correction according to how well it reflects that teacher’s post-training specialization. Across four distillation settings, ExpertAlign achieved the strongest overall performance; on unlabeled training data, it improved the overall score by 5.88 points over mean aggregation.
Recommended citation: Xu, T., Zheng, Y., Zhang, Z., Yu, Y., Ma, C., Zhu, J., Wu, L., Ye, L., Liu, P., Dong, B., Zhu, H., Huang, R., & Yu, G. (2026). "MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation." arXiv:2609.30837.
Download Paper