ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry
Published in Conference on Language Modeling (COLM), 2026
Co-first author (listed first). Accepted at COLM 2026. Introduces the first benchmark for evaluating Deep AI Research Systems (DARS) on frontier AI research: 65 questions from authentic research settings spanning 35 AI subjects and 3 task types, paired with expert-rubric insight evaluation and citation faithfulness/groundedness assessment. Leading DARS reached 69–70% insight coverage, outperforming other DARS by 20–30 points, while exposing an open-consulting strength and a high-faithfulness/low-groundedness pattern. Dataset and code are released.
Recommended citation: Xu, T., Lu, P., Ye, L., Hu, X., & Liu, P. (2025). "ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry." Accepted at COLM 2026. arXiv:2507.16280.
Download Paper