Paper
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Published May 15, 2023 · Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu
ArXiv
537
Citations
67
Influential Citations
Abstract
New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.
C-Eval is a comprehensive Chinese evaluation suite that assesses advanced knowledge and reasoning abilities of foundation models, highlighting significant room for improvement in current language models for Chinese users.
Full text analysis coming soon...