多维计算机化自适应测验中在线标定方法的开发及应用

Multidimensional item response theory (MIRT) is an outgrowth of both factor analysis and unidimensional IRT (Reckase, 2009), and is gaining more attention recently due to the increased interest in testing for diagnosis (Wang, Boughton, & Chang, 2011). Building adaptive tests based on MIRT, called multidimensional computerized adaptive testing (MCAT), features a combination of tailored testing and multi-trait estimation which shows great potential to support formative assessments (Wang & Chang, 2011). As an important component of MCAT, item bank is the prerequisite for administrating the MCAT test (Reckase, 2009). Just like all regular CAT and cognitive diagnostic CAT (CD-CAT), some items in the MCAT item bank maybe overexposed or obsolete or flawed as time goes on and they should be replaced by new items (Wainer & Mislevy, 1990). Thus, item replenishing is essential for item bank maintenance and management in MCAT. Calibration of new items is a technical difficulty in item replenishing, and the precision of calibration directly impacts the accuracy of the estimation of examinees' abilities. In regular CAT and CD-CAT, online calibration technique is commonly employed to calibrate new items. Nevertheless, discussions of online calibration in MCAT have been absent from the current literature. This project builds an analytical foundation to extend the three classical online calibration methods (Method A, OEM and MEM) originally proposed for regular CAT to MCAT scenario (denoted as M-Method A, M-OEM and M-MEM, respectively), and develops innovative online calibration methods that meet the prefixed calibration accuracy based on the structures and features of MCAT and online calibration technique. On the other hand, this project investigates the possibility to generalize some representative online calibration designs (random design, adaptive design and sequential design) used in regular CAT and CD-CAT to MCAT case. In addition, the ability estimation methods (e.g., MLE, EAP, MAP and WMLE) and item selection strategies (e.g., D-Optimality, A-Optimality and KL information index) commonly used in MCAT are compared by conducting a Monte Carlo simulation study and the best method and strategy are selected to participate in the online calibration process of new items. Based on this, an online MCAT software platform is developed to implement a large-scale assessment of the academic quality of the 3-6 grade mathematics learners in China. This project is a starting point of introducing online calibration in the realm of MCAT, and the results can enrich the functions of item bank maintenance and management in MCAT. Furthermore, this project will provide scientific support to enhance exam quality and better serve our educational testing and evaluation.

基于多维项目反应理论的多维计算机化自适应测验 (MCAT) 将自适应测验与多维能力估计相结合，在准确提供学生学业水平信息、诊断学生优缺点方面具有巨大潜力。题目增补对MCAT的题库建设与维护至关重要，而新题标定作为题目增补过程中的技术难点，其精度直接影响被试能力估计的准确性。目前在线标定技术广泛应用于传统CAT的新题标定中，在MCAT领域还未见到相关研究。本课题拟探讨将传统CAT中常用的在线标定方法/设计推广到MCAT的可能性，通过模拟研究选择性能最优的MCAT能力估计方法与选题策略参与新题的在线标定；并在此基础上，结合我国国情，开发适用于对我国3~6年级学生数学学业质量进行大规模评价的MCAT在线测验系统。本研究第一次将在线标定技术应用于MCAT领域，所得结果将从理论上丰富MCAT的题库维护与管理功能。本研究将为提升考试质量、更好地服务于我国教育考试与评价的相关需求提供科学支撑。

计算机化自适应测验(CAT)在连续使用一段时间后，题库中的某些题目会因为过度曝光、内容过时等原因不再适合被继续使用。这时就需要定期开发新题，然后标定新题的参数。在整个题目增补过程中，对新题的准确标定既是技术重点也是技术难点。目前，在线标定技术被广泛应用于单维CAT(UCAT)的新题标定中，但在多维CAT(MCAT)领域还未见相关报导。本项目围绕该主题进行一系列原创性研究，主要研究内容及重要结果总结如下：.1. 在线标定方法的拓展. 对MCAT中常用选题策略(比如D最优、A最优以及互信息)与能力向量估计方法(MLE、EAP与MAP)进行全面比较，模拟研究表明：贝叶斯A最优与EAP的组合可获得最准确的被试参数返真性。在此基础上，成功将UCAT中3种有代表性的在线标定方法(“方法A”[Method A]、“只有一个EM循环的方法”[OEM]以及“有多个EM循环的方法”[MEM])推广至MCAT，新方法分别记为M-Method A、M-OEM和M-MEM。模拟研究表明：所有3种新方法都能准确标定新题。.2. 新在线标定方法的开发. 为了克服M-Method A存在的理论缺陷(将被试能力向量估计值视为能力向量真值)，分别将全功能极大似然估计方法(FFMLE)和修改的Lord偏差校正方法的误差校正思路融入M-Method A/Method A(新方法称为FFMLE-M-Method A和MLE-LBCI-Method A)，从理论上对MLE能力向量估计误差进行校正。结果表明：在几乎所有模拟条件下，FFMLE-M-Method A和MLE-LBCI-Method A较传统M-Method A/Method A都可得到更为准确的题目参数估计结果。. 另外，为了解决M-MEM在“能力维度间相关较高”和/或“标定新题样本量较少”情境下经常迭代不收敛的问题且更为准确地标定新题，将贝叶斯众数估计方法与M-OEM和M-MEM相结合，以期在标定新题的过程中充分利用来自新题参数的先验信息。模拟研究表明：考虑新题参数的先验信息有助于改进M-MEM的标定精度与效率，但对M-OEM的改进不大。. 本项目将多项统计与测量技术应用于MCAT的在线标定研究，所得结果不仅可以从理论上丰富MCAT的题库维护与管理功能，而且可以为测量研究者在选择在线标定方法时提供科学依据。