评价科研产出质量的新定量方法研究

Nowadays citation based indicators have been widely applied for research output quality assessment. However there is a heated debate about the shortcomings of the application of this approach for assessment. In this work, we are going to introduce the new quantitative assessments for research output quality, which can overcome some of the limitations of citation based indicators. In this work, we start from the definition of research raised from the Research Excellence Framework (REF) proposed by the High Education Funding Council for England (HEFCE). Under this framework, research output quality can be further divided into two dimensions: the significance of new insights and effectiveness of sharing. It is widely accepted that citation based indicators are mainly aimed to measure the effectiveness of sharing, and not much work has been carried out on the measurements of significance of new insights. ..Thus we will attempt to discuss this issue following similar ideas in the labor theory of value. Like a product, a research publication has its value which is contingent upon how much labor (up to certain quality standards) was necessarily exerted in producing it - we will refer this labor to workload of a paper. How to judge value of a research publication? We here argue that the originality and use value, which are the two dimensions of the new insights, both positively correlated with the workload. Thus it is plausible to estimate quality of academic publications through estimating the contained workload. The pilot study also show that it is possible to further estimate relative workload spent on completing manuscripts using journal acceptance rate in one subject area, and then in turns to produce a proxy indicator for relative significance of new insights generated in research publications. This leads to Equivalent Paper Number (EPN) and other EPN based indicators, which are the new defined indicators for the measurements of new insights. ..Followed from the framework, we argue that a more complete and rational assessment for research output quality can be achieved by combining information from these two dimensions. Therefore, we are going to undertake different approaches such as the simple mathmatical combination, DEA combination and ordinal regression analysis, for the combinations of the measurements of the two dimensions, and their effective will be studied by comparing with other well recognized expert review based research assessment results, such as the ABS journal rank and the RAE2008 result.

引文数为基础的文献计量学指标已经被普遍应用于科研产出质量的评价，但前人的工作已经充分揭示了该类指标的缺陷。本课题试图构造一系列的新定量指标弥补该类指标的部分缺陷。本研究首先从英格兰高等基金委员会(HEFCE)提出的科研机构研究卓越性评估(REF)框架出发，并从中分解出研究产出质量的两个维度：即研究产出的学术价值和研究影响力。目前已知的文献计量学指标大部分都用于测量研究的影响力，而监测研究学术价值的定量指标并不多，且各有弊端。基于此，本课题尝试通过对论文中包含的劳动量进行估量，关联论文的学术价值，首先发展一系列基于期刊接收率的新指标，即EPN类指标，用于测量研究的学术价值，进而开展结合引文类指标与EPN类指标的方法的研究，通过理论研究与实例应用相结合，研究并筛选适用于不同情况的指标，最后提出评价科研产出卓越性方面宜于实际操作且可适性优良的新定量方法。

科研评价是科技管理领域的重要问题。目前评价方法多以基于定性的同行评议和基于定量的文献计量学为主，其中定量评价方法多采用引文数据作为主要指标。但引文数据实际上评价的是科研产出的学术影响力，而非科研产出质量。因此，泛用甚至滥用引文数据和相关指标，对科研评价和管理会造成一定的混乱与误导。本项目围绕科研管理领域的科研评价问题，借助英国REF 理念，从科研产出的学术价值和研究影响力两方面出发，开展了对科研产出质量评估方法的研究。研究首先创建知识创新过程模型，在知识产生和知识传播模块，考察其与科研产出质量评价的关系，进而提出以期刊接收率为核心的学术价值评价指标，以及学术影响力等级自然划分法和分区影响因子等影响力评价方法，最后通过实证研究验证了该方法的有效性。一方面，研究在很大程度上拓展了以往的以同行评议为主的定性评价方法和以文献计量学为主的定量评价方法，修正了基于引文数据的科研产出质量评价指标，有助于完善我国的科研评价体系。另一方面，本研究提出的评价方法可以节约大量的科研评价时间和资源，并可以纳入第三方科研评价，为后续基于大数据的科研评价方法研究打下基础。.项目研究三年期间，参与组织了1次国际会议，参加国内外相关学术会议10余次；协助指导研究生1人，毕业1人；共发表学术论文10篇，其中国际合作论文10篇，人民大学报刊复印中心全文转载2篇；其中，学术论文主要发表在科学计量学顶尖SCI期刊诸如Scientometrics, Information Processing & Management, Journal of Informetrics以及基金委管理学部核心期刊。发表在Scientometrics上的关于分区影响因子评价研究工作从知识积累角度出发，同时在考察期刊内部引用分布规律基础上提出，与常用的期刊影响力因子影响力评价方法相比，更科学合理。该工作得到国内外知名学者的认同，国际科学计量学会前主席Ronald Rousseau教授更是主动邀请项目负责人开展合作研究，建立合作关系。