Vallee-Tourangeau, Gaelle [Presenter], Wheelock, Ana, Vandrevala, Tushna and Harries, Priscilla (2021) Applying social judgment theory to better understand what peer reviewers pay attention to when evaluating fellowship grant proposals. In: Centre for Decision Research (CDR) Seminar Series; 03 Mar 2021, Leeds, U.K. (Held online). (Unpublished)

Abstract

While research has revealed startling inconsistencies and lack of agreement in reviewers’ evaluations of grant proposals (Pier et al., 2018), the reasons why such inconsistencies arise remain unclear. In this talk, I will present results from the first phase of the “Towards Outstanding Research Reviews” (TORR) project” (TORR) project, which aims to understand how individual peer reviewers use and combine information in written grant proposals to make their recommendations for funding. Informed by the Social Judgment Theory (SJT) framework (Hammond, 1977), we conceived reviewers’ recommendations as resulting from the application of a professional “judgement policy” and we first sought to infer what research quality factors expert reviewers pay attention to while reviewing a proposal for funding in a semi-naturalistic setting. To do so, we recorded think aloud protocols from 7 reviewers who were re-examining a grant application for the Wellcome Trust’s Humanities and Social Sciences Fellowship funding scheme, which they had recently reviewed. The think aloud protocols were adapted from the Critical Decision Making method of interviewing (Klein, 1989) and the Cognitive Event Analysis framework (Steffensen et al., 2016). Following a content analysis of the transcripts, we identified a total of 56 possible quality criteria mentioned by reviewers, which were organised around 5 areas of quality. Next, we conducted a survey where a further 28 expert reviewers were asked to rank the quality criteria in order of importance. Contrary to earlier findings suggesting lack of consensus between peer-reviewers, we uncovered ten quality factors which were consistently rated as most important for assessing the quality of a fellowship application. Upon presenting the factors identified, I will discuss implications advancing the understanding of peer-review judgements, the peer-reviewers’ dilemmas we uncovered through this process, as well as the challenges and opportunities involved in meta-research studies.

Actions (Repository Editors)

Item Control Page