Peer review indicators
reliability of a reciprocal anonymous review of masters proposals
DOI:
https://doi.org/10.18617/liinc.v9i1.531Keywords:
Peer review, Reliability, Agreement among referees, Intraclass correlation coefficient, Peer review in learningAbstract
Peer review lacks indicators, which is incongruent with its importance. This article reports the reliability of mutual, anonymous review of masters research proposals. Twelve masters students in Information Science refereed anonymously their peers' research proposals according to 7 evaluation items, using a Likert-6 scale. Reliability was expressed as intraclass correlation indices between 0.500 and 0.202, with 9 positive and 3 negative values - lower than usual in professional processes. Reliability is not an absolute measure of quality, but calculating reliability allows for a systematic study of the quality of peer review.
References
BENOS, D. J. et al. The ups and downs of peer review. Advances in Physiology Education, v. 31, n. 1, p. 145-152, 2007.
BORNMANN, L. Scientific peer review. Annual Review of Information Science and Technology, v. 45, n. 1, p. 199-245, 2011.
CASADEVALL, A.; FANG, F. C. Is peer review censorship? Infection and Immunity, v. 77, n. 4, p. 1273-1274, 2009.
CASTRO, R. C. F.; NEGRÃO, M. B.; ZAHER, C. R. Procedimentos editoriais na avaliação de artigos para publicação em periódicos de ciência da saúde da América Latina e Caribe. Ciência da Informação, v. 25, n. 3, p. 352-356, 1996.
COSTA, S. M. S. Controle de qualidade em periódicos científicos eletrônicos disponibilizados na Internet: a questão do julgamento pelos pares. Revista de Biblioteconomia de Brasília, v. 20, n. 2, p. 227-236, 1996.
DAVYT GARCÍA, A.; VELHO, L. A Avaliação da Ciência e a Revisão por Pares: passado e presente. Como será o Futuro? História, Ciência, Saúde – Manguinhos, Rio de Janeiro, v. 7, n. 1, p. 93-116, mar./jun. 2000.
EGGHE, L.; BORNMANN, L. Fallout and Miss in journal peer review. Journal of Documentation (to appear), 2013.
FISHER, R. A. Statistical methods for research workers. New York: Hafner Press, 1970. 362 p.
HACKETT, E. J.; CHUBIN, D. E. Peer review for the 21st Century: applications to education research. Prepared for a National Research Council workshop, Washington, D.C. Final version, July 31, 2003.
HORROBIN, D. F. Peer review of grant applications: a harbinger for mediocrity in clinical research? The Lancet, v. 348, p. 1293-1295, 1996.
HORROBIN, D. F. Referees and research administrators: barriers to scientific research? British Medical Journal, v. 2, p. 216, 1974.
HORROBIN, D. F. Something rotten at the core of science? Trends in Pharmacological Sciences, v. 22, n. 2, p. 51-52, February 2001.
JAYASINGHE, U. W.; MARSH, H. W.; BOND, N. A new reader trial approach to peer review in funding research grants: An Australian experiment. Scientometrics, v. 69, n. 3, p. 591–606, 2006.
JENNINGS, C. G. Quality and value: The true purpose of peer review. What you can't measure, you can't manage: the need for quantitative indicators in peer review (Nature Peer Review Debate). Nature, 2006.
KERN, V. M. et al. Growing a peer review culture among graduate students. In: TATNALL, A.; JONES, A. (Orgs.) Education and technology for a better world. IFIP Advances in Information and Communication Technology (AICT) Series. Proceedings of WCCE. Berlin, Heidelberg, New York: Springer, 2009. p. 388-397.
KHAN, K. Head to head: Is open peer review the fairest system? No. British Medical Journal, v. 341, 2010.
LAUREANO, G. H. da C. Coeficiente de correlação intraclasse: Comparação entre métodos de estimação clássicos e bayesianos. 2011. 69 f. Monografia (Graduação em Estatística) - Instituto de Matemática, Universidade Federal do Rio Grande do Sul, Porto Alegre, 2011. Disponível em: <http://www.lume.ufrgs.br/bitstream/handle/10183/36714/000818152.pdf?sequence=1>. Acesso em: 29 jan. 2013.
MARSH, H.; BOND, N.; JAYASINGHE, U. Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid. Australian Psychologist, v. 42, n. 1, p. 33-38, 2007.
MCGRAW, K. O.; WONG, S. P. Forming inferences about some intraclass correlation coefficients. Psychological Methods, v. 1, n.1, p. 30-46, 1996.
MUELLER, S. P. M. A seleção de artigos científicos para publicação em revistas brasileiras: um levantamento de práticas e procedimentos adotados pelas revistas científicas brasileiras financiadas pelo CNPq e INEP, 1995-1996. Revista de Biblioteconomia de Brasília, v. 21, n. 2, p. 229-250, 1997.
MULLIGAN, A.; HALL, L.; RAPHAEL, E. Peer review in a changing world: An international study measuring the attitudes of researchers. Journal of the American Society for Information Science and Technology, v. 6, n. 1, p. 132-161, 2013.
PAVAN, C.; STUMPF, I. R. C. Avaliação pelos pares nas revistas brasileiras de ciência da informação: procedimentos e percepções dos atores. Encontros Bibli, v. 14, n. 28, p. 73-92, 2009.
PESSANHA, C. Critérios editoriais de avaliação científica: notas para discussão. Ciência da Informação, v. 27, n. 2, p. 226-229, 1998.
SEATON, A. V. Blowing the whistle on tourism referees. Tourism Management, v. 17, n. 6, p.397-399, 1996.
SHROUT, P. E.; FLEISS, J. L. Intraclass correlations: Use in assessing rater reliability. Psychological Bulletin, v. 86, n. 2, p. 420-428, 1979.
SMITH, A. J. The task of the referee. Computer, v. 23, n. 4, p. 65–71, 1990.
STEHBENS, W. E. Basic philosophy and concepts underlying scientific peer review. Medical Hypotheses, v. 52, n. 1, p. 31–36, 1999.
STUMPF, I. R. C. Avaliação pelos pares nas revistas de comunicação: visão dos editores, autores e avaliadores. Perspectivas em Ciência da Informação, v. 13, n. 1, p. 18-32, 2008.
WELLER, A. C. Editorial peer review: Its strengths and weaknesses. Second printing (ASIST monograph series). Medford, New Jersey (EUA): Information Today, Inc. 2002. 342 p.
WOOD, M.; ROBERTS, M.; HOWELL, B. The reliability of peer reviews of papers on information systems. Journal of Information Science, v. 30, n. 1, p. 2-11, 2004.
YANKULOV, K.; COUTO, R. (2012), Peer review in class: Metrics and variations in a senior course. Biochemistry and Molecular Biology Education, v. 40, n. 3, p. 161-168, 2012.
Downloads
Additional Files
Published
Issue
Section
License
Copyright (c) 2025 Marta Denisczwicz, Vinícius Medina Kern
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
Authors retain copyright and grant Liinc em Revista the right of first publication with the work simultaneously licensed under a Creative Commons Attribution 4.0 International License.
The authors have permission and are encouraged to deposit their manuscripts and versios of record (VoR) in their personal web pages or institutional repositories, generic repositories etc., before (pre-print) or after (post-print) the publication in Liinc em Revista, according to its open access depositing policy registered in the Directory of Editorial Policies of Brazilian Journals (DIADORIM), kindly providing a link to the article published on Liinc's website.
Liinc em Revista, published by Instituto Brasileiro de Informação em Ciência e Tecnologia, is licensed under a Creative Commons Attribution 4.0 International License – CC BY 4.0 
