An overview of assessing the quality of peer review reports of scientific articles

Assuring the quality control of publications in the scientific literature is one of the main challenges of the peer review process. Consequently, there has been an increasing demand for computing solutions that will help to maintain the quality of this process. Recently, the use of Artificial Intelligence techniques has been highlighted, applied in the detection of plagiarism, bias, among other functions. The assessment of the reviewer’s review has also been considered as important in the process, but, little is known about it, for instance, which techniques have been applied in this assessment or which criteria have been assessed. Therefore, this systematic literature review aims to find evidence regarding the computational approaches that have been used to evaluate reviewers' reports. In order to achieve this, five online databases were selected, from which 72 articles were identified that met the inclusion criteria of this review, all of which have been published since 2000. The result returned 10 relevant studies meeting the evaluation requirements of scientific article reviews. The review revealed that mechanisms to rank review reports according to a score, as well as the word analysis, are the most common tools, and that there is no consensus on quality criteria. The systematic literature review has shown that reviewers’ report assessment is a valid tool for maintaining quality throughout the process. However, it still needs to be further developed if it is to be used as a resource which surpass a single conference or journal, making the peer review process more rigorous and less based on random choice.