Reliability of meta-analyses in ecology and evolution: (mostly) good news from a case study on sexual signals

Meta-analyses are powerful synthesis tools that are popular in ecology and evolution owing to the rapidly growing literature of this field. Although the usefulness of meta-analyses depends on their reliability, such as the precision of individual and mean effect sizes, attempts to reproduce meta-analyses’ results remain rare in ecology and evolution. Here, we assess the reliability of 41 meta-analyses on sexual signals by evaluating the reproducibility and replicability of their results. We attempted to: (i) reproduce meta-analyses’ mean effect sizes using the datasets they provided; (ii) reproduce meta-analyses’ effect sizes by re-extracting 5703 effect sizes from 246 primary studies they used as sources; (iii) assess the extent of relevant data missed by original meta-analyses; and (iv) replicate meta-analyses’ mean effect sizes after incorporating re-extracted and relevant missing data. We found many discrepancies between meta-analyses’ reported results and those generated by our analyses for all reproducibility and replicability attempts. Nonetheless, we argue that the meta-analyses we evaluated are largely reproducible and replicable because the differences we found were small in magnitude, leaving the original interpretation of these meta-analyses’ results unchanged. Still, we highlight issues we observed in these meta-analyses that affected their reliability, providing recommendations to ameliorate them.