Phylotranscriptomic analyses reveal the reticulate evolution in Dennstaedtiaceae

Abstract Dennstaedtiaceae, a heterogeneous family of ferns with 11 genera and about 270 species, has a global distribution. While substantial progress has been made in elucidating the intergeneric relationships within Dennstaedtiaceae, certain nodes, particularly within Hypolepidoideae, remain controversial. To date, no phylogenomic investigation of nuclear genes has been conducted for Dennstaedtiaceae, and the biological processes underlying its complex evolution remain largely unknown. In this study, we generated transcriptome sequences from nine species in Dennstaedtiaceae and combined them with publicly available data sets from 13 species and one outgroup. By utilizing the 23 transcriptomic data sets, representing nine out of the 11 genera in Dennstaedtiaceae, we successfully resolved the intergeneric relationships within the family and established a fundamental phylogenetic framework to investigate its evolutionary history. By combining the analyses of rate‐adjusted K s ‐based age distributions and phylogenetic reconciliation approaches, we found evidence of at least one round of whole‐genome duplication (WGD) that is shared by all Dennstaedtiaceae species prior to their divergence. Extensive gene tree discordance was found across the backbone of Dennstaedtiaceae, with the most significant discordance within Hypolepidoideae. The results of incomplete lineage sorting (ILS) simulation revealed that ILS is a substantial contributor to these conflicts. Evidence from phylogenetic networks and introgression tests indicates the occurrence of gene flow among the clades of Paesia , Hiya , and Histiopteris , potentially explaining the observed cytonuclear discordance in Hypolepidoideae. Our phylotranscriptomic study of Dennstaedtiaceae provides novel insights into its complex reticulate evolutionary history, paving the way for future studies aimed at unraveling the mechanisms underlying its diversification and adaptation.