The probability of joint monophyly of all species in an arbitrary species tree

Abstract Monophyly is a feature of a set of genetic lineages in which every lineage in the set is more closely related to all other members of the set than it is to any lineage outside the set. Multiple sets of lineages that are separately monophyletic are said to be reciprocally monophyletic, or jointly monophyletic. The prevalence of reciprocal monophyly, or joint monophyly, has been used to evaluate phylogenetic and phylogeographic hypotheses, as well as to delimit species. These applications often make use of a probability of joint monophyly under models of gene lineage evolution. Studies in coalescent theory have computed this joint monophyly probability for small numbers of separate groups in arbitrary species trees, and for arbitrary numbers of separate groups in trivial species trees. Here, generalizing existing results on monophyly probabilities under the multispecies coalescent, we derive the probability of joint monophyly for arbitrary numbers of separate groups in arbitrary species trees. We illustrate how our result collapses to previously examined cases. We also study the effect of tree height, sample size, and number of species on the probability of joint monophyly. The result also enables computation of relatively simple lower and upper bounds on the joint monophyly probability. Our results expand the scope of joint monophyly calculations beyond small numbers of species, subsuming past formulas that have been used in simpler cases.