Maintenance of species boundaries despite rampant hybridization between three species of reef fishes (Hexagrammidae): implications for the role of selection

Hybridization and introgression are antagonistic to the process of speciation. If hybrids are viable and backcross, introgression will slow or prevent the evolution of reproductive isolation between populations. If species hybridize upon secondary contact, introgression will ultimately erode species boundaries. If hybridization is rampant and gene flow is high, the proportion of hybrids will approach values expected with random mating, and introgression will overcome species boundaries, homogenizing populations. Alternatively, if species are genetically distinct, the proportion of hybrids would be significantly less than expected with random mating, gene flow would be interrupted by selection against hybrids, and inviability or sterility of F1 hybrids or backcrosses would be expected. Here, we investigate a system with characteristics expected by both of these opposing scenarios. Hybridization occurs between three species of reef fishes in the genus Hexagrammos at unexpected high frequencies in a zone of distributional overlap. Backcrossed individuals are detected, indicating F1 hybrids are viable and capable of reproducing. Yet, these species are genetically distinct at multiple loci. To study this apparent paradox, we estimate the relative proportions of hybrids, patterns of symmetry, inviability, and cytonuclear disequilibria using one mitochondrial and two nuclear markers. We invoke selection against hybrids, at various life history stages, in the maintenance of species boundaries in this system.