Although the house mouse (Mus musculus) is a single species, fertility is reduced in crosses between geographic populations (Wikipedia - Bolid74)
It's widely believed that mating between two species never produces fertile offspring. Untrue. Interspecific mating sometimes produces fertile offspring, and the genomes of closely related species often show gene flow from one to the other. It would be more accurate to say that such crossings are usually compromised in one way or another—incompatibilities arise during development, which lead to miscarriage, sterile offspring, or less viable offspring.
There is a similar misunderstanding about mating within a species. It isn't true that mating within a species is uniformly possible. Even below the species level, genetic incompatibilities can arise between different geographic populations. Keep in mind the arbitrariness of all these terms: “geographic population,” “race,” “subspecies,” and “species.” These are points on a continuum of increasing reproductive isolation and genetic incompatibility.
Various problems have been observed in crossings between geographic populations of the same species (Presgraves 2010; Turner et al. 2011). In the case of crossings between two subspecies of the house mouse (M. m. domesticus and M. m. musculus), the problems have no single cause. There is not only reduced fertility but also reduced immune function, as shown by higher loads of intestinal parasites in hybrids. Even the reduction in fertility has multiple causes: "The type and severity of fertility defects observed depends on the geographic origin of the strains and also varies among individuals within regions. This variability suggests that multiple genetic incompatibilities contribute to hybrid male sterility" (Turner et al. 2011). Multiple loci seem to be responsible, and incompatibilities that cause sterility likely act in combination with each other and with other incompatibilities (Turner et al. 2011).
In general, as genetic distance increases, so does the risk of faulty interactions between different gene loci:
The Dobzhansky-Muller model proposes that populations diverging independently in allopatry will accumulate differences through drift or selection, but will maintain coadaptation between divergent loci within each single population. If secondary contact and hybridization occurs between these populations, these divergent loci from the previously allopatric populations may interact deleteriously in hybrids leading to lowered hybrid fitness (Leppälä and Savolainen 2011)
Icelandic and Danish studies on outbreeding
The above model is consistent with two recent studies on humans, one in Iceland and the other in Denmark. The Icelandic study used a database of couples born between 1800 and 1965. In the authors' own words, "the advantage of using the Icelandic data set lies in this population being small and one of the most socioeconomically and culturally homogeneous societies in the world, with little variation in family size, use of contraceptives, and marriage practices" (Helgason et al. 2008). The authors found that fertility rose and then fell with increasing genetic distance between husband and wife. Specifically, the relationship was:
[...] an n-shaped curve from the relatively low reproductive success of couples related at the level of second cousins or closer, to the maximum for couples related at the level of third and fourth cousins, after which there is a steady decrease in reproductive success with diminishing kinship between spouses. A similar picture emerges when the number of grandchildren per couple is examined (Helgason et al. 2008)
The Icelandic study was criticized in a tweet by Jayman:
Doubt it. The Icelandic study is likely confounded in various ways so I don't think genetic relatedness is the thing directly affecting fertility.
He was retweeted by hbdchick, but neither of them identified the "various" confounds. Nor did I find this criticism in the 140 papers that cited the study on Google Scholar. So we're not in the realm of common knowledge.
The original paper did identify a possible confound. From 1800 to 1965, there was a decrease in mean fertility and an increase in marriages with more distantly related individuals. The relationship between fertility and kinship may thus be an historical confound:
These results are based on couples born during a period of almost 200 years, in the course of which there was a marked decline both in the mean fertility and in kinship between couples. (Helgason et al. 2008)
To eliminate the confound, the authors broke the data down by 25-year intervals:
Nonetheless, the same general relationship between kinship and reproductive outcome was observed within each 25-year subinterval (fig. S2). We evaluated the correlation between the standardized variables of kinship and reproductive outcome for all couples and for each time interval separately (Table 2), adjusting for the impact of geographical differences in the kinship and fertility of couples within Iceland (10). Each test revealed as significant association with kinship, with correlation coefficients of 0.063 (P = 1.5 × 10-129) for the number of children, 0.045 (P = 3.6 × 10-66) for the number of children who reproduced, and 0.042 (P = 7.6 × 10 -58) for the number of grandchildren. (Helgason et al. 2008)
There might be other confounds. That's why the Danish study is important. It went through a number of socioeconomic factors: "education, family income, urbanicity, mother's age at first birth, and six variables representing proximity to kin [maternal radius, i.e., distance between mother's and child's birthplace; paternal radius; and presence of each grandparent in the child's birth parish or neighboring parish" (Laboriau and Amorim 2008). Controlling for those factors did not change the findings.
Davenport's study on outbreeding in Jamaica
Jayman also criticized Davenport's study, apparently in reference to the finding that between 5 and 30% of the "brown" children did worse than predicted.
As for mixed race children, you have to keep in mind that people don't enter those relationships randomly - i.e., also confounded
Jayman is arguing that the parents of the brown children were atypical, apparently in the sense of being below-average. In its early days, Jamaica did have many poor whites, but most of them left because of competition from slave labor. European ancestry entered the Jamaican population largely from landowners and businessmen, particularly people of Scottish descent:
Jamaica has more people using the Campbell surnames than the population of Scotland itself, and it also has the highest percentage of Scottish surnames outside of Scotland. Scottish surnames account to about 60% of the surnames in the Jamaican phone books. The first Jamaican inhabitants from Scotland were exiled "rebels". Later, they would be followed by ambitious businessmen who spent time between their great country estates in Scotland and the island. As a result, many of the slave owning plantations on the island were owned by Scottish men, and thus a large number of mixed-race Jamaicans can claim Scottish ancestry. (Wikipedia 2020).
Today, "brown" Jamaicans are generally middle class, and they were even more so during the time of Davenport's study. It is indeed strange to intimate that his biracial subjects were the offspring of below-average whites and blacks.
Conclusion
Davenport's study probably corresponds to the highest degree of outbreeding possible. Even in that case, adverse mental effects were observable only in a minority of offspring. The most widespread effect is probably lower fertility and perhaps a higher risk of testicular cancer. Joffe (2009) suggests that an increase in outbreeding might explain a century-long decrease in semen quality and a corresponding increase in testicular cancer:
One implication of the proposed pathogenesis is that as D&D [duplications and deletions of genetic material during meiosis] accumulate, mating between individuals who are genetically unalike would be associated with lower reproductive success as pairing at the start of meiosis would be more likely to be impaired. On the face of it, this contradicts the earlier observation that hybrid vigour would tend to increase fertility in unrelated individuals. Yet these two ideas may be compatible: both extremes—genetic similarity (inbreeding) and genetic distance (D&D accumulation)—could decrease fertility, so that an intermediate degree of relatedness would be associated with the highest degree of fertility. This could explain the evidence from Iceland that the greatest reproductive success, measured as the number of grandchildren, was observed in couples who were third or fourth cousins (Helgason et al., 2008). Comparable findings have been reported from elsewhere, for example Denmark (Labouriau and Amorim, 2008).
I realize this may be a sensitive subject. If you feel offended, please read something else.
References
Davenport, C.B. and M. Steggerda. (1928). Race Crossing in Jamaica. Washington: Carnegie Institution, Publication no. 395.
http://www.velesova-sloboda.info/archiv/pdf/davenport-race-crossing-in-jamaica.pdf
Helgason, A., S. Pálsson, D.F. Guðbjartsson, þ. Kristjánsson, K. Stefánsson. (2008). An association between the kinship and fertility of human couples. Science 319(5864): 813-816.
http://facelab.org/debruine/Teaching/EvPsych/files/Helgason_2008.pdf
Joffe, M. (2009). What has happened to human fertility? Human Reproduction 25(2): 295-307.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1003.7270&rep=rep1&type=pdf
Labouriau, R., and A. Amorim. (2008). Comment on "An Association Between the Kinship and Fertility of Human Couples." Science 322(5908): 1634
http://science.sciencemag.org/content/322/5908/1634.2.full
Leppälä, J. and O. Savolainen. (2011). Nuclear-cytoplasmic interactions reduce male fertility in hybrids of Arabidopsis lyrata subspecies. Evolution 65(10): 2959-2972
https://onlinelibrary.wiley.com/doi/full/10.1111/j.1558-5646.2011.01361.x
Presgraves, D.C. (2010). The molecular evolutionary basis of species formation. Nature Reviews Genetics 11:175-180.
https://www.nature.com/articles/nrg2718
Turner, L.M., D.J. Schwahn, and B. Harr. (2011). Reduced male fertility is common but highly variable in form and severity in a natural house mouse hybrid zone. Evolution 66(2): 443-458.
https://onlinelibrary.wiley.com/doi/full/10.1111/j.1558-5646.2011.01445.x
Wikipedia (2020). Jamaica
https://en.wikipedia.org/wiki/Jamaica
