Vanishing American writes:

Another reprehensible myth that is all too widely accepted by many Whites is the idea that ''most blacks have some White ancestry'' -- a notion that I think is exaggerated --and moreover, that this infusion of White DNA into the black gene pool is the result of 'rapist slave owners.' Somehow everybody seems to accept the allegation that slave-owning aristocrats routinely forced their 'attentions' on female slaves. There is little evidence to back up this notion. Presuming that the White genes were introduced back during slavery, why assume that the slave owner was the source? Why not overseers, White field hands, or others at lower levels of society?
An analysis of census data tends to support VA:
Evidence is also available indirectly from the censuses of 1850 and 1860. The enumerators of these censuses listed the color of slaves as black or mulatto. The percentage of mulattoes reported in the slave population was 7.70 in 1850 and 10.41 in 1860. [. . .]

Historians have implicated all social classes in miscegenation. Slaves lived in closest contact with owners and overseers, and unmarried owners and overseers on absentee estates may have been most heavily involved. According to Stampp, "Men of the nonslaveholding class were responsible for much of the miscegenation . . . Female slaves were quite accessible to both rural and urban nonslaveholders." Contacts were usually casual but relationships sometimes evolved into concubinage that lasted sev- eral years and occasionally for life. [. . .]

The estimated regression is used to calculate the probabilities that a slave child was mulatto. The probabilities are an index of the relative frequency of sexual relations. The chances that a child was a mulatto declined with the size of the holding and the number of slaves per dwelling, and they increased with the proportion mulatto among adults aged 15-49 on the plantation, the proportion white among males aged 15-49 in the county, and with city size. The probabilities were relatively higher on sugar plantations as compared with cotton plantations, in urban areas, in the slave exporting states, and on small holdings where no slave dwellings were listed. The probability was lower on rice plantations as compared with cotton plantations.

Several explanatory variables are positively related to the chances that a slave woman would have encountered a white who did not live on the holding, which suggests that a high proportion of sexual contacts were not attributable to the owner. Increasing the proportion white among males aged 15-49 in the county from .45 to .55 nearly triples the probability that a child was mulatto. The probability was also low in rice agriculture where the density of white settlement was low, and was high in urban areas where the density of white settlement was high.

[Richard H. Steckel. Miscegenation and the American Slave Schedules. Journal of Interdisciplinary History, Vol. 11, No. 2 (Autumn, 1980), pp. 251-263.]
That American blacks average 18-22% white genes suggests only a very low incidence of miscegenation per generation:
one out of every four Negroes living in a southern city was a mulatto. But among rural slaves, who constituted 95 percent of the slave population, only 9.9 percent were mulatto in 1860. For the slave population as a whole, therefore, the proportion of mulattoes was just 10.4 percent in 1860 and 7.7 percent in 1850. Thus it appears that travelers to the South greatly exaggerated the extent of miscegenation because they came into contact with unrepresentative samples of the Negro population. They appear to have had much more contact with the freedmen and slaves of the urban areas than with slaves living in the relative isolation of the countryside. Far from proving that the exploitation of black women was ubiquitous, the available data on mulattoes strongly militates against that contention.

The fact that during the twenty-three decades of contact between slaves and whites which elapsed between 1620 and 1850, only 7.7 percent of the slaves were mulattoes suggests that on average only a very small percentage of the slaves born in any given year were fathered by white men. [. . .]

Measurements of the admixture of "Caucasian" and "Negro" genes among southern rural blacks today indicate that the share of Negro children fathered by whites on slave plantations probably averaged between 1 and 2 percent.

[Fogel and Engerman. Time on the cross: the economics of American Negro slavery.]