At its peak in 1860, about 393,975 people were listed as slaveholders, roughly 1.25% of the total U.S. population of 31.4 million. In the slaveholding states, about 24.9% of households owned enslaved people, and nearly half of slaveholders held fewer than five enslaved people. The rate varied by state, with higher concentrations in Mississippi and South Carolina.
Not to mention, all races have been slaves in the world, including White people.