Some of the most historically influential arguments for not treating groups of people are inferior are religious in nature. They say that everyone was created by God, Jesus died for everyone, therefore, no person, in virtue of being a member of a group, should be treated as inferior.
On the other hand, if you assume the secular view that human beings are in the groups they are in through evolution, then you have to deal with the idea that creatures are entitled to the survival advantages that evolution has provided them. Thus if being white, or male, gives you a position of power that makes you, (as opposed to others) better able to pass on your (selfish) genes than other people, then why shouldn't you take advantage of that superior power?
I realize that the former kind of argument has not prevented religious people from exploiting others, nor has the second argument prevented secularists from being nonexploiters. But what would be the secularist rebuttal to the latter argument?