What makes a motive selfish? Usually we develop words in order to distinguish between one type of action and another. In other words, without an idea of what it would be for us not to be selfish, the word "selfish" doesn't mean anything. So if you think all actions are really selfish, we would have to have some idea of what it would be unselfish, but then in fact we are never that way. I've never seen a psychological egoist explain what it would be like if we weren't always selfish.
When we call an action selfish, I think what we have in mind is where people do something in their own interest when the interests of others could and should have been considered. Consider someone who hogs the ball and attempts to maximize their own scoring statistics when presumably they could pass the ball and improve their team's chances thereby. Here you have one course of action (hog the ball and shoot when you can) and another course (pass it to help the team) and you choose the selfish action.
If this kind of contrast is present every time we use the word selfish, the it looks like we aren't ever going to come up with a defensible theory that makes all actions "really" selfish. What could that possibly mean?