Some would make this argument. The parts of morality that are productive for society are those parts that religious and nonreligious people agree with (murder, theft, etc.). The parts of morality that religious people accept and nonreligious people reject are the parts of morality that are really harmful (such as opposition to homosexuality). Therefore whenever religion adds anything, it adds something counterproductive.
How would you respond to this argument?