Lots of good answers here, but one I haven't seen is that some people have different value systems. They would be the ones that say "yes, human rights would be nice, but at what cost?"
Typically, as everyone here has pointed out, they value their own well being and comfort. "We can't end child slave labor because then a KitKat would cost $20." They might cite economic priorities, national or personal security, religious beliefs, or civic pride (see: China).
It always boils down to the same core belief: “It's OK for some humans to suffer and die in indignity.” The cruelty is the point. Yes, it's a different value system, but usually it's about putting something other than humanity above everything else. Be it money, religion or ideology, it's always about the idea that some material or conceptual object is more valuable than human life and dignity. The other face of that coin is dehumanization, which is the idea that, “yes all humans deserve basic rights, but did you see what they did? They obviously are animals who don't deserve rights.”
Another common argument I've heard is regarding crime and criminals. Some politicians in my country tout that you cannot arrest criminals and we spend a fortune maintaining jails as if they were 5 star hotels. This is of course not true, but it does rail up the tough on crime crew who believe the only reason crime is rampant it is because it somehow allowed by human rights.
Since human rights are international agreements, it makes for an easy scape goat for those who believe in crazy conspiracy theories
So, if children aren't used as slave laborers, everything becomes more expensive.
Isn't that just what things cost? I mean, I could get free produce if I stole it from the local produce market. There are plenty of morally gray areas where reasonable people can have legitimate disagreements, but we can all get behind making child slaves illegal.
But you'll say it's a slippery slope. First the child slaves, then maybe all the slaves? What about the sex slaves? Are we just supposed to stop letting rich people have sex with unwilling humans? Where doe it end? If everyone has rights and freedom and access to food, shelter, medical care, and education, how will a tiny fraction of the population amass mountains of wealth and power? How will they manage to orgasm without squeezing the life from a poor immugrant who has been forcibly hooked on drugs?
No, I don't think boycotting cheap goods will create human rights in China and India. People will need to fight for human rights, and we, the privileged few who can afford to vote with our dollars, should demand fair trade foreign policies from our elected officials. We should vote for people who are for human rights everywhere. We should support policies that promote equality everywhere. And if that means we can't buy cheap jeans at Walmart, we should be prepared to accept that as an inconvenience.