I suppose you could sew a L and an R into each one to distinguish left from right, or even just an X on one of them so you could tell them apart, then you'd know which one to swap to the other foot, but ain't nobody got time for that.
Mine even themselves out by fluffing up in the washer/dryer. Are you wearing them a second time before washing them? Or maybe if you hang them dry they don't get fluffed?