Why would you say that?

As a non-native speaker I encounter this phrase from time to time (in podcasts and such) and I'd like to understand the use (beside the literal meaning which is obvious).

Why would you say that? or sometimes Why do you say that?

To me, that sounds almost rude, like rebuking the questioner. However, the context usually leads me to conclude that this sentence is to be understood neutrally, in the sense of "I am interested in the background of the question".

How should the sentence usually be used? Or does it depend solely on the tone the phrase is used?