I actually might be able to provide some insight to this. My wife is Balkan and the first time I called her a "Silly Goose" she became madly upset. It turns out in her language the phrase "Glupova Gusko" (Stupid Goose) is a common insult. It is considered incredibly harsh in her culture. My guess is that the phrase "Silly Goose" is borrowed from a Slavic Language and lost its harshness when it moved into English.
It’s a common insult in many languages - with a lot of theories about origins.
You can track medival usage of this all over Europe, but I have no knowledge about other continents.
Insults in general hold more power in honor-cultures. Your grandparents might have reacted the same way she did, while you might not feel the same.
I know of at lest Nordic languages, german languages, slavic languages, Urdu, Yiddish
Some are not literal translations, but have the same meening. Like silly as a goose - not stupid. Or just beeing refered to as a goose implying simplicity, stupidity or sillyness.