Maybe I should say "Madam" first then "Ma'am"

Pronouncing "ma'am" with the vowel sound from "are" might help reduce the ambiguity.