That... is something that I've never wondered about, but now you've got me contemplating the flavor profile of various mineral aggregates.
I can tell you what coal tastes like, not that I've eaten any. But when you burn it for heat, it gets everywhere. It's bitter and kind of earthy but not like rich topsoil. More like you've been tailgating a school bus on a dirt road for hours. And the dirt is charred.
I know what granite tastes like. I love putting a granite pebble in my mouth to suck on while trail hiking. I have been doing it since I was little, at this point it's just nostalgia. Granite, after the first minute, just tastes like almost nothing really. At first though, it tends to be rather a dusty earthy flavor.