They contribute texture, sure. And with enough maillard reaction, potato and bread can pull their own weight. I just hate the idea of filler ingredients. I had to make my food stretch when I was younger, and now I don’t have to (for now), so I’d much rather just have more of the genuinely tasty parts and leave out the bland bulk. Every rice dish is better to me without the rice, for instance. Same flavor, but more of it per bite.
That's like saying "I had to wait until the end of the meal to eat dessert when I was younger, and now I don't have to! Since dessert is the best part of the meal, I eat nothing but dessert now!"
I was also raised to believe that foods like rice were "filler". Refined grains might not have much to contribute, but it's been a lifechanging experience to have learned better that whole grains are nutritional and culinary cornerstones.