Fun fact: rollerblading was part of the X Games since their inception, but if I'm not mistaken it was removed due to the stigma around it (mainly homophobia and elitism from skateboarders).
I'm not sure where the stereotype came from, but for a long time you were automatically "gay" if you rollerbladed. It's where the term "fruitbooting" comes from (fruity + boot, because rollerblades are typically boots). I've heard some people say that the stigma is still very much alive, claiming to be called slurs or even attacked while rollerblading, though that probably depends on where you are.
The other issue has to do with rollerblading being perceived as "easier" than skateboarding since the wheels are attached to your feet, making it easier to jump. The tradeoff that people forget, however, is that the wheels are attached to your feet. You can bail on a skateboard by shoving it away and rolling/sliding out of it (if you know how to safely bail), but it's much, much harder to bail with rollerblades because they're heavy, they roll, and the wheels are attached to your feet.