yeah I have the same position. I would also give this kind of ironic answer to avoid answering the question, because I don't really know if I'd pull the lever or not
If the lever reflects a power dynamic, killing a slaveowner to free ten slaves, then that's a lever that should exist and I'd pull it without thinking. You can't wear white gloves while doing it but the slaveowners wouldn't be there if not by choice. There's a clear greater good option that breaks a dialectic which shouldn't exist. I'll always pull a lever with a liberation option and thank the trolley driver for their courage.
If the lever is killing generic people, it's a false dilemma. That just makes it a stupid fuck-fuck game being played by the person with power and I instinctively hate that person. That's the trolley company which is okaying this and every functionary of that company shares moral responsibility for it. When democrats tell me to vote for 99% Hitler because 100% Hitler will do four genocides, the only moral calculus there is to reject it for the same reason socialists are against all wars but class wars.
I agree on the first point. It depends on who these people are
tho I think this thought experiment is meant to be super abstract in order to gauge someone's ethics. Personally, I'm leaning toward not pulling the lever because i'd probably feel more personally responsible for killing this one person than letting five people die