Personally, I find him irksome because I get a strong vibe from him that he thinks of himself as a very smart person, looking down on the intellectual peasants. Part of why I perceive him that way is because this is how I used to think, as an autistic nerd who built much of my identity up around being smart. That's also why behaviour of the sort that shows up on /r/iamverysmart (such as many of NDT's posts) makes me cringe so much.
Dissecting this a bit further, it's not necessarily that I think he thinks he's better than other people — rather the opposite: some of the most intellectually arrogant people I have known are, at their core, deeply insecure and feeling the need to justify their interests by presenting themselves in a certain way.
Yeah that's exactly the vibe I get but for some reason it makes me feel more charitable towards him. Maybe because it doesn't trigger like an echo cringe in me because I'm not like that.