[HN] LLMs up to 4x Faster With Latest NVIDIA Drivers on Windows
[HN] LLMs up to 4x Faster With Latest NVIDIA Drivers on Windows
blogs.nvidia.com Large Language Models up to 4x Faster on RTX With TensorRT-LLM for Windows | NVIDIA Blog
Generative AI on PC is getting up to 4x faster via TensorRT-LLM for Windows, an open-source library that accelerates inference performance.
[ comments | sourced from HackerNews ]
2
comments
Why just windows?
Two reasons:
Linux is the preferred choice because it is traditionally more performant than windows regarding these things, meaning windows users get gains Linux users already have.
And Nvidia has a different codebase for Linux, so it might take time to get some of those efficiency gains over to Linux.
Additionally as a little aside there's no way Apple is ever dealing with NVIDIA again, and they might dump AMD all together in the coming years.