How much gpu do i need to run a 90b model
You may try Harbor. The description claims to provide an OpenAI-compatible API.
In case of LLM's you should look at AirLLM. I suppose there is no conviniet integrations to local chat tools, but issue at Ollama already started.