vLLM

vLLM is an open-source inference server for large language models (LLMs).