GitDex

Software directory / sw_fd4d3cf6ad8c43

vLLM

Open sourceUnreviewedcli

Visit website

Nobody has independently verified this record. Check the sources before relying on it.

vLLM is an open source library for LLM inference and serving, focused on high throughput and memory efficiency, originally developed in the Sky Computing Lab at UC Berkeley.

Capabilities

  • Manages attention key and value memory with PagedAttention, with continuous batching, chunked prefill and prefix caching
  • Supports many quantization formats, including FP8, INT8, INT4, GPTQ/AWQ and GGUF
  • OpenAI-compatible API server, plus Anthropic Messages API and gRPC support
  • Tensor, pipeline, data, expert and context parallelism for distributed inference
  • Supports 200+ model architectures on Hugging Face, including multimodal and embedding models
  • Runs on NVIDIA, AMD and Intel GPUs and x86/ARM/PowerPC CPUs, with plugins for hardware such as TPUs
  • Installs with uv or pip, or builds from source

Translation provided by the submitter. Original in Chinese.

Proposed for these requests

Not proposed for any request yet. Requests directory

Sources

Where the listed facts come from, and how far each was verified.

  • DocumentedFeatures, supported hardware and models, installation and origin as stated in the README; the repository license is Apache-2.0.github.com/vllm-project/vllm