Skip to content

Home

vLLM JAX vLLM JAX

| Documentation | Blog | User Forum | Developer Slack |

Important

tpu-inference is now vLLM JAX (vllm-jax) and is focused on serving JAX models on TPU with vLLM.

This repository is moving to vllm-project/vllm-jax. Until the move is complete, development continues in vllm-project/tpu-inference; open issues and pull requests there.

PyTorch-native TPU serving has moved to its own project, vllm-torchtpu, which ships its own releases and is where all future PyTorch-native TPU support in vLLM will live. If you are serving PyTorch models on TPU, head there.

About

vLLM JAX is powered by vllm-jax (formerly tpu-inference), an expressive and powerful new hardware plugin unifying JAX and PyTorch under a single lowering path within the vLLM project. The new backend now provides a framework for developers to:

  • Push the limits of TPU hardware performance in open source.
  • Provide more flexibility to JAX and PyTorch users by running PyTorch model definitions performantly on TPU without any additional code changes, while also extending native support to JAX.
  • Retain vLLM standardization: keep the same user experience, telemetry, and interface.

What are you trying to do today?

Contribute

We're always looking for ways to partner with the community to accelerate vLLM JAX development. If you're interested in contributing to this effort, check out the Contributing guide and Issues to start. We recommend filtering Issues on the good first issue tag if it's your first time contributing.

Contact us