vLLM & Custom Endpoints

+15 Mana ✨
OpenClaw Local Models(DOCUMENTATION)

Introduction

vLLM is a high-performance inference server for large language models. OpenClaw can connect to vLLM and any other OpenAI-compatible endpoint through its custom provider system. This lets you use self-hosted inference servers, LiteLLM as a unified gateway, or any API that follows the OpenAI chat completions format.

Key Concepts

  • vLLM: A fast, memory-efficient inference engine for LLMs that exposes an OpenAI-compatible API.
  • OpenAI-Compatible Endpoint: Any HTTP API that implements the /v1/chat/completions contract.
  • LiteLLM: A proxy that provides a unified OpenAI-compatible API across 100+ model providers.
  • Custom Provider: An OpenClaw provider configuration that points to any OpenAI-compatible URL.

Real World Context

A machine learning team hosts their fine-tuned model on a vLLM server in their private cloud. They want OpenClaw agents to use this custom model for domain-specific tasks while falling back to Claude for general queries. The custom endpoint provider makes this seamless.

Deep Dive

To connect OpenClaw to a vLLM server, configure a custom provider:

json
{
  "providers": {
    "vllm-server": {
      "type": "openai-compatible",
      "baseUrl": "http://gpu-server.internal:8000/v1",
      "apiKey": "${VLLM_API_KEY}",
      "models": ["custom-codegen-34b"]
    }
  },
  "agents": {
    "code-agent": {
      "model": {
        "primary": "vllm-server:custom-codegen-34b",
        "failover": ["anthropic:claude-sonnet-4-20250514"]
      }
    }
  }
}

The openai-compatible type tells OpenClaw to use the standard OpenAI API format. The baseUrl points to your vLLM server, and the models array lists available model names. The agent uses the vLLM model as primary and falls back to Claude if the vLLM server is unavailable.

LiteLLM adds another layer of flexibility by proxying requests to multiple providers:

bash
# Start LiteLLM proxy
litellm --model gpt-4 --model claude-3-opus --port 4000

Then point OpenClaw to the LiteLLM proxy:

json
{
  "providers": {
    "litellm": {
      "type": "openai-compatible",
      "baseUrl": "http://localhost:4000/v1",
      "apiKey": "${LITELLM_KEY}"
    }
  }
}

LiteLLM handles provider routing, rate limiting, and fallover internally, while OpenClaw sees a single unified endpoint.

Common Pitfalls

  1. Mismatched model names — The model name in your OpenClaw config must exactly match what the endpoint expects. Check the endpoint's /v1/models listing.
  2. Missing API key for authenticated endpoints — Some vLLM deployments require authentication. Always set the apiKey even if your server does not currently require it.

Best Practices

  1. Use health checks — Configure health check endpoints so OpenClaw can detect when a custom server is down and trigger failover.
  2. Version your model endpoints — When deploying new model versions, use a new endpoint URL rather than replacing the existing one. This allows gradual rollout.

Summary

  • vLLM and any OpenAI-compatible endpoint can be connected via the openai-compatible provider type
  • LiteLLM provides a unified proxy across 100+ model providers
  • Custom providers support failover chains for high availability
  • Model names must exactly match what the endpoint expects
  • Always configure health checks and failover for production deployments
✓ Completed