vLLM

vLLM is an open-source server for running models on your own hardware. Its API follows OpenAI’s format, so you wrap it with the OpenAI client and the SDK meters it with the OpenAI recipe.

Set up

Point the OpenAI client’s baseURL at vLLM and wrap it:

import OpenAI from 'openai';

const vllm = stigg.wrap(
  new OpenAI({ baseURL: 'http://localhost:8000/v1', apiKey: process.env.VLLM_API_KEY }),
);

const response = await vllm.chat.completions.create({
  model,
  messages,
  stiggContext: { customerId: 'customer-acme', entitlement: { currencyId: 'credits' } },
});

Use the address of your own vLLM server. http://localhost:8000/v1 is vLLM’s default.

What is reported

vLLM returns a usage object in OpenAI’s format with the number of tokens each call consumed. The calls go through the OpenAI client, so they are reported under OpenAI event names. The wrapper reads the counts and reports them to Stigg as the metrics below.

Method	Event name
`chat.completions.create`	`openai.chat.completions.create`

Chat Completions

Metric	What it counts
`prompt_tokens`	Input tokens, including cached tokens
`completion_tokens`	Output tokens
`cached_tokens`	Input tokens served from the cache, when the vendor reports them
`total_tokens`	All input and output tokens, as reported by the vendor

total_tokens includes the other token counts, so price the total or the parts, never both. To tell vLLM apart from other vendors on the same event name, add a dimension such as provider to the stiggContext. Every usage field the provider returns is also included, with a raw. prefix. See How usage is reported. For details on vLLM’s API, see vLLM’s OpenAI-compatible server docs.

Introduction

Get started

Usage metering

Credits

Entitlements and access enforcement

Governance

AI agents

Architecture and deployment

Pricing and packaging

Customers, subscriptions and contracts

Workflows

Widgets

Developer tools

Account management

Security & compliance

Set up

What is reported

​Set up

​What is reported

Set up

What is reported