# Kazuo Router

> OpenAI-compatible local AI gateway: route requests across multiple providers with auto-fallback, key rotation, model mapping and usage stats.

| | |
|---|---|
| Version | 1.0.0 |
| Category | AI & Agents |
| Permissions | network.remote, network.localhost |

# Kazuo Router

OpenAI-compatible local AI gateway. Runs an HTTP server on `127.0.0.1:<port>` that
speaks the OpenAI REST shape (`/v1/chat/completions`, `/v1/models`, …) and forwards
each request to one of several configured upstream providers — with auto-fallback,
round-robin key rotation, model mapping and per-provider usage stats. Point any
OpenAI-compatible tool (Cursor, Cline, SDKs, …) at `http://127.0.0.1:20128/v1`.

**Type:** `backend` — see `.claude/skills/kazuo-create-plugin/references/plugin-examples.md`

## Files

- `manifest.json` — metadata, `defaultSettings` (server/apiKey/routing/providers/modelMap), `defaultWidgetConfig`
- `widget.html` / `js/widget.js` / `css/widget.css` — dashboard: server control, provider CRUD, routing + model map, live usage
  - has its own "⚙" button + settings overlay (server host/port/auth/autostart) built into widget.html/widget.js
- `settings.html` / `js/settings.js` / `css/settings.css` — same server fields as the widget overlay (intentionally duplicated), used by the Plugin Manager screen and the pop-out window
- `locales/en.json5` / `locales/vi.json5` — UI strings, `widget.*` / `server.*` / `providers.*` / `routing.*` / `stats.*` / `settings.*`
- `main/index.ts` (+ compiled `main/index.js`) — main-process entry: `activate`/`deactivate`, IPC handlers, key generation, auto-start
- `main/router-server.ts` (+ compiled `main/router-server.js`) — the OpenAI-compatible HTTP server + routing/fallback/rotation/stats
- `main/claude-cli-bridge.ts` (+ compiled `main/claude-cli-bridge.js`) — `claude-sub` provider: runs the real Claude Code CLI via claude-agent-sdk, keeps live sessions, and hooks client tool calls (park/resume) for OpenAI function-calling
- `main/openai-anthropic-bridge.ts` — `openai-anthropic` provider: the reverse direction, serving Anthropic Messages from an OpenAI upstream (upstream call, SSE consumption, tool-call reassembly, `<think>` splitting)
- `main/anthropic-to-openai.ts` — request translation (system/messages/tool_result fan-out/schema sanitising/tool allowlist/context budget)
- `main/anthropic-wire.ts` — Anthropic SSE emission with a content-block **index allocator** (thinking → text → tool_use)
- `main/sse-parse.ts` — incremental SSE parser (streaming `TextDecoder`, so multi-byte characters split across chunk boundaries survive)

## How it works

- The main process runs one `http.createServer` bound to `server.host:server.port`.
- Config is read **fresh from plugin settings on every request**, so provider edits in the widget apply live without a restart.
- Auth: when `server.requireApiKey` is on, clients must send `Authorization: Bearer <apiKey>` (or `x-api-key`, or `?key=`). The key is generated on first activation.
- Routing for `POST /v1/chat/completions` (and other `/v1/*`):
  1. Parse the requested `model`; resolve it through `modelMap` (alias → forced provider and/or real model).
  2. Order candidate providers: forced provider first, then providers that declare the model, then the rest — sorted by `priority` (`priority-fallback`) or rotated (`round-robin`).
  3. Try each candidate; pick one of its keys round-robin. On network error / 429 / 5xx (before any bytes are sent) fall through to the next candidate.
  4. Stream (SSE) or buffer the winning response back to the client; record request + token stats.
- `GET /v1/models` aggregates the models declared by enabled providers plus mapping aliases.
- `POST /v1/messages` (Anthropic Messages API) is accepted for `claude-sub` providers, so a Claude Code CLI (e.g. the ai-chat plugin) can point `ANTHROPIC_BASE_URL` at this router and use it as a hub — and for `openai-anthropic` providers, which translate it down to `/v1/chat/completions` on a local OpenAI-compatible server.
- Errors are emitted in the caller's dialect: requests under `/v1/messages` get the Anthropic
  `{type:'error', error:{…}}` envelope, everything else keeps the OpenAI shape. An Anthropic
  SDK handed the OpenAI shape reports an opaque parse failure instead of the real problem.

## Default settings

- `server` — `{ port: 20128, host: "127.0.0.1", autoStart: false, requireApiKey: true }`
- `apiKey` — local access key (auto-generated on first activation)
- `routing` — `{ strategy: "priority-fallback" }` (or `"round-robin"`)
- `providers` — `[]` (each: `{ id, name, type, baseUrl, keys[], models[], priority, enabled }`)
  - `type: "claude-sub"` runs through the **real Claude Code CLI** via `@anthropic-ai/claude-agent-sdk` (`main/claude-cli-bridge.ts`) — the SDK spawns `claude.exe`, which uses the machine's Claude subscription login (OAuth; `ANTHROPIC_BASE_URL`/`AUTH_TOKEN` are stripped from the spawned env so it uses the subscription directly and can't loop back), so it is **not rate-limited** like a forged OAuth request. No `baseUrl`/`keys` needed. Serves **both** protocols so it can be a hub for OpenAI clients AND Claude Code CLIs: `POST /v1/chat/completions` (OpenAI) and `POST /v1/messages` (Anthropic). Function-calling passthrough: the client's tools are registered as in-process MCP tools (the spawned CLI's own built-in tools are disabled so only the client's tools exist); when the model calls one, the bridge does not run it — it emits a `tool_calls`/`tool_use` response and PARKS the session, then resumes (matched by the minted id) once the client posts the tool result back. `models[]` is just the client-facing list; the CLI maps them to `sonnet`/`opus`/`haiku`.
  - `type: "openai-anthropic"` is the **mirror image of `claude-sub`**: it serves the Anthropic Messages API *from* an OpenAI-compatible upstream (Ollama, vLLM, llama.cpp), so the real Claude Code CLI — and therefore ai-chat's Agent mode — can drive a **local model with tools**. Extra fields: `ctxTokens` (upstream context window) and `allowTools[]` (see below). `POST /v1/messages` is translated; `POST /v1/chat/completions` falls through to the plain relay, so one provider entry serves both protocols. `/v1/messages/count_tokens` is answered with an estimate — returning 404 there breaks the CLI's context meter and `/compact`.
- `modelMap` — `[]` (each: `{ alias, providerId, model }`)

### Running a local model in Agent mode (`openai-anthropic`)

Two constraints dominate this path, both measured rather than assumed:

**1. Tool schemas do not fit.** Claude Code advertises ~24 tools totalling ~24k tokens of
JSON Schema; with its ~8k-token system prompt, a bare `hello` costs **~34k tokens** before
the user types anything. `allowTools[]` filters which tool definitions reach the model —
the CLI still offers all of them, the model is simply never shown the rest. The default
(`Bash, Read, Edit, Write`) brings the same request down to **~15k**. This lives in the
router, so **ai-chat needs no code changes**.

**2. Prompt prefixes must be byte-stable.** llama.cpp reuses the KV cache when the leading
bytes of a prompt are identical. Measured on a 27B Q4: a stable ~15k-token prefix prefills
in **15.0s cold and 1.1s warm**. The system prompt plus tool schemas are most of every
request, so any nondeterminism in translation — a timestamp, a re-sorted tool list — costs
~14 seconds on *every* turn. `anthropic-to-openai.ts` never sorts or injects varying data.

Also worth knowing:

- An over-long prompt returns **400**, never a truncated request. Ollama silently discards
  the front of an oversized prompt, which presents as a model that ignores its tools with
  nothing in any log to explain why.
- Client disconnect aborts the upstream generation. Without it an abandoned 3k-token reply
  pins the GPU for minutes and blocks the next request.
- Reasoning is read from `delta.reasoning` / `delta.reasoning_content`, falling back to
  scanning inline `<think>` tags. Note that Ollama sends reasoning chunks with an **empty
  `content` string** alongside — testing `content` for truthiness rather than length opens
  a text block before the thinking block and corrupts block order for the whole message.
- Tool arguments are parsed and repaired before emission, not relayed as upstream
  fragments: one malformed `partial_json` kills the CLI mid-turn, and truncated or
  trailing-comma JSON is routine from a quantised model.

Wiring it to ai-chat is configuration only — add an env profile under
`providers['claude-code'].envProfiles` pointing `ANTHROPIC_BASE_URL` at this router.
Set every model variable (`ANTHROPIC_MODEL`, `ANTHROPIC_DEFAULT_{SONNET,OPUS,HAIKU}_MODEL`,
`ANTHROPIC_SMALL_FAST_MODEL`) to the same id, or the CLI will call a "small fast model" for
background work like conversation titles and that request will miss your provider.

## IPC channels (`plugin:kazuo-router:*`)

`start`, `stop`, `status`, `stats`, `clearStats`, `regenKey`, `testProvider` — all take the plugin id as the first arg (`ensureIsSelf`).

Events on `plugin:kazuo-router:event`: `{type:'status'|'request'|'stats', ...}`.

## Test

Open Kazuo AI → Plugin Manager — this plugin appears automatically → Install → Enable.
Add a provider (pick a preset, paste one or more API keys), press **Start**, then point a
client at `http://127.0.0.1:20128/v1` using the shown API key.

After editing `main/*.ts`, recompile before testing: `npx tsc -p tsconfig.plugins.json` (run from the project root).