llmCLI — Roxabi

01

What it is

Serving a local model usually means babysitting a server process — picking ports, juggling VRAM, restarting to change weights. llmCLI wraps that into a single command: pull a GGUF, serve it on an OpenAI-compatible endpoint, hot-swap to another without touching the callers. Everything runs on hardware you own — no cloud keys, no per-token billing, no model that vanishes when a provider deprecates it.

It is the third sibling in the local-first CLI family. voiceCLI does speech, imageCLI does pixels, llmCLI does tokens — the same operational shape applied to the inference layer that the others lean on.

The durable contract is the OpenAI endpoint, not the engine. Callers speak one API; the model behind it swaps freely.

02

A catalog of models

Models are config, not code. The host settings live in one TOML; each model is a single flat file in a models/ directory — name the file, name the model. Drop a new one in and llmcli list picks it up immediately, no restart.

# ~/.roxabi/llmcli/models/qwen3-14b-q5.toml
engine   = "llamacpp"
repo     = "Qwen/Qwen3-14B-GGUF"
file     = "qwen3-14b-q5_k_m.gguf"
port     = 8092
vram_gib = 11
flags    = ["-ngl", "99", "-c", "8192", "-fa", "on", "--jinja"]

The catalog lets each host carry what its card can hold — a 16 GB workstation pins the heavy models, a 10 GB always-on box keeps small ones resident. llmcli swap hot-swaps the running engine; llmcli status reports engines, ports, VRAM, and uptime at a glance.

03

One endpoint, every caller

The server exposes the OpenAI chat-completions API, so anything that speaks that protocol works unchanged — your own scripts, an SDK, an agent. The point is uniformity: a local Qwen and a hosted model are the same call shape to whatever sits in front.

Command	Does
`llmcli pull <name>`	Download a model into the shared HF hub cache.
`llmcli serve [name]`	Start the daemon and serve a model.
`llmcli swap <name>`	Hot-swap the running model in place.
`llmcli status` · `list`	Engines, ports, VRAM, uptime · catalog + running state.
`llmcli chat <name> "…"`	One-shot call, bypassing the proxy.

04

Proxy & worker

Two deployment surfaces ship as Podman Quadlet units. The LiteLLM proxy runs on every host and gives the whole LAN one address — Claude Code, agents, and tools all route through it, mixing local engines with cloud models behind a single config. The NATS worker runs only where a GPU lives, exposing local inference on the message bus.

llama-server :PORT   ◄── llmcli serve (catalog-driven hot-swap)
      ▲
      │ OpenAI API
      ├── agents          (LiteLLM library, per-agent config)
      └── litellm proxy   ◄── claude-code, LAN tools

That is how llmCLI plugs into roxabi-factory: the worker registers on the bus, so any agent can reach a local model without knowing which machine holds the GPU.

05

Run it

Requirements: Python 3.12, a CUDA GPU for local serving, and the uv package manager. The proxy alone runs anywhere; the engine wants the GPU.

uv sync
cp models/*.toml ~/.roxabi/llmcli/models/   # seed the catalog
llmcli pull qwen3_6-35b-a3b-tq3             # fetch a model
systemctl --user start llmcli               # proxy on :18091
llmcli status

Engines, catalog, proxy, and the NATS worker are all in the open.

The repo · github.com/Roxabi/llmCLI →