mini_ghost

Replay Engine for LLM Cost Optimization

Extract production LLM prompts from SigNoz, replay them against cheaper models via Oxlo.ai, and find safe swap candidates without touching your app.

View on GitHub

What it does

LLM teams overpay for expensive models when cheaper alternatives often produce equivalent output. mini_ghost turns SigNoz from a passive cost tracker into an active cost optimizer.

original: gpt-4o
$0.12 · 2.1s
"The capital of France is Paris."
deepseek-v3.2
$0.03 · 0.9s
Safe to swap · 94%
"The capital of France is Paris."
llama-3.3-70b
$0.05 · 1.4s
Safe to swap · 91%
"The capital of France is Paris."

See it in action

How it works

  1. Extract - Query SigNoz ClickHouse for LLM spans by trace_id. Pulls gen_ai.prompt and completion from live traffic.
  2. Replay - Fire the prompt concurrently against 2-3 cheaper ghost models via Oxlo.ai.
  3. Score - Compute semantic similarity with BGE-Large embeddings. Calculate cost and latency deltas.
  4. Inject - Send ghost traces back to SigNoz via OTLP/HTTP so you can explore them natively.

Quick start

uv sync
source .venv/bin/activate
python scripts/seed.py --count 3
uvicorn mini_ghost.web:app --reload --port 8080

Then open http://localhost:8080/ghost?trace_id=<id> to compare.