spark-utilities

Tooling and config for the home DGX Spark (spark-9d33): model router, self-hosted models, pi integration, cost tracking, and benchmarking.

.
├── ansible/                   # idempotent Spark deployment playbook
│   └── deploy.yml
├── litellm/                   # Spark-local LiteLLM proxy and systemd deployment
│   ├── config.yaml             # local model registrations and cost rates
│   ├── spark-litellm.service   # Docker-backed proxy service (:4000)
│   └── postgres.service        # persistent LiteLLM database
│   └── spark-litellm.env.example
├── router/                    # llama-swap front door + model recipes
│   ├── config.yaml            #   llama-swap config (source of truth; deployed to Spark ~/llama-swap/)
│   ├── llama-swap.service     #   systemd unit for the Spark
│   ├── recipes/
│   │   ├── qwen3-vl-8b-sglang/recipe.yaml          # vision model (Frigate), active
│   │   └── qwen3-vl-30b-a3b-sglang.recipe.yaml     # upgrade option (won't fit alongside ds4)
│   └── sparkrun/clusters/     # sparkrun cluster defs (deployed to ~/.config/sparkrun/clusters/)
├── pi/                        # pi (coding agent) client config
│   ├── models.json            #   deployed to ~/.pi/agent/models.json
│   └── settings.json          #   deployed to ~/.pi/agent/settings.json
├── scripts/
│   ├── bench.py               #   single-user + concurrency throughput benchmark
│   ├── vision_check.py        #   qwen3-vl vision pipeline sanity test
│   ├── recompute_cost.py      #   rewrite historical session costs (dry-run by default, --apply, --skip-active)
│   ├── session_cost.py        #   report tokens + cost per session/model at current rates
│   └── cost_amortization.py   #   electricity cost model + idle amortization math
└── tracking/
    └── session-costs.md       # running log of session costs

Architecture

pi / Open WebUI / Frigate / scripts
        │  model: deepseek-v4-flash | qwen3-coder-next | qwen3-vl
        ▼
  spark-swapper (spark-9d33:8081)   ← public queue/scheduler
        │
  llama-swap  (spark-9d33:8091)      ← request-level model switching
        │  spawns on demand (sequential hot-swap — 121GB unified RAM fits ONE model at a time)
        ▼
  deepseek-v4-flash  → ds4-server (custom llama.cpp fork, GGUF IQ2XXS, DSpark drafter, ctx 64K)
  qwen3-coder-next   → sparkrun → SGLang (Qwen3-Coder-Next-FP8, 200K ctx)
  qwen3-vl           → sparkrun → SGLang (Qwen3-VL-8B-Instruct-FP8)
  qwen3.6-27b        → sparkrun → vLLM (Qwen3.6-27B-FP8)
  qwen3.6-35b-a3b-nvfp4 → sparkrun → vLLM (NVFP4, on-demand sequential hot-swap)
  qwen2.5-coder:1.5b → sparkrun → SGLang (Qwen2.5-Coder-1.5B-Instruct, 8K ctx)

• Switch models = change model: in any OpenAI-compatible request. Point everything at http://spark-9d33:8081/v1 (spark-swapper queues and schedules; it forwards to llama-swap on :8091), or LiteLLM on :4000/v1.
• Hard limit: the Spark's 121GB unified memory cannot run two models at once (dmesg NV_ERR_NO_MEMORY). Sequential swap is the design.
• DeepSeek cloud access stays client-side (https://api.deepseek.com/v1) — llama-swap is local-only.

Deploy / reference notes

• Desktop ~/llama-swap/ was the working copy; router/ here is now the source of truth. Files deploy to the Spark as rssnyder@spark-9d33:
  • config.yaml → ~/llama-swap/config.yaml
  • llama-swap.service → /etc/systemd/system/ (sudo systemctl enable --now llama-swap)
  • recipes → ~/.config/sparkrun/services/<name>/
  • clusters → ~/.config/sparkrun/clusters/
  • pi/models.json → ~/.pi/agent/models.json
• pi/spark-litellm/ → a separate pi config directory for the LiteLLM gateway
• litellm/ → /home/rssnyder/litellm/ and /etc/systemd/system/ on the Spark
• spark-swapper is the scheduler in front of llama-swap. It owns :8081, while llama-swap uses :8091; do not bind llama-swap to :8081.
• Gotchas learned the hard way (all fixed in the configs):
  • ds4-server has no /health → checkEndpoint: /v1/models
  • systemd drops supplementary groups → SupplementaryGroups=docker in the unit
  • sparkrun recipe lookup fails under systemd env → use full recipe paths in cmds
  • Qwen3-VL FP8 config ships pad_token_id: None → patched cached config to 151645
  • DeepGEMM FP8 crashes on GB10 → fp8_gemm_backend: cutlass in VL recipe
  • llama-swap execs without a shell → absolute paths, folded cmd: scalars
  • earlyoom -s 80 + stale swap kills python (sparkrun) → clear swap on overloads

LiteLLM on the Spark

LiteLLM listens on http://spark.r.ss:4000/v1 and fronts the single llama-swap endpoint on 127.0.0.1:8081. All local models—including qwen3.6-27b, qwen3.6-35b-a3b-nvfp4, and qwen2.5-coder:1.5b—are managed by llama-swap's one-model-at-a-time hot-swap controller. LiteLLM's per-token rates mirror the measured Spark electricity rates in the pi config.

The service uses Docker and requires a root-owned, mode-600 /etc/litellm/spark-litellm.env containing LITELLM_MASTER_KEY and a database password. Ansible creates the database password once, runs a persistent PostgreSQL container for LiteLLM's UI/users/spend data, and preserves both secrets thereafter. Deployment is idempotent and uses Ansible for LiteLLM, PostgreSQL, and Qwen3.6:

export LITELLM_MASTER_KEY='sk-...' # only needed on a fresh Spark
./scripts/deploy_litellm.sh

Override the target with SPARK_SSH_HOST=rssnyder@192.168.2.209. Existing remote keys are preserved; Ansible creates a key only when the file is absent.

Hot-swap model setup

To add a model, define it in router/config.yaml under models and add its ID to the appropriate routing group. The spark-memory group is exclusive; the spark-small group uses swap: false for the Qwen3.6/VL combination. Qwen2.5-Coder remains in the exclusive group because its SGLang reservation uses ~36 GB even though its weights are small. The command must listen on ${PORT} and stay alive while llama-swap checks readiness. For detached sparkrun workloads, use a small launcher like router/qwen36-run.sh that starts with --ensure and then stays alive. Set:

• checkEndpoint to the backend's real readiness path (/health or /v1/models)
• useModelName when the friendly router ID differs from the upstream model ID
• timeouts.responseHeader for slow first responses
• ttl: 0 only for models that must remain warm; otherwise use an idle TTL

Do not run a second standalone systemd model service: it bypasses the swap controller and can consume the Spark's entire 121 GiB memory. The scheduler keeps a one-minute minimum hold before switching models; this is configured in router/spark-swapper.toml. Deploy and restart with ./scripts/deploy_litellm.sh, then test each route through LiteLLM using its model ID.

Pi is configured globally in ~/.pi/agent/models.json and ~/.pi/agent/settings.json; pi now defaults to the synced spark-litellm provider. The sync extension queries LiteLLM's /v1/models at startup. Restart Pi or run /reload after changing models in LiteLLM. Verify with curl http://spark.r.ss:4000/health/liveliness and curl -H 'Authorization: Bearer <key>' http://spark.r.ss:4000/v1/models.

Cost model (electricity)

Measured: 12W idle, ~56W under load, ~25 tok/s decode, 11,300 tok/s prefill. At $0.08/kWh → **$0.05/M output** marginal. Idle amortization at current usage (~29K output tok/day) → all-in ~$0.84/M; break-even vs cloud DeepSeek ($0.28/M) is ~100K output tok/day. See scripts/cost_amortization.py for the full model.

pi/models.json uses electricity rates for spark models and real API prices for cloud. tracking/session-costs.md logs per-session spend.

Instructions for the next pi session

1. Record the previous session's cost (see tracking/session-costs.md): run python3 scripts/session_cost.py --by-session --dir ~/.pi/agent/sessions/<PROJECT>/, append the finished session's row, commit.
2. Anything measured on the Spark (power, tok/s, benchmarks) — update scripts/cost_amortization.py defaults and this README.
3. Config changes → edit here, run ./scripts/deploy_litellm.sh, commit.