Local Providers¶

Run AI models locally for privacy, cost savings, and offline use.

Overview¶

Local providers offer:

• Privacy: Data stays on your machine
• Cost savings: No API fees
• Offline use: Work without internet
• Full control: Model selection and tuning

Quick Start: Zero To Local Coding¶

SuperQode bundles a guided path from "I want local coding" to a harness you can inspect, run, and improve on your repo. You pay once in hardware, not forever in token bills. Local is slower than frontier labs and quality depends on your model and machine, so SuperQode focuses on measurement, control, and ownership of the harness.

Run one command from inside your repository:

superqode local init --repo .

local init will:

1. Detect your hardware tier and installed local engines.
2. Recommend trusted models (sourced from models.dev Labs and vetted communities only).
3. Run a non-destructive smoke test against the running server.
4. Write a transparent harness to superqode.local.yaml.
5. Print the next command to run.

If no server is running yet, start one first, then rerun init:

superqode local serve ollama
superqode local init --repo .

Once init reports the harness is ready, start coding:

superqode --harness superqode.local.yaml

Warmup And First Response Latency¶

The first generation from a local model can be slow because the server may need to load weights, allocate KV cache, or initialize runtime paths. When you connect a local model from the TUI, SuperQode sends a tiny best-effort warmup request before your first real prompt. If warmup fails or times out, the connection still succeeds and the first prompt may pay the cold-start cost.

Disable automatic TUI warmup when you want full manual control:

SUPERQODE_LOCAL_WARMUP=0 superqode

Increase the timeout for large local models:

SUPERQODE_LOCAL_WARMUP_TIMEOUT=60 superqode

You can also warm a running server explicitly:

superqode local warm ollama --model qwen3:8b

Do not treat any shipped harness pack as sacred. Pick your model, pick or generate a harness, pick your memory and permissions, then customize the YAML for your project. SuperQode's Gemma, Qwen, GLM, MiniMax, Laguna, DS4, Devstral, and gpt-oss packs are getting-started defaults; your smoke tests and evals are what turn them into a project harness.

Who Starts The Server?¶

SuperQode does not secretly start model servers from readiness checks. Commands such as local init, local build, local migrate, and local smoke diagnose the setup and print next steps.

When you explicitly run superqode local serve <engine> or :local serve <engine>, SuperQode starts a managed background server, records pid/log data in ~/.superqode/servers/, and can stop only servers it started. Existing servers are adopted for status display, not restarted. In the TUI, selecting an MLX or llama.cpp model asks before launching because those engines serve one model per process. Type manual at that prompt to run the printed command yourself.

For MLX, missing Hugging Face weights are not downloaded unless you explicitly pass --allow-download.

Find The Right Model To Download¶

You download models with each engine's own tool (ollama pull, Hugging Face, a GGUF). To find the right one for your hardware without guessing or wasting a download, search the trusted catalog:

superqode local setup qwen3-coder --repo .
superqode local search qwen
superqode local search qwen3-coder --json

Prefer the TUI for first setup:

:local setup qwen3-coder --repo .

local setup is guidance-only. It recommends a model, download command, server command, context length, harness pack, and smoke test, but it does not download weights or start a server. That keeps large downloads and long-running local processes explicit.

Add --hub to also query the Hugging Face Hub live, filtered to trusted publishers (the model labs plus vetted quant communities like mlx-community and lmstudio-community), so you see the latest releases:

superqode local search glm --hub
superqode local search minimax --hub
superqode local search qwen3-coder --hub --gguf

In the TUI, type :hub to enter model-search mode, then type a model name without the :local search prefix. :hub <name> performs a one-shot search, and :hub off exits. Add --hub on a line for live Hugging Face results.

For each match it lists the real native download command for every engine the model can run on, plus a one-line superqode models download alternative that auto-picks the engine:

Qwen3-Coder 30B-A3B  [~17.4 GB · likely fits]
    ollama      ollama pull qwen3-coder:30b-a3b
    llama.cpp   llama-server -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF
    LM Studio   lms get https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF
    MLX         hf download lmstudio-community/Qwen3-Coder-30B-A3B-Instruct-MLX-4bit
    SuperQode   superqode models download lmstudio-community/...-MLX-4bit  (any engine)

MLX and GGUF repositories are retrieved from trusted publishers through the live Hugging Face API. Download commands use hf download, llama-server -hf <repo>, and lms get <full HF URL>. The deprecated huggingface-cli syntax is not emitted.

Each result includes an estimated model size, local availability, and an approximate memory-fit assessment. Size estimates are derived from parameter count and quantization. Without network access, the command returns catalog results only. Run the printed command with the corresponding model tool, then connect with :connect local.

Verify Readiness Anytime¶

local smoke runs the same non-destructive readiness probe on demand. It never reads or edits your repo:

superqode local smoke --repo .

It checks that the server is reachable, a chat model (not an embedding model) is loaded, the context window is detected, and that a tiny prompt returns clean tool-call and patch output. It also measures TTFT and decode speed. The verdict is ready, usable with warnings, or not ready yet, and every failure prints the exact next command to run.

Common Failure Messages¶

local init and local smoke report diagnostic information with errors. The following table lists common messages and corrective actions:

Supported Providers¶

Before adapting an existing project, run a migration dry-run:

superqode local migrate --repo . --model MiniMaxAI/MiniMax-M1

It inventories prompts, skills, role files, existing harnesses, and config, then prints the local-model adaptation work without changing files.

DwarfStar (DS4)¶

DS4 runs Laguna S 2.1 and DeepSeek V4 Flash locally and exposes OpenAI-compatible, Responses, and Anthropic-style endpoints. SuperQode treats it as a local provider named ds4, so it can be used from the CLI, TUI, provider doctor, and model recommendation flow.

Use the model-specific DwarfStar branch when it provides optimized support for your model. SuperQode preserves DwarfStar's thinking blocks across tool turns and uses its compact tool profile.

Prerequisites¶

• A working DS4 checkout or release directory.
• The ds4-server binary available in that directory or on PATH.
• A compatible GGUF file, such as laguna-s-2.1-Q4_K_M.gguf or ds4flash.gguf.

See the upstream project for installation and model details: antirez/ds4.

Start DS4¶

For Laguna S 2.1, download Poolside's official Q4_K_M GGUF once:

hf download \
  poolside/Laguna-S-2.1-GGUF \
  laguna-s-2.1-Q4_K_M.gguf

This stores the file in Hugging Face's standard cache. SuperQode's laguna-s-2.1 alias resolves the cached Poolside snapshot without making a network request. Start that one cached artifact with either DwarfStar or llama.cpp:

# Optimized DwarfStar path. --build checks out the Laguna support branch
# when ds4-server is not already installed.
superqode local serve ds4 --model laguna-s-2.1 --ctx 32768 --build

# General llama.cpp path, using the exact same GGUF file.
superqode local serve llama.cpp --model laguna-s-2.1 --ctx 32768

Reproducible Laguna Setup and Test¶

Every machine uses the same standard-cache download command:

hf download \
  poolside/Laguna-S-2.1-GGUF \
  laguna-s-2.1-Q4_K_M.gguf

Do not add --local-dir; Hugging Face chooses the snapshot location under its normal cache. HF_HOME and HF_HUB_CACHE are supported when a machine relocates that cache. SUPERQODE_LAGUNA_GGUF is needed only for a GGUF outside Hugging Face's cache and the other discovered model locations.

Optionally verify the completed cache entry:

hf cache verify poolside/Laguna-S-2.1-GGUF

For an existing DwarfStar checkout, verify the required branch:

git -C ~/oss/ds4 branch --show-current
# expected: laguna-s2.1

The following command performs an incremental make even when ds4-server already exists, preventing an old binary from being mistaken for a Laguna-ready build:

superqode local serve ds4 \
  --model laguna-s-2.1 \
  --ctx 32768 \
  --build

Once the server is ready, test discovery, a real completion, and tool calling:

superqode providers doctor ds4 --live
superqode providers smoke ds4 \
  --model laguna-s-2.1 \
  --run \
  --prompt "Inspect the current directory and summarize it."

Stop DwarfStar before testing the same cached GGUF with llama.cpp:

superqode local stop ds4
superqode local serve llama.cpp --model laguna-s-2.1 --ctx 32768
superqode providers doctor llamacpp --live
superqode providers smoke llamacpp \
  --model laguna-s-2.1 \
  --run \
  --prompt "Inspect the current directory and summarize it."
superqode local stop llama.cpp

Run only one of these servers at a time on a 128 GB Mac. Override the shared file location with SUPERQODE_LAGUNA_GGUF=/absolute/path/model.gguf. A direct GGUF path also works with both commands. The managed llama.cpp command enables Jinja templates and reasoning preservation for Laguna automatically.

Laguna requires a current llama.cpp build with Poolside model support. Upstream support landed after older builds such as Homebrew build 9430, which reject the model or do not recognize --reasoning-preserve. Upgrade before using the managed launcher:

brew update
brew upgrade llama.cpp
llama-server --version

The Poolside llama.cpp fork is also supported. For a manual compatibility test, start with the minimal flags below and replace the model argument with the absolute path printed by hf download:

llama-server \
  -m /absolute/path/to/laguna-s-2.1-Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8081 \
  --ctx-size 32768 \
  --jinja \
  --alias laguna-s-2.1

In the SuperQode TUI, run :connect local and choose either DwarfStar 4 or llama.cpp. When the shared GGUF is present, Laguna appears as a selectable model even if the server is stopped. Select it and confirm the launch; the TUI uses the same shared path, 32K starting context, and runtime-specific flags. When DwarfStar is not installed, the confirmed managed launch builds its Laguna support branch before starting the server.

After DwarfStar starts, its connection screen shows three intentional API variants backed by the same GGUF:

• Poolside Laguna S 2.1 lets each request control thinking.
• Poolside Laguna S 2.1 Chat disables thinking.
• Poolside Laguna S 2.1 Reasoner enables thinking.

These are behavior aliases, not three copies of the model. The llama.cpp provider opens a dedicated model picker instead of appending discovered models to the provider list.

SuperQode-managed start:

superqode local serve ds4 --ctx 32768
superqode local servers
superqode local stop ds4

local serve ds4 starts the DS4 binary from ~/oss/ds4 or PATH, uses a safe 32K context by default, and enables disk KV cache under ~/.superqode/ds4-kv. SuperQode prints the exact command before launching it.

Manual start from the directory that contains ds4-server and the model file:

./ds4-server --ctx 32768 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

For long local-agent sessions, raise context only when memory headroom allows:

./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

Think Max needs at least 393,216 context tokens:

./ds4-server --ctx 393216 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 16384

If you launch the server from another directory, pass the DS4 checkout path so runtime files resolve correctly:

./ds4-server --chdir /path/to/ds4 --ctx 32768 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

By default, SuperQode expects DS4 at:

http://127.0.0.1:8000/v1

If your DS4 server runs somewhere else, set:

export DS4_HOST=http://127.0.0.1:8000/v1

DS4 is moving quickly. To use a new upstream server flag that SuperQode has not promoted yet, either start ./ds4-server manually or pass it through:

superqode local serve ds4 --ctx 32768 --extra=--ssd-streaming --extra=--ssd-streaming-cache-experts --extra=32GB

Check SuperQode Connectivity¶

superqode providers guide ds4
superqode providers models ds4
superqode providers recommend local
superqode doctor

providers guide ds4 checks whether the local server is reachable. If DS4 is not running, SuperQode will show a setup hint instead of treating the provider as ready.

Run a Headless Coding Task¶

superqode -p --provider ds4 --model deepseek-v4-flash "summarize this repository"

For a harness run, use the DS4 example or template:

superqode harness run --spec examples/harnesses/ds4.yaml --prompt "review this repository"
superqode harness init my-ds4 --template ds4-fast-local

Use deepseek-chat when you want the DS4 non-thinking/direct alias:

superqode -p --provider ds4 --model deepseek-chat "review the current git diff"

Connect From The TUI¶

In the SuperQode TUI command input, open the local provider picker and select DS4 by number:

:connect local

You can also jump straight to the DS4 model list:

:connect local ds4

Direct model selection is still supported:

:connect local ds4/deepseek-v4-flash

Notes¶

• DS4 is local, so no API key is required.
• SuperQode supplies a placeholder OpenAI API key for OpenAI-compatible local clients that require one.
• SuperQode uses DS4's Anthropic-style /v1/messages path for direct local runs so tool and thinking blocks stay in the shape DS4 expects.
• DS4 uses a smaller DS4-specific tool profile and disables parallel tool execution by default.
• DS4 uses direct tool gating by default: SuperQode sends tools for repo, file, test, command, and code-change requests, but skips tools for ordinary questions and standalone code-generation prompts. This reduces unnecessary agent iterations.
• deepseek-v4-flash is the recommended default for coding and long-context local work.
• deepseek-chat is useful when you want the non-thinking mode exposed by DS4-compatible clients.

DS4 Tool Mode¶

The default DS4 tool mode is auto. Override it when you need different behavior:

# Default: send tools only for project/file/codebase work
export SUPERQODE_DS4_TOOL_MODE=auto

# Restore eager tool use
export SUPERQODE_DS4_TOOL_MODE=always

# Disable tools for DS4
export SUPERQODE_DS4_TOOL_MODE=never

Cold Start & Warm-up¶

DS4 mmaps a large GGUF (the IQ2XXS DeepSeek V4 Flash build is ~81GB) and pays a one-time cost paging it in from disk on the first inference. Once warm, responses are fast (sub-second for short prompts). To keep your first real prompt fast, SuperQode warms the model on connect: it sends a tiny 1-token request and shows a live elapsed-time indicator, then reports when DS4 is warm.

✓ DS4 server ready at http://127.0.0.1:8000/v1
⏳ Loading model into memory (first start can be slow on a cold cache)...
   ...still loading the model (10s)
✓ DS4 ready (warm) - 24s

Tips to avoid cold starts:

• Keep ds4-server running between sessions (the OS page cache stays warm).
• Use the disk KV cache (--kv-disk-dir) so prompt prefixes survive restarts.

Disable the connect-time warm-up with:

export SUPERQODE_DS4_WARMUP=0   # 0/false/no/off - skip warm-up on connect

Local Code Search (No Web Access)¶

Local models have no internet access, so web_search is intentionally not part of the DS4/local tool profile - and asking a local model to "search the web" will not work. Instead, local harnesses should ground the model in local code using repo_search (broad, ranked files + content + symbols in one pass), grep, code_search (symbols/definitions/references), and read_file.

To let a local model search a repo you downloaded outside your project (for reference, API examples, etc.), point SuperQode at it with SUPERQODE_SEARCH_ROOTS (os.pathsep-separated - : on macOS/Linux):

export SUPERQODE_SEARCH_ROOTS="$HOME/refs/react:$HOME/refs/linux"
superqode -p --provider ds4 "how does this project's router compare to react's? search the react ref"

• Search and read tools (repo_search, grep, glob, code_search, read_file, list_directory) may access those roots read-only.
• Writes/edits/shell stay confined to your working directory - reference repos cannot be modified.
• Address a reference repo by its absolute path. The configured roots are listed in the local model's system prompt so it knows they are available.

Ollama¶

The easiest way to run local models.

Installation¶

macOS:

brew install ollama

Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows: Download from ollama.com

Quick Start¶

# Start Ollama
ollama serve

# Pull a model
ollama pull qwen3:8b

# Connect in SuperQode
superqode connect local ollama qwen3:8b

Recommended Models¶

These are recommendations, not a model store. Keep them constrained to models.dev Labs, LM Studio Community, or MLX Community provenance so users do not get a biased or confusing grab bag of arbitrary model names.

Configuration¶

providers:
  ollama:
    base_url: http://localhost:11434
    type: openai-compatible
    recommended_models:
      - qwen3.6:35b-a3b
      - glm-4.5-air
      - gemma4:e4b

LM Studio¶

GUI-based local model runner.

Installation¶

1. Download from lmstudio.ai
2. Install and open LM Studio
3. Download a model (search for "qwen", "glm", or "gemma")
4. Load the model
5. Start Local Server (port 1234)

Connect¶

superqode connect local lmstudio local-model

Configuration¶

providers:
  lmstudio:
    base_url: http://localhost:1234
    type: openai-compatible

Tips¶

• Keep LM Studio running in background
• Load model before connecting
• Check "Local Server" tab for status

MLX (Apple Silicon)¶

Optimized for M1/M2/M3 Macs.

Installation¶

uv pip install mlx-lm

Quick Start¶

# Download model
mlx_lm.download mlx-community/Qwen2.5-Coder-3B-4bit

# Start server yourself (in a separate terminal)
mlx_lm.server --model mlx-community/Qwen2.5-Coder-3B-4bit

# Connect in SuperQode
superqode connect local mlx mlx-community/Qwen2.5-Coder-3B-4bit

You can also ask SuperQode to manage the server explicitly:

superqode local serve mlx --model mlx-community/Qwen2.5-Coder-3B-4bit
superqode local servers
superqode local stop mlx

Recommended Models¶

MLX recommendations use the vetted mlx-community namespace or a model family also present in models.dev Labs.

MLX Commands¶

# List available models
superqode providers mlx list

# Show suggested models
superqode providers mlx models

# Check installation
superqode providers mlx check

# Full setup guide
superqode providers mlx setup

Configuration¶

providers:
  mlx:
    base_url: http://localhost:8080
    type: openai-compatible

Limitations¶

• One server per model
• Single request at a time
• MoE models not supported

vLLM¶

High-performance inference for production.

Installation¶

uv pip install vllm

Quick Start¶

# Start server
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-Coder-7B-Instruct \
  --port 8000

# Connect in SuperQode
superqode connect local vllm Qwen/Qwen2.5-Coder-7B-Instruct

Configuration¶

providers:
  vllm:
    base_url: http://localhost:8000
    type: openai-compatible

Benefits¶

• High throughput
• Continuous batching
• PagedAttention

SGLang¶

Fast structured generation framework optimized for complex prompts.

Installation¶

uv pip install "sglang[all]"

Quick Start¶

# Start server
python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-Coder-7B-Instruct \
  --port 30000

# Connect in SuperQode
superqode connect local sglang Qwen/Qwen2.5-Coder-7B-Instruct

Configuration¶

providers:
  sglang:
    base_url: http://localhost:30000/v1
    type: openai-compatible

Features¶

• RadixAttention: Fast KV cache reuse for better performance
• Compressed FSM: Efficient structured output generation
• OpenAI-compatible API: Drop-in replacement for OpenAI endpoints

Benefits¶

• Faster inference for complex prompts
• Efficient structured generation
• Good for code analysis tasks

Recommended Models¶

TGI (Text Generation Inference)¶

HuggingFace's production-grade inference server with multi-GPU support.

Installation¶

# Using Docker (recommended)
docker run --gpus all \
  -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id Qwen/Qwen2.5-Coder-7B-Instruct

# Or using Python
uv pip install text-generation

Quick Start¶

# Using Docker
docker run -d --gpus all \
  -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id Qwen/Qwen2.5-Coder-7B-Instruct \
  --port 80

# Connect in SuperQode
superqode connect local tgi Qwen/Qwen2.5-Coder-7B-Instruct

Configuration¶

providers:
  tgi:
    base_url: http://localhost:8080
    type: huggingface

Features¶

• Flash Attention & Paged Attention: Memory-efficient attention
• Continuous Batching: Efficient request handling
• Tensor Parallelism: Multi-GPU support
• Token Streaming: Real-time token output
• Tool/Function Calling: Built-in tool support

Benefits¶

• Production-ready server
• Multi-GPU scaling
• Memory efficient
• Good for high-load scenarios

Multi-GPU Setup¶

docker run --gpus all \
  -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id Qwen/Qwen2.5-Coder-7B-Instruct \
  --num-shard 4  # Use 4 GPUs

llama.cpp¶

CPU/GPU inference server for GGUF format models.

Installation¶

# Clone and build
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make

# Or use pre-built server
# Download from releases

Quick Start¶

# Start server
./llama-server \
  -m models/qwen2.5-coder-7b.Q8_0.gguf \
  --port 8080 \
  --host 0.0.0.0

# Connect in SuperQode
superqode connect local llamacpp local-model

Configuration¶

providers:
  llamacpp:
    base_url: http://localhost:8080/v1
    type: openai-compatible

Features¶

• GGUF Format: Efficient model format
• CPU/GPU Support: Works on both CPU and GPU
• Low Memory: Efficient memory usage
• OpenAI-compatible: Standard API interface

Benefits¶

• Runs on CPU efficiently
• Works with quantized models
• Low resource requirements
• Good for older hardware

Model Format¶

llama.cpp uses GGUF format models:

# Convert model to GGUF
python convert.py --outfile model.gguf --outtype f16 model/

# Quantize model
./quantize model.gguf model-q8_0.gguf Q8_0

Quantization Levels¶

Performance Tips¶

1. Choose Right Model Size¶

2. Use Quantized Models¶

Quantized models (4-bit, 8-bit) use less RAM:

# MLX example
mlx_lm.download mlx-community/Qwen2.5-Coder-7B-4bit  # vs 8bit

3. Keep Server Running¶

Start servers before SuperQode sessions to avoid startup delays.

4. Use Appropriate Context Length¶

Shorter context = faster inference:

# Ollama example with context length
ollama run qwen3:8b --num-ctx 4096

Troubleshooting¶

Connection Refused¶

[INCORRECT] Connection failed: Connection refused

Solution: Ensure server is running:

# Ollama
ollama serve

# MLX
mlx_lm.server --model <model>

# LM Studio
# Check Local Server tab

Model Not Found¶

[INCORRECT] Model 'qwen3:8b' not found

Solution: Pull/download the model first:

# Ollama
ollama pull qwen3:8b

# MLX
mlx_lm.download mlx-community/Qwen2.5-Coder-3B-4bit

Out of Memory¶

[INCORRECT] CUDA out of memory / MPS out of memory

Solutions: - Use smaller model - Use quantized model - Close other applications - Reduce context length

Slow Inference¶

Solutions: - Use quantized models - Reduce context length - Use GPU acceleration - Consider faster hardware

Comparison¶

Next Steps¶

• BYOK Providers - Cloud alternatives
• Provider Commands - CLI reference