Local models

PentAGI can run entirely on models you serve yourself. The agents' prompts, the tool output they read and the findings they report then stay on your servers, and a long flow costs GPU time instead of tokens. Two kinds of model servers are supported: Ollama, the quickest to set up, and vLLM or any other server with an OpenAI-compatible API, for higher throughput.

Ollama

Point PentAGI at the Ollama server in .env:

OLLAMA_SERVER_URL=http://ollama-server:11434
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-qwen332b-fp16-tc.provider.yml
OLLAMA_SERVER_PULL_MODELS_ENABLED=false   # true downloads missing models at startup
OLLAMA_SERVER_LOAD_MODELS_ENABLED=false   # true lists the models the server already has

The config file assigns a model and its parameters to each agent type; without one, OLLAMA_SERVER_MODEL sets the default model. The image ships ready configs in /opt/pentagi/conf/, among them ollama-qwen332b-fp16-tc.provider.yml and ollama-qwq32b-fp16-tc.provider.yml.

Ollama's default context window is far smaller than a flow needs: agents typically use around 64K tokens, and PentAGI is set up for 110K. Ollama sets the context size only when it creates a model, so build a variant from a Modelfile:

FROM qwen3:32b-fp16
PARAMETER num_ctx 110000
PARAMETER temperature 0.3
PARAMETER top_p 0.8
PARAMETER min_p 0.0
PARAMETER top_k 20
PARAMETER repeat_penalty 1.1

ollama create qwen3:32b-fp16-tc -f Modelfile

vLLM and other OpenAI-compatible servers

vLLM, LocalAI and other servers with an OpenAI-compatible API connect as the custom provider:

LLM_SERVER_URL=http://vllm-server:8000/v1
LLM_SERVER_KEY=local                      # any non-empty value when the server has no key
LLM_SERVER_MODEL=Qwen/Qwen3.6-27B-FP8
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/vllm-qwen3.6-27b-fp8.provider.yml

The image ships configs for several Qwen builds on vLLM; the -no-think variants turn reasoning off. Agents work through tool calls, so start the server with the tool-call parser and chat template recommended for your model: with a wrong parser, tool-call arguments come back broken and a flow stops after a few steps.

Choosing a model

• Give your strongest model to primary_agent, pentester and adviser in the config file; lighter ones can serve searcher, enricher and summarization.
• For models under 32B parameters, turn on execution monitoring and task planning. They roughly double result quality at the cost of two to three times more tokens and time.
• Plan for the memory: QwQ 32B in FP16, for example, needs about 71 GB of GPU memory.

Keep the rest of the stack local