# Local models > Run PentAGI on models served from your own hardware, so prompts, tool output and findings stay on your servers. PentAGI can run entirely on models you serve yourself. The [agents](https://pentagi.com/docs/glossary/#agent)' prompts, the [tool](https://pentagi.com/docs/glossary/#tool) output they read and the [findings](https://pentagi.com/docs/glossary/#finding) they [report](https://pentagi.com/docs/glossary/#report) then stay on your servers, and a long [flow](https://pentagi.com/docs/glossary/#flow) costs GPU time instead of tokens. Two kinds of model servers are supported: Ollama, the quickest to set up, and vLLM or any other server with an OpenAI-compatible API, for higher throughput. ## [Ollama](https://pentagi.com/docs/deployment/local-models/#ollama) Point PentAGI at the Ollama server in `.env`: ```bash OLLAMA_SERVER_URL=http://ollama-server:11434 OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-qwen332b-fp16-tc.provider.yml OLLAMA_SERVER_PULL_MODELS_ENABLED=false # true downloads missing models at startup OLLAMA_SERVER_LOAD_MODELS_ENABLED=false # true lists the models the server already has ``` The config file assigns a model and its parameters to each [agent type](https://pentagi.com/docs/glossary/#agent-type); without one, `OLLAMA_SERVER_MODEL` sets the default model. The image ships ready configs in `/opt/pentagi/conf/`, among them `ollama-qwen332b-fp16-tc.provider.yml` and `ollama-qwq32b-fp16-tc.provider.yml`. Ollama's default context window is far smaller than a flow needs: agents typically use around 64K tokens, and PentAGI is set up for 110K. Ollama sets the context size only when it creates a model, so build a variant from a Modelfile: ```dockerfile FROM qwen3:32b-fp16 PARAMETER num_ctx 110000 PARAMETER temperature 0.3 PARAMETER top_p 0.8 PARAMETER min_p 0.0 PARAMETER top_k 20 PARAMETER repeat_penalty 1.1 ``` ```bash ollama create qwen3:32b-fp16-tc -f Modelfile ``` ## [vLLM and other OpenAI-compatible servers](https://pentagi.com/docs/deployment/local-models/#vllm-and-other-openai-compatible-servers) vLLM, LocalAI and other servers with an OpenAI-compatible API connect as the [custom provider](https://pentagi.com/docs/configuration/llm-providers/#custom-and-openai-compatible-endpoints): ```bash LLM_SERVER_URL=http://vllm-server:8000/v1 LLM_SERVER_KEY=local # any non-empty value when the server has no key LLM_SERVER_MODEL=Qwen/Qwen3.6-27B-FP8 LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/vllm-qwen3.6-27b-fp8.provider.yml ``` The image ships configs for several Qwen builds on vLLM; the `-no-think` variants turn reasoning off. Agents work through [tool calls](https://pentagi.com/docs/glossary/#tool-call), so start the server with the tool-call parser and chat [template](https://pentagi.com/docs/glossary/#template) recommended for your model: with a wrong parser, tool-call arguments come back broken and a flow stops after a few steps. ## [Choosing a model](https://pentagi.com/docs/deployment/local-models/#choosing-a-model) * Give your strongest model to `primary_agent`, `pentester` and `adviser` in the config file; lighter ones can serve `searcher`, `enricher` and summarization. * For models under 32B parameters, turn on [execution monitoring and task planning](https://pentagi.com/docs/concepts/agents/#safeguards). They roughly double result quality at the cost of two to three times more tokens and time. * Plan for the [memory](https://pentagi.com/docs/glossary/#memory): QwQ 32B in FP16, for example, needs about 71 GB of GPU memory. ## [Keep the rest of the stack local](https://pentagi.com/docs/deployment/local-models/#keep-the-rest-of-the-stack-local) | Part | Setting | | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Embeddings | `EMBEDDING_PROVIDER=ollama` with an embedding model in `EMBEDDING_MODEL` uses the same Ollama server; a local OpenAI-compatible server works with `EMBEDDING_PROVIDER=openai`, its address in `EMBEDDING_URL` and any non-empty `EMBEDDING_KEY` | | Web search | Search sends the agents' queries to the internet. DuckDuckGo is on by default; turn it off with `DUCKDUCKGO_ENABLED=false` and keep only the [engines](https://pentagi.com/docs/configuration/search-engines/) you accept, such as your own SearXNG | | [Knowledge graph](https://pentagi.com/docs/glossary/#knowledge-graph) | Off by default. When on, Graphiti calls OpenAI unless `GRAPHITI_LLM_CLIENT_TYPE=custom` sends it to the server in `LLM_SERVER_URL`, which it then also asks for embeddings with the model named in `EMBEDDING_MODEL`, or `text-embedding-3-small` when that is empty. If that server does not serve that model, set `GRAPHITI_SEPARATE_EMBEDDING=true`: Graphiti then takes `EMBEDDING_URL`, `EMBEDDING_KEY` and `EMBEDDING_MODEL`, the values PentAGI uses, and they must point at an OpenAI-compatible server — see [Knowledge graph](https://pentagi.com/docs/concepts/memory/#knowledge-graph-optional) | | Observability | The bundled Langfuse and Grafana stacks run on your host — see [Observability](https://pentagi.com/docs/configuration/observability/) | | Update check | `UPDATE_CHECK_INTERVAL=0` turns it off; [Security](https://pentagi.com/docs/configuration/security/#update-check) lists what it sends | | Images | Docker pulls the [sandbox](https://pentagi.com/docs/glossary/#sandbox) images from Docker Hub with its own settings; in a closed network, point the Docker daemon at a registry mirror |