Ollama Beyond the Basics: Model Management, Custom Models, and Optimization

You Pulled a Model. Now What?

Congratulations. You ran ollama pull gemma4 and had a conversation with a locally running LLM. You felt like a wizard. That’s valid.

But that was the tutorial, the training-wheels version. Ollama’s real muscle shows up after the pull: shaping model behavior, squeezing performance out of whatever GPU you have (or don’t), and wiring it into actual tools. The quickstart glosses over what matters: Modelfiles, quantization trade-offs, GPU offloading, context tuning, the REST API, and how to stop your model from cold-starting every request. Practical examples throughout, minimal fluff.

Modelfiles: Baking Your Personality In

A Modelfile is Ollama’s version of a Dockerfile: a plain text recipe defining how a model behaves. Layer a custom system prompt on top of any base model, tune parameters, and save the whole thing as a named model you can call anytime.

FROM gemma4

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096

SYSTEM """
You are a grumpy but accurate Linux sysadmin named Dave.
You answer questions correctly, but you always complain about how the user
should have Googled it first. Keep responses under 200 words.
"""

ollama create grumpy-dave -f Modelfile
ollama run grumpy-dave

You now have a persistent, named model with Dave’s whole personality baked in. No pasting a system prompt every session.

The four directives you actually need:

• FROM: the base model. A model name (gemma4, qwen3-coder) or a path to a GGUF file if you’re loading something you downloaded manually.
• PARAMETER: tuning knobs. temperature is the creativity dial (0.1 is a boring accountant, 1.5 is a caffeine-fueled conspiracy theorist, 0.7 is the sweet spot for most tasks). top_p controls vocabulary diversity, leave it at 0.9 unless you have a reason not to. num_ctx is the context window in tokens, more below. num_gpu is how many layers to load onto the GPU, the real performance lever.
• SYSTEM: your system prompt, verbatim. This is how you give the model a job title, constraints, or a persona. Use triple quotes for multiline.
• TEMPLATE: the prompt format wrapper. You rarely need to touch this, Ollama handles it per model, but a raw GGUF pulled from Hugging Face may need the chat template set manually so it knows how to parse turns.

System Prompts That Actually Matter

Your system prompt shapes everything. Three that work in the real world:

Structured output (JSON only):

You are a JSON generator. Respond ONLY with valid JSON, no other text.

Use this when piping output to a script. No more parsing hallucinated prose out of what should be a clean payload.

Ruthless summarizer:

You are a ruthless editor. Distill the key points into 3 to 5 bullets.
Ignore filler. Be specific. Include numbers if present.

Code review persona:

You are a senior engineer who has been burned by technical debt before.
Review this code for maintainability, not just correctness.
Point out what will hurt in 6 months, not what's broken today.

System prompts are free. Make them specific to what you’re actually building instead of generic.

Quantization: The Trade-Off Nobody Explains Clearly

When you pull a model, you’re usually pulling a quantized version. Quantization shrinks weights from 32-bit floats down to lower precision: smaller files, faster inference, some accuracy cost.

Q4_K_M is where most people should live: small, fast, and the quality loss is hard to notice in casual use. Q8_0 earns its extra headroom for coding and reasoning tasks where precision counts. F16 is for researchers with A100s; if you’re reading this blog, skip it. The extreme end (IQ1_M, IQ2_XXS) squeezes a 7B-class model into 1 to 2 GB with a real quality hit, good for testing whether something fits at all, bad for anything you’d ship.

Pull a specific quantization directly:

ollama pull qwen3-coder:q8_0

Check what’s available for a model at ollama.com/library.

GPU Layer Offloading: The Real Performance Lever

Here’s where people go wrong: they install Ollama, they have a GPU, and inference is still painfully slow. Nine times out of ten, they haven’t checked how many layers are actually hitting the GPU.

Ollama offloads model layers to GPU memory automatically, but it can only fit what fits. If your model is 8 GB and your VRAM is 6 GB, some layers stay on CPU and everything slows to a crawl. Control it explicitly:

PARAMETER num_gpu 35

A 7B-class model typically has 32 to 33 transformer layers plus a few more, so num_gpu 35 covers all of it. Setting num_gpu 99 is the “just put everything on the GPU” shorthand; Ollama caps it at whatever fits.

To see what’s actually happening:

ollama ps

This shows running models, VRAM usage, and processor. If you see 100% CPU next to your model, that’s your problem.

The bitter truth: running a 70B-class model on 8 GB of VRAM isn’t a configuration problem you can fix, it’s a hardware problem. A Q4_K_M quantization of a model that size still runs roughly 40 GB. You need a bigger GPU, multiple GPUs, or the patience of a monk. For 8 GB of VRAM, stay in the 7B to 13B range at Q4_K_M or Q5_K_M. That’s where inference actually feels snappy.

Context Length: Don’t Just Max It Out

num_ctx controls how many tokens the model can see at once, your conversation history plus the current prompt. More context means the model remembers more, but it also costs VRAM and inference speed.

Ollama’s own runtime default is 4096 tokens as of recent releases (it used to be 2048). That’s the runtime default, not a claim about what any specific model was trained on, some models ship with a native window well past that. Don’t assume, check:

ollama show gemma4

That prints the model’s actual trained context length straight from its metadata, which is more reliable than trusting whatever number a tutorial quoted last year.

A practical guide:

• Chat and Q&A: 2048 to 4096 is fine.
• Summarizing documents: 8192 to 16384.
• Long-form analysis: 32768 or higher, if your hardware allows it.

Cranking context to 32768 because a tutorial did it is how you accidentally push layers off the GPU and into slow CPU territory. A 32K window can run 1.5 to 2x slower than an 8K one. Only extend when you genuinely need it: reviewing a large file, summarizing a long document.

PARAMETER num_ctx 8192

You can also override it per request in the API if different tasks need different windows.

The REST API: Ollama Isn’t Just a CLI

Ollama exposes a REST API at http://localhost:11434 by default. This is how you integrate it with applications, scripts, and other tools. Two endpoints matter most.

/api/generate is single-turn completion, good for one-shot tasks and scripts:

curl http://localhost:11434/api/generate \
  -d '{"model": "gemma4", "prompt": "Explain VRAM in one sentence.", "stream": false}'

/api/chat is multi-turn, you maintain conversation history yourself and send it with each request:

curl http://localhost:11434/api/chat \
  -d '{
    "model": "gemma4",
    "messages": [
      {"role": "user", "content": "What is a Dockerfile?"},
      {"role": "assistant", "content": "A Dockerfile is a recipe for building a container image..."},
      {"role": "user", "content": "How is that different from a docker-compose.yml?"}
    ],
    "stream": false
  }'

For anything beyond a quick curl test, hit the same endpoints from Python:

import requests

response = requests.post(
    "http://localhost:11434/api/generate",
    json={"model": "gemma4", "prompt": "What's the capital of France?", "stream": False},
)
print(response.json()["response"])

Streaming for real-time output, useful for a chat UI or CLI progress feel:

import json
import requests

response = requests.post(
    "http://localhost:11434/api/generate",
    json={"model": "gemma4", "prompt": "Write a haiku about containers", "stream": True},
    stream=True,
)
for line in response.iter_lines():
    chunk = json.loads(line)
    print(chunk["response"], end="", flush=True)

Both endpoints stream by default; set stream: false when scripting and you just want the complete response back in one shot.

Concurrent Requests and Keeping Models Warm

By default, Ollama unloads a model from memory after 5 minutes of inactivity. The next request reloads it, costing several seconds. Annoying if you’re building something people actually use.

Keep a model loaded indefinitely via the API:

curl http://localhost:11434/api/generate -d '{"model": "gemma4", "keep_alive": -1}'

Or from the CLI, which is often more convenient for a quick test:

ollama run gemma4 --keepalive 10m "list Docker commands"

keep_alive: -1 means never unload; keep_alive: 0 force-unloads immediately. Long timeouts mean higher peak memory, short timeouts mean slower first-token latency on the next request, pick based on how bursty your traffic actually is.

Running two models at once is simple arithmetic once you check what’s loaded:

$ ollama ps
NAME                ID              SIZE      PROCESSOR
qwen3-coder:latest  d1234567890ab   4.2 GB    GPU
gemma4:9b           a5678901234cd   8.5 GB    GPU

4.2 GB plus 8.5 GB is 12.7 GB. If your card has 24 GB, you’re comfortable. If it has 8 GB, one of those models is getting pushed to CPU whether you planned for it or not.

For pinning concurrency explicitly rather than letting Ollama decide:

OLLAMA_NUM_PARALLEL=4 ollama serve

This lets Ollama handle 4 simultaneous requests to the same model. Useful for a shared instance. Tune it against your VRAM: more parallel requests means more memory consumed per request.

Plugging Into the Ecosystem

Open WebUI is a full ChatGPT-style interface that connects to Ollama out of the box: self-hosted, Docker-deployable, multiple models, conversation history, file uploads.

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  ghcr.io/open-webui/open-webui:main

Continue.dev brings Ollama into VS Code as a local coding assistant. Point it at your instance in ~/.continue/config.yaml:

models:
  - name: Qwen3 Coder (local)
    provider: ollama
    model: qwen3-coder
    apiBase: http://localhost:11434

Highlight code, ask questions, get completions from your own hardware instead of someone else’s API bill.

LiteLLM is a proxy that makes Ollama look like OpenAI’s API. If you already have code talking to OpenAI and want to swap in local models without rewriting anything, LiteLLM is the bridge. And Ollama itself ships an OpenAI-compatible layer at /v1/, so most OpenAI SDK code works against http://localhost:11434/v1/chat/completions with just a base URL change.

Troubleshooting Slow Inference

Before you blame the model, blame the setup.

Check GPU usage first:

ollama ps
watch -n1 nvidia-smi  # or rocm-smi for AMD

If the model’s running on CPU, you have a layer-fit problem: reduce num_ctx or switch to a smaller quantization. If it’s reloading every request, you need keep_alive: -1 or a longer timeout. If the first token takes forever but the rest streams fine, that’s load time, use ollama ps to confirm it’s actually resident in memory. And on Linux with an NVIDIA card, make sure the NVIDIA container toolkit is installed if you’re running Ollama in Docker, without it the container can’t see the GPU at all and you’re doing CPU inference while your GPU sits idle.

Pull It All Together

Ollama goes from neat demo to actual tool once you treat it like infrastructure. Modelfiles let you version and share model configurations. Quantization choices fit the right model into the hardware you actually have. GPU tuning removes the biggest bottleneck. The API opens it up to everything else in your stack.

Your 8 GB GPU isn’t going to run a 70B-class model at useful speeds. That’s fine. A well-tuned Gemma 4 or Qwen3 Coder at Q5_K_M with a solid system prompt handles most practical tasks at genuinely usable speeds. Optimize for what you have, not what you wish you had. Dave the grumpy sysadmin would tell you that you should have figured this out yourself, but at least now you’ve got somewhere to start.

Related Reading

• API vs Self-Hosted LLMs: The Real Cost
• LLM Backends: vLLM vs llama.cpp vs Ollama
• Continue.dev + Ollama: Local Code Assistant for Cheap
• Function Calling in Local LLMs
• The Embedding Model Choice Nobody Explains