Llamafile: Single-Binary LLMs That Actually Just Run

Your Colleague Is Not Installing Python

You’ve been there. You want to show someone a local LLM demo. Maybe it’s a manager, a non-technical teammate, or your dad who still calls every app “the internet.” You start explaining: “So first install Python, then pip, then we need CUDA drivers, then—”

Their eyes glaze over. The demo never happens.

Llamafile exists to fix this. It’s Mozilla’s project that takes llama.cpp — the workhorse C++ LLM runtime — and bundles it with Cosmopolitan libc to produce a single executable binary that runs on Linux, macOS, Windows, FreeBSD, NetBSD, and OpenBSD. One file. Double-click or ./run. Done.

No containers. No virtual environments. No installer wizard with 47 “Next” buttons.

How the Magic Works

Cosmopolitan libc is the genuinely weird piece here. It produces what Jart Arora (the creator) calls an Actually Portable Executable (APE) — a binary that’s simultaneously valid as an ELF (Linux), Mach-O (macOS), PE (Windows), and a shell script. The OS loader picks up whatever format it speaks, and the binary adapts at runtime.

Llamafile wraps this around llama.cpp and also embeds the GGUF model file directly into the binary using a ZIP append trick. The result: Mistral-7B-Instruct.llamafile is the runtime and the model in one ~4 GB file.

On Linux/macOS you just need to mark it executable once:

chmod +x Mistral-7B-Instruct-v0.2.Q4_K_M.llamafile
./Mistral-7B-Instruct-v0.2.Q4_K_M.llamafile

On Windows, rename it to .exe and double-click. That’s it. The server starts, opens a browser tab with a chat UI, and exposes an OpenAI-compatible API on http://localhost:8080.

Your 2 AM self will appreciate not having to debug a Python version mismatch at this step.

Grabbing a Llamafile

Mozilla maintains pre-built llamafiles on HuggingFace. The usual suspects are there — Mistral 7B, Llama 3.2 3B, Phi-3, Gemma 2. Pick one based on how much RAM your machine has:

For the “just hand someone a file” use case, Llama-3.2-3B is the sweet spot. It fits on a USB drive and runs on a potato.

# Grab the 3B — fast and fits on a thumb drive
wget https://huggingface.co/Mozilla/Llama-3.2-3B-Instruct-llamafile/resolve/main/Llama-3.2-3B-Instruct.Q6_K.llamafile

chmod +x Llama-3.2-3B-Instruct.Q6_K.llamafile
./Llama-3.2-3B-Instruct.Q6_K.llamafile

Server starts. Browser tab opens at http://localhost:8080. Chat away.

The OpenAI-Compatible API

The built-in server speaks the OpenAI API format on :8080. That means anything that talks to OpenAI — LangChain, Continue.dev, curl, your own scripts — can point at llamafile with a one-line change.

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ignored-but-required",
    "messages": [{"role": "user", "content": "Explain Docker volumes in one sentence."}],
    "max_tokens": 100
  }'

Want to use it from Python? Same deal — swap base_url and use a dummy API key:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="local-model",
    messages=[{"role": "user", "content": "What is Kubernetes?"}]
)
print(response.choices[0].message.content)

Zero other dependencies needed on the machine that runs the client — just the llamafile server running.

Bring Your Own GGUF

The pre-built llamafiles are convenient but large. If you already have a GGUF model from a previous Ollama or llama.cpp setup, you can use the bare llamafile runtime binary (no model embedded) and point it at your file:

# Download just the runtime (much smaller)
wget https://github.com/Mozilla-Ocho/llamafile/releases/latest/download/llamafile
chmod +x llamafile

# Point at any GGUF you already have
./llamafile -m ~/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf --host 0.0.0.0 --port 8080

This is the move if you’re embedding llamafile into a tool or CI pipeline. Ship the runtime separately, mount the model wherever. The binary is only ~5 MB without a model baked in.

You can also use --cli flag to skip the server entirely and get a terminal prompt — useful for quick one-shot queries in scripts:

./llamafile -m model.gguf --cli -p "Summarize this in 3 bullet points: $(cat notes.txt)"

The Warts (Because There Are Always Warts)

Llamafile is genuinely impressive engineering, but it’s not Ollama and doesn’t pretend to be.

File size. A full llamafile with a 7B Q4 model is ~4.5 GB. That’s fine for a USB demo but not something you’re distributing over a slow corporate VPN. “Just send the file” has limits.

GPU support. On macOS, Metal acceleration works out of the box — Apple Silicon in particular runs these fast. On Linux/Windows with NVIDIA, it’s trickier. CUDA support exists but requires a one-time compilation step or using pre-built CUDA variants when available. The default binary falls back to CPU if no GPU backend is detected. For pure CPU inference, performance is reasonable. For serious throughput, you’ll want Ollama with proper CUDA drivers instead.

Windows path lengths. Windows has a 260-character path limit by default. Long file names plus deep directory paths can cause the APE extraction to fail silently. The fix is enabling long paths in Windows (Group Policy or registry), but that’s not something you can assume your tech-allergic colleague has done.

No model management. Llamafile doesn’t have an Ollama-style registry, pull command, or model library. You find GGUFs, you download them, you run them. That’s more manual than Ollama but also more transparent — you always know exactly what model file you’re using.

Llamafile vs Ollama vs LM Studio

Honest comparison, no fanboy nonsense:

Ollama is the right answer for your home lab server that stays on 24/7. LM Studio is the right answer for the person who wants a nice GUI and doesn’t touch a terminal. Llamafile is the right answer for the scenarios where neither of those fit.

When Llamafile Is the Right Answer

Air-gapped environments. Security teams, industrial networks, classified setups — if the machine can’t reach the internet, you can’t ollama pull. You can hand someone a USB drive. Llamafile is the move.

One-off demos. You’re giving a talk. You want to show a local LLM running in 30 seconds without doing a live install. Drop the llamafile in your presentation folder, chmod +x, run. Audience sees it start immediately. No Docker daemon, no Python env, no “let me just install this real quick” dead air.

Embedding in applications. You’re building a tool that needs an LLM backend. Shipping ollama as a dependency means telling users to install Ollama first. Shipping a llamafile means bundling the runtime into your release and having your app exec it on startup. One fewer external dependency.

Ephemeral environments. CI pipelines, throwaway VMs, systems you’re not going to touch again. Don’t install anything permanently — just drop the binary, run, discard.

The USB drive scenario. This is the one I keep coming back to. A 4 GB USB drive with a llamafile on it is a fully self-contained AI assistant that runs on whatever laptop someone hands you. No install. No internet required. It’s the kind of thing that would’ve seemed like science fiction five years ago and now it just… works.

Quick Reference

# Run with built-in model + web UI
./model.llamafile

# Run on custom port, accessible on LAN
./model.llamafile --host 0.0.0.0 --port 11434

# CLI mode, no server
./llamafile -m model.gguf --cli -p "Your prompt here"

# Specify context size and thread count
./model.llamafile -c 4096 -t 8

# List available API endpoints
curl http://localhost:8080/v1/models

The project is actively maintained at github.com/Mozilla-Ocho/llamafile. Mozilla’s involvement gives it staying power — this isn’t a weekend project that’ll disappear in six months.

Honestly, the fact that a 4 GB file can run a capable language model on six operating systems with no configuration is the kind of thing that should make you stop and appreciate how far we’ve come. Your 2 AM air-gapped demo machine will thank you.

Related Reading

• Agentic Browsers: Browser-Use & Skyvern Reviewed
• AnythingLLM as Knowledge Base
• API vs Self-Hosted LLMs: The Real Cost
• Continue.dev vs Cody vs Tabby: AI Code Help Without the Cloud
• Dify: Visual Agent Workflows