API vs Self-Hosted LLMs: The Real Cost

Self-Hosting an LLM Won’t Save You Money. Probably.

APIs win for almost everyone. Self-hosting only wins at high volume (100M+ tokens a month) if your ops time is free, meaning nobody’s billing hours to babysit the GPU. Pay for that labor and the API wins even at that scale, thanks to September 2026 hardware prices. Below that volume, or the moment privacy or hard latency requirements matter more than the bill, cost stops being the deciding factor anyway.

Look, I get it. You’ve got a beefy GPU gathering dust. You’ve read the pricing pages. You’ve done the mental math. Surely running Llama locally is cheaper than paying Anthropic $10 per million output tokens? But you’re only counting half the bill.

Everyone who self-hosts ends up on this journey. You start with righteous cost calculations, wire up Ollama on a nice GPU, and for about six weeks you feel smug. Then reality sets in, the electricity bill, the hardware you had to buy, the time spent debugging CUDA on a Tuesday night, the fact that you’ve now become the on-call support for a latency SLA that nobody cares about except you.

By the end of 2026, the API vs self-hosted decision is less about “which is cheaper” and more about “what am I actually buying?” Cost is ONE axis. Privacy, latency, model quality, and the operational tax of keeping a GPU humming are equally real.

Let’s break down the actual numbers.

So Which One Should You Actually Run?

• Pick API if your usage is under 100M tokens a month, bursty, or you need top-tier model quality without ops overhead.
• Pick self-hosted if you’re already running 100M+ tokens a month and the ops time is free (already staffed, or you’re doing it yourself as a hobby), or you need offline or private inference, or you have hard latency requirements.
• Pick hybrid if you want a local model handling routine work and API access for the reasoning-heavy slice.

The API Cost: Token Math

APIs have gotten aggressively cheap. Pricing as of September 2026:

Anthropic Claude:

• Claude Haiku 4.5: $1 / 1M input, $5 / 1M output
• Claude Sonnet 5: $2 / 1M input, $10 / 1M output
• Claude Opus 5: $5 / 1M input, $25 / 1M output

OpenAI:

• GPT-4o mini: $0.15 / 1M input, $0.60 / 1M output
• GPT-4o: $2.50 / 1M input, $10 / 1M output

Groq (extreme latency play):

• Llama 3.3 70B: $0.59 / 1M input, $0.79 / 1M output (yes, really)

For a realistic workload (say, 10 million tokens per month of mixed input/output at Sonnet quality), you’re looking at:

Anthropic Sonnet 5: (6M input @ $2) + (4M output @ $10) = $52/month
OpenAI GPT-4o: (6M input @ $2.50) + (4M output @ $10) = $55/month
Groq Llama 3.3 70B: (6M input @ $0.59) + (4M output @ $0.79) = $6.70/month (!)

For most people doing creative work, coding assistance, or research, 10M tokens/month is generous. If you’re just using it occasionally, you’re closer to 1M, 2M.

The math is very friendly to APIs right now.

The Self-Hosted Cost: The Full Bill

Now let’s talk about what it actually costs to run Ollama in your basement.

Hardware (one-time, amortized). A 2026 AI-driven VRAM shortage roughly doubled used GPU prices industry-wide. Figures below are current listings as of September 2026:

• Used RTX 4090: $2,900 to 3,000
• Used RTX 3090: $1,350 to 1,520
• Used A6000 (Ampere): $4,100 to 4,800
• M1/M2 Mac Mini: Already own it? $0. Buying one? $600+.

Let’s assume a used RTX 4090 at $3,000 (the going rate in September 2026). Over 3 years, that’s $1,000/year or $83/month.

Buy new instead and the gap gets worse, not better: NVIDIA stopped making the RTX 4090 in late 2024, and remaining retail stock now runs $3,700 to 4,500 on Amazon, well above even the inflated used price. There’s no “just buy it new to skip the used-market hassle” option here. Used is the only sane path to a 4090 in September 2026.

Electricity (continuous):

• RTX 4090 pulls ~450W under load, 100W idle
• A6000 pulls ~300W under load, 80W idle
• Assuming average 40% utilization (you’re not running inference 24/7):

RTX 4090 @ 40% avg: (450W × 0.4 × 730 hours/month) = 131.4 kWh/month
US average electricity (September 2026): $0.18/kWh
Cost: 131.4 kWh × $0.18 = $23.66/month

RTX 4090 in San Francisco (PG&E median rate, ~$0.42/kWh): 131.4 × $0.42 = $55.19/month

A6000 @ 40% avg: (300W × 0.4 × 730) = 87.6 kWh/month = $15.77/month (US)

The model quality gap:

• Qwen3-Coder / Llama 3.1 405B (self-hosted): Excellent for coding, math, knowledge work. Missing some reasoning finesse vs Claude Opus.
• Gemma 4 / Qwen 3.6 (self-hosted): Solid all-around. Noticeably worse than Sonnet for nuanced tasks.
• Mistral Large (self-hosted): Good. Still trailing Claude/GPT on reasoning tasks.

If you’re self-hosting, you’re probably running something in the 27B to 405B range. That’s good, but it’s not “Opus-grade.” You’re trading intelligence for cost and control.

The operational tax (the part people skip):

• NVIDIA driver updates break things. Welcome to May 2026, where the latest driver changed CUDA initialization.
• You’ll model-swap. “Oh, let me try this new thing.” That’s idle waiting around.
• You become the system admin. When the GPU gets stuck, you fix it. At 2 AM. Because that’s when it always breaks.
• Security updates for the inference server (vLLM, Ollama, whatever).
• Network security: you either lock it behind a reverse proxy or accept that your local LLM is accessible to whoever’s on your network.

None of that shows up on an invoice. It shows up as evenings. Budget a few hours a month and be honest with yourself about what your evenings are worth.

Break-Even Math: When Self-Hosting Wins

Let’s build a real scenario.

Scenario: You’re a solopreneur developer using LLMs for coding assistance.

API spend (reasonable estimate):

• 5M tokens/month at Sonnet quality = ~$26/month
• Annual: $312

Self-hosted (RTX 4090, ~$3,000 used as of September 2026):

• Hardware amortized: $83/month = $1,000/year
• Electricity (US average, $0.18/kWh): $23.66/month = $284/year
• Total: $1,284/year

Self-hosted is LOSING by about $972/year before you even count your time. Add 10 hours of operational overhead annually (driver updates, debugging, etc.) at $50/hour consultant rates, and self-hosting costs about $1,784/year vs $312/year for the API.

But wait, it gets more interesting than that.

Scenario 2: You’re a research org running high-volume inference (100M tokens/month).

API (Anthropic Sonnet 5):

• 60M input @ $2 + 40M output @ $10 = $520/month = $6,240/year

Self-hosted (A6000 cluster, 2 GPUs at ~$4,500 each, September 2026 pricing):

• Hardware: 2 × $4,500 = $9,000 total → $3,000/year amortized
• Electricity: 2 × $15.77/month = $31.53/month = $378/year
• Operational overhead: ~100 hours/year @ $75/hr = $7,500/year (dedicated infra engineer time)
• Total: ~$10,878/year

Self-hosted loses by about $4,638/year once you pay a dedicated engineer for that 100 hours. Drop the labor line, meaning the ops work is already someone’s job or you’re eating it yourself as a hobby, and self-hosted saves about $2,862/year on hardware and electricity alone. Either way, you own the inference pipeline. You can optimize. You control the latency. You don’t depend on anyone’s API availability.

The break-even happens when:

1. You’re running high volume (100M+ tokens/month) and the ops time doesn’t cost you anything extra, OR
2. You value privacy over everything else, OR
3. You already have the hardware and electricity cost is your only variable, OR
4. Latency is a hard requirement (local inference is 10-100ms; API round-trip is 500ms-2s)

The Privacy Axis

APIs send your prompts to someone else’s servers. Even if you trust OpenAI or Anthropic (and they have strong data policies), the fact remains: your data leaves your house.

For most people, this is fine. For some, healthcare, legal, proprietary code, competitive research, this is a dealbreaker. Self-hosting gives you the property of “it never leaves my network.”

This isn’t a cost in dollars. But it’s real cost in risk. Sometimes that risk is worth $100 to 200/month to eliminate.

The Latency Axis

Calling an API: 500ms-2s round-trip if you’re in the US and their servers are responsive. Could be worse depending on congestion.

Local inference on a 405B model: 5-50 tokens per second. A ~300-token response takes 6-60 seconds, but it’s deterministic. You control it. No surprise spikes.

This matters for interactive work (chatbots, real-time co-pilots). It’s irrelevant for batch jobs. For most dev tasks, “6 seconds locally” feels slower than “1 second API round-trip,” even if the API-to-token-generation is slower.

A Real Cost Comparison Table

The numbers:

The Hybrid Sweet Spot

Here’s what actually makes sense for most people in 2026:

Run a local 8B, 70B model (Gemma 4, Qwen 3.6, Mistral Large) for:

• Coding suggestions (fast, good enough, offline)
• Writing helpers (grammar, rephrasing)
• Summarization (you don’t need genius here)
• Brainstorming (speed matters more than perfection)

Use API for the hard stuff:

• Reasoning-heavy tasks (Opus tier)
• Novel problem-solving (Claude/GPT-4o tier)
• Anything where quality > speed

Cost breakdown for this hybrid:

• Self-hosted (single RTX 3090, ~$1,500 used as of September 2026, 350W TDP at 40% avg utilization): ~$42/mo hardware amortized + $18/mo electricity ($0.18/kWh US average) = $60/mo
• API budget for the 20% of work that needs it: $50 to 100/mo
• Total: $110 to 160/mo, with better results than either alone

You’re not saving money vs pure API (which would run about $52/mo for this volume). You’re buying:

1. Offline inference
2. Sub-100ms latency for routine work
3. The satisfaction of control (worth something to some people)
4. Privacy for draft work

When Each Wins

Use APIs (Claude, GPT-4o, Groq):

• You’re not sure if you’ll use it long-term (no hardware investment yet)
• Your workload is bursty (inconsistent token usage month-to-month)
• You need top-tier model quality (Opus reasoning, GPT-4o vision)
• Your time is expensive and infrastructure overhead is a drag
• You want zero operational burden
• You’re integrating into production and need SLA guarantees

Self-host (Ollama + Llama/Mistral):

• You’ve already got the GPU hardware (used market is your friend)
• Your usage is consistent and high-volume (100M+ tokens/month) and the ops work is already covered, not billed separately
• Privacy is non-negotiable
• Latency requirements are hard constraints
• You enjoy the ops work (or have a team to handle it)
• You want to run edge inference (on-device, no cloud at all)

Hybrid (local 70B + API fallback):

• You want best of both without betting everything
• You’re a developer (coding assistance from local Llama, reasoning from API)
• You’re cost-conscious but not obsessive
• Latency matters for some tasks, not others

The Real Talk

If you’re reading this thinking “I’m gonna self-host and save money,” go back and re-read the operational overhead section. That’s the part nobody talks about until it’s 2 AM and your GPU driver is corrupted.

Self-hosting makes sense if you:

1. Already own the hardware, or
2. Are running at serious scale, or
3. Value privacy/latency/control over money

For everyone else? APIs in 2026 are cheap enough that the math loses to the headache ratio.

But hey, if you love tinkering, own a nice GPU, and enjoy the autonomy of a local model, do it. Some things aren’t about cost. They’re about ownership.

Your 2 AM self will either thank you for running local inference (no dependency on anyone else), or curse you for the CUDA driver debugging.

Flip a coin. Pick the one that makes you happy.

Related Reading

• Ollama: Powerful Language Models on Your Own Machine
• LLM Backends: vLLM vs llama.cpp vs Ollama
• Running Gemma 4 Locally with Ollama
• Gemma 4 vs Qwen3.6
• Local Vision LLMs Worth Running in 2026