Tag: ai

Constrained decoding makes local LLMs emit valid JSON in llama.cpp, Ollama, and vLLM. It guarantees syntax, not truth. Working curl, Pydantic, and retry code.

A 5.8M-parameter GPT scores 1.626 bits per byte on Holmes. Gemma 4 E4B scores 0.953 untrained, and a 34-minute QLoRA run on an 8 GB GPU gets it to 0.865.

The KV cache sped a 5.8M-parameter GPT up 4.4x on a laptop CPU and did almost nothing for the RTX 3070 at batch 1. Then sampling settings decide how it reads.

Train a 5.8M-parameter GPT on Sherlock Holmes: 59 seconds on an RTX 3070 vs 48 minutes on its CPU, and the overfitting that starts near step 1,250 of training.

Letting a coding agent run npm install and arbitrary shell? Compare gVisor, Firecracker, and Kata on startup, KVM needs, Docker fit, and network egress.

Build causal self-attention and RoPE in plain PyTorch, then measure naive vs fused attention on an 8GB GPU: 6400 MiB vs 8.3 MiB of VRAM at 16k tokens.

Build a byte-level BPE tokenizer in about 100 lines of Python, then compare it to tiktoken's gpt2, cl100k_base and o200k_base on prose, code, YAML and German.

Cloudflare's Clef decision model tied Jev on accuracy in a private benchmark, abstained on junk better, and cost 4.2x more per decision. Real numbers inside.

Prompt-level defenses against injection are probabilistic and lose to a patient attacker. Here's the capability-level design that holds, with working configs.

Loading model weights, not math, is your GPU bottleneck. Continuous batching with llama-server serves many concurrent requests for close to the cost of one.

A 2026-09-22 patch left a self-hosted model's weights untouched but made it more confident on nonsense input. Six systems compared, real numbers only.

Docker MCP Toolkit sandboxes AI agent tools in containers. Here's exactly what that isolation stops, where it silently breaks, and how to configure it safely.