Tag: llm

Dify vs Open WebUI: Chat or App Builder?

Dify and Open WebUI both front your local LLMs, but they solve different problems. Compare footprint, RAG, auth, APIs and licenses, and see how to run both.

Agent Memory Without a Vector DB

Give a local LLM agent long-term memory with Markdown files or SQLite FTS5. Working code, token math, and the point where you finally need embeddings.

Local LLM JSON Output That Doesn't Lie

Constrained decoding makes local LLMs emit valid JSON in llama.cpp, Ollama, and vLLM. It guarantees syntax, not truth. Working curl, Pydantic, and retry code.

Crawl4AI: Feed Your RAG From the Web

Crawl a docs site to clean markdown with Crawl4AI, embed it with Ollama, store it in Qdrant, and query it. A working local RAG ingest pipeline, with politeness.

Train a Tiny GPT, Part 5: Fine-Tuning

A 5.8M-parameter GPT scores 1.626 bits per byte on Holmes. Gemma 4 E4B scores 0.953 untrained, and a 34-minute QLoRA run on an 8 GB GPU gets it to 0.865.

Train a Tiny GPT, Part 4: Inference

The KV cache sped a 5.8M-parameter GPT up 4.4x on a laptop CPU and did almost nothing for the RTX 3070 at batch 1. Then sampling settings decide how it reads.

Train a Tiny GPT, Part 3: Training

Train a 5.8M-parameter GPT on Sherlock Holmes: 59 seconds on an RTX 3070 vs 48 minutes on its CPU, and the overfitting that starts near step 1,250 of training.

Train a Tiny GPT, Part 2: Attention

Build causal self-attention and RoPE in plain PyTorch, then measure naive vs fused attention on an 8GB GPU: 6400 MiB vs 8.3 MiB of VRAM at 16k tokens.

Train a Tiny GPT, Part 1: Tokenizers

Build a byte-level BPE tokenizer in about 100 lines of Python, then compare it to tiktoken's gpt2, cl100k_base and o200k_base on prose, code, YAML and German.

Clef vs Jev: Cloudflare's 4x Tie

Cloudflare's Clef decision model tied Jev on accuracy in a private benchmark, abstained on junk better, and cost 4.2x more per decision. Real numbers inside.

Prompt Injection vs Your Coding Agent

Prompt-level defenses against injection are probabilistic and lose to a patient attacker. Here's the capability-level design that holds, with working configs.

Batch Inference: One Model, 100 Users

Loading model weights, not math, is your GPU bottleneck. Continuous batching with llama-server serves many concurrent requests for close to the cost of one.