Multi-GPU LLM Inference on Consumer Cards
Two used 3090s or a mismatched pair can run a 27B model. Compare llama.cpp layer split with vLLM tensor parallelism, PCIe limits, and a verdict on which to run.
Tag: gpu
All the articles with the tag "gpu".
Frigate Beyond Coral: GPU and OpenVINO
·
Updated:
The Coral USB TPU runs out of headroom past a few cameras. GPU and OpenVINO detector numbers for Frigate, config gotchas, and what to buy for six cameras.
Immich Hardware Acceleration: Stop Cooking Your CPU
·
Updated:
Wire up QSV, VAAPI, or NVENC in Immich so face recognition and thumbnails run on the GPU, not your CPU. Step-by-step for Intel, AMD, and Nvidia.
GPU Memory Math: Will This Model Actually Fit?
·
Updated:
Before you download a 70B model, calculate if it fits. The formulas, the gotchas, and a quick calculator you can actually use.
LLM Fine-Tuning for Mortals: LoRA, QLoRA, and Your Gaming GPU
·
Updated:
Fine-tune an LLM on your gaming GPU with LoRA and QLoRA: dataset prep, Hugging Face, Unsloth, VRAM math, a full walkthrough, and mistakes that waste your afternoon.