I Built My First LLM and Agents Without Third-Party APIs: Here's What Happened

Recently I made a decision that felt crazy: stop relying on third-party cloud APIs and see what happens when I build my own language models and agents completely from scratch on my desk workstation. No fine-tuning someone else's weights. No API keys. No monthly bill. Just PyTorch, a consumer GPU, and an unreasonable amount of stubbornness.

Here's what came off the bench: the Vivid86 Model Family — a series of custom SLMs trained on my own hardware, with a 91M parameter model running at 671 queries per second and a 220M parameter coding model hitting 369 QPS, both compiled to TensorRT. The entire runtime costs nothing per month beyond electricity.

Was it easy? Absolutely not. Did it work? Yes — and the lessons go way beyond inference speed.

The macro picture is this: API dependency is a business liability. Pricing changes, model deprecations, and rate limits are outside your control. The developers who survive the next phase of AI commoditization will be the ones who own their inference stack. Self-hosted models are not a hobbyist curiosity — they are the foundation of a defensible AI business.

The technical reality: a modern LLaMA-style decoder (RoPE, SwiGLU, RMSNorm, GQA) trained with CPU-offloaded AdamW on a RTX 5070 12GB can achieve production-grade throughput when compiled to TensorRT FP16. The key insight was keeping optimizer states in system RAM (32GB DDR5) while GPU handles only forward/backward passes — VRAM drops from 10.9GB to under 2GB. Sub-3ms latency is achievable without cloud infrastructure.

The idea that small models are inherently inferior is a recent and contestable assumption. Research from 2023–2025 consistently demonstrates that domain-specialized models with 100M–500M parameters match or exceed general-purpose models 10x their size on targeted tasks (see Phi, MiniCPM, SmolLM literature). The history of compute efficiency suggests specialized small models are the correct architectural bet for edge and cost-constrained deployments.

What everyone ignores: a 220M parameter model with a 1,024-token context window cannot replace frontier reasoning for genuinely complex tasks. The honest constraint is context depth, not raw parameter count. If your use case requires holding 50 pages of documentation in memory while writing code, self-hosted small models are not the answer yet. The honest move is knowing exactly where the ceiling is — and building workflows that respect it.