Local multi-agents with Ollama, LiteLLM and vLLM
Running agent orchestration without paid APIs: latency and VRAM trade-offs, and when it pays to go beyond local.
Running LLMs locally is no longer just a hobbyist curiosity. With models like Llama 3, Mistral and Qwen in sizes from 7B to 70B, it is possible to build agent systems without sending sensitive data to external APIs and without end-of-month billing surprises.
The stack I use is simple: Ollama for local prototyping and inference, LiteLLM as a model abstraction layer, and vLLM when I need high throughput on GPU. LangGraph or CrewAI handle orchestration. The same code that runs locally can run in production just by swapping the provider in LiteLLM.
The main trade-off is latency versus cost. A local 8B model on CPU can take 5-10 seconds for a complete response. On GPU, it drops to 1-2 seconds. For asynchronous tasks — document processing, code analysis, report generation — that latency is acceptable. For synchronous user chat, it may not be.
VRAM is the second bottleneck. A 13B model in FP16 needs about 26 GB of VRAM. With 4-bit quantization, it drops to ~8 GB, with small quality loss on technical tasks. Plan memory allocation the same way you plan CPU: it is finite and easy to exhaust.
When is it worth leaving local? When you need larger models than you can run economically, when latency is critical, or when the cost of owned hardware exceeds API costs. My practical rule: local for sensitive data and predictable workloads; API for spikes, frontier models and fast experimentation.
Working on something similar? Tell me about your project.