vLLM, Ollama, or llama.cpp? Picking Your On-Premise Inference Server by Concurrency, Not Hype
Most UAE SMEs pick an on-premise LLM server off a demo with one user, then hit a wall when three staff members query it at once. At a single request the gap between Ollama and vLLM is invisible. At ten requests it is catastrophic. Here is the one number that should drive the whole decision: count your concurrent users at peak. Below five, Ollama is fine. Above five, you need vLLM, and no amount of tuning closes that gap. This guide maps each server to the real concurrency load of UAE clinics, law firms, and brokerages using measured throughput, not vendor claims.
The Concurrency Gap That Catches Everyone Off Guard
Run a one-user demo and Ollama actually wins. It does roughly 45 tokens per second against vLLM's 38 on a Llama 3.1 8B model. That single-request number is what teams see, and it's why they keep reaching for Ollama in production.
The trouble starts the moment a second and third user show up. Ollama processes requests sequentially by default, so each one waits behind the last, and the effect compounds fast as load climbs. At 8 concurrent requests, vLLM produces 187 tokens per second to Ollama's 82, a 2.3x gap. At 10 concurrent users it's 485 versus 148. Push to 50 concurrent users on a single GPU and vLLM reaches about 920 tokens per second while Ollama flattens out near 155. Independent 2026 benchmarks put the gap as wide as 16x to 29x at very high concurrency, so if anything the numbers here are conservative.
The crossover sits at roughly 5 simultaneous users. A clinic running one front-desk system alongside one clinical notes assistant stays under that line, and Ollama will hold up. A 20-person law firm where paralegals and partners hammer a document assistant all day does not, and there vLLM becomes a functional requirement rather than an upgrade you can defer. None of this is projection. It's how each server's architecture handles concurrent attention computation, measured on real hardware.
Why the Numbers Diverge: PagedAttention Versus a FIFO Queue
The architecture explains the whole gap. vLLM implements PagedAttention, which splits the KV cache, the memory holding attention state for each token, into fixed-size non-contiguous blocks, an idea borrowed straight from operating-system virtual memory paging. That one design choice kills KV cache fragmentation. It recovers up to 55% of the VRAM that naive implementations waste on padding and over-allocation, and it makes continuous batching trivial.
Continuous batching is on by default, so vLLM works tokens from several requests in the same forward pass instead of draining one request before it touches the next. Adding concurrent users then costs you 10 to 20 percent of throughput up to moderate load, rather than dropping it off a cliff.
Ollama works the other way. It wraps llama.cpp internally and serialises requests by default, a FIFO queue, with OLLAMA_NUM_PARALLEL auto-selecting 1 or 4 based on available memory and OLLAMA_MAX_QUEUE holding the rest at a default of 512. Set OLLAMA_NUM_PARALLEL=4 explicitly and throughput rises three to four times, but per-request latency climbs 20 to 40 percent and you still don't approach vLLM's concurrent efficiency. The worse problem is model eviction. Under mixed-model traffic Ollama unloads and reloads models, and latency spikes by several seconds. Picture a voice-AI system in a Dubai clinic freezing for three seconds mid-sentence while a doctor is with a patient. That alone takes Ollama off the table for any multi-model production deployment.
The Server the Original Verdict Left Out: SGLang for RAG, Voice, and Agents
The "vLLM above five users, full stop" rule was written for generic traffic. The systems this firm actually builds are not generic. A RAG pipeline, a voice agent, and a WhatsApp booking bot all send the same large block of text at the front of every single request: the system prompt, the tool and function definitions, and frequently the same retrieved context window. That repeated front block is the prefix, and on prefix-heavy traffic one server beats vLLM. That server is SGLang.
SGLang's edge is RadixAttention. It stores the KV cache in a radix tree and reuses any shared prefix across concurrent requests, computing that block once instead of recomputing it for every caller. On prefix-heavy workloads the payoff is steep: up to 5x in the original LMSYS measurement and 6.4x in the later RadixAttention prefix-caching result, with roughly 29% higher aggregate throughput than vLLM on a comparable H100 running Llama 3.1 8B, about 16,200 tokens per second versus 12,500. Mean time-to-first-token lands at 79ms against 103ms, a 23% reduction, with inter-token latency of 6.0ms versus 7.1ms. Read the caveat plainly. That gap appears on prefix-heavy traffic, not on every workload, and on flat single-shot requests the two are close.
One correction is worth stating, because it gets gotten wrong constantly. SGLang's advantage is not that vLLM lacks prefix reuse. Since the V1 engine, vLLM enables automatic prefix caching by default, and V1's implementation causes under 1% throughput loss even at a 0% cache-hit rate, which is exactly why it ships on. vLLM gets cross-request prefix reuse out of the box too. SGLang's real differentiators are the radix-tree eviction strategy, its structured-generation and agent features, and the measured throughput delta on heavy shared prefixes.
So the practical rule is a refinement, not a reversal. vLLM stays the broad default: widest model and hardware coverage, the strongest docs, no compile step. Reach for SGLang when your traffic is multi-turn chat, RAG, or agentic with a heavy shared system prompt, which is the exact shape of a UAE clinic's WhatsApp triage bot or a law firm's contract-analysis agent. Two servers you can skip. TGI from HuggingFace sits in maintenance mode in 2026, and HuggingFace itself points to vLLM and SGLang, so it is not a new-build choice. TensorRT-LLM is the squeeze-every-last-token option that costs one to two weeks of setup and locks you to NVIDIA, worth it only when you are wringing a fixed GPU budget for everything it has.
The Metric That Decides a Voice Deployment Is Not Tokens Per Second
Aggregate tokens per second is the wrong yardstick for a voice agent. What decides whether a phone line feels human is time-to-first-token: how long the caller sits in silence before the first word comes back. Twilio's late-2025 benchmarks put the LLM time-to-first-token target at 375ms with a 750ms upper limit, inside a full conversational turn-gap budget of about 885ms (1,100ms upper limit). Miss that and the caller hears dead air, then starts talking over the agent.
Sustained throughput is rarely the constraint once the first token lands. A human reads at roughly 4 to 5 tokens per second, and a model streaming at 50 to 100 tokens per second is already faster than anyone can comfortably follow; about 10 tokens per second is the floor to keep up. Speech synthesis begins the moment the first tokens arrive. The binding number is first-token latency, not the total rate.
This is exactly where Ollama's FIFO queue is disqualifying for real-time conversation. The second concurrent caller's time-to-first-token becomes the time it takes to finish caller one — latency that is invisible in a single-user demo and fatal on a live line. vLLM holds time-to-first-token roughly flat as callers are added, because continuous batching and chunked prefill (always on in the V1 engine) schedule a new request's first prefill pass without draining the queue. SGLang's cache-aware scheduling holds per-request rate even more stably at high concurrency. Put it in UAE terms. A clinic reception line or a WhatsApp booking bot handling three simultaneous conversations is already past the point where Ollama's per-request latency is acceptable for voice, even while its aggregate tokens per second still looks fine on a dashboard.
Same Model, Different Quantization: What Changes When You Move to vLLM
The article keeps citing Q4_K_M without saying what a vLLM box actually runs, and that matters. Moving from Ollama to vLLM changes the quantization, and a buyer needs to know whether quality survives the move. Q4_K_M is a GGUF quant, the native format for Ollama and llama.cpp, and it is excellent for single-user CPU or GPU-offload work.
vLLM production serving uses something else: AWQ at INT4, or FP8. AWQ-via-Marlin is the 2026 default for GPU serving, the recommended quality-and-throughput sweet spot, and FP8 is recommended for most scenarios because the quality cost is near nothing. On a current vLLM build, FP8 W8A8 delivers roughly 15% to 27% higher throughput than BF16 while MMLU stays flat and GSM8K moves within 0.15 to 0.83 points. On an MMLU-Pro comparison FP8 scored 69.64% against BF16's 70.24%, a 0.6-point gap. The move to vLLM is a throughput gain, not a quality cut. The quantization change is a one-time conversion to AWQ or FP8, not a downgrade.
Do not run GGUF on vLLM expecting its real numbers. vLLM can technically load GGUF, but its own docs label that path "highly experimental and under-optimized" and point you to llama.cpp instead; in practice GGUF on vLLM runs around 93 tokens per second with a time-to-first-token near 958ms, far off AWQ or FP8 territory.
There is a genuine exception that keeps llama.cpp in the picture. On a VRAM-tight box, say a 32B model that barely fits inside 24GB, GGUF Q4 on llama.cpp or Ollama can leave more headroom than FP8 on vLLM, because the 4-bit weights are simply smaller. When the card is the binding constraint, that headroom is a real reason to stay on GGUF rather than chase vLLM's throughput.
Hardware Requirements and Where llama.cpp Belongs
vLLM needs NVIDIA hardware: driver 525 or above, CUDA 12.1 minimum, and GPU compute capability 7.0 or higher. That covers the V100 forward, including the RTX 20, 30, and 40 series, A10G, A100, and H100. VRAM scales with model size. An 8GB card runs a 7B to 8B model at Q4_K_M quantisation; 16 to 24GB handles 14B to 32B models; 70B models want 48 to 96GB. Budget another 500MB to 2GB of VRAM for framework overhead before the model even loads, because it's easy to undersize a GPU by forgetting it.
Those ranges describe a single user. At concurrency the arithmetic flips. The KV cache, not the model weights, dominates VRAM, and this is the sizing mistake that catches the most teams. A Llama 3.1 70B model at 32k context needs roughly 10 to 11GB of FP16 KV cache per active sequence (about 5GB at FP8). Put 10 concurrent users on it and you are looking at on the order of 100 to 112GB of FP16 KV cache, around 56GB at FP8, sitting on top of the weights. No single card holds that. A real multi-user 70B deployment runs multi-GPU tensor parallelism via --tensor-parallel-size, and that flag must be a power of 2 that evenly divides the model's attention-head count, so 1, 2, 4, and 8 work while odd values like 3, 5, or 6 typically fail. The "48 to 96GB" line sizes a single user. A roomful of them on a 70B model is a two-, four-, or eight-GPU build.
If your UAE site has no NVIDIA GPU at all, llama.cpp is the answer. Think of a branch office on a workstation with an integrated GPU, or an edge box at a remote clinic. On an NVIDIA GPU it does roughly 122 to 128 tokens per second on Llama 3 8B Q4_K_M. On an Apple M4 Max through Metal it manages about 75. On pure CPU with AVX-512 or ARM NEON it falls to 10 to 15 tokens per second, fine for one person summarising a document, useless for anything real-time.
The strength of llama.cpp is reach, not concurrency. It runs on CUDA, ROCm, Intel SYCL, Apple Metal, and Vulkan. Use it for CPU-only edge deployments and developer machines. Do not put it in front of a roomful of staff sharing one system.
The Decision Matrix and UAE Compliance Reality
It comes down to four cases now, not three. Development, demos, and single-user testing go to Ollama: setup is under ten minutes, GGUF quantised models load with one command, and sequential throughput is plenty. Ollama clears a second, narrower bar honestly too. A small, bursty internal tool with a fixed model set and no eviction, where OLLAMA_NUM_PARALLEL plus a capped OLLAMA_MAX_QUEUE is genuinely good enough, makes paying for vLLM ops a false economy. Know its failure modes before you lean on it. A full queue returns a 503, the default loaded-model cap is 3 per GPU, and a cold model reload costs 10 to 60 seconds. Any production deployment serving three or more concurrent users goes to vLLM, and that's most of the clients we see: a five-doctor clinic running separate sessions for front desk and clinical documentation, a law firm where two paralegals and a partner query a contract analysis system at once, a brokerage with agents pulling property summaries during client meetings. Prefix-heavy production — RAG, voice, or agentic traffic with a heavy shared system prompt — goes to SGLang for the throughput and time-to-first-token edge. CPU-only or mixed-hardware edge sites go to llama.cpp, either as a single-user endpoint or embedded in a pipeline with pre-processing to shed load.
That ops gap is half the decision and it usually goes unstated. Ollama is a single signed binary with no built-in metrics and no horizontal scaling. vLLM in production means Docker or Kubernetes, CUDA and Python dependency patching, HuggingFace auth, and GPU-memory tuning, and in return it ships a Prometheus /metrics endpoint and scales horizontally. You are buying observability and concurrency with operational burden. Price both sides before you commit.
UAE regulation sharpens all of this. The federal Personal Data Protection Law, Federal Decree-Law No. 45 of 2021, covers all processing of personal data belonging to UAE residents, so patient records, case files, and client transaction data running through an LLM inference server are in scope. Be precise about its current teeth. As of 2026 the law's federal Executive Regulations have not yet been issued and the UAE Data Office is not yet fully operational, so enforcement is still limited and largely self-directed. The obligations are binding; the detailed enforcement machinery is mid-transition. That is the honest state, and it is exactly why on-premise is the conservative posture: it holds regardless of how the regulations land.
For DIFC-registered entities the picture is more concrete. DIFC Regulation 10 on processing through autonomous and semi-autonomous systems was enacted in late 2023, is enforceable now, with full enforcement commencing in 2026. It requires transparency notices at first use of an AI system, a register of system use-cases and processing activities, risk-and-impact assessments and human-intervention triggers for high-risk processing, system certification, and, for high-risk processing by deployers and operators, appointment of an Autonomous Systems Officer, an officer with status and competencies similar to a Data Protection Officer. The cost of getting data protection wrong in DIFC rose in the 8 July 2025 amendments to the DIFC Data Protection Law. A DPIA failure now carries an administrative fine up to USD 50,000, an unlawful public-authority disclosure up to USD 50,000, and a failed annual DPO assessment up to USD 25,000, inside a Schedule 2 range running roughly USD 10,000 to 100,000. Run a clinic, law firm, or brokerage inference server against personal data and a DPIA is part of the deployment, not an afterthought.
Running the inference server on-premise in the UAE removes cross-border transfer questions entirely and gives you a clean posture for any audit. For a clinic it intersects health-data residency directly: patient records flowing through an LLM should not leave the jurisdiction, and the three national health exchanges (NABIDH in Dubai, Malaffi in Abu Dhabi, and Riayati federally, now deeply integrated nationwide) are a reason the inference server lives on local hardware rather than a managed cloud endpoint with no UAE region. A brokerage or fund admin operating in ADGM faces a parallel obligation under its own data-protection regime rather than an identical one, which only reinforces the point: where the inference server physically runs is a compliance decision, not just a performance one. So the server choice carries past performance. It decides whether the deployment clears a DPIA review at all.
Questions about your setup?
We help UAE SMEs build AI systems that are compliant, on-premise, and actually useful. Free initial conversation.