Private AI infrastructure: air-gapped LLM deployment guide

On-premises deployment means running infrastructure in your own data centre rather than a cloud provider's — but the infrastructure may still have internet connectivity for management, updates, and telemetry. Private cloud deployment (e.g., a VPC in AWS GovCloud or Azure Government) means logically isolated cloud infrastructure where your data doesn't share compute with other tenants, but the physical infrastructure is still in a cloud provider's facility and connected to internet-routed networks. Air-gapped deployment means physical network isolation — no network interface connected to any externally routable network. The air gap provides the strongest security guarantee for certain threat models (insider threat with network access, nation-state network-layer attacks) but at significant operational cost. Most organisations with data sensitivity requirements are adequately served by on-premises or private cloud deployment; air-gapped deployment is reserved for the highest classification levels and most sensitive operational contexts.

For most enterprise use cases in 2026, the gap between top open-weight models (Llama 3.3 70B, Mistral Large 2) and frontier closed models has narrowed substantially. On standard benchmarks and practical enterprise tasks — document summarisation, information extraction, structured output generation, code assistance — 70B+ parameter models deliver performance that satisfies most enterprise requirements. The gap is most pronounced for complex multi-step reasoning, nuanced instruction following on ambiguous tasks, and cutting-edge coding capability — tasks where GPT-4 class models maintain a meaningful edge. For air-gapped deployments, the question is not 'is it as good as GPT-4?' but 'is it good enough for our use cases?' — and for a large majority of enterprise NLP tasks, the answer is yes. Running a parallel evaluation of your specific use cases with local models before committing to air-gapped deployment is strongly recommended.

NVIDIA H100 and H200 GPU lead times have improved from the extreme shortage of 2023–2024 but remain 8–16 weeks for standard commercial procurement through Dell, HP, and Supermicro server configurations. Classified procurement through government channels may have different lead times and may require additional export control review for H100/H200 in certain geographies. AMD MI300X availability has improved and represents a 4–8 week lead time through AMD's enterprise channel. For organisations with urgent requirements, the secondary market for A100 servers provides shorter lead times (2–6 weeks) with performance adequate for most 70B model serving use cases. Factor GPU procurement lead times into your deployment timeline from the start — hardware delays are the most common cause of air-gapped LLM deployment timeline slippage.

Fine-tuning in an air-gapped environment requires all training infrastructure to be available locally: the base model weights, training data (classified within the enclave), and the fine-tuning framework. The standard approach uses parameter-efficient fine-tuning methods — LoRA (Low-Rank Adaptation) or QLoRA — which require significantly less GPU memory than full fine-tuning and can be completed on 1–2 A100 GPUs for 7B–13B parameter models. The HuggingFace Transformers library and PEFT library are fully self-hostable and require no external connectivity after the initial transfer of code and model weights into the enclave. Training data curation, model evaluation, and deployment promotion all occur within the air-gapped environment. The operational challenge is that fine-tuned models need the same secure transfer and version management processes as base model updates — establish these processes before beginning fine-tuning work.

Air-gapping eliminates network-based exfiltration risks but does not eliminate all AI-specific security risks. Prompt injection attacks — where malicious instructions embedded in user-supplied documents manipulate the LLM into taking unintended actions — remain a risk in air-gapped RAG and agent deployments. Model weight provenance is a concern: open-weight models downloaded from the internet before transfer into the enclave must be verified against known-good hashes; compromised model weights (supply chain attack) could embed backdoors that are activated by specific inputs. Insider threat remains: authorised users can attempt to extract sensitive information through carefully crafted prompts, and LLM responses to sensitive queries may need audit review. Output redaction — automatically screening LLM outputs for sensitive information that shouldn't leave the enclave through the user interface — is an additional control worth considering for the highest-sensitivity environments.

The minimum viable configuration for a useful production capability (serving 7–13B parameter models to 10–20 concurrent users) is 1–2 servers with NVIDIA A100 40GB or 80GB GPUs, 256–512GB system RAM, NVMe SSD storage (4TB+ for model weights and operating system), and 100GbE networking within the enclave. This supports Llama 3.1 8B in FP16 or a 13B model in INT4 quantisation with reasonable throughput. For 70B model serving, 2–4 A100 80GB GPUs are required. The minimum viable configuration for development and testing is lower: a single NVIDIA A100 40GB (or consumer RTX 4090 for non-classified development) running quantised models is sufficient to validate use cases and develop applications before production GPU hardware is procured. Consumer GPUs (RTX 4090, RTX 3090) are not appropriate for classified or production use but work well for capability prototyping within an isolated development environment.

RAG in air-gapped environments requires all components to be self-hosted: the vector database (Chroma, Qdrant, Weaviate, Milvus — all available as self-hosted deployments), the embedding model (open-weight embedding models like nomic-embed-text, BGE, or E5 run locally on CPU or GPU), the document ingestion pipeline, and the LLM itself. The architecture is identical to cloud RAG except every component runs within the enclave. Performance considerations: embedding generation is compute-intensive and may require dedicated GPU capacity separate from the inference server; vector database performance scales with index size and query volume; and document preprocessing (PDF parsing, OCR for scanned documents) requires additional software components that must also be air-gapped. The end-to-end air-gapped RAG stack is mature and well-tested in 2026 — all required components are available as open-source software with no mandatory external connectivity.

Ongoing costs for air-gapped LLM infrastructure are dominated by three categories: power and cooling (GPU servers consume 3–15kW per server at load — a 4-GPU H100 server draws ~10kW; a 10-server cluster costs £50,000–150,000 annually in power and cooling at typical data centre rates), staff costs (1–2 FTE specialised in AI infrastructure, model management, and security operations is typically required for a production air-gapped LLM environment), and hardware refresh (GPU hardware has a 3–5 year useful life for AI workloads before performance falls meaningfully behind current-generation models; a full hardware refresh cycle for a 4-server cluster represents £500K–2M capital cost depending on GPU tier). Software costs are minimal — the open-source serving stack has no licence fees — but security accreditation and audit costs for classified environments add £20–100K annually depending on classification level and audit frequency. Total cost of ownership for a modest air-gapped LLM environment runs £200–600K annually in the UK or equivalent markets.