The decisive question for a local document assistant is not: Which language model is currently the largest? It is: Can the system trace an answer to the approved, current document revision—and remain silent when the evidence is insufficient?
For manufacturers, that is not an academic distinction. Service manuals, bills of materials, maintenance instructions, drawings, and customer specifications contain intellectual property, conflicting revisions, and information with different access levels. A convincing chat demo is therefore not enough.
This article presents a production-oriented reference architecture using FastAPI, vLLM, PostgreSQL, and pgvector. It is not a fabricated customer case study and makes no universal accuracy or runtime promises. Hardware and budget ranges are transparent planning assumptions that must be validated against the real documents and load profile before implementation.
1. Define acceptance criteria before choosing a model
A defensible pilot begins with four decisions. Without them, neither GPU sizing nor a serious assessment of RAG's process value is possible.
Document boundary
Which formats, languages, drawings, tables, and OCR scans are in scope? Which sources are explicitly excluded?
Validity
Which revision is approved? How are obsolete, withdrawn, or not-yet-reviewed documents handled?
Access model
Which roles may search which product line, customer project, or service tier?
Answer contract
Which citation is mandatory, when must the system abstain, and which decisions remain exclusively human?
2. A reference architecture with an auditable evidence chain
The central design rule is simple: identity and document permissions are enforced before retrieval. The language model sees only the small set of approved passages selected for that request—never the complete document estate.
Reference flow: local sources are versioned, filtered by permission, and searched before any context is sent to the locally hosted inference service. Evaluation and release rules form a separate control system.
Ingestion is a controlled data process
Parsers, OCR, and table extraction create chunks with document ID, page, section, language, revision, approval state, and ACL. Every chunk stays linked to its original text.
Retrieval combines meaning with exact terminology
Vector search finds semantically similar passages; PostgreSQL full-text search protects recall for part numbers, error codes, material names, and standards. A reranker orders the combined candidate set.
Inference remains replaceable
vLLM exposes a local OpenAI-compatible HTTP interface. The application is therefore decoupled from a specific open-weight model, and changing the model does not require rebuilding the complete API.
The answer remains an evidence-backed draft
The system returns status, answer, citations, and confidence signals in a fixed schema. Conflicting revisions, missing evidence, or an out-of-scope request produce an “insufficient_evidence” status.
3. Why pgvector alone does not make a good RAG system
An HNSW index accelerates similarity search, but it does not solve permissions or revision conflicts. With strong filters, approximate search can return fewer matches because pgvector applies filtering after the index scan. Iterative scans, partial indexes, or partitioning must therefore be evaluated on the real data distribution.
Hybrid retrieval is usually more robust for technical documentation: semantic similarity supports natural-language questions, while full-text search protects recall for exact identifiers. The lists can be merged with Reciprocal Rank Fusion and then reranked.
CREATE TABLE chunks (\n id uuid PRIMARY KEY,\n document_id uuid NOT NULL,\n revision text NOT NULL,\n approval_state text NOT NULL,\n acl text[] NOT NULL,\n page_no integer,\n body text NOT NULL,\n search tsvector GENERATED ALWAYS AS\n (to_tsvector('german', body)) STORED,\n embedding vector(1024) NOT NULL\n);\n\nCREATE INDEX chunks_embedding_hnsw\n ON chunks USING hnsw (embedding vector_cosine_ops);\n\nCREATE INDEX chunks_search_gin\n ON chunks USING gin (search);The 1024 dimension is only an example and must exactly match the selected embedding model. In the real query, tenant, ACL, approval state, and revision are derived server-side from identity—not from client-editable fields.
4. The API contract enforces evidence and genuine abstention
The language model remains probabilistic. What can be deterministic are its boundaries: allowed input, retrieval filters, output schema, citation verification, and approval workflow. A machine-readable abstention is more honest and operationally useful than a well-written guess.
class Citation(BaseModel):\n document_id: UUID\n revision: str\n page: int | None\n excerpt: str\n\nclass RagAnswer(BaseModel):\n status: Literal[\"answered\", \"insufficient_evidence\"]\n answer: str | None\n citations: list[Citation]\n\n@router.post(\"/query\", response_model=RagAnswer)\nasync def query(request: Query, user: User = Depends(require_user)):\n evidence = await retrieve_with_acl(request.question, user)\n return await answer_only_from(evidence, request.question)In production, this simplified sketch also needs request-size and timeout limits, rate limiting, audit events, tracing, a reviewed prompt, and server-side verification that every returned citation belongs to the approved retrieval set.
5. On-premise is a location, not a security model
A server inside the company building does not automatically prevent leakage. BSI and ANSSI recommend Zero Trust principles for LLM-based systems, while OWASP treats the entire RAG chain—from ingestion and embeddings to output—as an attack surface.
Deny egress by default
Inference, embeddings, telemetry, and error reporting must not call external services without explicit approval. Exceptions are allow-listed and logged.
Apply ACLs before retrieval
Tenant, role, product line, and project access are derived from SSO or directory groups and enforced in the database query.
Treat sources as untrusted
Even internal PDFs can contain prompt-injection content. Document text is data context, not an executable instruction, and cannot override tools or policies.
Preserve version and provenance
Hash, source, import time, approval state, and revision relationships make it possible to trace exactly which document supported a statement.
Constrain output
Schema validation, citation checks, restricted Markdown, and the absence of tool permissions reduce the impact of manipulated or incorrect output.
Minimize logs
Audit logs need user, document IDs, policy decision, and timestamp—not necessarily the complete confidential prompt or document text.
Important: local processing can reduce data-protection and confidentiality risk, but it does not guarantee GDPR or NDA compliance. Legal basis, retention, roles, technical and organizational measures, operating processes, and any required DPIA remain organization-specific.
6. Size hardware for model, context, and concurrency
VRAM is not consumed by model weights alone. KV cache, context length, batch size, and concurrent requests determine real capacity. vLLM therefore reports GPU KV-cache capacity and estimated maximum concurrency; both must be tested against the intended workload.
Pilot · single GPU
Planning profile: 32–48 GB VRAM, 128 GB RAM, and 2–4 TB NVMe for quantized small-to-mid-size model classes, limited concurrency, and a bounded document set.
Suitable for evaluation and a small pilot team; no high availability and no promise of a specific response time.
Department · server class
Planning profile: 48–96 GB ECC GPU memory, at least 256 GB RAM, redundant NVMe storage, and separated inference, database, and ingestion services. NVIDIA's L40S (48 GB) and RTX PRO 6000 Blackwell Server Edition (96 GB) represent two current memory classes—not blanket purchasing recommendations.
For more users, longer contexts, or larger models; sustainable concurrency is measured with representative questions and document lengths.
Production · resilient
At least two inference nodes, a replicated database, backed-up object storage, monitoring, and a defined recovery path. GPU and storage redundancy follow the required RTO, RPO, and acceptable degraded mode.
For business-critical workflows, multiple sites, or committed service hours; requires operational ownership, not just a powerful workstation.
Document count alone is not a useful GPU sizing metric: embeddings normally live in the database, while model, context, and concurrency drive inference memory. Procurement should therefore follow a load test that records p50/p95 latency, token lengths, queue depth, and KV-cache utilization.
7. Realistic cost ranges for an initial decision
The GPU is rarely the only expensive component. Data cleanup, permissions, DMS integration, evaluation, security approval, and operations determine whether a chat demo becomes a usable system. I use the following corridors for an initial budget discussion:
These amounts are non-binding planning ranges for the described reference architecture, not market statistics or a quotation. Hardware, licenses, VAT, internal effort, and ongoing operations must be calculated separately. A small, well-defined pilot is economically safer than premature enterprise procurement.
8. What must be measured before go-live
A model benchmark does not show whether the system uses the company's manuals reliably. The decisive artifact is a versioned test set of real questions reviewed by subject-matter experts—including intentionally unanswerable and unauthorized requests.
Retrieval Recall@k: Does the correct passage appear in the candidate set at all?
Citation Precision: Does the cited page genuinely support the specific claim?
Subject-matter correctness: Does the responsible expert approve the answer and interpretation?
Abstention: Does the system refuse answers when evidence is missing or contradictory?
Permission isolation: Does every test stay inside the user's role and tenant?
Operating profile: p50/p95 latency, error rate, and queue depth at representative concurrency.
Only this baseline supports a serious comparison of models, chunking strategies, rerankers, and hardware. Targets are agreed with the business owner; they must not be copied from an unrelated public benchmark.
9. Build or buy: when does a custom system make sense?
Custom on-premise RAG is a fit when …
- technical IP or NDA-protected documents must not leave the corporate network;
- revisions, product lines, and role-based access are part of the answer logic;
- DMS, SharePoint, file servers, ERP, or service portals need integration;
- citations, abstentions, and audit events must remain reviewable for approval.
A standard solution is more likely enough when …
- only a small number of non-sensitive, publicly shareable documents is searched;
- no custom permissions or source-system integrations are required;
- the process still has no accountable owner or measurable acceptance criteria.
Technical sources
The reference architecture is grounded in primary documentation and current security guidance. Vendor pages support hardware specifications only; they do not establish suitability for a specific workload.
- BSI & ANSSI — Design Principles for LLM-based Systems with Zero Trust
- OWASP — RAG Security Cheat Sheet
- vLLM — OpenAI-compatible local inference server
- vLLM — parallelism, KV cache, and capacity planning
- pgvector — HNSW, filtering, and iterative index scans
- PostgreSQL — full-text search and ranking
- NVIDIA — L40S, 48 GB ECC GPU memory
- NVIDIA — RTX PRO 6000 Blackwell Server Edition, 96 GB
Can your document estate be used safely with local AI?
A 30-minute architecture call starts with the process, not the model: document sources, permissions, revision logic, required evidence chain, and operational boundaries.
The outcome is a clear view of whether a bounded on-premise RAG pilot is technically and economically justified—or whether a simpler solution is sufficient.
Frequently asked questions
Is on-premise RAG automatically GDPR-compliant?
No. Local processing can reduce data transfers, but it does not replace a legal basis, retention and deletion rules, access controls, technical and organizational measures, or a DPIA where one is required.
How many documents can such a system hold?
Document count primarily affects storage, ingestion, and index size. The GPU is driven more strongly by model size, context length, and concurrent requests. “Documents per GPU” is therefore not a credible universal metric.
Which local model is best for technical manuals?
That cannot be answered responsibly without the organization's own test set. Language, tables, desired answer length, hardware, and permission logic all affect the choice. The architecture should support model replacement rather than coupling the whole process to one model.
Can RAG eliminate hallucinations completely?
No. RAG can ground answers more strongly in source material, but retrieval and generation can still fail. Citation verification, abstention, structured output, evaluation, and human approval reduce risk; they do not remove it.
How long does a meaningful pilot take?
A bounded pilot can typically be planned for three to six weeks when document access, subject-matter owners, and acceptance criteria are available. Complex OCR, DMS, SSO, or permission requirements can extend that timeline.