Retrieval-Augmented Generation (RAG) has become the go-to architecture for enterprise AI applications. The promise is simple: combine the power of large language models with your company's proprietary data. The reality is often far messier.
The Problem With Most RAG Systems
When we started working with CyPro, a London-based cybersecurity firm, they were facing a common challenge. Third-party due diligence questionnaires (sometimes containing hundreds of questions) were taking up to four days to complete. Each questionnaire required manual cross-referencing of internal documentation, careful redaction, and review.
They'd tried using ChatGPT and other AI tools directly, but the results were inconsistent. Hallucinations were frequent. Answers lacked the specific context of their security policies. And sensitive data couldn't be sent to external APIs.
Our Approach: Knowledge-First Architecture
The key insight was treating this as a knowledge management problem, not just an AI problem. We built a system with three core principles:
- All data stays on-premises with no external API calls for sensitive information
- Chunking strategy optimised for Q&A retrieval, not general search
- Confidence scoring to flag answers that need human review
The Technical Stack
We deployed the entire system on Azure VMs within CyPro's existing infrastructure. The core components:
- n8n for workflow orchestration (self-hosted)
- PostgreSQL with pgvector for vector storage
- Azure OpenAI for embeddings and generation
- React frontend with real-time chat interface
The system reduced questionnaire response time from 4 days to approximately 3 minutes. More importantly, the accuracy was high enough that staff trusted the outputs.
Lessons Learned
Building RAG systems that work in production requires more than connecting an LLM to a vector database. You need to understand the specific retrieval patterns of your use case, build confidence mechanisms, and create clear human-in-the-loop workflows for edge cases.
The technology is mature enough for real enterprise use, but only if you're willing to invest in getting the architecture right. If you're budgeting for a build like this, we've broken down what AI automation actually costs in the UAE.
Frequently asked questions
Why do most RAG systems fail in production?
Most RAG systems fail because they treat retrieval as a search problem rather than a knowledge management problem. Poor chunking strategy, no confidence scoring, and no human-in-the-loop workflow for edge cases lead to hallucinated answers that staff quickly stop trusting.
Can RAG work with sensitive company data?
Yes, if the architecture keeps data on your own infrastructure. We deployed the entire system on Azure VMs inside the client's environment, so no sensitive information was ever sent to external AI APIs.
What stack do you recommend for an enterprise RAG system?
We used self-hosted n8n for orchestration, PostgreSQL with pgvector for vector storage, Azure OpenAI for embeddings and generation, and a React chat frontend. Boring, proven components beat exotic ones in production.