RAG for codebases: answering questions about your codebase

RAG for codebases is a system that indexes your source code and documentation into a searchable vector database, then retrieves relevant context to ground LLM answers about your codebase. It can answer questions like: "How does our authentication middleware work?"; "Show me examples of how we use the PaymentService class"; "What's the correct way to add a new API endpoint in this project?"; "Why was the retry logic changed last year?" (from git history); "Find all places where we connect to the database"; and "How do we handle errors in the async job processor?" — questions impossible to answer accurately without actual codebase context.

For production codebase RAG, Voyage Code 2 (from Voyage AI) is currently the state-of-the-art dedicated code embedding model, significantly outperforming general text embedding models on code retrieval benchmarks (CodeSearchNet and similar). It supports 20+ programming languages and was specifically trained for code semantic similarity tasks. OpenAI's text-embedding-3-large is a strong second choice with good code performance if you are already in the OpenAI ecosystem. For self-hosted or privacy-sensitive deployments where code cannot leave your infrastructure, Nomic Embed Code provides a strong open-source option that runs locally on GPU or CPU.

For codebases under approximately 100,000 lines (or ~150K tokens), full-context loading with a 200K token model (Claude 3.5 Sonnet, Gemini 1.5 Pro) is often simpler and more accurate than RAG — the model sees the complete codebase and can reason about relationships that RAG retrieval might miss. For larger codebases, RAG is necessary as they exceed context window limits. Many tools (Cursor, Claude Code) use a hybrid approach: a repo map summarising the codebase structure always in context, with RAG retrieval for specific relevant files when needed. RAG also enables sub-second responses for specific questions, while full-context loading of a large codebase takes seconds and costs significantly more per query.

For a self-built codebase RAG system: LlamaIndex or LangChain provide the RAG orchestration framework with code-specific loaders; Qdrant or Weaviate are recommended vector stores with good hybrid search support; Voyage Code 2 or OpenAI embeddings for code embedding generation; tree-sitter for language-aware AST-based code parsing and chunking; and a cross-encoder reranker (Cohere Rerank, ColBERT) to improve retrieval precision. For pre-built solutions: Sourcegraph Cody, Continue.dev, and Cursor all implement codebase RAG internally. GitHub Copilot Chat uses RAG over your repository for contextual answers. For enterprise deployments requiring self-hosting with access control, LlamaIndex's Enterprise RAG framework is worth evaluating.

Access control in codebase RAG must be implemented at the retrieval layer, not just the application layer. Each indexed chunk should be tagged with its repository, file path, and the access control group that can see it (mirroring the repository's actual permissions). At query time, the retrieval query is filtered to only return chunks accessible to the querying user. This requires your vector store to support metadata filtering (Qdrant, Pinecone, Weaviate, Chroma all support this). Never retrieve and include code chunks that the user doesn't have permission to see in the LLM context — the LLM may summarise or reveal information from those chunks in its answer even if you don't display the source chunks directly.

Hybrid search combines vector semantic search (finding code that is conceptually similar to the query) with keyword-based BM25 search (finding exact symbol names, error messages, or identifiers). Code queries span both types: "how does authentication work?" benefits from semantic search (finding auth-related code even without those exact words); "find all uses of UserRepository" benefits from keyword search (exact symbol name matching). Hybrid search with reciprocal rank fusion (RRF) combines scores from both retrieval methods, producing results that capture both semantic similarity and keyword relevance. Benchmarks consistently show hybrid search outperforming either approach alone for code retrieval, making it the recommended default for production codebase RAG systems.

Effective code chunks for RAG include more than just the raw code: prepend the file path and module name to each chunk; include the function signature and docstring even if the implementation is chunked separately; for method chunks, include the class name and class-level docstring; add language and framework metadata tags; and for multi-file concepts, include cross-reference metadata (this function is called by X, Y, Z). Some implementations use a "context window" approach: the chunk contains the target function plus N lines of surrounding context (imports, adjacent functions) to help the LLM understand the code's environment. Richer chunk context consistently improves answer quality at the cost of larger index size and higher retrieval cost.

Common codebase RAG failure modes include: stale index (code changes but the index is not updated — answers reference deleted functions or old APIs; fix with automated reindexing on merge); retrieval misses (relevant code is not retrieved because the query doesn't match the embedding well — use hybrid search and query expansion to mitigate); context window overwhelm (too many retrieved chunks fill the LLM's context without fitting the most relevant code — use a reranker to prioritise top-3 most relevant chunks); hallucinated code (LLM generates plausible but non-existent internal APIs — enforce citation requirements and ground answers in retrieved chunks); and poor chunking (functions split at wrong boundaries lose context — use AST-based chunking to preserve semantic units).