Regular databases excel at exact matching: find the row where id=42, or where name='Alice'. But RAG systems need a fundamentally different operation: find the rows whose vector representation is closest to this query vector. This is approximate nearest neighbor (ANN) search, and it requires specialized data structures and algorithms that traditional B-tree indexes cannot provide.
Vector databases solve this problem. They store high-dimensional vectors alongside metadata and support fast similarity search using indexes like HNSW (Hierarchical Navigable Small World) and IVF (Inverted File Index). Some are standalone services (Qdrant, Milvus), some are embeddable libraries (FAISS, Chroma), and one is an extension to PostgreSQL (pgvector) that lets you add vector search to your existing relational database.
The choice of vector database shapes your RAG architecture: how you deploy, how you scale, how you filter, and how fast your retrieval step runs. This guide compares the major open-source options and shows you how to use each one with working code.
pgvector is the simplest path to vector search if you already use PostgreSQL. The HNSW index provides sub-millisecond search up to a few million vectors. The key advantage is SQL filtering: you can combine WHERE clauses with vector search, which standalone vector databases handle differently. The <=> operator computes cosine distance (lower is more similar).
Chroma runs in-process with no server to manage. The embedding function integration means you pass raw text and Chroma handles embedding transparently. PersistentClient writes to disk so data survives restarts. The where clause supports filtering by metadata fields before vector search. Chroma is ideal for prototyping and small-to-medium datasets.
Qdrant is a purpose-built vector database designed for production. It runs as a Docker container, supports payload filtering during search (not just post-filtering), and handles on-disk storage for datasets that exceed memory. The recreate_collection call is for demos -- in production, use create_collection with error handling.
FAISS gives you direct control over the index type. Flat is brute-force (perfect recall, O(n) search). IVF clusters vectors and only searches nearby clusters (fast, but requires training). HNSW builds a graph (best recall/speed trade-off, no training needed). FAISS is a library, not a database -- there is no built-in persistence, replication, or metadata filtering.
This comparison framework helps you choose the right vector store for your use case. pgvector wins when you want to avoid infrastructure complexity. Chroma wins for fast prototyping. Qdrant wins for production with rich filtering. FAISS wins for raw speed in batch workloads. Milvus wins for billion-scale distributed search.
Choosing a standalone vector database when pgvector would suffice, adding unnecessary infrastructure complexity, deployment costs, and data synchronization challenges.
If your dataset is under 5 million vectors and you already use PostgreSQL, start with pgvector. It handles metadata filtering via standard SQL, requires no additional service, and is sufficient for most RAG applications.
Using a flat (brute-force) index in production with millions of vectors, resulting in search times that scale linearly with dataset size.
Switch to an HNSW or IVF index once your dataset exceeds a few thousand vectors. HNSW offers the best recall-speed trade-off for most workloads. Flat indexes are only appropriate for datasets under 10,000 vectors or when you need exact results.
Not tuning HNSW parameters (m, ef_construction, ef_search). The defaults are conservative and may not match your recall and latency requirements.
Benchmark with your actual data. Start with m=16, ef_construction=64, ef_search=40. Increase ef_search to improve recall at the cost of latency. Increase m and ef_construction when building the index if your recall target is above 95%.
Treating FAISS as a database -- loading the entire index into memory on every application restart, with no built-in persistence or metadata filtering.
FAISS is a search library, not a database. Use faiss.write_index() and faiss.read_index() for persistence. For metadata filtering, maintain a separate mapping from FAISS IDs to document metadata. If you need built-in persistence and filtering, use Chroma, Qdrant, or pgvector instead.
Ignoring metadata filtering and doing post-filtering on vector search results. This returns k results from the full corpus, then discards the ones that do not match the filter, leaving fewer than k relevant results.
Use pre-filtering (filter before vector search) or integrated filtering (Qdrant and pgvector support this natively). Pre-filtering ensures you always get k results that match both the semantic query and the metadata constraints.
Vector databases are purpose-built for similarity search over high-dimensional embeddings. pgvector extends PostgreSQL with vector columns and HNSW indexes, keeping your vectors alongside your application data. Chroma runs embedded for local development. Qdrant provides a production-grade standalone service with rich filtering. FAISS is a raw search library for maximum throughput. Choose based on your scale, existing infrastructure, and filtering requirements -- not based on hype. For most RAG applications, pgvector or Qdrant covers the need, and HNSW is the indexing algorithm you want.
Interactive lessons and challenges on Stanza, practice in VS Code, Cursor, or the web.
Interactive lessons and challenges, right in your code editor.
Check the free courses. No credit card.