Skip to content

Vector Databases & Embeddings: The Foundation of RAG

Overview

Embeddings convert text into numerical vectors that capture semantic meaning. Vector Databases store and retrieve embeddings efficiently. Together, they enable semantic search: finding documents similar in meaning, not just keywords.

  • Embeddings: Text → fixed-size vector (1536 dimensions typical)
  • Vector Database: Stores millions of vectors, retrieves by similarity
  • Key Advantage: Semantic search (finds meaning, not just keywords)
  • Adoption: Pinecone, Weaviate, Qdrant, Milvus, FAISS

Embeddings: Converting Text to Vectors

How Embeddings Work

Text → Embedding Model → Vector

Example: OpenAI's text-embedding-3-small

Input text:
"The cat sat on the mat"

Embedding process:
  - Tokenize: ["The", "cat", "sat", "on", "the", "mat"]
  - Convert to vectors (per token)
  - Aggregate (average, cls token, etc.)
  - Output: [0.123, -0.456, 0.789, ..., 0.234]  (1536 dimensions)

Different text, similar meaning:
"A feline rested on the carpet"
→ [0.125, -0.450, 0.792, ..., 0.232]  (very similar vector!)

This is semantic similarity!

Model                  Dimensions  Speed   Quality   Cost
────────────────────────────────────────────────────────
text-embedding-3-small   1536      Fast    Good      Cheap
text-embedding-3-large   3072      Medium  Excellent Moderate
BAAI/bge-large-en       1024      Fast    Excellent Free
BAAI/bge-small-en       384       Very Fast Good     Free
all-MiniLM-L6-v2       384       Very Fast Fair      Free
all-mpnet-base-v2      768       Fast    Good      Free

Recommendation:
  - Production: text-embedding-3-small (best balance)
  - Cost-sensitive: BAAI/bge-small-en (free, good quality)
  - High quality: text-embedding-3-large (best, slower)
  - Self-hosted: bge-large-en (best free option)

Cost comparison (1M documents):

OpenAI embedding-3-small:
  - Cost: $0.02 per 1M tokens
  - 1M docs × 100 tokens avg = 100M tokens
  - Cost: $2 (cheap!)

Self-hosted bge-large:
  - Cost: $0 (except compute)
  - Setup: More complex
  - For high volume: Worth it

Embedding Dimensions & Trade-offs

Dimension size affects:

Small dimensions (384):
✅ Fast search
✅ Less memory
✅ Cheap storage
❌ Less expressive

Large dimensions (3072):
✅ Higher quality
✅ More expressive
❌ Slower search
❌ More memory

Comparison:

1M documents, each 1536 dims (8 bytes per float):
  - Size: 1M × 1536 × 8 = 12GB
  - Search latency: ~50ms per query
  - Cost: $100/month on cloud

Same documents, 384 dims:
  - Size: 1M × 384 × 8 = 3GB (4x smaller!)
  - Search latency: ~10ms per query (5x faster!)
  - Cost: $25/month (4x cheaper!)

Quality trade-off:
  - 384-dim: 95% quality of 3072-dim
  - Most applications don't need 3072
  - Sweet spot: 768-1536 dimensions

Vector Databases

Architecture

Traditional database:
- ┌─────────────────────────────────┐
    - Document: "Blood clotting..."   │
  - ┤
    - SQL row:                        │
    - ID | Title | Content | Tags    │
  - ┘

Search: "SELECT * WHERE title LIKE '%clot%'"
Result: Exact keyword match only

Vector database:
- ┌──────────────────────────────────────────────────┐
    - Document: "Blood clotting..."                    │
  - ┤
    - Embedding: [0.12, -0.45, 0.78, ..., 0.23]       │
    - (1536 dimensions, semantic meaning)             │
  - ┘

Search: "Find similar to: [0.11, -0.46, 0.79, ...]"
Result: Top-K semantically similar documents!

Database      Type        Scale      Latency   Cost    Self-hosted
─────────────────────────────────────────────────────────────────
Pinecone      Managed     Billions   <100ms    $$      No
Weaviate      Self-hosted Billions   <100ms    $       Yes
Qdrant        Self-hosted Billions   <50ms     $       Yes
Milvus        Self-hosted Billions   <50ms     $       Yes
Chroma        In-memory   Millions   <10ms     $       Yes
FAISS         Library     Billions   <10ms     $       Python
Elasticsearch Hybrid      Billions   100-500ms $       Yes

Selection criteria:

Use Pinecone if:
✅ Want managed solution (no ops)
✅ Don't mind monthly fees
✅ Need enterprise support

Use Weaviate if:
✅ Need self-hosted
✅ Want GraphQL interface
✅ Need vector + text search

Use Qdrant if:
✅ Need self-hosted
✅ Want simplicity
✅ Need fast performance

Use FAISS if:
✅ Building research project
✅ Have data < 1B vectors
✅ Want pure Python

Vector Database Operations

from pinecone import Pinecone

# Initialize
pc = Pinecone(api_key="...")
index = pc.Index("medical-docs")

# Step 1: Upsert (store documents with embeddings)
documents = [
    {"id": "doc1", "text": "Blood clotting...", "embedding": [0.12, -0.45, ...]},
    {"id": "doc2", "text": "Hemostasis is...", "embedding": [0.13, -0.44, ...]},
    # ... more documents
]

for doc in documents:
    index.upsert([
        (doc["id"], doc["embedding"], {"text": doc["text"]})
    ])

# Step 2: Query (find similar documents)
query_embedding = [0.11, -0.46, ...]  # Embedding of user question
results = index.query(
    vector=query_embedding,
    top_k=3,
    include_metadata=True
)

# Results:
for match in results["matches"]:
    print(f"Doc: {match['metadata']['text']}")
    print(f"Similarity: {match['score']}")  # 0-1, higher is more similar

# Step 3: Update (modify stored document)
index.update(
    id="doc1",
    values=[0.12, -0.46, ...]  # New embedding
)

# Step 4: Delete (remove document)
index.delete(ids=["doc1"])

Similarity Metrics

Distance Calculations

How vectors are compared:

1. Cosine Similarity (most common)
  - Measures angle between vectors
  - Range: -1 to 1 (higher = more similar)
  - Formula: cos(θ) = A·B / (|A||B|)
  - Advantage: Works regardless of magnitude
  - Best for: Text embeddings

2. Euclidean Distance (L2 norm)
  - Straight-line distance in space
  - Range: 0 to infinity (lower = more similar)
  - Formula: √(Σ(A_i - B_i)²)
  - Best for: Dense vector spaces

3. Manhattan Distance (L1 norm)
  - City-block distance
  - Faster to compute than L2
  - Best for: Sparse vectors

Example (2D for visualization):

Vector A: [1, 0]  (query)
Vector B: [0.9, 0.1]  (document 1 - very similar)
Vector C: [0, 1]  (document 2 - different)

Cosine similarity:
  - A vs B: 0.995 (very similar!)
  - A vs C: 0.0 (orthogonal, unrelated)
  - Correctly ranks B > C

Recommendation:
  - Use cosine similarity for embeddings!
  - It's what embedding models are trained for

Scaling Considerations

Latency at Scale

Query latency breakdown:

Small index (1M documents):
  - Embedding query: 1ms
  - Vector search: 5ms
  - Metadata fetch: 1ms
  - Total: ~7ms (very fast!)

Medium index (100M documents):
  - Embedding query: 1ms
  - Vector search: 20ms (more candidates to search)
  - Metadata fetch: 2ms
  - Total: ~23ms (still fast!)

Large index (1B documents):
  - Embedding query: 1ms
  - Vector search: 50ms (even more candidates)
  - Metadata fetch: 5ms
  - Total: ~56ms (acceptable)

Optimizations:

1. Approximate search (HNSW, IVF)
  - Don't search exact nearest neighbors
  - Search nearby clusters
  - Trade: Small accuracy loss for speed
  - Typical: >98% recall at 10x speedup

2. Quantization
  - Store vectors as int8 instead of float32
  - Reduce memory 4x
  - Minor quality loss

3. Sharding
  - Split vectors across multiple servers
  - Query multiple shards in parallel
  - Linear scaling with shards

4. Caching
  - Cache frequent queries
  - Avoid re-embedding

Cost Analysis

Storage & Compute

Cost breakdown for 1M documents:

Option 1: Pinecone (managed, standard index)
  - Storage: 0.25 index units × $0.25/month = $6.25/month
  - Queries: ~$0.001 per 1000 queries
  - Example (1M queries/day): ~$30/month
  - Total: ~$36/month

Option 2: Self-hosted (Qdrant on AWS)
  - Instance type: r6i.2xlarge (64GB RAM)
  - Cost: $0.65/hour = ~$475/month
  - Storage: 12GB vectors + 50GB snapshots = $5/month
  - Total: ~$480/month

Breakeven analysis:
  - Pinecone: $36/month (small scale)
  - Self-hosted: $480/month (large scale)
  - Breakeven: ~100M queries/day
  - Most applications stay with managed below this!

Recommendation:
  - Small to medium: Use Pinecone or Weaviate Cloud
  - Large scale: Self-host Qdrant/Milvus
  - Trading off: Simplicity vs cost

Key Takeaways

🔢 Embeddings: Text → dense vectors capturing meaning
🔍 Vector databases: Fast semantic search on millions of vectors
⚡ Cosine similarity: Best metric for text embeddings
📊 Approximate search: Trade tiny accuracy for 10x speed
💰 Managed services cheap for small scale, self-host for large