Vector Databases & Embeddings

Overview

Embeddings convert text into numerical vectors that capture semantic meaning. Vector Databases store and retrieve embeddings efficiently. Together, they enable semantic search: finding documents similar in meaning, not just keywords.

• Embeddings: Text → fixed-size vector (1536 dimensions typical)
• Vector Database: Stores millions of vectors, retrieves by similarity
• Key Advantage: Semantic search (finds meaning, not just keywords)
• Adoption: Pinecone, Weaviate, Qdrant, Milvus, FAISS

-

Embeddings: Converting Text to Vectors

How Embeddings Work

Text → Embedding Model → Vector

Example: OpenAI's text-embedding-3-small

Input text:
"The cat sat on the mat"

Embedding process:
 - Tokenize: ["The", "cat", "sat", "on", "the", "mat"]
 - Convert to vectors (per token)
 - Aggregate (average, cls token, etc.)
 - Output: [0.123, -0.456, 0.789,..., 0.234] (1536 dimensions)

Different text, similar meaning:
"A feline rested on the carpet"
→ [0.125, -0.450, 0.792,..., 0.232] (very similar vector!)

This is semantic similarity!

Popular Embedding Models

Model Dimensions Speed Quality Cost
────────────────────────────────────────────────────────
text-embedding-3-small 1536 Fast Good Cheap
text-embedding-3-large 3072 Medium Excellent Moderate
BAAI/bge-large-en 1024 Fast Excellent Free
BAAI/bge-small-en 384 Very Fast Good Free
all-MiniLM-L6-v2 384 Very Fast Fair Free
all-mpnet-base-v2 768 Fast Good Free

Recommendation:
 - Production: text-embedding-3-small (best balance)
 - Cost-sensitive: BAAI/bge-small-en (free, good quality)
 - High quality: text-embedding-3-large (best, slower)
 - Self-hosted: bge-large-en (best free option)

Cost comparison (1M documents):

OpenAI embedding-3-small:
 - Cost: $0.02 per 1M tokens
 - 1M docs × 100 tokens avg = 100M tokens
 - Cost: $2 (cheap!)

Self-hosted bge-large:
 - Cost: $0 (except compute)
 - Setup: More complex
 - For high volume: Worth it

Embedding Dimensions & Trade-offs

Dimension size affects:

Small dimensions (384):
Fast search
Less memory
Cheap storage
Less expressive

Large dimensions (3072):
Higher quality
More expressive
Slower search
More memory

Comparison:

1M documents, each 1536 dims (8 bytes per float):
 - Size: 1M × 1536 × 8 = 12GB
 - Search latency: ~50ms per query
 - Cost: $100/month on cloud

Same documents, 384 dims:
 - Size: 1M × 384 × 8 = 3GB (4x smaller!)
 - Search latency: ~10ms per query (5x faster!)
 - Cost: $25/month (4x cheaper!)

Quality trade-off:
 - 384-dim: 95% quality of 3072-dim
 - Most applications don't need 3072
 - Sweet spot: 768-1536 dimensions

---

Vector Databases

Architecture

Traditional database:
- ┌─────────────────────────────────┐
 - Document: "Blood clotting..." │
 - ┤
 - SQL row: │
 - ID| Title| Content| Tags │
 - ┘

Search: "SELECT * WHERE title LIKE '%clot%'"
Result: Exact keyword match only

Vector database:
- ┌──────────────────────────────────────────────────┐
 - Document: "Blood clotting..." │
 - ┤
 - Embedding: [0.12, -0.45, 0.78,..., 0.23] │
 - (1536 dimensions, semantic meaning) │
 - ┘

Search: "Find similar to: [0.11, -0.46, 0.79,...]"
Result: Top-K semantically similar documents!

Popular Vector Databases

Database Type Scale Latency Cost Self-hosted
─────────────────────────────────────────────────────────────────
Pinecone Managed Billions <100ms $$ No
Weaviate Self-hosted Billions <100ms $ Yes
Qdrant Self-hosted Billions <50ms $ Yes
Milvus Self-hosted Billions <50ms $ Yes
Chroma In-memory Millions <10ms $ Yes
FAISS Library Billions <10ms $ Python
Elasticsearch Hybrid Billions 100-500ms $ Yes

Selection criteria:

Use Pinecone if:
Want managed solution (no ops)
Don't mind monthly fees
Need enterprise support

Use Weaviate if:
Need self-hosted
Want GraphQL interface
Need vector + text search

Use Qdrant if:
Need self-hosted
Want simplicity
Need fast performance

Use FAISS if:
Building research project
Have data < 1B vectors
Want pure Python

Vector Database Operations

from pinecone import Pinecone

# Initialize
pc = Pinecone(api_key="...")
index = pc.Index("medical-docs")

# Step 1
documents = [
 {"id": "doc1", "text": "Blood clotting...", "embedding": [0.12, -0.45,...]},
 {"id": "doc2", "text": "Hemostasis is...", "embedding": [0.13, -0.44,...]},
 #... more documents
]

for doc in documents:
 index.upsert([
 (doc["id"], doc["embedding"], {"text": doc["text"]})
])

# Step 2
query_embedding = [0.11, -0.46,...] # Embedding of user question
results = index.query(
 vector=query_embedding,
 top_k=3,
 include_metadata=True
)

# Results:
for match in results["matches"]:
 print(f"Doc: {match['metadata']['text']}")
 print(f"Similarity: {match['score']}") # 0-1, higher is more similar

# Step 3
index.update(
 id="doc1",
 values=[0.12, -0.46,...] # New embedding
)

# Step 4
index.delete(ids=["doc1"])

-

Similarity Metrics

Distance Calculations

How vectors are compared:

1. Cosine Similarity (most common)
 - Measures angle between vectors
 - Range: -1 to 1 (higher = more similar)
 - Formula: cos(θ) = A·B / (|A||B|)
 - Advantage: Works regardless of magnitude
 - Best for: Text embeddings

2. Euclidean Distance (L2 norm)
 - Straight-line distance in space
 - Range: 0 to infinity (lower = more similar)
 - Formula: √(Σ(A_i - B_i)²)
 - Best for: Dense vector spaces

3. Manhattan Distance (L1 norm)
 - City-block distance
 - Faster to compute than L2
 - Best for: Sparse vectors

Example (2D for visualization):

Vector A: [1, 0] (query)
Vector B: [0.9, 0.1] (document 1 - very similar)
Vector C: [0, 1] (document 2 - different)

Cosine similarity:
 - A vs B: 0.995 (very similar!)
 - A vs C: 0.0 (orthogonal, unrelated)
 - Correctly ranks B > C

Recommendation:
 - Use cosine similarity for embeddings!
 - It's what embedding models are trained for

---

Scaling Considerations

Latency at Scale

Query latency breakdown:

Small index (1M documents):
 - Embedding query: 1ms
 - Vector search: 5ms
 - Metadata fetch: 1ms
 - Total: ~7ms (very fast!)

Medium index (100M documents):
 - Embedding query: 1ms
 - Vector search: 20ms (more candidates to search)
 - Metadata fetch: 2ms
 - Total: ~23ms (still fast!)

Large index (1B documents):
 - Embedding query: 1ms
 - Vector search: 50ms (even more candidates)
 - Metadata fetch: 5ms
 - Total: ~56ms (acceptable)

Optimizations:

1. Approximate search (HNSW, IVF)
 - Don't search exact nearest neighbors
 - Search nearby clusters
 - Trade: Small accuracy loss for speed
 - Typical: >98% recall at 10x speedup

2. Quantization
 - Store vectors as int8 instead of float32
 - Reduce memory 4x
 - Minor quality loss

3. Sharding
 - Split vectors across multiple servers
 - Query multiple shards in parallel
 - Linear scaling with shards

4. Caching
 - Cache frequent queries
 - Avoid re-embedding

-

Cost Analysis

Storage & Compute

Cost breakdown for 1M documents:

Option 1: Pinecone (managed, standard index)
 - Storage: 0.25 index units × $0.25/month = $6.25/month
 - Queries: ~$0.001 per 1000 queries
 - Example (1M queries/day): ~$30/month
 - Total: ~$36/month

Option 2: Self-hosted (Qdrant on AWS)
 - Instance type: r6i.2xlarge (64GB RAM)
 - Cost: $0.65/hour = ~$475/month
 - Storage: 12GB vectors + 50GB snapshots = $5/month
 - Total: ~$480/month

Breakeven analysis:
 - Pinecone: $36/month (small scale)
 - Self-hosted: $480/month (large scale)
 - Breakeven: ~100M queries/day
 - Most applications stay with managed below this!

Recommendation:
 - Small to medium: Use Pinecone or Weaviate Cloud
 - Large scale: Self-host Qdrant/Milvus
 - Trading off: Simplicity vs cost

---

Key Takeaways

Embeddings: Text → dense vectors capturing meaning Vector databases: Fast semantic search on millions of vectors Cosine similarity: Best metric for text embeddings Approximate search: Trade tiny accuracy for 10x speed Managed services cheap for small scale, self-host for large

-

Related Notes in RAG Subdirectory

• Rag Fundamentals - Overview of RAG
• Retrieval Strategies - Different retrieval approaches
• [Chunking & Document Preparation](/01-modeling/03-inference/03-knowledge-integration/rag/(chunking-document-preparation/) - Preparing documents
• Scaling Rag Systems - Production considerations