Memory Fundamentals

Overview

Memory is the system that stores and retrieves information over time, enabling agents to:

• Maintain context across interactions
• Learn from past experiences
• Make better decisions based on history
• Improve performance over time

---

The Memory Problem

Without Memory

User: "What's my favorite color?"
Agent: "I don't know. What is it?"

User: "Blue"
Agent: [Remembers for this conversation]

[Later, new conversation]
User: "What's my favorite color?"
Agent: "I don't know. What is it?"
[Forgot everything from before]

With Memory

User: "What's my favorite color?"
Agent: "I don't know. What is it?"

User: "Blue"
Agent: [Stores: "User's favorite color = blue"]

[Later, new conversation]
User: "What's my favorite color?"
Agent: "Blue" [Retrieved from memory]

-

4 Memory Types (Cognitive Science Inspiration)

1. Short-Term/Working Memory

What: Information currently in focus (current task)

Duration: Seconds to minutes (single conversation)

Capacity: Small (like context window of LLM)

Storage: LLM context window, current execution state

Purpose:

• Track current goal
• Store intermediate results
• Hold reasoning steps
• Manage current state

Example:

class ShortTermMemory:
 def __init__(self):
 self.current_context = ""
 self.working_state = {}
 self.recent_events = []

 def add_to_context(self, text):
 """Add to current working context"""
 self.current_context += text

 def get_context(self):
 """Get current context for LLM"""
 return self.current_context

 def clear(self):
 """Clear at end of task"""
 self.current_context = ""

---

2. Episodic Memory

What: Memories of specific events that happened

Duration: Hours to days to lifetime

Capacity: Large (can grow indefinitely)

Storage: Vector database, time-indexed

Purpose:

• Remember what happened before
• Recall similar past events
• Learn from past experiences
• Provide grounding and context

Example:

class EpisodicMemory:
 def __init__(self):
 self.events = [] # List of past events
 self.vector_db = VectorStore()

 def add_event(self, event: dict):
 """Store an event"""
 event["timestamp"] = now()
 self.events.append(event)

 # Add to vector DB for semantic search
 embedding = embed(event["description"])
 self.vector_db.add(embedding, event)

 def recall(self, query: str, k=5):
 """Retrieve similar past events"""
 query_embedding = embed(query)
 return self.vector_db.search(query_embedding, k)

 def get_recent(self, hours=24):
 """Get recent events"""
 cutoff = now() - timedelta(hours=hours)
 return [e for e in self.events if e["timestamp"] > cutoff]

-

3. Semantic Memory

What: General knowledge and facts (not tied to specific events)

Duration: Persistent (lifetime)

Capacity: Very large

Storage: Knowledge graphs, knowledge bases, embeddings

Purpose:

• Store facts and knowledge
• Don't need to remember when/where learned
• Support reasoning and decision-making
• Enable transfer learning

Example:

class SemanticMemory:
 def __init__(self):
 self.knowledge_base = KnowledgeGraph()
 self.facts = {}

 def add_fact(self, subject, predicate, object):
 """Store a fact"""
 # Example: add_fact("GPT-4", "is_a", "language_model")
 self.knowledge_base.add_triple(subject, predicate, object)
 self.facts[f"{subject}:{predicate}"] = object

 def query(self, subject, predicate):
 """Query for a fact"""
 return self.knowledge_base.query(subject, predicate)

 def get_related(self, entity):
 """Get all facts about an entity"""
 return self.knowledge_base.get_all(entity)

---

4. Procedural Memory

What: Knowledge of how to do things (skills, procedures)

Duration: Persistent

Capacity: Large

Storage: Learned patterns, policies, neural weights

Purpose:

• Remember how to accomplish tasks
• Improve performance through practice
• Store successful strategies
• Enable skill learning

Example:

class ProceduralMemory:
 def __init__(self):
 self.procedures = {}
 self.policies = {}
 self.success_rates = {}

 def learn_procedure(self, task_type: str, procedure: Callable):
 """Learn how to do a task"""
 self.procedures[task_type] = procedure
 self.success_rates[task_type] = 0.0

 def record_success(self, task_type: str):
 """Update success rate for procedure"""
 current = self.success_rates.get(task_type, 0.0)
 self.success_rates[task_type] = current + 0.1 # Improve

 def get_best_procedure(self, task_type: str):
 """Get most successful procedure for task"""
 if task_type in self.procedures:
 return self.procedures[task_type]
 return None

-

Memory Architecture

Hierarchical Memory System

graph TD
 A["LLM Context Window<br/>(Short-Term Memory)<br/>• Current goal<br/>• Recent observations<br/>• Working state<br/>Capacity: 100K-1M tokens"]

 B["Retrieval-Augmented System"]

 C["Vector DB<br/>(Episodic)<br/>Similar past events<br/>to retrieve"]

 D["Knowledge Base<br/>(Semantic)<br/>Facts and<br/>relationships"]

 E["Policy Store<br/>(Procedural)<br/>Learned procedures<br/>& success patterns"]

 F["Persistent Storage Layer<br/>• Database<br/>• Long-term archives"]

 A --> B
 B --> C
 B --> D
 B --> E
 B --> F

 C -.->|Retrieval| A
 D -.->|Retrieval| A
 E -.->|Retrieval| A

• • File system │
• • Cloud storage │
• ┘

-

## Memory Operations

### Core Operations

#### 1. **Store/Add**
```python
memory.add(event, embedding, metadata)

Save information to memory

2. Retrieve/Recall

results = memory.retrieve(query, k=5)

Get relevant information back

3. Update

memory.update(id, new_content)

Modify stored information

4. Forget/Delete

memory.forget(id)

Remove information (retention policy)

5. Consolidate

memory.consolidate()

Compress/reorganize memory

-

Memory Levels

Level 1: Immediate (LLM Context)

Capacity: 100K-1M tokens
Duration: Single turn
Access: Fast (already in context)
Use: Current reasoning

Level 2: Conversation (Session Memory)

Capacity: 10M-100M tokens
Duration: Single conversation (hours)
Access: Fast (can fit in context)
Use: Conversation history

Level 3: Long-Term (Persistent)

Capacity: Unlimited
Duration: Weeks/months/years
Access: Medium (retrieval needed)
Use: Cross-session learning

-

Memory Retrieval Strategies

1. Semantic Similarity (Vector Search)

# Find memories similar to query
query_embedding = embed("Find research on climate change")
similar = vector_db.search(query_embedding, k=5)
# Returns

Best for: Conceptual matching

2. Keyword/BM25 Search

# Find memories with matching keywords
results = memory.search("climate change global warming", k=5)

Best for: Exact matching

3. Time-Based Retrieval

# Find recent memories
recent = memory.get_since(hours=24)

Best for: Temporal relevance

4. Metadata Filtering

# Find memories by properties
results = memory.filter(
 source="research_paper",
 year=2024,
 confidence=0.9
)

Best for: Structured queries

5. Hybrid Retrieval

# Combine multiple strategies
results = memory.hybrid_search(
 query="climate research",
 filters={"year": 2024},
 recent_weight=0.3
)

Best for: Complex retrieval

---

Memory Retention Policies

Not all memories should be kept forever. Design retention policies:

Policy 1: Time-Based

class TimeBasedRetention:
 def apply(self, memory_item):
 age = now() - memory_item.created_at

 if age < 1_day:
 return True # Keep
 elif age < 7_days:
 return random() < 0.8 # 80% chance
 elif age < 30_days:
 return random() < 0.2 # 20% chance
 else:
 return False # Delete

Policy 2: Importance-Based

class ImportanceBasedRetention:
 def apply(self, memory_item):
 if memory_item.importance > 0.8:
 return True # Keep high-importance
 elif memory_item.access_count > 5:
 return True # Keep frequently accessed
 elif memory_item.relevance_score > 0.6:
 return True # Keep relevant
 else:
 return False # Delete low-value

Policy 3: Space-Based (LRU)

class SpaceBasedRetention:
 def __init__(self, max_size_gb=10):
 self.max_size = max_size_gb

 def apply(self, memory):
 if memory.current_size() > self.max_size:
 # Delete least recently used
 lru = memory.get_least_recently_used()
 memory.delete(lru)

-

Memory Challenges

Challenge 1: Context Window Limit

Problem: 
 • Agents generate lots of text (actions, reasoning, results)
 • LLM context windows finite (100K-1M tokens)
 • Can't fit all history in context

Solution:
 • Use short-term memory for current task only
 • Use vector DB for retrieval-augmented context
 • Compress/summarize old conversations

Challenge 2: Semantic Drift

Problem:
 • Memories get less relevant over time
 • What was important may become stale
 • Semantic meaning can change

Solution:
 • Update embeddings periodically
 • Re-rank by relevance
 • Use time-based decay

Challenge 3: Hallucination/Contamination

Problem:
 • LLM might modify memories
 • Incorrect information stored
 • Corrupts future decisions

Solution:
 • Validate before storing
 • Version control memories
 • Separate facts from reasoning

Challenge 4: Scalability

Problem:
 • Memory systems grow over time
 • Search becomes slow (O(n) or worse)
 • Storage costs increase

Solution:
 • Index vectors for fast search
 • Partition by time or domain
 • Implement compression

---

Memory System Design Pattern

class AgentMemory:
 def __init__(self):
 # Short-term: current context
 self.context_window = []

 # Medium-term: current session
 self.conversation_history = []

 # Long-term: persistent storage
 self.episodic_store = VectorDB() # Events
 self.semantic_store = KnowledgeBase() # Facts
 self.procedural_store = PolicyStore() # Procedures

 def remember(self, event: dict, memory_type: str):
 """Store event in appropriate memory level"""
 if memory_type == "immediate":
 self.context_window.append(event)
 elif memory_type == "session":
 self.conversation_history.append(event)
 elif memory_type == "episodic":
 self.episodic_store.add(event)
 elif memory_type == "semantic":
 self.semantic_store.add_fact(event)
 elif memory_type == "procedural":
 self.procedural_store.add_procedure(event)

 def recall(self, query: str, memory_type: str = "all"):
 """Retrieve relevant memories"""
 results = []

 if memory_type in ["all", "immediate"]:
 results.extend(self._search_context(query))
 if memory_type in ["all", "session"]:
 results.extend(self.conversation_history)
 if memory_type in ["all", "episodic"]:
 results.extend(self.episodic_store.search(query))
 if memory_type in ["all", "semantic"]:
 results.extend(self.semantic_store.query(query))
 if memory_type in ["all", "procedural"]:
 results.extend(self.procedural_store.find(query))

 return self._rank_results(results, query)

 def consolidate(self):
 """Clean up and compress memories"""
 # Apply retention policies
 self._apply_retention_policies()

 # Compress old conversations
 self._compress_conversations()

 # Update indices
 self._update_indices()

---

Memory Best Practices

1. Be Explicit About What to Remember

# Good
memory.remember(
 {
 "type": "fact",
 "subject": "user",
 "content": "User prefers concise responses",
 "importance": "high"
 },
 memory_type="semantic"
)

# Bad
memory.remember({"data": "something"})

2. Index for Fast Retrieval

# Good
memory.create_index("user_id")
memory.create_index("timestamp")
memory.create_index("importance")

# Bad
memory.search(query) # O(n) scan each time

3. Have a Retention Policy

# Good
memory.apply_retention_policy(
 "delete_if_old_and_low_importance",
 days=30,
 importance_threshold=0.3
)

# Bad
memory.remember(everything)

4. Monitor Memory Health

# Good
print(f"Memory size: {memory.size_gb()}")
print(f"Most retrieved: {memory.most_accessed(k=10)}")
print(f"Search latency: {memory.avg_search_time_ms()}")

# Bad
# Don't know if memory is healthy

-

Memory vs Knowledge

Memory (Episodic)

- Specific events that happened
- Time-stamped and contextual
- "I helped a user analyze financial data on Jan 15"
- Used for: Learning from experience

Knowledge (Semantic)

- General facts and concepts
- Not tied to specific time/place
- "Financial analysis involves trend analysis"
- Used for: Understanding and reasoning

Both are needed:

• Memory → Learn from specific experiences
• Knowledge → Transfer learning across situations

---

Key Takeaways

1. 4 memory types - Short-term, Episodic, Semantic, Procedural
2. Hierarchical architecture - From LLM context to persistent storage
3. Retrieval critical - Memory only useful if you can find what you need
4. Retention policies - Can't keep everything forever
5. Challenges exist - Context limits, semantic drift, hallucination
6. Design for scale - Start simple, add indexing as needed

-

Next Steps

• Read Short Term Memory - Current context management
• Read Long Term Memory - Persistent storage
• Read Vector Stores - Semantic search

-

Last Updated: August 9, 2026