Short-Term Memory¶
Overview¶
Short-term memory is the "working memory" of an agent—what's currently in focus. For LLM-based agents, this is essentially the context window they're operating within.
The challenge: Finite context + Infinite possible information = Must choose carefully
What is Short-Term Memory?¶
Definition¶
Information agent is currently working with, typically measured in tokens and usually fits in the LLM's context window.
Characteristics¶
- Duration: Seconds to minutes (single conversation turn)
- Capacity: 100K-1M tokens (depending on model)
- Speed: Instant access (already in context)
- Scope: Current task only
- Volatility: Cleared after task completion
Example¶
Agent working on: "Analyze these 5 research papers"
Short-term memory currently holds:
Goal: "Analyze papers"
Papers retrieved: [paper1, paper2, paper3, paper4, paper5]
Analysis so far: [analysis of papers 1-3]
Next action: "Analyze paper 4"
Past conversations: [NOT in short-term]
General knowledge: [Only relevant parts in context]
The Context Window Problem¶
The Dilemma¶
LLM Context Window: 100K tokens
│
- System prompt: 5K tokens
- Current goal: 1K tokens
- Instruction prompt: 3K tokens
- Available for data: 91K tokens
But agent needs:
- Previous turns: 20K tokens
- Retrieved documents: 30K tokens
- Reasoning trace: 15K tokens
- Current observations: 10K tokens
- Total needed: 75K tokens Fits!
But what if the conversation is longer?
Previous turns: 100K tokens
Doesn't fit!
Context Window Strategies¶
Strategy 1: Summarization¶
Original conversation (100K tokens):
User: "Tell me about climate change"
Agent: [Long response about causes, effects, solutions]
User: "What about the economic impact?"
Agent: [Another long response]
User: "Compare solutions"
...
Summarized (5K tokens):
Summary of earlier discussion:
- Climate change caused by greenhouse gases
- Economic impacts: $2T annual costs
- Solutions: Renewable energy, efficiency, carbon capture
Implementation:
class SummarizationStrategy:
def compress_history(self, history, target_tokens=5000):
"""Summarize old conversation turns"""
# Keep recent turns fully
keep_recent = 3 # Last 3 turns
to_summarize = history[:-keep_recent]
# Summarize older turns
summarized = self.llm.summarize(
to_summarize,
max_tokens=target_tokens
)
return summarized + history[-keep_recent:]
Strategy 2: Selective Retention¶
Keep:
Recent interactions (last 5)
Important facts mentioned
User preferences stated
Task-critical information
Discard:
Intermediate reasoning
Verbose explanations
Repeated information
Off-topic comments
Implementation:
class SelectiveRetention:
def filter_history(self, history):
"""Keep only essential information"""
essential = []
for turn in history:
if self._is_essential(turn):
essential.append(turn)
return essential
def _is_essential(self, turn):
"""Judge if turn should be kept"""
if turn.is_recent:
return True # Keep recent
if turn.contains_fact:
return True # Keep facts
if turn.affects_task:
return True # Keep task-relevant
if turn.is_preference:
return True # Keep user preferences
return False
Strategy 3: Windowing¶
Maintain sliding window of recent context:
- Turn 1: ────┐
- Turn 2: ├─ [Window 1: turns 1-5]
- Turn 3: │
- Turn 4: │
- Turn 5: ────┘
- Turn 6: ┐
- Turn 7: ├─ [Window 2: turns 6-10]
- Turn 8: │
- Turn 9: │
- Turn 10:────┘
- Turn 11: ┐
- Turn 12: ├─ [Window 3: turns 11-15]
...
Implementation:
class WindowingStrategy:
def __init__(self, window_size=5):
self.window_size = window_size
self.history = []
def get_context(self):
"""Get current window of context"""
start_idx = max(0, len(self.history) - self.window_size)
return self.history[start_idx:]
def add_turn(self, turn):
"""Add new turn to history"""
self.history.append(turn)
if len(self.history) > self.window_size:
# Old turns fall out of window
# But should be stored in long-term memory
pass
-
Short-Term Memory Architecture¶
- ┌─────────────────────────────────────────────┐
- LLM Context Window │
- (Short-Term Memory) │
- ┤
│ │
- System Prompt │
- Task description │
- Behavioral guidelines │
- Output format spec │
│ │
- Recent Context │
- Last 5 conversation turns │
- Relevant retrieved documents │
- Current goal/subtask │
- Working state/progress │
- Recent observations/results │
│ │
- Instruction Prompt │
- "You are currently at step 3/5" │
- "The user just said:..." │
- "Respond with JSON:..." │
│ │
- ┘
Implementing Short-Term Memory¶
Basic Implementation¶
class ShortTermMemory:
def __init__(self, max_tokens=90000):
self.max_tokens = max_tokens
self.current_tokens = 0
self.items = []
def add(self, content: str, priority: int = 0):
"""Add content to short-term memory"""
tokens = count_tokens(content)
# Check if fits
if self.current_tokens + tokens > self.max_tokens:
# Make space by removing low-priority items
self._make_space(tokens)
self.items.append({
"content": content,
"priority": priority,
"timestamp": now(),
"tokens": tokens
})
self.current_tokens += tokens
def _make_space(self, needed_tokens):
"""Remove items to make space"""
# Sort by priority (keep high-priority)
self.items.sort(key=lambda x: x['priority'], reverse=True)
# Remove low-priority items until we have space
while self.current_tokens + needed_tokens > self.max_tokens:
removed = self.items.pop()
self.current_tokens -= removed['tokens']
def get_context(self):
"""Get formatted context for LLM"""
return "\n".join([
item['content'] for item in self.items
])
def clear(self):
"""Clear memory at end of task"""
self.items = []
self.current_tokens = 0
# Usage
memory = ShortTermMemory(max_tokens=90000)
memory.add("Goal: Analyze market trends", priority=10)
memory.add("Retrieved data: Q1 results...", priority=8)
memory.add("Current finding: Growth +15%", priority=9)
context = memory.get_context()
# Pass to LLM
Advanced: Prioritization¶
Priority Scoring¶
class PriorityScorer:
def score(self, content, context):
"""Score content importance"""
score = 0
# Factor 1: Recency (0-30 points)
age_seconds = (now() - content.timestamp).total_seconds()
if age_seconds < 60:
score += 30
elif age_seconds < 300:
score += 20
elif age_seconds < 3600:
score += 10
# Factor 2: Task Relevance (0-40 points)
if content.mentions_goal:
score += 40
elif content.is_observation:
score += 30
elif content.is_reasoning:
score += 15
# Factor 3: Frequency (0-20 points)
if content.access_count > 5:
score += 20
elif content.access_count > 2:
score += 10
# Factor 4: User-provided (0-10 points)
if content.from_user:
score += 10
return score
-
Common Patterns¶
Pattern 1: Prompt Recycling¶
def build_prompt_with_context(memory: ShortTermMemory, task: str):
"""Build prompt including current context"""
system_prompt = """
You are a helpful research assistant.
Complete the task below using the context provided.
"""
context = memory.get_context()
user_prompt = f"""
Context (previous findings):
{context}
Current task:
{task}
Respond with your findings.
"""
return system_prompt, user_prompt
Pattern 2: Progressive Enrichment¶
class ProgressiveContext:
def __init__(self):
self.history = []
self.observations = []
def execute_step(self, step_num, total_steps):
"""Execute one step, building context"""
# Add progress to memory
self.history.append(f"Step {step_num}/{total_steps}")
# Get relevant context
context = self.build_context()
# Execute
result = self.llm.process(context, step_num)
# Record result
self.observations.append(result)
return result
def build_context(self):
"""Build context with completed steps"""
return {
"steps_completed": self.history,
"findings_so_far": self.observations,
"tokens_used": self.estimate_tokens()
}
Pattern 3: Sliding Window¶
class SlidingWindowMemory:
def __init__(self, window_size=5):
self.window_size = window_size
self.all_history = []
def add_turn(self, turn):
"""Add turn to history"""
self.all_history.append(turn)
def get_context_window(self):
"""Get recent window for LLM"""
start = max(0, len(self.all_history) - self.window_size)
return self.all_history[start:]
def get_full_history_for_longterm(self):
"""Get full history for storage"""
return self.all_history
Token Budgeting¶
Allocating Tokens¶
Context Window: 100K tokens total
Fixed overhead (30%):
- System prompt: 5K
- Instruction prompt: 5K
- Output format spec: 5K
- Few-shot examples: 5K
- Reserved for output: 5K
- Subtotal: 25K
Available for content (70%):
- Current goal: 2K (required)
- Recent context: 30K (recent turns)
- Retrieved documents: 20K (for task)
- Working state: 5K (current progress)
- Buffer: 8K (safety margin)
- Subtotal: 65K
Implementation:
class TokenBudgeter:
def __init__(self, context_size=100000):
self.context_size = context_size
# Allocate fixed overhead
self.system_prompt_budget = 5000
self.instruction_budget = 5000
self.output_budget = 5000
self.buffer = 5000
# Available for content
self.content_budget = (
context_size -
self.system_prompt_budget -
self.instruction_budget -
self.output_budget -
self.buffer
)
def allocate_content(self):
"""Allocate tokens to different content types"""
return {
"goal": int(0.05 * self.content_budget),
"context": int(0.50 * self.content_budget),
"documents": int(0.35 * self.content_budget),
"state": int(0.10 * self.content_budget),
}
-
Best Practices¶
1. Explicit Token Counting¶
# Good
from tiktoken import encoding_for_model
enc = encoding_for_model("gpt-4")
tokens = len(enc.encode(text))
# Bad
estimated = len(text.split()) // 0.75 # Rough estimate
2. Plan for Overflow¶
# Good
if current_tokens + new_content_tokens > MAX:
# Compress, summarize, or discard
make_space(needed_tokens)
# Bad
memory.add(new_content) # May overflow
3. Preserve Task-Critical Info¶
# Good
memory.prioritize([
goal, # Keep: defines task
user_preferences, # Keep: affects output
current_state, # Keep: tracks progress
# Lower priority: verbose explanations
])
# Bad
memory.add(everything)
4. Use Retrieval for Reference¶
# Good
# In context
# Retrieve full details only if needed
# Bad
# Context
-
Debugging Short-Term Memory¶
def debug_context(memory: ShortTermMemory):
"""Debug what's in current context"""
print(f"Total tokens used: {memory.current_tokens}")
print(f"Max capacity: {memory.max_tokens}")
print(f"Usage: {memory.current_tokens/memory.max_tokens*100:.1f}%")
print("\nItems in memory (by priority):")
sorted_items = sorted(
memory.items,
key=lambda x: x['priority'],
reverse=True
)
for item in sorted_items:
print(f" [{item['priority']:2d}] {item['tokens']:5d} tokens: "
f"{item['content'][:50]}...")
print("\nLow-priority candidates for removal:")
low_priority = [i for i in memory.items if i['priority'] < 3]
for item in low_priority:
print(f" {item['tokens']:5d} tokens: {item['content'][:50]}...")
Key Takeaways¶
- Context window is finite - Must choose what to include
- Prioritization critical - Keep task-relevant, discard noise
- Strategies exist - Summarization, windowing, selective retention
- Token budget - Allocate wisely to system, content, output
- Overflow handling - Plan for when content exceeds capacity
- Measure everything - Count actual tokens, don't guess
-
Next Steps¶
- Read Long Term Memory - Persistent storage
- Read Vector Stores - Semantic retrieval
-
Last Updated: August 9, 2026