Safety Fundamentals¶

Overview¶

Safety is not about preventing all harm—it's about knowing what can go wrong and designing systems to minimize, detect, and recover from failures.

What Makes Systems Unsafe?¶

1. Unbounded Scope¶

# DANGEROUS
agent = Agent(tools=[delete_database_tool])

# SAFE
agent = Agent(
 tools=[delete_test_database_tool],
 constraints={
 "allowed_databases": ["test_db_*"],
 "protected_databases": ["production", "archive"]
 }
)

2. No Input Validation¶

# DANGEROUS
user_query = request.query
result = agent.run(user_query)

# SAFE
user_query = request.query
if not validate_input(user_query):
 return error("Invalid input")
result = agent.run(sanitize(user_query))

3. Missing Error Handling¶

# DANGEROUS
result = database.query(sql) # Might fail

# SAFE
try:
 result = database.query(sql)
except DatabaseError as e:
 log_error(e)
 return {"error": "Query failed", "code": e.code}

Safety Properties¶

1. Containment¶

Agent actions stay within defined boundaries.

2. Auditability¶

Every decision is logged and traceable.

3. Graceful Degradation¶

Failures don't cascade; system recovers.

4. Observability¶

Issues detected quickly, not by users.

Risk Assessment Framework¶

class RiskAssessment:
 def assess(self, action):
 # Analyze impact if action fails
 impact = self.estimate_impact(action)

 # Estimate probability of failure
 probability = self.estimate_probability(action)

 # Calculate risk
 risk = impact * probability

 # Determine response
 if risk > CRITICAL_THRESHOLD:
 return "escalate_to_human"
 elif risk > HIGH_THRESHOLD:
 return "require_approval"
 else:
 return "proceed"

3 Safety Warnings¶

Warning 1: Safety Theater¶

# WRONG
if agent.has_safety_enabled: # Just a flag!
 agent.run(request)

# RIGHT
if agent.passes_safety_checks(request):
 if agent.calculate_risk(request) < SAFE_THRESHOLD:
 agent.run(request)

Warning 2: Complexity Explosion¶

# WRONG
if not (
 (check_a() and check_b() and not check_c()) or
 (check_a() and not check_d()) or
 (check_e() and check_f() and check_g())
):
 # Policy enforced?
 pass

# RIGHT
policies = [
 "no_delete_production",
 "no_access_personal_data",
 "no_external_api_calls"
]

if any(violates_policy(action, p) for p in policies):
 return "Denied"

Warning 3: Forgot About Humans¶

# WRONG
agent.run(user_request) # Fully autonomous

# RIGHT
if calculate_risk(user_request) > HIGH_THRESHOLD:
 require_human_approval(user_request)
else:
 agent.run(user_request)

Production Incident Case Study¶

Incident: Agent deleted 10,000 customer records

Root Cause Analysis:
- No scope limitation ← Agent could delete anything
- No audit trail ← Didn't know WHAT was deleted
- No escalation ← High-impact action not reviewed
- No verification ← Didn't verify success before reporting

Fixes:
1. Scope: Only allow delete on archived records
2. Audit: Log every deletion with reason
3. Escalation: Require approval for bulk deletes
4. Verification: Verify deletion with second query

Best Practices¶

Design for safety first - Don't bolt it on later
Multiple layers - No single point of failure
Clear policies - Easy to understand and enforce
Audit everything - Log decisions and outcomes
Test failure modes - Don't just test happy path

-

Last Updated: August 9, 2026