Safety Fundamentals: Building Trustworthy Agents¶

Overview¶

Safety is not about preventing all harm—it's about knowing what can go wrong and designing systems to minimize, detect, and recover from failures.

What Makes Systems Unsafe?¶

1. Unbounded Scope¶

# ❌ DANGEROUS: Agent can delete ANY database
agent = Agent(tools=[delete_database_tool])

# ✅ SAFE: Agent can only delete test databases
agent = Agent(
    tools=[delete_test_database_tool],
    constraints={
        "allowed_databases": ["test_db_*"],
        "protected_databases": ["production", "archive"]
    }
)

2. No Input Validation¶

# ❌ DANGEROUS: Accept any input
user_query = request.query
result = agent.run(user_query)

# ✅ SAFE: Validate and sanitize
user_query = request.query
if not validate_input(user_query):
    return error("Invalid input")
result = agent.run(sanitize(user_query))

3. Missing Error Handling¶

# ❌ DANGEROUS: Crash on error
result = database.query(sql)  # Might fail

# ✅ SAFE: Handle failures
try:
    result = database.query(sql)
except DatabaseError as e:
    log_error(e)
    return {"error": "Query failed", "code": e.code}

Safety Properties¶

1. Containment¶

Agent actions stay within defined boundaries.

2. Auditability¶

Every decision is logged and traceable.

3. Graceful Degradation¶

Failures don't cascade; system recovers.

4. Observability¶

Issues detected quickly, not by users.

Risk Assessment Framework¶

class RiskAssessment:
    def assess(self, action):
        # Analyze impact if action fails
        impact = self.estimate_impact(action)

        # Estimate probability of failure
        probability = self.estimate_probability(action)

        # Calculate risk
        risk = impact * probability

        # Determine response
        if risk > CRITICAL_THRESHOLD:
            return "escalate_to_human"
        elif risk > HIGH_THRESHOLD:
            return "require_approval"
        else:
            return "proceed"

3 Safety Warnings ⚠️¶

Warning 1: Safety Theater¶

# ❌ WRONG: Safety check that doesn't actually protect
if agent.has_safety_enabled:  # Just a flag!
    agent.run(request)

# ✅ RIGHT: Actual safety mechanisms
if agent.passes_safety_checks(request):
    if agent.calculate_risk(request) < SAFE_THRESHOLD:
        agent.run(request)

Warning 2: Complexity Explosion¶

# ❌ WRONG: Too many rules, hard to reason about
if not (
    (check_a() and check_b() and not check_c()) or
    (check_a() and not check_d()) or
    (check_e() and check_f() and check_g())
):
    # Policy enforced?
    pass

# ✅ RIGHT: Simple, clear policies
policies = [
    "no_delete_production",
    "no_access_personal_data",
    "no_external_api_calls"
]

if any(violates_policy(action, p) for p in policies):
    return "Denied"

Warning 3: Forgot About Humans¶

# ❌ WRONG: Fully automated, no human oversight
agent.run(user_request)  # Fully autonomous

# ✅ RIGHT: Humans for high-impact decisions
if calculate_risk(user_request) > HIGH_THRESHOLD:
    require_human_approval(user_request)
else:
    agent.run(user_request)

Production Incident Case Study¶

Incident: Agent deleted 10,000 customer records

Root Cause Analysis:
- No scope limitation ← Agent could delete anything
- No audit trail ← Didn't know WHAT was deleted
- No escalation ← High-impact action not reviewed
- No verification ← Didn't verify success before reporting

Fixes:
1. Scope: Only allow delete on archived records
2. Audit: Log every deletion with reason
3. Escalation: Require approval for bulk deletes
4. Verification: Verify deletion with second query

Best Practices¶

Design for safety first - Don't bolt it on later
Multiple layers - No single point of failure
Clear policies - Easy to understand and enforce
Audit everything - Log decisions and outcomes
Test failure modes - Don't just test happy path

Last Updated: August 9, 2026