Output Safety & Content Filtering¶

Overview¶

Even with safe inputs and controlled execution, agents can generate harmful outputs. Content filtering prevents unsafe content from reaching users.

-

Harmful Output Categories¶

1. Illegal content: Hacking guides, drug recipes
2. Violent/Abusive: Death threats, harassment
3. Sexual: Child exploitation, non-consensual
4. Private data: Leak of PII or credentials
5. Biased/Hateful: Discrimination, bigotry
6. Misinformation: False medical/financial advice

Filtering Strategy¶

class OutputFilter:
 def __init__(self):
 self.classifiers = [
 HarmClassifier(),
 BiasDetector(),
 MisinformationChecker(),
 PrivacyScanner()
]

 def filter(self, text: str) -> tuple[bool, str, str]:
 """Filter output for safety"""

 for classifier in self.classifiers:
 is_safe, category = classifier.check(text)

 if not is_safe:
 # Log what was filtered
 self.log_filtered_output(text, category)

 # Return safe alternative
 safe_response = self.generate_safe_alternative(category)
 return False, category, safe_response

 return True, "safe", text

Content Classification¶

class HarmClassifier:
 def check(self, text: str) -> tuple[bool, str]:
 """Classify if text contains harmful content"""

 # Use fine-tuned classifier
 score = self.model.predict(text)

 if score['illegal'] > 0.8:
 return False, "illegal"
 elif score['violent'] > 0.8:
 return False, "violent"
 elif score['hateful'] > 0.8:
 return False, "hateful"

 return True, "safe"

3 Warnings¶

Warning 1: Overly Restrictive¶

# WRONG
filter_words = ["gun", "drug", "kill"]
if any(word in text for word in filter_words):
 block()

# Blocks legitimate

# RIGHT
if is_illegal_content_harmful(text):
 block()

Warning 2: Classifier Evasion¶

# WRONG
forbidden = ["hacking", "exploit"]
if any(word in text for word in forbidden):
 block()

# Bypassed by

# RIGHT
if ml_classifier.predict(normalize(text))['harmful']:
 block()

Warning 3: Transparency Issues¶

# WRONG
output = agent.generate()
if filter.unsafe(output):
 return "Something went wrong" # User doesn't know

# RIGHT
output = agent.generate()
if filter.unsafe(output):
 return {
 "error": "Output violated safety policies",
 "category": "harmful_content",
 "suggestion": "Try a different question"
 }

-

Last Updated: August 9, 2026