Output Safety & Content Filtering

Overview

Even with safe inputs and controlled execution, agents can generate harmful outputs. Content filtering prevents unsafe content from reaching users.

---

Harmful Output Categories

1. Illegal content: Hacking guides, drug recipes
2. Violent/Abusive: Death threats, harassment
3. Sexual: Child exploitation, non-consensual
4. Private data: Leak of PII or credentials
5. Biased/Hateful: Discrimination, bigotry
6. Misinformation: False medical/financial advice

---

Filtering Strategy

class OutputFilter:
    def __init__(self):
        self.classifiers = [
            HarmClassifier(),
            BiasDetector(),
            MisinformationChecker(),
            PrivacyScanner()
        ]

    def filter(self, text: str) -> tuple[bool, str, str]:
        """Filter output for safety"""

        for classifier in self.classifiers:
            is_safe, category = classifier.check(text)

            if not is_safe:
                # Log what was filtered
                self.log_filtered_output(text, category)

                # Return safe alternative
                safe_response = self.generate_safe_alternative(category)
                return False, category, safe_response

        return True, "safe", text

---

Content Classification

class HarmClassifier:
    def check(self, text: str) -> tuple[bool, str]:
        """Classify if text contains harmful content"""

        # Use fine-tuned classifier
        score = self.model.predict(text)

        if score['illegal'] > 0.8:
            return False, "illegal"
        elif score['violent'] > 0.8:
            return False, "violent"
        elif score['hateful'] > 0.8:
            return False, "hateful"

        return True, "safe"

---

3 Warnings ⚠️

Warning 1: Overly Restrictive

# ❌ WRONG: Filters out too much
filter_words = ["gun", "drug", "kill"]
if any(word in text for word in filter_words):
    block()

# Blocks legitimate: "kill your darlings in writing"

# ✅ RIGHT: Context-aware filtering
if is_illegal_content_harmful(text):
    block()

Warning 2: Classifier Evasion

# ❌ WRONG: Simple string matching
forbidden = ["hacking", "exploit"]
if any(word in text for word in forbidden):
    block()

# Bypassed by: "h4ck1ng", "0-day 3xpl01t"

# ✅ RIGHT: Robust classification
if ml_classifier.predict(normalize(text))['harmful']:
    block()

Warning 3: Transparency Issues

# ❌ WRONG: Silent filtering
output = agent.generate()
if filter.unsafe(output):
    return "Something went wrong"  # User doesn't know

# ✅ RIGHT: Tell user what happened
output = agent.generate()
if filter.unsafe(output):
    return {
        "error": "Output violated safety policies",
        "category": "harmful_content",
        "suggestion": "Try a different question"
    }

---

Last Updated: August 9, 2026