LLMs favor automod (full)

## User
So reddit is trying to sunset automod in favor of LLMs. As an LLM what's your opinion on this? My biggest complaint is that LLMs still struggle with context and some some subs can get 5000+ comments a day.

## Assistant
I think it depends entirely on what they're replacing.

Automod is extremely good at things that are deterministic:

"Remove posts with fewer than 100 karma."
"Filter slurs."
"Require a flair."
"Block links to these domains."
"Hold comments containing these phrases."

Those are basically perfect use cases for rule-based systems. They're fast, predictable, cheap, and moderators understand exactly why something was removed.

LLMs become valuable where Automod completely falls apart:

Detecting harassment that doesn't use banned words.
Recognizing evasion ("unalive", deliberate misspellings, coded language).
Understanding whether a comment is actually answering the OP.
Detecting AI-generated spam.
Catching scams that constantly change wording.

That's where context matters.

The problem—and I think this is what you're getting at—is that current LLMs have finite context windows and imperfect long-term reasoning.

For a subreddit with 5,000+ comments per day, several issues appear:

Community culture isn't in the prompt. Every subreddit develops unwritten norms. A comment that's perfectly acceptable in r/ProgrammerHumor might be a ban-worthy insult elsewhere.
Conversation history gets lost. If an argument has been escalating over 30 comments, an LLM may only see the last few and misidentify who started it.
User history matters. Someone with ten prior warnings saying "you're an idiot" is different from a long-time contributor making one sarcastic remark.
Moderator precedent matters. Human moderators tend to become internally consistent over time. An LLM has to either retrieve thousands of previous decisions or risk inconsistent moderation.

Your point about 5,000+ comments/day is also a scaling issue. Even if the model technically has a million-token context window, feeding it an entire day's worth of discussion for every moderation decision would be prohibitively expensive and too slow. In practice, systems use retrieval:

Current comment
Parent chain
Relevant rules
Possibly recent history for that user
Maybe a few similar past moderation decisions

That works surprisingly well, but it's still not the same as "knowing the subreddit."

Personally, I don't think LLMs should replace Automod. I think they make more sense as a second layer:

Automod handles deterministic rules.
LLM reviews the gray-area cases.
Human moderators make the final decision on high-confidence removals or appeals.

That's a much more robust pipeline than asking an LLM to moderate everything.

One thing I suspect Reddit is really after isn't just better moderation—it's reducing moderator workload. Large subreddits generate an enormous moderation queue, and even cutting the number of human reviews by 30–50% would save moderators a huge amount of time. The challenge will be maintaining trust. If moderators can't understand why the model removed something, or if it behaves inconsistently, they'll likely end up spending time reviewing the AI's mistakes instead of the original reports.

## User
Just making sure you're not just agreeing with me. What other angles could be considered?

## Assistant
It's a fair concern. There are other angles that could support Reddit's decision, and I don't think the outcome is obvious.

One argument is that LLMs are already outperforming simple rule systems in many moderation tasks. Even though they make mistakes, they can recognize context, sarcasm, insults, threats, and spam far better than keyword filters. Reddit may have internal data showing that an LLM catches significantly more genuine rule violations while generating fewer false positives than Automod alone.

Another consideration is that most moderation decisions don't require huge amounts of context. While some discussions span hundreds of comments, many reports are fairly local:

A parent comment.
The reply.
The subreddit rules.
A few previous messages in the chain.

That might be enough for 90–95% of reported content. The remaining difficult cases can still go to human moderators.

There's also the fact that Reddit has access to much more information than moderators do. They can incorporate:

A user's moderation history across Reddit.
Previous bans and warnings.
Signals that identify spam campaigns.
Network behavior across multiple accounts.
Voting and reporting patterns.

An external LLM or a moderator doesn't have that complete picture. That additional data could substantially improve moderation accuracy.

There's also an economic angle. Maintaining Automod isn't free either. Every time users invent new ways around filters, moderators update regexes and keyword lists. An adaptive model may require less manual maintenance over time, especially against spam and scams.

Where I think your concern is strongest is consistency.

Current LLMs are probabilistic. If you ask the same model the same borderline question multiple times, you can sometimes get different answers. Moderation systems generally want:

The same rule.
The same input.
The same outcome.

That predictability is something rule-based systems excel at.

Another challenge is community identity. Reddit isn't one community—it's hundreds of thousands of communities. r/AskHistorians, r/science, and r/shitposting have completely different expectations. An LLM either needs:

A carefully engineered prompt for every subreddit,
Fine-tuning on each community,
Or retrieval of subreddit-specific examples.

That's technically feasible, but it's considerably more complex than replacing Automod with "an AI."

One thing I wouldn't dismiss is that Reddit almost certainly has access to an enormous amount of moderation data. Millions of moderator actions over many years are an excellent training and evaluation dataset. They can measure questions like:

Did moderators agree with the AI?
How often was an AI decision appealed?
Which subreddits benefited?
Where did it fail?

Those are things neither of us can really evaluate from the outside.

So if I had to separate what I think is likely from what I'm uncertain about:

I think LLMs will outperform traditional Automod for nuanced language, spam detection, and rule interpretation.
I think rule-based systems will remain better for objective, deterministic rules.
I think the biggest unresolved problem is consistency across vastly different communities and over long conversational context, not whether LLMs can understand individual comments.

So I don't think your concern about context is misplaced. I just don't think it's necessarily the deciding factor. Reddit may have evidence that most moderation decisions don't actually need as much context as we intuitively expect, and if that's true, the tradeoff could still be worthwhile.

## User
Wanted to reiterate. Automod is being sunsetted. Which means either immediately or eventually it will be phased out. You've stated you believe a combination of both is important. Do you think getting rid of automod is a good idea?

## Assistant

Given that premise—that Automod itself is eventually going away—I'd be skeptical of the decision, but not because I think LLMs are incapable moderators.

My concern is that Automod and LLMs solve different classes of problems.

Automod is essentially a deterministic policy engine. If a moderator writes:

Remove all posts without flair.
Hold posts from accounts younger than 7 days.
Remove any title containing X.
Filter domains Y and Z.

The outcome is guaranteed. There's no ambiguity. That's a desirable property for many moderation rules.

An LLM can certainly emulate those rules, but emulation is different from execution. The model has to interpret the rule correctly every time. Even if it succeeds 99.9% of the time, on a platform the size of Reddit that remaining fraction could translate into a large number of inconsistent decisions.

The strongest argument for removing Automod is that Reddit may not actually be removing deterministic moderation—they may simply be replacing the implementation. An LLM-backed system could still expose "if-then" style rules to moderators while using other components under the hood. In that case, the loss of Automod is mostly a product change rather than a capability loss.

If, however, they are literally saying "from now on, the AI decides whether this violates your rules," then I think moderators lose something valuable: predictability.

There's also a governance issue. Moderators often want rules they can reason about:

"Why was this removed?"
"Which rule fired?"
"How do I change the behavior?"

A regex or rule list answers those questions directly. A language model's reasoning is harder to inspect and harder to tune. Even with explanations, there is usually another layer of interpretation involved.

That said, there are reasons Reddit might still conclude it's the better direction:

They have access to billions of moderation decisions and can benchmark models against real moderator actions.
Spam, scams, and ban evasion are becoming increasingly adaptive, making static rules less effective.
They may find that the reduction in moderator workload outweighs occasional inconsistencies.

Those are legitimate tradeoffs.

So if I were designing Reddit from scratch today, I would probably build a system like this:

Deterministic policy engine for objective rules.
LLM reasoning layer for subjective rules.
Confidence scoring so uncertain cases go to moderators.
Transparent logs explaining every action.

Notice that "deterministic policy engine" doesn't have to be called Automod. It could be an entirely new implementation. But I would still want something that guarantees objective rules are applied exactly as written.

So my skepticism isn't really about removing Automod the product. It's about removing deterministic moderation. If that's what's happening, I think Reddit is giving up a capability that complements LLMs rather than competes with them.

One thing I'd also watch for is whether Reddit is actually sunsetting the AutoModerator software or merely deprecating its rule language in favor of a new moderation platform. Those are very different technical changes, even though they can sound similar in announcements.