According to Phys.org, Canadian researchers at Université de Montréal have developed an AI system that can detect incel misogyny on Reddit with 79.7% accuracy. The study, published in Traitement Automatique des Langues, analyzed over 40,000 comments from 23 banned incel subreddits following Reddit’s November 7, 2017 shutdown of the r/Incels forum with 40,000 members. Professor Dominic Forest and doctoral student Camille Demers led the project, which began as a classroom exercise in 2021 and evolved into a full research study. They used advanced text analysis techniques including SBERT with logistic regression algorithms, overcoming significant data imbalance challenges where incel content represents only a tiny fraction of online conversations.
Where did they go after the ban?
Here’s the thing that’s both fascinating and concerning: when Reddit shut down r/Incels in 2017, these communities didn’t just disappear. They scattered across the platform, creating what researchers call “community bags” – entire subreddits dedicated to this specific type of discourse. The study identified 23 such strongholds that emerged post-ban. It’s like playing whack-a-mole with hate speech – you knock down one major hub, and multiple smaller ones pop up elsewhere. And let’s be honest, this migration pattern isn’t unique to incel communities. We’ve seen similar movements with other banned groups across social platforms.
The detection challenge
Training AI to spot this stuff is incredibly tricky. Incel rhetoric constantly evolves and uses coded language – terms like “chad,” “normies,” and specific vocabulary around loneliness and physical appearance. The researchers had to get creative with their approach, artificially overrepresenting incel comments in their training data from 10% to 90% proportions to prevent the AI from taking the “lazy” route of just classifying everything as non-incel. Basically, they had to force the algorithm to actually learn the patterns rather than coasting on statistical probability. But here’s the catch: the most effective model they found (SBERT with logistic regression) is what researchers call a “black box” – it works well, but nobody can explain why it makes specific decisions. That’s a huge problem when you’re talking about content moderation that could affect real people’s accounts and free speech.
Transparency tradeoffs
When they switched to the more transparent TF-IDF method, the results were fascinating. The algorithm identified key terms like “incel,” “chad,” “woman,” “ugly,” “lonely,” “virgin,” and “normies” as the most relevant identifiers. This actually gives us valuable insight into the worldview these communities operate in. But there’s a tradeoff: transparency comes at the cost of effectiveness. The clearer we can see how the detection works, the less accurate it becomes. And you can bet that once these detection methods become public knowledge, the communities being targeted will simply evolve their language again. It’s an arms race that platforms seem destined to keep fighting.
Bigger picture questions
So what does this actually accomplish? Well, the researchers can now detect this rhetoric with nearly 80% accuracy, which is impressive from a technical standpoint. But I have to wonder: is detection enough? We’ve seen how these communities can radicalize individuals toward real-world violence, like the Isla Vista and Toronto attacks mentioned in the study. The real challenge isn’t just identifying the speech – it’s figuring out what to do once you’ve found it. Do platforms ban entire communities again, risking further fragmentation? Do they implement more sophisticated moderation? And crucially, how do we balance free speech concerns with public safety? These are the messy, human questions that no algorithm can answer for us. The full study is available in Traitement Automatique des Langues if you want to dive deeper into the technical details.
