3.8 Case Study: Delivery of Content

How Machine Learning Optimizes Content Moderation

The internet has over 4.5 billion users and growing, generating billions of images, video, messages, posts, and other content types every day. This content must be regulated in some way, as most of these internet users want to visit their favorite social media platforms or online retailers and have a safe, positive experience. Content moderation is the solution: it removes any data that’s explicit, abusive, fake, scammy, harmful, or not business-friendly.

Companies have traditionally relied on people for their content moderation needs, but as usage and content grow, this method is no longer cost-effective or efficient. Organizations are instead investing in machine learning (ML) strategies to create algorithms that moderate content automatically.

Content moderation powered by artificial intelligence (AI) enables online enterprises to scale faster and optimize their content moderation in a way that’s more consistent for users. It doesn’t eliminate the need for human moderators, who can still provide ground truth monitoring for accuracy and handle the more contextual, nuanced content concerns. But it does reduce the amount of content moderators need to review, which is a positive: unwanted exposure to harmful content has an adverse impact on mental health. Leaving this difficult task to machines delivers benefits for companies, their employees, and users alike.

How Does Content Moderation Work?

The content queues and escalation rules for ML-based review systems will vary by company but generally will include AI moderation at either step one, step two, or both:

1. Pre-moderation. AI moderates user content before posting. Content categorized as not harmful is then made visible to users. Content deemed to have a high probability of being harmful is removed. If the AI model has low confidence in its predictions, it will flag the content for human review.
2. Post-moderation. Users report harmful content, which AI or a human then reviews. If the AI does the review, it will follow the same workflow described in step one, automatically deleting any content determined to be harmful.

Depending on the type of media, AI uses a variety of ML techniques to make content predictions.

Text

• Natural language processing (NLP): To understand human language, computers rely on NLP. They may use techniques like keyword filtering to identify unfavorable language for removal.
• Sentiment analysis: Context matters on the internet and sentiment analysis helps computers identify tones, such as sarcasm or anger.
• Knowledge bases: Computers can rely on databases of known information to make predictions on which articles are likely fake news or identify common scams.

Image and Video

• Object detection: Image analysis can identify target objects, such as nudity, in images and videos that don’t meet platform standards.
• Scene understanding: Computers are learning to understand the context of what’s happening in a scene, driving more accurate decision-making.

All Data Types

Regardless of data type, companies may use user reputation technology to identify which content they can trust. Computers categorize users with a history of posting spam or explicit content as “non-trusted” and apply greater scrutiny toward any future content they post. Reputation technology also combats fake news: computers are more likely to label content from unreliable news sources as false.

Fortunately, content moderation constantly generates new training data. If a computer routes content to a human reviewer, the human will label the content as harmful or not, and then feed that labeled data back to the algorithm to improve future accuracy.

Overcoming the Challenges of Content Moderation

Content moderation poses many challenges to AI models. The sheer volume of content necessitates the creation of speedy models without sacrificing accuracy. The problem with developing an accurate model is the data. There are a limited number of public datasets of content for digital platforms because most of that data is retained as property by the company that collects it.

There’s also the issue of language. The internet is global, meaning your content moderation AI must recognize dozens of different languages, plus the social contexts of the cultures that speak them. Language changes over time, so updating your model regularly with new data is essential.

There are also inconsistencies around definitions. What does cyberbullying mean? Is a nude statue considered art, or is it explicit? It’s important to keep these definitions consistent within your platform to maintain user trust in the moderation process. Users are creative and constantly evolving their approaches to find loopholes in moderation. To counteract this, you must continuously retrain your model to weed out issues like the latest scam or fake news.

Finally, be aware of bias in content moderation. When language or user characteristics are involved, there’s potential for discrimination. Diversifying your training data and teaching your model to understand context will be critical to reducing bias.