Sulat.com / Writing on AITuesday, November 17, 2026
Artificial Intelligence/17 November 2026

OpenAI releases a free CSAM classifier to vetted organisations and adds ChatGPT suspensions

OpenAI says its new classifier scores images for suspected CSAM, including AI-generated material, and is free to vetted organisations. ChatGPT now removes flagged chats and pauses the account for 24 hours.

Glossy 3D OpenAI logo beside a glass shield and a grid of frosted tiles, a teal light sweeping across and a few tiles glowing amber.

At a glance

  • OpenAI says its new CSAM image classifier returns two scores from 0 to 1, one for suspected CSAM and one for suspected AI-generated CSAM, and was built with assistance from NCMEC.
  • OpenAI estimates the classifier would flag 48% fewer images than its existing tool while finding roughly the same number of abusive images, and says the images it flags are about twice as likely to contain abuse.
  • Approved organisations get API access at no cost, and ROOST says OpenAI connected the classifier to its free, self-hosted Coop console, which adds review queues and NCMEC reporting.
  • ChatGPT now shows a warning, removes the flagged conversation and pauses the account for 24 hours, with longer restrictions, closure or reports to authorities for repeat violations, OpenAI says.

OpenAI has put a CSAM image classifier into the hands of vetted organisations for free, along with new ChatGPT interventions that pause an account for 24 hours when someone asks for the material. The classifier scores a picture for suspected child sexual abuse material (CSAM), AI-generated or not, and OpenAI announced it on 16 November in a post that also covers developer guidance and tools for child-safety investigators.

The National Center for Missing and Exploited Children (NCMEC), which runs the US CyberTipline for online exploitation reports, helped OpenAI train, evaluate and design the classifier. OpenAI calls it the first CSAM classifier with broad availability to carry a specific signal for AI-generated material; TechCrunch covered the launch too.

How the classifier works

Send it a single still image of up to 20 MB and it returns two scores between 0 and 1, one for suspected CSAM and one for suspected AI-generated CSAM. An image can score high on both. Developers call the moderations endpoint with the model omni-moderation-csam-latest; inline moderation and the Batch API aren’t supported.

The model card describes an internal CLIP image encoder (the part that boils an image down to a numerical summary) feeding a three-layer multilayer perceptron, a small neural network that turns the summary into scores. Training used roughly 505,000 positive and 913,000 negative images from access-controlled, human-labelled sets. Whatever customers send to the API is processed without retention and isn’t used for training or evaluation.

Four steps: image in, classifier scores it, trained people review, report to NCMEC.
The classifier gives reviewers two scores; people decide what gets reported.

A high score isn’t a verdict. The scores aren’t calibrated probabilities, a flag doesn’t establish that an image is CSAM or illegal where the reviewer works, and the AI-generated score isn’t proof of where an image came from. The classifier reports nothing to authorities by itself. Per the model card, anyone filing NCMEC or hotline reports from its output still needs trained, authorised human review, and the classifier is meant to run next to known-image hash matching and other abuse detection.

What the tests found

In OpenAI’s comparison on user-uploaded images, the new classifier would flag 48% fewer images than the existing tool while finding roughly the same number of abusive ones, and what it flagged was about twice as likely to contain abuse. Three early-access partners ran their own checks, listed below.

MeasureResultReported by
Images flagged, against OpenAI’s existing tool48% fewer, for about the same abuse foundOpenAI
Flagged images that contain abuseAbout twice as likelyOpenAI
High-confidence flags confirmed as novel CSAM72%fal, on human-reviewed production images
Drawings and animations that sexualise children detected90%SafetyKit
Flagged cases that led to a banOne in fourinvideo AI
Figures from OpenAI’s announcement; the 48% is OpenAI’s estimate.

The model card also has held-out numbers. On roughly 46,849 labelled images, the default threshold of 0.996324 on the suspected-CSAM score found 80.2% of the positives, 5,490 of 6,849. The AI-generated score uses 0.90 by default and found 86.4%, or 1,966 of 2,275. OpenAI also ran about 1.25 million open-web images from the Re-LAION-5B-research dataset through it: the first score flagged five and the second flagged none.

Stat cards: 48% fewer images flagged, twice as likely to contain abuse, 72% confirmed, 90% of drawings caught, one in four led to a ban.
OpenAI estimates 48% fewer flagged images for about the same abuse found.

Who can get it

To get in, an organisation applies through OpenAI’s intake form, and there’s no charge for approved applicants. The Tech Coalition and ROOST are helping OpenAI share the tool, and OpenAI backs others’ work as well, including Thorn’s Safer detection system. It has also joined the Tech Coalition’s Lantern programme, in which member companies swap signals about online child exploitation across platforms.

ROOST, the nonprofit behind the free, self-hosted Coop console, says OpenAI connected the classifier to Coop at launch. Coop bundles review queues, automated enforcement rules and NCMEC reporting, so a small team can detect, review and report in one tool. ROOST also says OpenAI checks each applicant for NCMEC reporting, manual review and current stack, and calls the model card the first for a CSAM classifier.

What ChatGPT does now

ChatGPT now answers a request for CSAM with a warning that sexualising children is harmful, even when the material is AI-generated, and can be illegal. The chat ends and is removed, and access to ChatGPT pauses for 24 hours. OpenAI’s help centre adds that a signed-in user’s conversation disappears from their history and can’t be reopened. Repeat violations can bring longer restrictions, account closure or a report to authorities.

A suspended user receives an email with the reason and an appeal link; appealing doesn’t restore access on its own. The warning also points to specialist support for people who want to stop, and the help centre names ReDirection, Troubled Desire and Stop It Now, plus the NO MORE Global Directory and Child Helpline International for people affected by abuse.

Five steps: warning, chat removed, 24-hour pause, longer limits for repeats, account closure or report to authorities.
A first violation means a warning, a removed chat and a 24-hour pause; repeats escalate.

The interventions run in Chat and Work mode on web and mobile for personal accounts, including signed-out visitors. They don’t reach ChatGPT Business, Enterprise, Edu or Healthcare workspaces yet, or voice mode, Codex and the desktop apps. The rollout is gradual, and the screens can differ by country.

OpenAI built the approach with Protect Children, the Lucy Faithfull Foundation, Dr Caoilte Ó Ciardha of the University of Kent and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of deterrent messages.

Guidance for developers and investigators

Developers get a set of CSAM guidance written with the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation. It covers rules, user reporting, detection tools and review and reporting processes, and explains how safety identifiers tie misuse to individual users. OpenAI says the guidance has already led more developers to adopt child safety tooling and register to report to NCMEC. Separately, OpenAI, Anthropic and Google jointly commissioned Tech Coalition guidance on handling disclosures of child sexual exploitation and abuse in AI systems.

Child-safety law enforcement units get new tools and training: dedicated plugins, investigative tools and complex data analysis, with access managed by programme administrators. The first tool reads through long chat logs and legal process returns, spots potential connections and builds timelines, each finding linked back to its source. Training starts as an overview and demonstration with a field guide, and access depends on agency authorisation and programme requirements.

Organisations that want the classifier apply through the intake form; OpenAI says it will keep improving the classifier with feedback from users.

Sources

Disclosure: I’m an OpenAI, Firecrawl and Devin ambassador.


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.