Building an AI Text Detection System for Content Moderation and Platform Trust
- ~96% detection accuracy on client data
- ~1.2% false positives on human writing
- <800 ms per document at p95
- ~14 hrs/week saved per moderator
Currently, publishers, user-generated content communities, and educational institutions pose the same question to each submission: Was this piece written by a person? We have developed AI content detection software that responds to this question sentence by sentence in multiple languages and provides moderators with a short reason.
Project Details
A global platform—which hosts user-generated content for a number of large online communities, produces editorial material for enterprise customers, and operates an LMS used by universities—had approached us because its moderation team was individually pasting suspected submissions into various free AI checkers and then debating the outcomes. The client is bound by a non-disclosure agreement. Their requirement was for an AI detector to be incorporated into their own process, providing them with scores that could be justified to a user, an author, or a dean.
Business Challenge: Detecting Synthetic Text at Platform Scale
The platform received about 1.4 million text submissions each week, including forum posts and long-form articles from the user-generated content side, press releases, product blogs and research papers from enterprise publishing customers, as well as student assignments via the learning management system. Over the course of one spring the proportion of items that the moderators suspected to be AI-generated rose from a negligible amount to around one in five. The fact that it could not be measured was the aspect that concerned the client most.
About 40 moderators were responsible for processing the queue. Each of them dealt with nearly 300 flagged items per week, spending between nine and eleven minutes on each one: first reading it, then copying it into a free tool, waiting, looking at the result as a percentage, copying that percentage into a second free tool since the first one seemed suspicious, and finally writing a note which no one further down the line could act upon. The median time it took to get an enterprise article approved was 19 hours. Two universities had already stated that they would not renew their contracts unless teachers were provided with something better than a number with no explanation attached. On the publishing side, readers were leaving comments such as “this reads like ChatGPT”, and in most cases the editors had no means of checking.
Telling Synthetic Patterns Apart From Human Writing
Free AI detectors flagged formal writing, in particular that of non-native speakers, about 9% of the time. They had even more difficulty with short posts, lists, and legal boilerplate, and when it came to detecting AI in text that had been paraphrased by AI, the situation was almost a coin flip.
Supporting Content Moderation Workflows
It's not a decision merely to assign a percentage without giving any reasons. Moderators needed sentence-level highlights, a confidence band, and the ability to disagree, all within the queue that they were already working in instead of having to use a separate browser tab that they kept open.
Aligning Detection With Policy and Compliance
For a university, a flagged essay is regarded as an academic integrity issue; for a publisher, it is seen as an editorial question. A single threshold, policy, and appeals procedure couldn't apply to both situations at the same time.
Technologies Behind the AI Content Detection System
Our AI Content Detection Software Solution
The question wasn’t ever ‘Is this AI?’ rather it was ‘What should a reviewer do in the next thirty seconds?’ This way of putting it determined most of the design. A simple probability is of no use to a teacher who has to read 120 academic essays before Monday and also of no help to an editor deciding whether to publish an article. What is needed are a number, a range, and highlighted sentences.
One score for two families of signals. The statistical component analyses approximately 60 language features extracted from the raw text, such as perplexity with respect to a reference model, burstiness, variation in sentence length, distributions of token ranks, the type-token ratio, and the use of function words which distinguish human writing from the smooth and even output of generative AI tools. This component takes about 40 milliseconds to run and does not require a GPU. The deep learning layer is a transformer classifier that has been fine-tuned on about 4.2 million documents, half of which are human-written and come from the client’s pre-2021 archives, the other half having been generated across around 30 model families and versions, including ChatGPT-type assistants, open-weight models, and three AI paraphrasing services. For details on training, validating, and deploying ML models, see our development software machine learning page.
No third-party benchmark decided whether the thing worked. The holdout set came from the client’s own traffic: 180,000 documents the moderation team had already ruled on, split by content type and language. The combined model reached roughly 96% accuracy on it, with a false-positive rate on human-written text near 1.2%. The client cared more about the second figure than the first, because a wrongly flagged student costs more than a missed bot.
The AI text detection software also knows when to stay quiet. Below about 120 words, the probability estimate becomes unreliable, so the engine returns a “too short to score” state instead of a guess. Reviewers told us during discovery that a confident wrong answer on a two-line comment did more damage than no answer at all. The AI software development work that followed kept coming back to that rule.
Statistical NLP Analysis
About 60 features per document (ranging from perplexity and burstiness to sentence structure and punctuation rhythm) are calculated in around 40 milliseconds. This layer detects the flat and uniform rhythm, which usually reveals AI-generated text before any neural network is even used.
Deep Learning Classifier of Writing Patterns
A transformer has been fine-tuned on about 4.2 million labeled documents in nine languages, such as French, Spanish, German, and Portuguese; it is able to interpret context beyond what the statistical layer can do, for example, by understanding how an argument unfolds across paragraphs, and it provides a per-sentence probability map.
Content Authenticity Scoring Engine
A calibration layer combines the two signals into a 0 to 100 authenticity score, dividing this score into three categories: likely human, mixed, and likely AI. Each score is given a version number so that a result obtained in March can be reproduced in September, for example, when a student makes an appeal or a publisher requests an audit.
Reviewer Feedback Loop
Moderators and teachers have the ability to cancel a flag with a single click and give the reason for doing so; around 6,000 of these corrections were used in the first six months of retraining, and the false-positive rate for non-native academic writing dropped by about one-third as a result.
Integration With Existing Workflows
There is a REST API for batch checks, a streaming API for live queues, a CMS plugin for the publishing side, and an LMS plugin which is placed alongside the plagiarism checker and grammar checker that teachers currently use. The scores end up in the moderation queue and no one opens a second tool.
Data Privacy, Security, and Governance
By default, submitted text is held for 30 days, while university tenants have it held for zero days under their contract. Each tenant has its own encryption keys and region; each score includes an audit trail, and anything that a user submits is never used to train a model for another customer.
— Scrum Methodology
Project Journey
The engagement, which lasted 14 months, was carried out in 28 sprints of two weeks each, with the first three focusing completely on building the corpus since an AI detector is only as effective as the human-written baseline upon which it was trained. Together with the client’s trust and safety lead, two university customers and one enterprise publisher, the criteria for what constituted a flag, what constituted an appeal, and how long any text could be stored were defined.
How the AI Content Detection System Works
- A post, article, or assignment arrives through the API, the CMS plugin, or the LMS plugin, and lands on a Kafka topic within a second of submission.
- The engine computes about 60 features, from perplexity and burstiness to sentence structure, and checks that the document is long enough to score at all.
- The fine-tuned transformer reads the full text in context and returns a probability for every sentence, in whichever of the nine languages it detects.
- A calibration layer merges both outputs into one 0 to 100 score, assigns a band, and highlights the sentences that drove the result.
- The score appears inside the moderation queue or the teacher's gradebook with its reasons. One click confirms the flag or overturns it, with a short note.
- Confirmed and overturned flags feed a weekly retraining job, so the detector tracks what moderators actually decide rather than last quarter's assumptions.
Scrum Process Flow
Since generative models evolve more rapidly than any annual release plan allows, a SaaS product of this type could not have been developed once and then handed over. Each sprint ended with a demonstration based on live submissions obtained that morning, and in three cases these demonstrations led to actual changes in the band thresholds.
How We Delivered AI Text Detection Software
- We agreed on content types, languages, the storage rules each tenant needed, and a 14-month plan with a pilot at month eight.
- Long-form articles and student assignments came first, since that is where a wrong call costs the most. Short comments and listings followed.
- Each sprint paired one model experiment, such as a new feature set or a new generator in the corpus, with one shippable piece of the product.
- Features were built against live traffic and checked by QA against the labeled regression set before any score reached a reviewer.
- Trust and safety, two universities, and one publisher reviewed every demo and signed off on thresholds, appeal flows, and retention terms.
- Pilot with five tenants, then the rest over ten weeks, with weekly retraining and a drift alert that fires when accuracy moves more than two points. These practices are part of our <a href="https://litslink.com/services/ai-development-services">AI development services</a>, which cover deployment planning, model monitoring, and ongoing improvement.
UI/UX Design: Making AI Detection Results Understandable
The real danger was adoption. Detection tools gain trust only on the condition that a person can see the reason why a sentence has been flagged, and a correct detector which is ignored by moderators is no more than another tab. To do this before producing the screen, the designer spent a week sitting down with six moderators and four teachers to observe the way they actually read. For another example of NLP in practice, explore our AI-powered sentiment analysis solution, which scores customer reviews and aggregates the results by product.
Submission and Results Screens
When the results view is opened, it shows the score, the band, and the text itself, with the sentences that have been flagged being shaded in three different shades rather than one uniform color. As the reviewers pointed out, a document that has a 40% chance of being AI doesn’t look at all like one that has a 90% chance, and the previous system masked this difference. While the cursor is over each line, the line’s individual probability is displayed, the word count is shown at the top since the ‘too short to score’ condition needs to be explained, and the entire document scrolls in place so that people don’t lose their position. Batch uploads involving up to 500 documents produce a sortable list using the same bands.
Explainability and Reviewer Tools
The top three reasons on every flag are given in plain language; for example, rather than saying “feature_22: 0.41”, they state “sentence length varies less than 95% of human writing in this category”. There is a side panel which shows how the score was arrived at, with the statistical signal and the neural signal displayed side by side, and there is an appeals tab that contains the full record of who confirmed or overturned each item and when. Teachers requested one additional feature – a comparison with the student’s previous submissions – and this was included in sprint 19 as a subtle line beneath the score.
Accessibility and Internationalization
Text labels are also included together with color, as one moderator raised the issue of colorblindness during the first week of shadowing. The dashboard complies with WCAG 2.1 AA, supports right-to-left for the Arabic pilot, and is available in six languages, while the detector supports nine. The date formats, number formats, and the phrasing of a flag all adhere to the tenant’s local preferences since a flag which sounds like an accusation in one culture is seen as a normal check in another.
Results
Before
- ✕~1.4M submissions a week, with suspect items pasted into free AI checkers by hand, 9 to 11 minutes each
- ✕~9% false positives on human-written content, and no way to tell which flags were wrong
- ✕Median approval time of 19 hours for enterprise articles; student work waited up to three days
- ✕~300 flagged items per moderator per week, 40 moderators, no shared record of decisions
- ✕Two university customers at risk of churn over unexplained scores
After
- ✔~96% document-level accuracy and ~1.2% false positives on the client's own holdout set
- ✔Scores in under 800 ms at p95, inside the queue, with highlighted sentences and reasons
- ✔Median approval of 40 minutes for enterprise articles; same-day turnaround on student assignments
- ✔~14 hours a week back per moderator, from ~45 hours of review to about 30, with the queue cleared daily
- ✔Both universities renewed, and four more institutions signed within two quarters
Impact on Trust, Compliance, and Operations
— Verified Reviews
Our Reputation on Top Platforms
LITSLINK holds a 4.8 rating on top platforms. Clients writing about our machine learning development work most often mention technical depth, steady communication across long engagements, and a willingness to say when the data does not support a requirement.
Have an AI Content Detection Project in Mind?
Need AI content detection software for a publishing platform, a UGC community, or an LMS, or an authenticity score that teachers and editors can defend? Tell us what you are working with, and a LITSLINK specialist will get back to you within 48 hours.
Thank you for your message. It has been sent.
