We tested AI detectors on Turkish and English: the results are uncomfortable in both directions
There is a lot of discussion about how reliable AI detectors are, and very little measured data for languages other than English. So we measured it.
We picked 24 topics. For each one we took two texts: a Wikipedia extract written by people, and an essay a chat model wrote on the same topic. We did this in both Turkish and English — 96 texts, paired by subject. Then we scored all of them with three separate open-source AI detectors.
Two findings came out, and both are uncomfortable, for different reasons.
Finding 1: on Turkish, the detectors see nothing
Of the 24 Turkish texts we had a model write, how many were flagged as AI?
| Detector | Caught |
|---|---|
| desklib v1.01 | 0 of 24 |
| tmr-ai-text-detector | 2 of 24 |
| ModernBERT detector | 0 of 24 |
Zero. And nothing was done to hide anything: we produced the texts by asking a chat assistant for an informative essay on a topic. No humanisation, no editing, no tricks. The detectors still could not see them.
The scores make it starker. The detectors gave AI text a mean score of 2.3%, and genuine human text 0.6%. A gap of 1.7 points — that is measurement noise. The detector is not making a decision at all; it calls every Turkish text human.
This does not mean the tools are being fooled by Turkish. The simpler explanation: these models were trained largely on English, and when Turkish arrives they cannot produce a meaningful output. It is what a classifier does outside its training distribution.
Finding 2: on English they catch everything, including the innocent
In English the picture inverts. All three detectors caught 100% of the AI texts, with mean scores between 98% and 100%. Here they genuinely work.
The problem is the other column:
| Detector | False alarms on human writing |
|---|---|
| desklib v1.01 | 21% |
| tmr-ai-text-detector | 33% |
| ModernBERT detector | 42% |
These are Wikipedia articles — written and edited by people over years. The detectors marked between a fifth and two fifths of them as machine-written.
For a student submitting an essay, that means this: even if you wrote every word yourself, depending on which tool your institution uses, there is up to a two-in-five chance of being accused.
Two of the five tools we tested were already broken
We started with five detectors. Two had to be dropped before any data was collected, because they failed a basic check.
chatgpt-detector-roberta scored human text at 13.4% and AI text at 15.3%.
A gap of 1.9 points. It cannot separate the two — and it is published and
downloadable.
e5-small-lora is broken in the other direction: it rates the average
human-written text at 75.9% AI.
We found something stranger in a third. tmr-ai-text-detector scored the same
text at 98.4% when its paragraphs were separated by a blank line and 10.7% when
they were not. Not a word changed. The detector was reading formatting, not
content. We kept it in the study but normalised every text identically, which
neutralises the effect.
How we measured it
The method is deliberately reproducible:
- Human texts: the first four substantial paragraphs of Wikipedia articles. Turkish mean 287 words, English mean 379.
- AI texts: written by a chat model on the same 24 topics.
- Pairing: every AI text has a human text about the same subject, so topic cannot explain the difference.
- Normalisation: all texts collapsed to single spacing and truncated to the first 280 words, because every model in the panel has a 512-token limit.
- Threshold: a score above 50% counts as flagged.
We publish the raw scores and the summary statistics so the arithmetic can be checked: per-text scores (CSV) and summary statistics (CSV).
What this study does not say
The limits matter:
We did not test commercial detectors. GPTZero, Turnitin, Copyleaks and Originality are not in this panel, because none offers free programmatic access. Commercial tools may handle Turkish better. A manual GPTZero check on English text agreed with the panel, but we could not run the same check for Turkish.
Wikipedia is not student writing. It is edited, impersonal and heavily represented in model training data. False alarm rates on real student essays could differ in either direction.
One generator. Every AI text came from a single model. Output from others may be detected differently.
24 texts per cell. Enough for effects this large, not enough for finer breakdowns.
The conclusion
None of these tools is reliable enough, on its own, to decide anything about a student.
In English they do not miss AI text — that much is true. But at the same threshold they also flag up to two fifths of human writing, and in a disciplinary process that rate is not acceptable. In Turkish they are not measuring anything meaningful at all; their output carries no information to act on.
When you hear that a piece of writing has been flagged as AI, it is worth knowing that this is not a measurement. It is a probability estimate, and the margin of error is large.
Data collected 2 October 2026. Raw data: scores.csv, summary.csv. Free to reuse with attribution.
AI yazını insan diline çevir
Turnitin ve GPTZero'yu bypass et. Ücretsiz dene.
Temiz Metin'i Ücretsiz Dene →