How AI Detectors Work (and the AI vs. Human Content Debate)

How AI Detectors Work cover image

Recently, I put together an SEO content study, looking at nearly 3,000 top 10 links for B2B finance keywords, and I discovered something unexpected. Each AI detector solution I tried offered vastly different results. 

For example, a GoCardless article on anti-money laundering, which ranked 3rd for the keyword money laundering life cycle, had 3 drastically different results on whether AI likely wrote it:

  • Originality.ai: 18% 
  • On-Page.ai: 0.20% 
  • KazanSEO: 86%

This article dropped 5 ranks (from position 3rd to 8th) after the March 2024 update.

But it wasn’t the only inconsistency. 

Looking at another keyword, global allocation fund, a T. Rowe Price investment fund page also displayed significant disagreement in AI-author likelihood:

  • Originality.ai: 38% 
  • On-Page.ai: 98.60% 
  • KazanSEO: 13%

In contrast, this page’s ranking went up—from rank 4 to 3 for its keyword.

So, it’s needless to say, I wanted to understand these tools better.

I may not be using AI content detectors as a writer (I know whether I am using AI or not). But clients, prospects, and competitors are. And as an SEO-friendly content writer, I wanted to know more about how AI (and its detectors) worked. 

In this article, I’ll summarize:

  • How AI detection works
  • What this means for our understanding of AI
  • What it adds to the Human vs. AI content discussion. 

How do AI Detectors Work?

First, each AI detector works differently. In this experiment, I used 3 programs:

Each solution runs on a different dataset and model. However, there is overlap in the overall technology.

AI detectors typically use the following to analyze a text:

  • Classifiers and labels to identify patterns
  • Embeddings to map words to data points for pattern recognition
  • Pattern perplexity to determine text unpredictability (for measuring humans)
  • Pattern burstiness to evaluate sentence variation, word diversity, and other stylistic elements

It’s also important to note that AI detectors offer a general confidence score. 

So, for example, if a text is said to be 40% likely to be written by AI, this does not mean that 40% of the text is AI-written. It’s a general probability that AI was involved in the entire article. 

In other words, AI and AI detectors analyze patterns in a text. The more words there are, the more it has to go on — but there are many complications. 

An image listing how ai detectors work. They: Classify data to identify patterns, map words to data points, analyze unpredictability for humanness, analyze burstiness for style and variation, use confidence scores

False positives tell us much about generative content, not just AI detection. For example, according to Originality.ai, false positives can be caused by:

  • Using any AI tool or word processor (Grammarly, ChatGPT, Microsoft Word)
  • Writing short content
  • Creating formulaic content
  • Use of public domain materials
  • Using academic content

When we understand that current AI and AI detection are built on pattern recognition, it becomes easier to understand how they can be fooled or misunderstood.

How to Beat AI Detectors: Being Human

For example, unpredictable or surprising patterns register as more human. These “surprising” results, however, can also be errors or poor word choice. One study by Top Content found that rough drafts received higher “human-ness” scores than proofread ones. But following grammar conventions and writing well isn’t unique to AI. A publishable piece shouldn’t suddenly be attributed to AI simply because it is error-free. 

Consider this: What AI detectors look for is what defines current generative AI. Therefore, doing the opposite of that can help avoid detection, even if you are using AI-driven content tools:

  1. Using any AI tool or word processor —> Don’t use direct suggestions; be selective at recommendations and rephrase the generated text
  2. Writing short content —> Write longer-form content or add to AI-generated content
  3. Creating formulaic content —> Rephrase and reformat to mix things up
  4. Use of public domain materials and academic content —> Paraphrase and cite public domain content when possible, but focus on including unique perspectives (such as expert quotes)

You may also want to avoid:

  • Content and word repetition
  • Jargon
  • Passive voice
  • Lack of sentence variety

It’s also essential to note tasks generative AI generally does not do, such as accurate sourcing or scanning for plagiarism. These may or may not tip off the AI detectors, but they will certainly look bad to your audience. 

What AI Tells Us About Human-Created Content

What AI detection tells us about artificial intelligence generates content that’s often:

  • Repetitive in topic and sentence/word variety
  • Overly complex or stale
  • Inaccurate or too general
  • Predictable

AI-generated solutions also need help with plagiarism since the text is pulled from a database, and the source is never cited. In addition, artificial intelligence lacks the ability to explain its decisions. While classifiers and labels can help AI broadly identify certain objects or ideas, the machine lacks a human understanding of the world to always provide accurate output. 

In his book Smart Until It’s Dumb, AI engineer and consultant Emmanuel Maggiori says:

“Machines learn statistical regularities in how people have done the job before, but this doesn’t give them the broad knowledge of the world required to truly excel at the tasks. So, machines may dupe us for a while because they’re good pretenders, but at some point, they end up making silly mistakes a human would never make. At that point, it becomes clear that they are functioning without a model of the world as we know it.” 

Humanness matters and that’s why so many AI developers reassert that they base their algorithms on the human brain. That’s true—to some extent. But to verify, I did some preliminary digging in neurolinguistics. 

Image that says: Generative AI is not the equivalent of human thought. Both may leverage predictability in comprehending data, but AI cannot understand the contextual meaning, among other limitations.

Lucky for me, Jonathan R. Brennan, an Associate Professor of Linguistics and Psychology at the University of Michigan, released a wonderful primer in 2022. Also, I majored in comparative linguistics and translation, so I understood basic linguistic principles. That helped.

Here are 3 fascinating points (I recommend reading the rest) I discovered from his book Language and the Brain: A Slim Guide to Neurolinguistics:

  1. Both AI and the human brain leverage predictability.
  2. But while AI uses unpredictability to suggest errors or human-ness, the brain uses plausibility to help process meaning.
  3. Predicting what comes next in a sentence does not equal understanding the content.

What we can pull from all of these details is that human content should:

  • Go beyond general information to provide human context
  • Create plausible sentences and ideas, even if they are not always predictable
  • Understand not just what each word or concepts mean but how they relate to each other
  • Fact-check sources and information 
  • Provide variety and narrative flow
  • Paraphrase and synthesize information to create unique ideas

Why It Matters: AI Doesn’t Necessarily Rank

AI and AI detectors are complicated. I don’t mean to downplay their importance. These solutions offer much in terms of productivity enhancement and streamlining operations. However, it’s clear that their limitations hinge on one factor:

Us.

Humans are complicated, too. One of the reasons there is such a high discrepancy between AI detectors and AI tools is that we simply haven’t mapped the human brain yet. There are too many variables for an algorithm to label content as human or not fully. And while I’m sure this will improve, it’s our human brain that still creates the most effective content.

According to the data I collected of all 2,834 scannable links, only 0.46% of likely AI-written articles ranked in the top 10 for their keyword. That’s considering the article was 100% likely AI-generated after averaging the AI detector results. When we go to 50% likely and above, the number is still only 7.80%.

And, looking at a subsection of 141 links after the March 2024 update, 26.43% of likely links were removed.  

If we believe that multiple AI detectors level themselves out, then yes, human content still rocks. 

But that doesn’t mean AI is going away, either.

You can tap into more insight in the full report.