Stopping the slop in the HG Wells Short Story Competition
Here at Prater Raines, we're proud to sponsor, and host the website for, the HG Wells Competition. The competition is an annual short story competition with junior (under 22) and senior prizes for stories based upon a specific theme (chosen by the organisers). All shortlisted entries are also published as a paperback anthology.
Over the past few years, however, improvements in AI have resulted in Large Language Models (LLMs) that have become extremely competent at generating text, including short stories. This has created a problem for the organisers. How can they ensure that the stories they shortlist are not AI generated? To solve that problem, they came to us.
Before doing so, the organisers had explored various services for detecting AI generated text but found these to be both time consuming and expensive. Such services are themselves built upon AI models, so they asked us if we had any experience working with AI and, as it happens, we did. The Open University's (OU) Computing and IT degree includes a module on AI and Gary Fuller, our Customer Support Manager, graduated from the OU in 2024.
Before exploring AI models, however, we first had to identify a process for checking each entry. Each story is uploaded by the individual entrant, so it made sense to check them at the point of upload rather than in bulk once entries had closed. That said, analysing each story during the upload process would have taken too long, so we decided to hand this off to another service then store the result once the process was complete.
Having decided to hand off the process, it felt logical to build an API to receive the story, analyse the text, and return the result to WordPress, on which their site is built. We decided to use API Platform with Symfony to build the API as it can handle HTTP requests, authorisation, and asynchronous jobs with very little setup. The API would then access a small Python service to interact with the AI model.
Here we hit our first constraint. The maximum length of an entry to the competition is 5,000 words. Analysing that amount of text would be impractical using a small open-weight model, and using a larger model would require enormous amounts of RAM. We therefore decided to break each story into chunks of approximately 500 words and produce an aggregate score for the entire document. The API would handle this, leaving the Python service to score each chunk.
/**
* Split file text into chunks with overlapping words
* @param string $text
* @param int $chunkSize
* @param int $overlap
* @return array
*/
private function chunkText(string $text, int $chunkSize = 500, int $overlap = 50): array
{
$words = explode(' ', preg_replace('/\s+/', ' ', trim($text)));
$totalWords = count($words);
$chunks = [];
if ($totalWords === 0) {
return $chunks;
}
$currentIndex = 0;
while ($currentIndex < $totalWords) {
$chunkWords = array_slice($words, $currentIndex, $chunkSize);
$chunks[] = implode(' ', $chunkWords);
$currentIndex += ($chunkSize - $overlap);
}
return $chunks;
}
The next challenge was choosing a model. Binoculars was considered first, but deemed to be too resource intensive as it relies on two 7B token LLMs. Next we explored ZipPy, which isn't a model, but instead relies on compression algorithms and AI generated text. Effectively, the more ZipPy can compress a text it is provided with, the more likely that text is AI generated. Unfortunately, we found this to be too unreliable for our requirements, although blazingly fast.
Next we explored models specifically targeted at text classification, such as RoBERTa and DeBERTa. Two such models vraj33/ai-text-detector-deberta and ogmatrixllm/glyph-v1.1 were considered, but were shown to struggle with text generated by newer LLMs. Next anon-review-meld-2026/meld was explored. Although much better, this model proved more resource intensive and complex to use. We therefore decided it was time to look at some benchmarks.
The RAID Benchmark Leaderboard lists various models used for detecting AI generated text, scoring them based on their ability to detect text generated by a range of LLMs, including more recent ones. This helped us identify Shushant/ADAL-detector-large as a potential candidate. Although it combined small size with excellent benchmark scores, we quickly realised that the model needed fine-tuning to achieve the kind of accuracy we required.
Fine-tuning a model is relatively straightforward with Python, using a combination of Hugging Face libraries and machine learning libraries such as scikit-learn. The key challenge was providing the model with an appropriate corpus of text on which to train. AI generated text was comparatively easy to source. We used the Open Router API and a range of prompts and models to generate 80 stories ranging from 1,500 to 5,000 words (costing around 50p).
MODELS = [
"anthropic/claude-3-haiku",
"google/gemini-2.5-flash",
"openai/gpt-4o-mini",
"meta-llama/llama-3.3-70b-instruct",
"mistralai/mistral-small-24b-instruct-2501"
]
GENRES = ["Contemporary Drama", "Sci-Fi", "Urban Fantasy", "Psychological Thriller", "Mystery", "Historical Fiction", "Slice of Life"]
PERSPECTIVES = ["First-person ('I')", "Third-person limited ('He/She')", "Third-person omniscient"]
TONES = ["melancholy and introspective", "fast-paced and tense", "witty and satirical", "lyrical and descriptive", "bleak and direct"]
def generate_initial_prompt():
genre = random.choice(GENRES)
perspective = random.choice(PERSPECTIVES)
tone = random.choice(TONES)
return f"""Write a highly detailed, slow-paced short story in the {genre} genre.
Style Guidelines:
- Perspective: {perspective}
- Tone: {tone}
- PACING: This is a long-form story. Do NOT rush the narrative. Take time to build the setting, develop character dialogue, and explore internal thoughts.
- Begin directly with the first sentence. No titles or meta-text."""
Human written stories proved more of a challenge. We started with stories sourced from Project Gutenberg using gutenbergpy. Project Gutenberg digitises works for which US copyright has expired and has over 75,000 free eBooks. While it is a great source for classic works, we quickly found that fine-tuning our model made it very good at delineating between older works and newer ones, rather than human written and AI generated stories.
At this stage we decided we needed some more modern text. Luckily, a dataset of stories taken from Reddit's WritingPrompts community proved perfect for our needs. The dataset in question was built over three years using Reddit's API by researchers at Facebook. Their research can be found at https://arxiv.org/pdf/1805.04833. With our improved dataset, we were able to fine-tune our model.
We created nine versions of the model overall. Each version resulted from changing the number of training epochs, the batch size, and the learning rate. The later versions were trained using a laptop with a discrete GPU, which reduced the training time from four hours to about fifteen minutes. We compared each model with results generated from popular AI text detection services using an existing corpus of stories from this year's competition. The table below shows what we found.
| Model Iteration | Accuracy | Precision | Recall | F1 Score |
| v9 | 89.6% | 100.0% | 78.5% | 88.0% |
| v6 | 82.7% | 84.6% | 78.5% | 81.4% |
| v5 | 79.3% | 72.2% | 92.8% | 81.2% |
| v8 | 72.4% | 68.7% | 78.5% | 73.3% |
| v4 | 65.5% | 66.6% | 57.1% | 61.5% |
With version nine of the model as a clear winner, we finished creating our Python service and connected it to our API. We then updated our WordPress plugin so that it passed each story to our API and stored the resulting AI probability score. We also tested this year's long list for the organisers, generating AI probability scores for each story to assist them with generating their final shortlist. Sadly, there were some AI generated stories in there but that should be significantly less likely in the future.