Creative writing for prizes and column inches with LLMs

Short story competitions are making it into the news for all the wrong reasons lately, it would seem. In July, the Guardian published an article on accusations that Jamir Nazir’s The Serpent in the Grove was written using AI. More recently they published an article detailing similar accusations, made in a piece written for the Atlantic, about Simbarashe Kundizeza's The Blue Fighter.

Both articles mention the service Pangram, which was used by their accusers to test the stories. Pangram has been shown to outperform other popular models at detecting AI use in academia. The study that produced this particular finding only tested content generated using one generative AI model (GPT Deep Research), however, as acknowledged in the Limitations section of the journal article. 

Of course, performing well with academic pieces is not the same as performing well with short stories, as we found when fine-tuning our own model for the HG Wells Short Story Competition. Pangram's own research suggests it outperforms popular models when testing creative writing examples. As we did, they appear to have sourced examples from Reddit WritingPrompts and Project Gutenberg, though it's not clear if they used them for training data.

Pangram's training data included 28 million human-written documents, of which 300,000 (just over 1%) were labelled as creative writing. Our fine-tuning dataset was smaller, but specifically focused on short stories, and included stories generated by mutltiple different models. With that in mind, we decided to see what our model said. When testing The Serpent in the Grove, our model returned a low probability that it was AI generated. For The Blue Fighter, it returned a high probability that the story was written using AI. 

Our model is a fine-tuned version on the ADAL model, which identifies 98% of AI generated texts when it is set up in such a way that 5% of human created texts are falsely classified as AI. This particular metric is used by the RAID Benchmark Leaderboard to test various AI text detection models. Sadly, Pangram doesn't seem to have submitted their model to the leaderboard making a direct comparison of the two models impossible.

It's highly unlikely that definitive proof that AI was used to create either story will be found, unless one (or both) of the authors states that they did in fact use AI. That creates a problem where short story competitions are concerned, as it means potentially giving prizes to people that have effectively cheated. Specialised models fine-tuned on a corpus of short stories, such as the one we created for the HG Wells Short Story Competition, may offer a way to help prevent this happening.