this post was submitted on 05 Aug 2026
19 points (66.1% liked)

Technology

43429 readers
154 users here now

A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.

Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.

Subcommunities on Beehaw:

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 4 years ago
MODERATORS
remington@beehaw.org
alyaza@beehaw.org
TheRtRevKaiser@beehaw.org
gyrfalcon@beehaw.org
rs5th@beehaw.org
coldredlight@beehaw.org
SemioticStandard@beehaw.org
19
AI-generated stories rated better quality than human-written ones, study finds (www.theguardian.com)
submitted 2 weeks ago by Powderhorn@beehaw.org to c/technology@beehaw.org
22 comments fedilink hide all child comments
 

In a study published in the journal Judgment and Decision Making, 1,682 adults were asked to read one of six short stories, three of which were written by humans and three by ChatGPT. Each AI story had a similar theme to one of the human-authored works.

The team told participants whether their story was written by a human or AI, but this information was not always correct. The researchers then asked participants to rate how absorbing and engaging they found the story, and its quality.

Participants who read an AI-generated story rated it as more absorbing and of higher quality than those who read a story written by a human. However, participants gave higher ratings to stories they had been told were written by people.

Dr Deena Skolnick Weisberg, a senior author of the research from Villanova University in Pennsylvania, said: “AI systems can already generate short stories that are seen as being at least as good as – if not better than – human-written stories. We should update our views of AI’s abilities accordingly.”

However, Weisberg, who is herself a creative writer, said that did not mean writing should be left to AI, noting that novels produced by tech were probably going to be different from those written by humans.

“We may need to make room for AI-generated novels, and for AI/human co-written novels, but that doesn’t mean that there’s no longer space for us to appreciate the process of human creativity,” she said.

all 23 comments
sorted by: hot top controversial new old
[–] ds12@beehaw.org 29 points 2 weeks ago (2 children)

Reading the journal paper this article referenced, I was a little disappointed that there weren't more variables accounted for regarding the participants and the texts. With the participants, factors I see commonly accounted for are levels of education, socioeconomic indicators (say income), or how many books the participants have read in the past month or something similar.

Regarding the text, some form of analysis to then account for or attempt to equalise the AI-generated texts so that they're similar to the author-written ones would have been nice - like average sentence lengths or average syllables per word.

However, this is still a data point (and a scary one) that it can be very hard to distinguish AI-generated content from human-generated ones..

[–] spit_evil_olive_tips@beehaw.org 5 points 2 weeks ago (2 children)

I was a little disappointed that there weren’t more variables accounted for regarding the participants

I think it's even worse than that. quoting from the study's abstract:

In Study 1, participants (1,682 adults recruited from Prolific)

...

In Studies 2 and 3, participants (905 adults recruited from Prolific)

hadn't heard of Profilic before, they describe themselves as:

"Prolific connects researchers, AI developers and product teams with real people who take part in studies online. It’s built for collecting high-quality, human data quickly and reliably, from training AI models to running behavioral research. Every participant is vetted, active and fairly paid, so you get trustworthy results without the usual noise."

so the common theme is that 100% of the study participants have opted-in to this "help evaluate AI as a paid side-hustle" thing. that is going to introduce a huge selection bias that is definitely not representative of the population as a whole.

[–] ds12@beehaw.org 3 points 2 weeks ago

Oof, I didn't even catch or look up the Prolific bit!

[–] BaroqueInMind@piefed.social 1 points 1 week ago

In this economy, I'm not going to sit down and do evaluation work for you to sell to another company unless you fucking compensate me for my time. Who is willing to help do this study for free?

[–] Onomatopoeia@lemmy.cafe 3 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

AI text can be very hard or nearly impossible to detect from human generated, at times, depending in the model used.

It's rather surprising.

[–] ByteSorcerer@beehaw.org 1 points 2 weeks ago

People are also way worse at distinguishing them than they think they are. And it's the same story with images.

[–] Brutticus@midwest.social 23 points 2 weeks ago

Look, man, I have been in writing groups and classes with people of all ages and skill levels. I have read some truly horrendous stories. Is the infinite word predictive machine going to write a better story than the worst, least skillful writer? Easily. Is it going to produce a better story than some of the more skillful writers? Absolutely not.

Worth noting also, I'm not saying someone who is noted for selling a lot of books, or even someone in an MFA program for writing. I am saying, LLMs are unlikely to produce a better work than me, and I am just some guy who is pretty skilled and practiced at writing. Maybe the one or two most skilled in a group of hobbyists who meet and share feedback and try to get better.

AI is always going to shoot for the absolute middle in terms of quality, and can definitionally produce nothing new or innovative.

[–] Kwakigra@beehaw.org 22 points 2 weeks ago

This is like comparing landscape paintings to photographs of the same landscape from the same perspective and asking random people to judge which one they think is better. This is an insane comparison.

[–] MagicShel@lemmy.zip 16 points 2 weeks ago (1 children)

I would love to know the prompts and parameters of their short stories, but no. I generate stories all the time as a sort of self-roleplay since I don't have a tabletop troupe to play with. The stories are terrible and get worse with length. I only tolerate it because roleplaying already has a lot of bad writing and retconning, and I'm not looking for depth or a coherent plot.

[–] p03locke@lemmy.dbzer0.com 6 points 2 weeks ago

I think a lot of it depends on the model and the amount of context it has. LLMs usually get worse with time because they are running out of context, or they don't have a good summarization of the story at large. For story-writing, it's best to have a full outline planned and set up as its own Markdown doc, along with critiques on things you think don't make sense or just don't like as a plot device. Then, start a new session with each section, keeping it in a Markdown doc as it goes. Some sort of character doc is probably a good idea, too. Give it as much reference material as possible.

It's just like planning features for code. You don't just say "gimme a story like blah". You have to set it up like writers write movies.

[–] eleijeep@piefed.social 15 points 2 weeks ago (3 children)

Here are the stories they used, if you want to decide for yourself. Although it won't be a blind test since in these documents they tell you which is which:

Story pair 1

Story pair 2

Story pair 3

[–] Kwakigra@beehaw.org 25 points 2 weeks ago (3 children)

Having read the compared stories now, I have to say I'm deeply offended at the results. The AI stories are quicker to read because they are chains of cliches which state and restate the theme of the story (as stated in the prompt) outright instead of demonstrating it through the charged and personal perspectives of the characters. The LLM generated text is meaningless word sausage formatted to be easier to read for people who don't read much. I have never read something more cynical, and I have read plenty of trash by authors who hate their audience and consider them all to be idiots.

[–] OneCardboardBox@lemmy.sdf.org 12 points 2 weeks ago

You nailed it. For me, the human-written stories were interesting because it's not immediately apparent where they're going. The prose evokes images and emotion, and I want to put the pieces together to identify the theme.

The AI stories blurt out the theme almost immediately, and then my attention wanes because there's nothing left in the text except for a base retelling of events.

[–] dom@lemmy.ca 5 points 2 weeks ago* (last edited 2 weeks ago)

Most people are dumb and are functionally illiterate. Of course the incredibly dumbed down text will appeal to them more.

[–] Crozekiel@lemmy.zip 2 points 2 weeks ago

Too many people love a piece of media that makes them feel smart without effort. They don't want to think, they want to believe they already know anything important already and they love to have that reinforced. They read something written by an adult human and their thinky meat gets all hurty, but if they read the dumbed down, stripped bare version that spoon-feeds the "answers" to them, they get all tingly in their bits.

[–] XLE@piefed.social 10 points 2 weeks ago

I like how story pair #1's AI just took the prompt and basically kept repeating "yeah, them's fish" over and over without any attempt at significance

[–] tmyakal@infosec.pub 8 points 2 weeks ago

I love when someone has already answered my question in the comments.

[–] ReasonablyStressed@beehaw.org 1 points 1 week ago

At first I read the title as "AI-generated stores" and was like "well, yeah, Amazon's design sucks, of course they're better rated!"

[–] BlameThePeacock@lemmy.ca 1 points 2 weeks ago (1 children)

People down voting a study. Hilarious.

Feels over reals. We're fucking doomed.

[–] DragonTypeWyvern@midwest.social 5 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

A study isn't valid simply because it's a study, and headlines telling you what to think of their results without at least a vague sense of the numbers are propagandistic by nature.

Rated better quality? By who and how much? Pretty basic details to include.

From what the guardian reports on the details, this study looks like garbage. From the posted stories, the results are definitely garbage, and whether that's the fault of the researcher or humanity will need a closer inspection.

I'll tell you right now, coming to such a broad conclusion because some American sophomore students liked three AI stories better than three selected human written stories sounds pretty fucking stupid.

[–] BlameThePeacock@lemmy.ca 0 points 2 weeks ago

Are you kidding me? This was a University Prof, doing a proper study, published in the Cambridge University Press, and involving 1700 participants.

Did you not even bother to look up the source before jumping down my throat?

This is even further evidence of why we're fucking doomed. Your feelings about this headline override any reasonability you have to the point where you won't even look at the source.

[–] thisisbutaname@discuss.tchncs.de 1 points 2 weeks ago

I don't feel that this matters much.

As something that uses statistics to generate its output, AI creates stuff that's average, and so appealing to the masses. It's not much different from mediocre books churned out by publishers chasing sales instead of art.

It's the McDonald's of art. Is it ok? Sure. Will it change your life? No fucking way.