this post was submitted on 29 Dec 2025
364 points (98.4% liked)
Technology
78482 readers
2476 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
TL;DW:
He wrote checksum verifier for ISBN and discovered AI generated content on Wikipedia with hallucinated sources. He used Claude to write the checksum verifier and the irony is not lost on him. He tracked down those who submitted the fake articles and determined many are doing so out of a misplaced desire to help, without an understanding of the limitations and pitfalls of using LLM gen content without verification.
What's the irony?
He used AI to write the anti-AI tool
To be fair, humans are excellent at building anti-human tools
Heads up he talks about this specifically at 26:30 for those who didn't take the time to watch the video.
But wasn't the issue with the AI stuff that it was false info, whereas the tool sounds like it worked as intended?
Here's the explanation of the irony in this situation from an LLM ;)
Side note: I'd only thought about the LLM generated code irony. I'd missed the 2nd irony of the editors trying to be helpful in providing useful accurate knowledge but achieving the opposite.
It just seems like the tool he is using is working though...
Yes, but the specific type of irony that this situation fits the definition of does not come from whether or not the tool they used worked for the intended purpose. The irony comes from the fact that they are relying on the output from LLM-generated content (ISBN checksum calculator) to determine the reliability of other LLM-generated content (hallucinated ISBN numbers).
Irony is a word that has a somewhat vague meaning and is often interpreted differently. If the tool they used did not work as intended and flagged a bunch of real ISBNs as being AI generated, the situation would (I think) be more ironic. They are still using AI to try and police AI, but with the additional layer of the outcome being the opposite of their intention.
But how does that diminish the irony? The story is still ironic as a whole, even though he achieved his goals.
I would feel like it would be ironic if it was after AI in general instead of the mistakes, I dunno
Right, but it's also the same/similar tool that's being used to damage the article with bad information. Like the LLM said, this is using the poison for the cure (also, amusing that we're using the poison to explain the situation as well).
Yes, he's using the tool (arguably) how it was designed and is working as intended, but so are the users posting misinformation. The tool is designed to take a prompt and spit out a mathematically appropriate response using words and symbols we interpret as language - be it a spoken or programming language.
Any tool can be used properly and also for malicious/bad via incompetent methods.
But in this case the tool actually works well for one thing and not so well for another. It doesn't feel that ironic to use a hammer to remove nails someone has hammered to the wrong place, if some sort of analogy is required here. You'd use a hammer because it is good at that job.
See, that's where you're wrong though. AI is about as competent at natural English as it is writing code.
I use it for both at times, since it can be an easy way to both rubber duck debug my code as well as summarize large projects/updates in ways that are easily digestible when I don't have the time to write out a proper summary manually. But in either case, I have to go back and fix a good bit of what's provided.
AI is not great at either option, and sucks at both in different ways. Saying AI is a hammer is not supwr helpful, because hammers have a defined use. LMMs are a solution looking for a problem. The difference between the posters and the researcher is that the researcher has an advantage that he both knows what he's doing and knows how to fix the turds he's provided to make it work, where the users are just trusting the output.
I don't know how to explain the irony any better in this scenario, but it's there. If the users actually fact checked their output, we wouldn't be having this discussion. Same as if the researcher chose not to validate his output. The issue isn't necessarily the use, but the usage. So this is akin to the posters using a hammer to put up a shelf, but they didn't look at the directions and saying "yep, that looks right"
But it did create a working tool to identify AI contributions with fake ISBN, didn't it? Are we assuming the tool from OP wasn't working?
So, how is your autism?
I'd be careful, people get banned for saying that sort of thing
Ok, but how do you know that other edits from Wikipedia aren't AI generated, but had users who actually validated the output? And can you explain to me the difference between the users who validated the AI output before updating Wikipedia, and the researcher who validated his AI output before making his talk?
The point you're missing is that both sides are using the same crappy tool, but you're only seeing an example of one side doing it wrong and the other right, and using that to make a conclusion that is not able to be validated. You appear to be saying it's better for code than language because of the example in front of us and naively extrapolating that to mean ai works better in one task than the other, when the difference is how the user handled the output, not the output itself.
I mean in this specific case it did make a working tool to identify hallucinated sources.
And yes it's definitely better at code than doing something requiring the sort of thinking that Wikipedia edits require. I'm not drawing that from this singular thing but rather when they've studied the correctness of AI output. Coding is sorta "easy" for a computer to do compared to language and information tasks in that sort of free-er form way.
Right, but it also outputs really bad code, so pointing to this specific example is pointless to the overall point. Both code and language output have to be validated and reworked, because an AI is just a stochastic parrot. The quality of the final product is dependent on the person using what the AI gives them.
Lol. Lmao even. Vibe coding is arguably worse than vibe talking, just ask the users of Tea. Or the devs of NX that had to stop an AI PR that would have run unsanitized shell commands. Or that idiot that fired everyone to vibe code his app that was immediately hacked.
If you think language requires more thinking than coding, I don't know what to tell you, considering that code is applied thinking/logic. AI is good at speeding things up by accelerating blocking things out, but absolutely sucks at producing a workable product. Just like with language processing.
But I'm done with this conversation, considering you've ignored the point I'm making to point back at this example multiple times now. Good day.
I mean it's not really my take,it's just when they've tested it out at different tasks. It does a lot better at coding than most other tasks.
Yes. Why are you fixated on this? LLMs are tools and they work, but you have to understand their abilities and limitations to use them effectively.
The guy who needed the anti-ai tool, did. The Wikipedia editors, didn't.
I feel like it would be a lot more ironic if the tool didn't work. It doesn't seem very ironic to use hammer to remove nails hammered into incorrect position with a hammer imo.
Good for you I guess? Idc
Thanks man, I appreciate it
I think the point is it would have been truly ironic if the AI itself was the authoritative fact checker instead of merely being a tool that built another tool.
If Claude was the fact checking tool instead of the ISBN validator, that's the real irony.
If in a messed up future, only an AI could catch a fellow AI, what's stopping the AI collective from returning false negatives? Who watches the watchers?
Which is kinda my favorite thing to do as of late, and what I prefer AI be used for.
I'm not taking about building a suite of black box tools. But tiny scripts to scrape, shape, and generate reports. Things I used to pull in a dozen node libraries to do and manually configure and patch up.
You know, busy work.