this post was submitted on 27 Mar 2026
592 points (97.1% liked)

Technology

83632 readers
3915 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots


founded 2 years ago
MODERATORS
 

The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, "Reasoning" models, and Agentic frameworks.

ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels.

Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what's next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations.

You can try the tasks yourself here: https://arcprize.org/arc-agi/3

Here is the current leaderboard for ARC-AGI 3, using state of the art models

  • OpenAI GPT-5.4 High - 0.3% success rate at $5.2K
  • Google Gemini 3.1 Pro - 0.2% success rate at $2.2K
  • Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K
  • xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K.

ARC-AGI 3 Leaderboard
(Logarithmic cost on the horizontal axis. Note that the vertical scale goes from 0% to 3% in this graph. If human scores were included, they would be at 100%, at the cost of approximately $250.)

https://arcprize.org/leaderboard

Technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf

In order for an environment to be included in ARC-AGI-3, it needs to pass the minimum “easy for humans” threshold. Each environment was attempted by 10 people. Only environments that could be fully solved by at least two human participants (independently) were considered for inclusion in the public, semi-private and fully-private sets. Many environments were solved by six or more people. As a reminder, an environment is considered solved only if the test taker was able to complete all levels, upon seeing the environment for the very first time. As such, all ARC-AGI-3 environments are verified to be 100% solvable by humans with no prior task-specific training

top 50 comments
sorted by: hot top controversial new old
[–] HaunchesTV@feddit.uk 136 points 2 weeks ago (3 children)

Grok Reasoning: 0%

Hilarious

[–] brsrklf@jlai.lu 78 points 2 weeks ago* (last edited 2 weeks ago)

Reasoning is woke propaganda anyway.

[–] Janx@piefed.social 22 points 1 week ago

Grok isn't designed to solve problems. It's designed to create sexually explicit images of children for Republicans...

load more comments (1 replies)
[–] RustyShackleford@piefed.social 57 points 2 weeks ago (16 children)

As a psychiatrist, I have a theory about what’s missing in AI. First, it lacks childhood dependency and attachments. Second, it struggles to overcome repeated pain and suffering. Third, it lacks regular eating and restroom breaks. Fourth, it struggles to accept loss in everyday situations. Finally, it lacks the concept of our inevitable death. Without these nagging memories and concepts, machines will simply revert to the simpler concepts we use them for in our recent times, such as stealing cryptocurrency. After all, we live in a world run by capitalism, so it’s only logical. ¯\(ツ)

[–] CosmicTurtle0@lemmy.dbzer0.com 106 points 2 weeks ago (7 children)

As a technologist, I have to remind everyone that AI is not intelligence. It's a word prediction/statistical machine. It's guessing at a surprisingly good rate what words follow the words before it.

It's math. All the way down.

We as humans have simply taken these words and have said that it is "intelligence".

[–] unpossum@sh.itjust.works 55 points 2 weeks ago (20 children)

As another technologist, I have to remind everyone that unless you subscribe to some rather fringe theories, humans are also based on standard physics.

Which is math. All the way down.

[–] HereIAm@lemmy.world 29 points 2 weeks ago

I agree, the maths argument is not a good one. While a neural network is perhaps closer to what a brain is than just a CPU (or a clock, as it was compared to in he olden days), it would be a very big mistake to equate the two.

load more comments (19 replies)
[–] silverneedle@lemmy.ca 12 points 2 weeks ago

As someone who knows a thing or two about biology I think LLMs strip away >90% of what makes animals think.

[–] Iconoclast@feddit.uk 12 points 1 week ago (2 children)

Few of countless dictionary definitions for intelligence:

  • The ability to acquire, understand, and use knowledge.
  • The ability to learn or understand or to deal with new or trying situations
  • The ability to apply knowledge to manipulate one's environment or to think abstractly as measured by objective criteria (such as tests)
  • The act of understanding
  • The ability to learn, understand, and make judgments or have opinions that are based on reason
  • It can be described as the ability to perceive or infer information; and to retain it as knowledge to be applied to adaptive behaviors within an environment or context.

There isn't even concensus on what intelligence actually means yet here you are declaring "AI is not intelligence" what ever that even means.

Artificial Intelligence is a term in computer science that describes a system that's able to perform any task that would normally require human intelligence. Atari chess engine is an intelligent system. It's narrowly intelligent as opposed to humans that are generally intelligent but it's intelligent nevertheless.

load more comments (2 replies)
[–] TherapyGary@lemmy.dbzer0.com 9 points 1 week ago

As a therapist, I can tell you the only thing holding LLMs back from true intelligence is having to pee and poop. Peeing and pooping is the foundation of all higher level operations. I poured water on my PC and the LLM I was running said "I think" right before committing suicide

load more comments (3 replies)
[–] msage@programming.dev 15 points 2 weeks ago (1 children)

Are you anthromorphizing word suggester into a being experiencing things?

load more comments (1 replies)
[–] sp3ctr4l@lemmy.dbzer0.com 12 points 1 week ago* (last edited 1 week ago)

Here is a way of describing what I see as 'the problem':

An LLM cannot forget things in its base training data set.

Its permanent memory... is totally permanent.

And this memory has a bunch of wrong ideas, a bunch of nonsensical associations, a bunch of false facts, a bunch of meaningless gibberish.

It has no way of evaluating its own knowledge set for consistency, coherence, and stability.

It literally cannot learn and grow, because it cannot realize why it made mistakes, it cannot discard or ammend in a permanent way, concepts that are incoherent, faulty ways of reasoning (associating) things.

Seriously, ask an LLM a trick question, then tell it it was wrong, explain the correct answer, then ask it to determine why it was wrong.

Then give it another similar category of trick question, but that is specifically different, repeat.

The closer you try to get it toward reworking a fundamental axiom it holds to that is flawed, the closer it gets to responding in totally paradoxical, illogical gibberish, or just stuck in some kind of repetetive loop.

... Learning is as much building new ideas and experiences, as it is reevaluating your old ideas and experiences, and discarding concepts that are wrong or insufficient.

Biological brains have neuroplasticity.

So far, silicon ones do not.

load more comments (13 replies)
[–] GreatBlueHeron@lemmy.ca 46 points 2 weeks ago (1 children)

It's fun to point at the crappy performance of current technology. But all I can think about is the amount of power and hardware the AI bros are going to burn through trying to improve their results.

[–] partofthevoice@lemmy.zip 21 points 1 week ago* (last edited 1 week ago)

Funnier yet will be if they continue to just train the model on that particular kind of test, invalidating its results in the process.

[–] SaraTonin@lemmy.world 42 points 1 week ago (3 children)

Tell me again how AGI is just around the corner, Sam

[–] Vupware@lemmy.zip 17 points 1 week ago (3 children)

When Sammy fuck says “we’re so close to AGI, I can just feel it. Like a tingle on the tip of my shrimpdick it’s getting so close to blossoming into something guys”, just ignore him. He’s crazy man!

load more comments (3 replies)
load more comments (1 replies)

It's almost as if a chatbot isn't actually thinking.

[–] mechoman444@lemmy.world 28 points 1 week ago (1 children)

I know lemmy's very anti-ai but this is really fascinating stuff.

[–] PhoenixDog@lemmy.world 9 points 1 week ago (18 children)

We're anti-AI because AI is fucking stupid. Both literally and figuratively.

load more comments (18 replies)
[–] ExLisper@lemmy.curiana.net 27 points 2 weeks ago

Can't wait for this to be the new captcha.

[–] Tetragrade@leminal.space 24 points 1 week ago* (last edited 1 week ago) (2 children)

This replay is the funniest shit lmao. Keep building that bridge Claude.

https://arcprize.org/replay/0964128b-a2f5-4c5b-886e-497d893f429d

Interesting that it seems to be perceiving the environment mostly accurately, and is just completely wrong about the purpose of all the game objects.

[–] bss03@infosec.pub 13 points 1 week ago (3 children)

I couldn't find replays. Are there more? Also, it is a bit funny that "building the bridge" which at one point seems to be Claude's "chosen goal" is just "running out of moves" and failing the task.

Task failed successfully, Claude. Task failed, successfully.

load more comments (3 replies)
load more comments (1 replies)
[–] fox2263@lemmy.world 24 points 1 week ago (5 children)

I can’t see AI actually being intelligent until they no longer need to send a built up prompt of guides and skills and the chat history on every submission.

It’s no different from Alexa 15 years ago with skills. Just a better protocol and interface and ability to parse the current user prompt.

In my opinion of course.

load more comments (5 replies)
[–] Bubbaonthebeach@lemmy.ca 20 points 1 week ago (1 children)

I tend to be anti-AI because it doesn't seem to me to be anything other than a super fast regurgitator of data. If a database can be searched for an answer, AI can do that faster than a human. However it doesn't to seem to be able to take some portion of that database, understand it, and then use that information to solve a novel problem.

[–] cmhe@lemmy.world 17 points 1 week ago (2 children)

Well... It cannot even search databases without errors.

LLMs just produce plausible replies in natural languages very quickly and this is useful in certain situations. Sometimes it helps humans getting started with a task, but as it is now, it cannot replace them. As much as the capital class want it, and sink our money into it.

load more comments (2 replies)
[–] UnrepentantAlgebra@lemmy.world 19 points 2 weeks ago (6 children)

If human scores were included, they would be at 100%, at the cost of approximately $250

Wait, why did it cost real humans $250 to pass the test?

[–] KairuByte@lemmy.dbzer0.com 22 points 1 week ago

I assume it’s an hourly wage or something. Just because humans can work for free if they choose, doesn’t mean they have no cost associated with them. Just like a company could choose to give away unlimited tokens, those tokens still have a standard cost.

[–] FrankFrankson@lemmy.world 13 points 1 week ago* (last edited 1 week ago) (1 children)

That is how much individual testing humans cost when you buy them in bulk.

load more comments (1 replies)
[–] aesopjah@sh.itjust.works 11 points 1 week ago (1 children)

it's also an odd metric since only 20-60% of the humans completed it. Very 60% of the time they complete it everytime energy.

Ideally they'd run the bots multiple times through (with no context or training of previous run), but I guess that is cost prohibitive?

[–] monotremata@lemmy.ca 11 points 1 week ago

Yeah, this is what I was going to call out. Calling it "100% solvable by humans" and saying "if human scores were included, they would be at 100%" when 20-60% of humans solved each task seems kinda misleading. The AI scores are so low that I don't think this kind of hyperbole is necessary; I assume there are some humans that scored 100%, but I would find it a lot more useful if they said something like "the worst-performing human in our sample was able to solve 45% of the tasks" or whatever. Given that the AIs are still scoring below 1%, that's still pretty dark.

[–] mapleseedfall@lemmy.world 8 points 2 weeks ago

Youd have to eat $250 worth of burgers to pass it.

load more comments (2 replies)
[–] Multiplexer@discuss.tchncs.de 16 points 2 weeks ago

Link to the recent Al Explained video mainly covering ARC-AGI-3:
https://www.youtube.com/watch?v=s4tptozUJ8Y

[–] arcine@jlai.lu 14 points 1 week ago (5 children)

Try spelling things phonetically (example: faux net tick alley), that's one of my benchmarks that AI fails almost every time.

If the input is at all long, or purposefully includes a lot of words about a specific unrelated theme to the coded message, it's impossible.

load more comments (5 replies)
[–] sunbeam60@feddit.uk 10 points 1 week ago* (last edited 1 week ago)

Ii can thoroughly recommend “A Brief History of Intelligence” (by Max Bennett), which explains how intelligence has taken steps through evolution, what those steps were etc.

Spatial intelligence requires spatial understanding and it’s not something that can be solved through a large language model, IMHO.

I’m excited to see how these are solved. And I’m terrified to see how these will be solved.

load more comments
view more: next ›