Waow that is so cool that LLMs are better at diagnosing LLMs pretending to be human than humans are. That'll be so revolutionary for the large number of LLMs pretending to be humans seeking healthcare
But the study could only imperfectly simulate the back and forth a doctor has with a patient by having DxO talk to a “gatekeeper agent” that “mediates all information requests drawing from the underlying case record.” Put another way, it’s an AI diagnosing a patient that’s played by an AI. And while there’s no doubt that AI models are powerful analyzers and synthesizers of information, human doctors already did the leg work of putting together these neat, highly detailed case files for the AI to pore over in the first place.