I think it could work as a minor gimmick, like terminal hacking minigame in fallout. You have to convince the LLM to tell you the password, or you get to talk to a demented robot whose brain was fried by radiation exposure, or the like. Relatively inconsequential stuff like being able to talk your way through or just shoot your way through.
Unfortunately this shit is too slow and too huge to embed a local copy of, into a game. You need a lot of hardware compatibility. And running it in the cloud would cost too much.
Yeah plenty of opportunities to just work it into the story.
I dunno what kind of local models you can use, though. If it is a 3D game then its fine to require a GPU, but you wouldn't want to raise minimum requirements too high. And you wouldn't want to use 12 gigs of vram for a gimmick, either.