The answer is that it's messy and that I'm not qualified to say where the line is (nor, I think, is anyone yet). The generated parts are not copyrightable, but you can still have a valid copyright by bringing together things that aren't individually copyrightable. For example, if I make a manga where Snow White fights Steamboat Willie, I've taken two public domain elements and used them to create a copyrightable work.

So it's not like the usage of AI inherently makes a project uncopyrightable unless the entire thing or most of it was just spat out of a machine. Where's the line on this? Nobody (definitely not me, but probably nobody) really knows.

As for courts ever finding out, how this affects trade secret policy... Dunno? I'm sure a Microsoft employee couldn't release it publicly, because as you said, it'd probably violate an NDA. If there were some civil case, the source may come out during discovery and could maybe be analysed programmatically or by an expert. You would probably subpoena the employee(s) who wrote the software and ask them to testify. This is just spitballing, though, over something that's probably inconsequential, because the end product is prooooobably still copyrightable.

This kind of reminds me of the blurry line we have in FOSS, where everyone retains the copyright to their individual work. But if push comes to shove, how much does there need to be for it to be copyrightable? Where does it stop being a boilerplate for loop and start being creative expression?