Got another chance to experience slop firsthand when the instructor for my electrician course was 'encouraged' to use the hallucinatron to help create our final exam on the NEC. Now given that the NEC is a dense technical document with a lot of minor but significant variation across its considerable length, this was clearly a perfect use case. Here's how it shook out:
-
It condensed 100 multiple choice questions from the input to 36
-
On one question "1-2 inches" was simplified to "12"
-
Units in general seem to have been dropped off a lot of questions and answer choices. Usually this didn't matter too much but it's a bad look
-
Another question asked about fill percentages for a 30 inch conduit. If you look around your office or he and see a >2ft diameter piece of PVC pipe let me know because the tables in the NEC only go up to 6 inches. This is actually a unit issue again because one of the questions on the input test referred to a 30mm conduit which, you know, does actually exist.
-
Other questions had a correct answer matching a generic part of the NEC, but had additional information added as a distractor that ended up matching to more specific elements that changes the relevant rule.
-
Several questions asked about the reasoning behind a certain rule. Notably the NEC rarely actually gets into that information, as it's already an incredibly long reference and policy document and would be made even more unweildy if it gave the justification for everything that you should be learning as part of becoming a licensed electrician.
-
However, this rarely mattered as the answer choices for those questions uniformly included an obviously correct answer about a generic safety risk and distractors about doing things for cost savings, aesthetic reasons, or arbitrarily.
Given that one of the challenges of this test is time management and looking things up, having to deal with the extra layer of "is this just slop or am I missing something" ended up adding an extra and unintended layer of difficulty onto the test. As always, no matter how egregious or obnoxious the errors introduced by AI, the biggest problem is the loss of trust: you can no longer assume that the text you're reading was put together with the intended purpose in mind rather than being generated to be statistically similar to text matching that purpose. Even if the differences are relatively small in scope, as they were for most questions on the test, they significantly harm the actual communication of information.
