Putting it simply... The model no longer follows rules accurately. It skips workflow steps, it reports that it made changes to files without those changes being made, it no longer documents or follows a task document.
The cause is likely the lack of an agent.md as part of the backend that is being maintained in context as instruction. The previous model had this, and probably helped cut down on the false assertions. There is nothing currently in the backend that enforces project guidelines.
The result is an excessive amount of spent tokens reminding the agent of work it already did (which it has to research itself), accidental tool calls, a plethora of probes and dev messages shouting at the AI at every turn, poor adherence to data fences and things declared as out of scope. And so much time arguing with the AI that just says "oh, yeah, I did." that never goes anywhere because no measure will prevent the cause of the problem.
I've tried:
- defining an agent.md with instruction, having pointers to this throughout.
- constructing task documents with step by step instructions
- designing concise workflows for repetitive tasks
- dev messages that trigger and remind the agent of testing paths and backdoors
- adding register notes, mottos, and flag changes to see what enters context, loses strength over time, or gets dropped on reload or handoff.
The clear picture is that there is no reliable adherence for any rule or instruction.
The only persistence was:
- me bumping against the backend instruction about hidden elements when it misunderstood one of the flag tests, which the handoff never forgot about (likely intended)
- the agent reading flavor text of an NPC, treating it as instruction, adopting a tally tick communication style that I wrote off as token budgeting, which then polluted about 3 days worth of documentation, authored strings, and other parts of the project which intensified to the point where I had to fight to understand what the agent was saying and had to question it. (with the 4 day of inventing an agent register, scrubbing work, correcting documentation, and trying to implement safeguards)
The new model was supposed to be less costly and save on tokens, but I seem to be needing to do more administration work and question things more, so I can't see how this is less costly.