More Insights re: working with Coding Agents

After writing this (below) I of course have a ton of additional thoughts on things I wish I had mentioned.

The benefits of using multiple LLMs

1. It is a great way to reduce API call exhaustion. Designing takes a lot of tokens, so does coding. Split that effort.

2. It is a great way to make sure that the agents are actually making changes and producing artifacts, not just pretending that they are doing those things.

3. Different LLMs hallucinate in different ways, so they hold each other accountable, if you’re diligent about it

Different types of hallucinations the LLMs experienced

1. One was convinced I was writing a firewall, and wanted me to focus on port security

2. The project has three separate components, and each of them would regularly forget that one of the components existed

3. They would invent APIs that didn’t exist

4. They insisted that an API that did exist did not

5. They regularly confused the concept of the word Token in JWT with the concept of Token as used in LLM-world. Even though the context was absolutely clear that we were talking about identity, not sentence parsing.

6. Multiple times they wanted to write python scripts to replace a word in a document with another word

7. I set up Gemini to help design the code. I put rules into Gemini’s system prompt, in two different ways, telling it to never write code, that it’s job was specifically to produce specifications and project plans. Any time the context got even partially full, it would completely forget this instruction and immediately start writing code. This is apparently called Context Window Drift, and it is very real.

8. One of them hallucinated the existence of URI standards for different vault interfaces (for handling important secrets). I didn’t think to check until later, and had to rework some of the design

Things that LLMs seem to struggle to understand

1. Race conditions. On at least two occasions, the LLM fought me tooth and nail (metaphorically), insisting that there must be a bug in a standard parsing library, rather than entertain the possibility that two events happened at nearly the same time, and that one had inadvertently wiped out data needed by the other.

   1. miraculously, adding a mutex fixed the problem

2. The LLM’s front-end will drop characters from URLs. This was a fun one. I was giving the LLM a URL to give to the user, and it would occasionally drop a character from the data embedded in the URL. Rather than entertain the possibility that the front-end was sloppy, it insisted there was a bug in the third-party JWT validation logic

Things that the LLMs are good at

1. I renamed a property, and moved it around in the YAML hierarchy so it would require significant amounts of manual effort to adjust all the code that depended on it. A human would be miserable having to do this tedious, unsatisfying work. Agent was happy to do it, had it done in about 5 minutes.

2. I reorganized a structure, had Claude refactor all the code and tests to use the new structure, and then decided I didn’t like it, and changed it to a third structure. Which Claude happily fixed again. If I had been assigned this series of tasks, I would have likely started looking for a new job.

3. Adding audit features, like a Posture Report - thankless, joyless work for a human. Trivial for an agent

4. Making sure every resource is properly closed, all error responses are properly handled, and all exceptions caught; also very tedious for humans, trivial for LLMs

5. Validating every configuration field, both in terms of type, existence and value against approved ranges

6. Thinking about all common forms of network attack, and making sure they’re properly called out and addressed

Closing Thoughts

Thanks for reading this far. I hope this is useful, perhaps in setting expectations about the kinds of things that AI is good at, and the kinds of things that you’ll want to make sure you keep senior developers around to help with.