Code profiling tools are everything now.

When I was staring at Thaura writing code, it was doing some wild tricks with python and grep and other unix tools that I’d never seen before.

So I told it to write tools to do simple kinds of code analysis, to help map out and index the code. I also wrote scoreboard database tools, to keep track of work to do, and work completed. I had it put the code analysis info into the database, too.

I also asked it to tell me about all the PHP static analysis tools, and enlisted them all into the programming process: linters, code stylers, static type analysis.

The LLM can use these tools the way we use them, to understand the code, and to demonstrate that code conforms to some idea of “order”.

The LLM also uses them as a kind of debugging aid, or a pre-run tester.

I also leaned on Playwright and the old reliable PHPUnit.

Illusions of Complete Coverage, and “AI Psychosis”

The problem with these code analysis tools is: they don’t enforce writing good code as much as assist with it. They mainly help when the code is big enough to forget parts of it.

They give you the illusion of total control, when you’re just keeping things together.

Likewise, the LLM will appear to be making great, sweeping changes over the code, but touching only a fraction of it, leaving lots of code untouched, and report back with a chat message that it did a lot of work.

The LLM straight up bullshitted the boss!

I totally fell for it.

The problems would trigger errors and warnings, and then the LLM would look at the errant code, and go fix it, with some chatter about how it was still doing the old thing, when it should be doing the new thing.

Then it was fixed, passed the test, looks good in the analysis tools, and the LLM moved on, and I moved on.