Testing to see how the models do writing Elm code. Fourth try.

I had Thaura read my AGENTS file, and critique it. I made the suggested changed, and also added more advice. Then, I started a fresh session and had it reorganize it.

Here it is as a download (I had trouble pasting it here).

Scoreboard

I used the Free versions in OpenCode, which has some free models included.

Token counts are approximate.

LLM	Tokens	Works?	Notes
Thaura (Qwen3.8)		Y	Had very few bugs, and they were corrected quickly.
Ling3.0	34K	Y	Layout is a little off. Right column under left.
MiMo2.5	26K	Y	Built it quickly, and the layout is correct.
MuseSpark1.2	35K	Y	Built it quickly, and the layout is correct.
MuseSpark1.3	31K	Y	Built it quickly, and the layout is correct.
Nemotron3	35K	Y	Took so long I could not monitor it.
Nemotron3.5		N	** See notes

For this round, I used an AGENTS.md file that was improved by asking Thaura to critique it. The critique was based on what happened during the LLMs coding and testing. I made changes, and then asked Thaura to organize it.

I figured this would help Thaura, which was already succeeding at writing the code. It breezed through the task, and appeared to use only a small amount of energy (Thaura converts tokens into energy usage, and said it was like driving under 500 meters).

It also seems to have helped the others as well.

For each build, I asked the agent to critique AGENTS.md, and write the critique to critique.md. Mimo failed to do that, but Ling and Muse Spark did it. So they can compare how their build went, and how the AGENTS.md file helped.

One thing that jumped out was that the LLMs asked for a more determinate test, with more specific layout details. They asked me to be more specific about what I wanted.

Nemotron 3.5

Again, Nemotron had problems.

I told it to read AGENTS.md, and got this:

Theytics motherellsellsamu inells this alongellsells from indigenousellsells Carn shred mill certifiedells
Code language: JavaScript (javascript)

It’s not having a good day.

Eventually, I interrupted it, and the layout was there, but the CSS wasn’t, and the Playwright tests weren’t done. So I gave it the “OK” to complete that. It still didn’t end up working.

Then it said:

It seems port 800 الخارجية-orientedे- extern需-exل-allow-its owners المشاكل mu로서 cơis(),
 lui. ث.
 starken-ise anno (h. (something
 also . ( weiteren
 شركة томnil

Performance Improvements

LLM	Tokens layout1	Tokens layout2	Tokens layout3	Tokens layout4	Effect on token use
Thaura (Qwen3.8)					suspect a reduction
Ling3.0	51K	42K + 26K	42K	34K	general reduction
MiMo2.5	19K	20K	37K	26K	general increase, but improves from 3 to 4
MuseSpark1.2	20K	42K	43K	35K	general increase, but improves from 3 to 4
MuseSpark1.3	16K	21K	27K	31K	increase
Nemotron3	24K	29K	34K	35K	increase
Nemotron3.5					oh well

Ignore the layout1 counts: some of these runs didn’t succeed. So those values aren’t all valid.

The layout2 counts are more valid, because they all produced a reasonable result, excepting Nemotron3.5.

The change from version 3 to version 4 of AGENTS.md was generally saving tokens across three (or four) of the 7 models. AGENTS.md 4 was much longer, and more comprehensive, so it consumed more context, but it prevented the models from making so many errors, and requiring more compile-debug cycles, saving both time and tokens.

Because I can’t see token counts on Thaura, I cannot be certain the count went down, but I know that the “miles driven” metric it provides in its stead plummeted, and it produced the code quickly, and it didn’t appear to be running the compiler as much. This wasn’t a surprise, because I asked Thaura to help me improve my AGENTS.md, and it listed all the ways it tripped up, and told me how it could have avoided the bugs.

We see some large drops in token counts, approximately 10k or 25% less for three models.

However, it’s obvious that my AGENTS.md is not cross-platform. A lot of it reads like a “guide to pitfalls and landmines” across a space that’s rough and rocky. It works for Thaura’s LLM to make this specific project, but it could also be a hindrance to a different project.

AGENTS.md Effectiveness

New Research Reassesses the Value of AGENTS.md Files for AI Coding reported that AGENTS.md, across 60,000 instances found on GitHub, seemed to produce a 4% improvement, and in some cases, made things worse.

My sense is that a detailed spec in AGENTS.md with advice about how to program initially costs more tokens, but as these are improved, performance will improve.

Code Quality

I can’t evaluate it because I don’t know Elm well enough – however, this process of discussing the problem code with LLMs has helped my learning considerably.

Conclusion

All the models basically completed the code without asking questions, but, once again, Nemotrion 3.5 Lighting Free failed.

Here is a ZIP file with all the generated code, and project files.