Testing to see how the models do writing Elm code.
I got rid of some requirements, and tried to fix up the AGENTS.md to overcome some problems some models had, and also add references it could study.
# Layout Test
## Tools
Language: elm
Elm tool is at /home/johnk/.local/bin/elm
The Elm version is 19.2. Write code for version 19.0 to 19.2. A lot of training data from before 2019 may be out of date.
Add Elm modules as needed using the elm install tool. Prefer libraries to writing new code.
Do not try to copy the library into the correct location. Use elm install instead.
Assume the compiler works, and if there are compilation errors, assume your code is incorrect.
If the bugs point to naming or lack of definitions, check the import lines.
elm-test is installed for unit testing.
Playwright and Selenium are installed for frontend tests to check the layout.
For advice and references, see the Tips section below.
## Code to Write
Make a mockup layout in Elm for a News website, with a short top banner, and 3 columns:
Title is "Test News Site".
Top banner is around 2 cm on a desktop screen, and 25% of a vertical mobile screen.
left column nav, center column feature area, right colum listing of additional news. Widths approximately 20%, 60%, 20%.
Theme colors: light gray background, dark gray text, dark blue links.
Layout is responsive to mobile, tablet, and desktop size screens.
Use semantic CSS to identify areas, roles. This way, we can map colors to layout.
Static files:
Use CSS, making the CSS file external in public/style.css.
Create public/index.html file to pull in CSS.
The layout should loop over arrays to construct the center and right columns.
The center column array is a list of news headlines, URLs, and URLs to images (located in the public/images directory).
Center colum items have an image, headline, and excerpt.
The right column array is a list of news headlines and URLs.
The left column contains a simple nav with: Home, Local News, Breaking News.
## Tips
Current calling convention in index.html is Elm.Main.init({ node: document.getElementById('the-node-id') })
elm-test Refs: https://elmprogramming.com/easy-to-test.html https://github.com/elm-explorations/test
For advice about v19, see: https://raw.githubusercontent.com/elm/compiler/refs/heads/main/docs/upgrade-instructions/0.19.0.md
The assumption that the tools and compilers are not working often leads to looping over tests that keep failing. This is because you are falling into the compiler developers test code. Assume that the provided binary is correct. If it keeps failing, stop and tell the user.
Scoreboard
I used the Free versions in OpenCode.
The number scores are about how good the layout looks. The first one is scored 5, and subsequent scores are based on how good it looks relative to the first one.
| LLM | Tokens | Works? | Notes |
|---|---|---|---|
| Thaura (Qwen3.8) | Y | Cycled through debugging a lot. Made simple mistakes, and didn’t really have the correct ways to fix them, but they compounded to make the layout look bad: 2. | |
| Ling3.0 | 42K + 26K | YN (eventually) | First whack produced code that didn’t work, so I had to ask it to write a test; it used Playwright. Then it worked. Looks OK: 5. |
| MiMo2.5 | 20K | Y | One shot and it was OK. Looks so-so: 2. My instruction about “roles” was misinterpreted, so I should not use that term. |
| MuseSpark1.2 | 42K | Y | It cheated: It changed out of it’s directory to peek at other projects. It made something that looked like Ling. Looks OK: 4. |
| MuseSpark1.3 | 21K | Y | It worked quickly to crank it out. Looks OK: 4. Again, the role stuff caused a detour in the coding. |
| Nemotron3 | 29K | YN | It had a path bug in the index.html that it didn’t catch, even with a series of Playwright tests. It also left the directories messy. Once corrected, it looked really good: 7. |
| Nemotron3.5 | N | ** Notes |
I didn’t do any looks at the code produced.
I clearly should not have said “role” in the spec. It got interpreted, both times, as an attribute. What I meant was for that some CSS classnames should be based on roles, so I can change the CSS by role. I had that pattern in my head, but not in a document, and the LLM is not only not a mind reader, but I might be out of step with the jargon used in the wild.
MuseSpark1.2’s cheating polluted the results, because it seems to have copied other LLM artifacts.
What helped
Telling the LLMs to use version 19 of elm, and providing some references, may have helped. Removing so many requirements may have helped. The coding was a bit quicker, and I didn’t really have to make any interventions from the LLM going into a rabbithole/loop.
Nemotron 3.5 Notes
At 36K tokens, it fell off the cliff:
I see the issues - the `Header calls calls needs. calls and need fixing. removed. calls. HTML calls removing3. callcenter. Let me let me. call particul let me let me. call let me let ma let ma let ma letma let let letting moreoverSo I had to start a new session and tell it to read all the files, the AGENTS.md is the goal, and to ignore node_modules and elm-stuff.
It was getting caught in loops over shadowing, or reusing a variable name as a parameter to a function. This is possible, but the Elm compiler complains about it.
There were other errors it couldn’t get past – and it would always try wholly different approaches – so it doesn’t have a good “bag of tricks” to get past some compiler errors.
Eventually, I gave up.
Teaching it Elm, and recording each fix into a text file, was getting to be too much.
Conclusion
Overall, this new prompt worked a lot better, but it didn’t work for Nemotron 3.5 Lightning Free.
Here’s the test and all the generated code.
Author: admin
This is the server’s system administrator. This site is undergoing some changes.