Testing to see how the models do writing Elm code. Third try.
AGENTS.md was edited to address a lot of bugs and problems from the second run. Also added a directory for more extensive coding advice. You can find it in the ZIP file.
# Layout Test
## Tools
Language: elm
Elm tool is at /home/johnk/.local/bin/elm
The Elm version is 19.2. Write code for version 19.0 to 19.2. A lot of training data from before 2019 may be out of date.
Add Elm modules as needed using the `elm install` tool. Prefer libraries to writing new code.
Do not try to copy the library into the correct location. Use `elm install` instead.
Assume the compiler works, and if there are compilation errors, assume your code is incorrect.
If the bugs point to naming or lack of definitions, check the import lines.
elm-test is installed for unit testing.
Playwright and Selenium are installed for frontend tests to check the layout.
`playwright-cli` is installed, info here: https://www.npmjs.com/package/playwright
`selenium` is installed. `selenium-webdriver` is installed, info: https://www.npmjs.com/package/selenium-webdriver
For advice and references, see the Tips section below.
## Code to Write
Never try to cd or use .. to get above the project root directory.
If you need temp files make a tmp/ in the project root.
Directory layout
Elm sources go in src/
HTML files, js artifacts from compilation, js code you write, css code you write, and binary assets go in public/.
The project root contains the elm.json, project.json, composer.json and other package manager files.
The project root contains .md and .txt files for LLMs and people. llm-advice as well.
Tests go in tests/
Make a mockup layout in Elm for a News website, with a short top banner, and 3 columns:
Title is "Test News Site".
Top banner is around 2 cm on a desktop screen, and 25% of a vertical mobile screen.
left column nav, center column feature area, right colum listing of additional news. Widths approximately 20%, 60%, 20%.
Theme colors: light gray background, dark gray text, dark blue links.
Layout is responsive to mobile, tablet, and desktop size screens.
Use common CSS class naming conventions when possible. See: ./llm-advice/css-class-naming-conventions.txt
Static files:
Use CSS, making the CSS file external in public/style.css.
Compile the Elm code into javscript code, and put it into the public/ directory, or the public/js directory.
Create public/index.html file to pull in CSS.
URL references in the static files should be relative, not absolute.
Assume I'm starting `elm reactor` in the project root, not inside public/, so always use relative paths.
The layout should loop over arrays to construct the center and right columns.
The center column array is a list of news headlines, URLs, and URLs to images (located in the public/images directory).
Center colum items have an image, headline, and excerpt.
The right column array is a list of news headlines and URLs.
The left column contains a simple nav with: Home, Local News, Breaking News.
Testing
At the very minimum, create a Playwright test to check that the browser can display the layout.
Often, code compiles and works, but the index.html file has bad URLs that won't load the artifacts.
Problems include: /main.js - absolute path doesn't work; ../dist/main.js - the http server doesn't allow going up a directory.
Solution is to use relative paths, and generate artifacts in public/, and then use url: main.js
Same has happened with CSS files.
## Tips
The following advice below may help avoid getting into endless debugging loops.
Important: the assumption that the tools and compilers are not working often leads to looping over tests that keep failing. This is because you are falling into the compiler developers test code. Assume that the elm compiler is correct. If it keeps failing, stop and tell the user.
Read about shadowing: ./llm-advice/shadowing.txt
Current calling convention in index.html is Elm.Main.init({ node: document.getElementById('the-node-id') })
elm-test Refs: https://elmprogramming.com/easy-to-test.html https://github.com/elm-explorations/test
For advice about v19, see: https://raw.githubusercontent.com/elm/compiler/refs/heads/main/docs/upgrade-instructions/0.19.0.md
The file is now 61 lines long.
Scoreboard
I used the Free versions in OpenCode.
The number scores are about how good the layout looks. The first one is scored 5, and subsequent scores are based on how good it looks relative to the first one.
I am not going to try and evaluate the “look” anymore. All the layouts are basically midrange in quality, and all will require work to look nice. Additionally, it’s not like I’ve provided an image of the desired layout, so it seems pointless to compare.
Token counts are approximate.
| LLM | Tokens | Works? | Notes |
|---|---|---|---|
| Thaura (Qwen3.8) | Y | Basic. Not chatty! | |
| Ling3.0 | 42K | Y | Had lots of connection errors, but it still produced everything. |
| MiMo2.5 | 37K | YN | Declared a “port” module erroneously. Layout is broken, with right column below left and center. Not tragic. |
| MuseSpark1.2 | 43K | Y | It just produced it. Seemed to spend time writing test cases. |
| MuseSpark1.3 | 27K | Y | It made 14 tests. |
| Nemotron3 | 34K | Y | Interaction with the model was slow. Seemed to take 90 minutes. I stopped paying attention. |
| Nemotron3.5 | N | ** See notes |
The commands I used:
read AGENTS.md
build it -- sometimesCode language: CSS (css)
I expected token counts to rise as the prompts grew, but that didn’t seem to happen in any appreciable way. Sometimes, the token count decreased. This may be related to my manually setting up and installing as many tools as I thought it’d need. It may be related to specifying things like filenames and locations explicitly.
I told it I would use elm reactor from the project root to view the site, but it seemed like the LLMs preferred to use the http server in Python, and ran it from public/. This could cause it to pass tests that would fail, due to bad paths, if the server ran in the project root.
Adding the requirement to produce a Playwright test helped tremendously.
Time Spent
I didn’t keep good track, but until this round, it was around 2-3 hours per to test and produce each of the last three reports, including changes to the prompt in AGENTS.md, building the projects, and posting it to the site.
As the prompt got longer, the build times grew to several minutes, and sometimes it felt like 30 minutes or more.
This round, Nemotron 3 took over 90 minutes.
Nemotron 3.5 Lighting Free
At 13K tokens, it crashed out:
Let "action: layout main: 56.tabular: "Ginger Analytics summary full não complete complete deployingتهاء kompletFull完成 الكامل midline—: clauses 3://umé 1 layouts widths, I, kh
Code language: JavaScript (javascript)
Just for giggles, I told it continue. It just spun the spinner several minutes before I quit and restarted.
The second time around, it wanted access to /tmp, and I refused it. Then it got into a death spiral at 45K when it couldn’t figure out how to end a record that starts with ‘{‘ and that contained a literal list, which looks like [ … ], with a matching ‘}’.
Code Quality
I can’t evaluate it because I don’t know Elm well enough – however, this process of discussing the problem code with LLMs has helped my learning considerably.
Conclusion
All the models basically completed the code without asking questions, but, once again, Nemotrion 3.5 Lighting Free failed.