Earth, I apologize for pulling an all-nighter teaching Thaura to manage my PDF archives.
I had this archive of around 300 PDFs, and maybe 5% had proper filenames. Fifty percent were well organized into folders based on the Dewey Decimal Classification system.
I asked for an archiving tool to check the files to see if they looked like 320 Trask, Haunani-Kay From a Native Daughter_ Colonialism and Sovereignty 1999.pdf.
Before long, I was asking it to fix the filenames that were close to correct. It did. Then I asked it to read each PDF and generate a proper filename from its contents. That worked as well.
Writing consistent filenames for PDFs is actually kind of hard.
Keeping things organized is even harder, so I decided to use the well-established — if flawed — Dewey Decimal Classification system, which public libraries still use widely today. The above book’s file path is Reading Materials/300 Social sciences/320 Political Science and Government/320 Trask, Haunani-Kay From a Native Daughter Colonialism and Sovereignty 1999.pdf.
I asked it to move the unsorted files into the proper locations. It did a pretty bad job, so I eventually got it to produce appropriate subdirectories based on the DDC, reclassify the files, and ensure correct DDC prefixes.
I’m not sure why, but telling it to make more subdirectories seemed to helped it classify the articles a little bit better.
I needed an INDEX.md of all the files. It tried to do it all at once, and failed. There were too many files — 300+ was simply too many.
I said, do it with depth-first recursion. Produce an index for the subdirectories first. This worked, until it hit a subdirectory with 30+ PDF files.
The big lessons learned so far:
- Use established classification systems and filing schemes instead of inventing your own.
- Break up large collections into smaller ones.
- Put the classification and other metadata into the filename.
- This LLM can’t handle the metadata text (which includes a synopsis) for 30+ documents at once.
Since the LLM couldn’t work with that much text, I told it to write JSON files for each document in the 30+ article subfolder. The data is externalized into a file, sparing memory.
JSON, unfortunately, caused a lot of error messages. The real issue wasn’t that Bash hates JSON — it’s that LLM-generated code juggling structured data inside shell strings is fragile regardless of format. Switching to YAML helped because it’s structurally simpler and easier to parse without heavy escaping.
I also had it create a .yaml file for each document, and leave it in the directory.
I noticed that the Bash code it produced was still having problems. Thaura was resorting to writing Python programs to do the processing.
So I told it to write that program to a file. It created index_builder.py, and it worked fine.
I could finally index the entire archive.
It took some tweaking, but it basically became a recursive index builder.
(Note: I think, if I externalized the program sooner, JSON would not have been a problem.)
Externalize the Memories
I don’t think that changed Thaura’s resource consumption, but it was good to have the Python program in a file, just in case I lost the chat.
Also, run a prompt like this to externalize some of the internal state:
Write an AGENTS.md file into the Reading Materials root that will help reconstruct all the work that's been done so far.
I didn’t look into the index_builder.py code, but it was clearly doing a lot of filename checking and cleanup, because previous efforts had left stray –, _ and other characters.
I refactored out filename checking and repair into another program. This improved performance, and reduced code size.
-rw-rw-r-- 1 johnk johnk 9893 Jul 7 13:34 filename_cleanup.py
-rw-rw-r-- 1 johnk johnk 18456 Jul 7 13:33 index_builder.py
I also broke out the synopsis generation into its own program.
-rw-rw-r-- 1 johnk johnk 11114 Jul 7 13:37 doc_extractor.py
-rw-rw-r-- 1 johnk johnk 9893 Jul 7 13:34 filename_cleanup.py
-rw-rw-r-- 1 johnk johnk 11071 Jul 7 13:37 index_builder.py
While the total amount of code is greater, each program is independent, so when Thaura needs to comprehend and edit, it’s less code.
(This feels like we’re back in the era of 8-bit computers with 64K of RAM or less. You can’t keep all your programs and data in memory.)
Synopsis Quality and Token Budget
Somewhere along the way, synopsis quality declined.
I think it’s related to using a local PDF extraction tool, which pulls raw text from PDFs.
When the scripts weren’t so ambitious about summarizing from extracted text, Thaura would fall back to using the web or the LLM’s training data to write a synopsis, especially for well-known books.
As I externalized the process into Python programs, it probably stopped doing that. The synopsis writing process became almost instantaneous (especially after I told it to stop extracting all PDFs to a single file, and make individual files per PDF).
Prior to creating Python programs, I think the LLM would resort to writing a synopsis from scratch, and this would take minutes.
That, presumably, meant the token budget was being burned up.
In retrospect, this was not necessary: almost my entire archive consists of research papers and books.
Research papers contain DOI URLs, and also contain a section titled “Abstract” that can be extracted, or copied from a website like PubMed.
Books have summaries on Goodreads, Amazon, at the publisher, and elsewhere.
All of that work Thaura undertook, prompted by a simple suggestion to find internet info or info in the model, was a huge waste of energy, and could easily produce less accurate synopses due to hallucination.
A well-tuned extraction and scanning program, combined with some clever search tools, could probably perform accurate file naming based on DOIs, abstracts, front matter in books, and ISBN numbers. Book front matter contains LOC classification numbers. Research papers contain categorization keywords, and are also, obviously, classified by the journal’s field.
Of course, this approach covers research papers and published books well — gray literature, reports, and materials lacking standard identifiers would still need manual attention.
Conclusion
While this programming/vibe coding excursion was a lot of fun, and I learned a lot, I was actually replicating a lot of work already performed by LIBRARIANS, who probably did a much better job than the LLM and programs did.
This begs the question: why don’t these kinds of files automatically stay organized? Do the search technology companies want us to hoard information in big messes, so they can find it for us?
Yeah, bro. It’s already organized.
Yes… this is definitely a DOGE-brain-is-screwed feeling.
Author: admin
This is the server’s system administrator. This site is undergoing some changes.