The Inference Ouroboros
Let’s talk about inference, and the ouroboros-like relationship it has with a couple broad project vision statements I made very early on.
Vision
In the past I’ve worked on systems that use GenAI to build narrative content in a fire-hose like setting: the attitude was more AI is better. The more I worked with LLMs the more I felt like this was a bad idea. Thus, when I got to build my own project from scratch, I wrote down a couple guiding stars:
- AI Should Expound, Not Invent
- Prompts Should Request Minimal Data
AI Should Expound, Not Invent
Asking AI to build from nothing, whehter that’s world building or writing elaborate narrative content, is a recipe for failure. Despite our perceptions, LLMs in my experience are quite terrible at creating something from nothing. They thrive on context. Give an LLM lots of direction or lots of grist for the mill, and then you’ll get something good.
Recently I was encountering a problem - I gave my NPCs motivations which were hand authored (good), and then used to generate quests for the player (bad). I realized I was asking the LLM to invent too much. The quests it created always felt like beginnings that lead nowhere. No two quests were really related. Occaisonally the LLM might invent something that felt like it was tying some ideas together, but it usually fizzled out and went nowhere.
This is because I wasn’t giving the LLM enough to go on. LLMs are in fact quite good at tying ideas together and adding richer detail to a rough skeleton, but I wasn’t even giving it a skeleton. Fortunately, I realized I had a feature on the roadmap that would solve the problem, I just had to bump up the priority of adding it in: Adventures. More on that later.
Prompts Should Request Minimal Data
The other big code smell in my recent problem was the output size of the prompts. I even went so far as to put some guidance on this into my CLAUDE.md, and I know it’s getting bad when Claude itself starts calling me out on transgressing this:
**Push back on large response designs.** If a feature plan calls for a prompt that returns a complex object (multiple arrays, nested fields, long strings), treat that as a design smell. Propose splitting it into smaller focused calls before implementing.
Yup, Claude itself started complaining at me that the data type of the response structures for my quest creation prompts were getting too big. It tried a couple times to break them down into multiple prompts chained together, but then I get nervous when I see it executing three or more prompts in rapid succession. Something was wrong here and I finally realized how to fix it: Adventures.
Adventures
Adventures are a data format that gives a loose outline of plot. Think of it like a D&D module writtein the sandbox style. It lists locations, items, and NPCs that must exist in the world, and injects inter-related goals for the players. It does not dictate plot, plot emerges from play.
My adventure data format is pretty high level - it might describe a town that is being raided regularly by goblins, but won’t specifically create quests to go kill the goblins or explore their caves. That said, it’s enough content that points in an obvious direction that the LLM happily takes the bait and pushes the story in that direction.
It’s a narrow difference from what I described above, the LLM is in fact still creating the quests, but it’s creating it from rich context that was hand-written, not whole cloth. I’ve found adventures do the thing I felt was really missing from the game - tie the content together into a larger narrative arc for the duration of a single game.
Inference
So waht does this mean for inference? Well, occaisonally I or one of my better-informed testers will call out that I’m using the smallest, cheapest model for this game: llama-3.1-8b-instant. Sure I could get better results if I just pushed to a bigger / more expensive model?
Probably. But now I use that as an indicator that I may be pushing against the early vision statements. If I stop and think about those statements and look at what I’m writing, the answer usually comes to me and it’s not get a bigger model.
This is good for the hopes and dreams of this project anyway. I’d love to be able to port this thing to a PC game with embedded inference. And if it stays a web game, you can bet I need to keep those hosted inference costs down. So I’m sticking to my guns and using the smallest, cheapest model I can get my hands on, and assuming if it’s not enough then that’s probably a design problem, not a budget problem.
Plus, I rather enjoy having that one giant ace in my back pocket. If I can make a fun game out of llama-3.1-8b-instant, just imagine what will happen if I ever flipped it over to a fancier modern model.