Prompt Tip: No Examples
I often use Claude Code to author or iterate on prompt files. This may feel kind of meta - prompting AI to write prompts for AI. I’ve found it pretty effective, especially as I’m usually using a much fancier model for authoring these than will consume it. That said, here’s a problem Claude loves to introduce:
The Problem
For some reason AI loves to suggest that you include examples in your prompts. Here’s an example from my own prompt on authoring quest text:
- Before asking the player to act on any person, item, or place that has not appeared in the conversation log, introduce it in one clause ("Behgol, the man who stole from my family" or "the Crown my father once kept"). Never assume the player already knows names that haven't come up in conversation yet.
The instructions here are good, but the presence of concrete examples is actually a huge mistake and a major point of introducing hallucination. Especially when you consider the tiny model I’m running these against.
The end result is this: I was seeing a lot of quests with thieves named Behgol and/or artifacts that turn out to be a crown held by the character’s father. The examples are well intentioned, but consider that these prompts also have all kinds of concrete game state injected into them. It’s easy to see how the model might confuse facts that are actually driven by concrete game data vs those that were included merely to illustrate rules.
The Solution
Obviously first and foremost rip out those examples! The above prompt can be reduced to:
- Before asking the player to act on any person, item, or place that has not appeared in the conversation log, introduce it in one clause. Never assume the player already knows names that haven't come up in conversation yet.
Also, I’ve go so far as to add the following to my project’s CLAUDE.md file to try and prevent Claude Code from introducing this problem in the first place:
- **No concrete examples.** Do NOT include concrete examples in prompts. The LLM cannot distinguish between specific named entities listed for the sake of example from actual game data that is desirable to include in the generated content. Examples in prompts will inevitably leak into actual game data and thus must be avoided.
This is actually part of a larger section in my CLAUDE.md about prompt design rules. But like most issues around LLM generated work, nothing beats human review. I do like letting Claude author and iterate on my prompts, but it does mean that ever now and then I find myself taking a full sweep through all my prompt files, reading every line, thinking critically about what effect it has on the game, and tuning it.