Groq Model Speeds
Hexcrawl currently uses groq for inference, mainly because it was the fastest thing for me to get up and running. Ultimately I’ve got it in the back of my head that I’d love to make this whole game entirely client driven with an embedded highly trained small model for inference. No internet connection needed, no ethically toublesome data centers being used, just a fun little single-player game with an LLM spinning it in fun directions.
But here’s the thing - the game is currently a prototype at best, a sandbox for me to play in to find the design space of working with an LLM for real time gameplay. And my mentality with prototypes has always been this: identify the riskiest element and focus your energy on that. Cut every other corner.
Right now, I think the design ideas are the biggest risk. I want to find out first if an LLM can even successfully drive the kind of gameplay I want. Once I’ve proved that, then I can focus on how to package and distribute it.
The Cost
Well, that was my theory at first anyway. But recently groq sent me an email to inform me that my model is going away. Llama 3.1 8B Instant is deprecated and will be completely decommissioned next month.
As you can see from the advertising, they recommend moving over to GPT OSS 20B. For now, that’s what I’ve done, and you may have noticed the difference. Not because the game has gotten better - because it’s gotten slower.
Benchmarks
Cursory research shows that GPT OSS 20B is faster. Here’s a nice chart showing how it clocks in at 931 tokens per second, way faster than Llama 3.1 8B’s 672 tokens per second. So what gives?
What gives is that these models are not apples to apples - GPT OSS 20B is a reasoning model. That means it goes back and forth with itself internally before spitting out an answer, and that back and forth chews up tokens, and time. The evidence is hiding in the stat “time to first answer token”, or how long does it take for the model to start spitting out output. GPT OSS 20B clocks in at 3.0 seconds, even with the reasoning parameter set to “low”. Llama 3.1 8B was scoring 1 second. But let’s look at the chart I really care about: end-to-end response time.
On average, GPT OSS 20B with low reasoning takes about 3.5 seconds to respond to a prompt. Llama 3.1 8B takes on average only 1.7 seconds.
Why Isn’t the Quality Better?
It may be, it’s a little hard to measure that honestly. But here’s the thing – I designed my game, my code, my prompts to work with a small, fast model. The whole point of my architecture was to keep token count and prompt complexity down. I wanted my inference to be as fast and cheap as possible, with the hope of keeping the door open to eventually creating a purely client-driven game with an embedded model as mentioned at the top.
To do that, I’d lean on standard game architecture (procedural generation, classic game-AI, etc) and let the LLM drive these with the smallest prompts possible. True, the LLM does also generate evocative text based on the mechanics its representing - perhaps the biggest lift I’m asking of it - but the actual mechanical inputs it gives tends to be on the order of “select one option from this list”. Given all that, I wouldn’t be surprised if tossing a fancier model at it doesn’t actually produce any noticeable improvement.
What About Cost?
Yeah, I did say “as fast and cheap as possible”, and it turns out GPT OSS 20B fails on that front too. Yes, I’m on the free tier right now, so this may be largely academic, but groq currently lists Llama 3.1 8B at $0.05/token input and $0.08/token output, while GPT OSS 20B is at $0.075/token input and $0.30/token output. That’s a pretty hefty increase.
Where Does This Leave Me?
For the short term, I’m prepared to switch over to GPT OSS 20B. I put in a few code changes required and it’s easy enough to switch between the two models. I want to continue to focus on gameplay and prompting problems, without everything coming to a grinding halt mid August.
UPDATE
The above is all lies! Well intentioned lies perhaps, but my silly brain just couldn’t handle not digging in deeper. I ended up making these two changes:
- I switched from groq to deepinfra. The latter has a ton more models to experiment with, including Llama 3.1 8B and some even smaller ones. Downside is it costs money, but so far I’ve yet to spend more than $0.001, so I think I can handle it.
- I’ve also been experimenting with a bunch of the smaller models in a local ollama instance, and it’s made me realize that they’re just not up to the task. So that’s an interesting problem to ponder – either this silly little game will have surprisingly high RAM/VRAM requirements, or I’ll have to swallow the bitter pill of using cloud inference.
Still all problems for another day. For now I can continue using my precious Llama 3.1 8B and get back to working on the more interesting encounter logic I’ve been wrestling with.
More to come!