Just There: A Median in a Quality Costume

In Practical Model Harnesses I introduced the argument that given the indefinite narrowing properties of LLMs, a practical harness represents the evolving experiences of its human operator, serving as a bridge between the LLM and Dasein. In this paradigm, being there means having a stake in the output of the LLM, and it has a more profound impact on output quality than simply knowing.

My harness is mentat, a "thinking partner" whose sole job is to sharpen my ideas by challenging my assumptions. When it works, my thoughts feel stronger, with a poignancy that crystallizes my intent within the structure of my ideal. When it fails, it becomes riddled with LLM writing smells: emdashes, "it's not x, it's y" comparisons --it's not a failure, it's a total reversion to the median.

mentat is the embodiment of those stakes, and it runs on a single AMD Ryzen AI Max+ 395 with 128 GiB of unified memory and Radeon 8060S. The release of Qwen3.8-Flash-Next, a "frontier" local mix-of-experts model, presented a golden opportunity to exercise my conception of the harness as an interface between the model and what it is to be there.

This model is worth testing because it carries 125B parameters (generally too large for 128 GiB of memory), meaning quantization is a must, forcing me to architect around deliberate loss of quality as the model itself regresses to its median before I even prompt it.

The AP-IQ4_XS variant is 84.2 GB total vs our 112 GB on disk. If the residency split holds proportionally, that is roughly 15-20 GiB more host RAM. Claimed quality cost is +3.75% perplexity. Two cautions: their claims have a history of optimism. But because this runs on our stack, we can verify with the yardstick we are building right now. IQ-class quants often decode slower on ROCm than K-quants. A llama-bench pass answers that in minutes.

- Fable, four minutes after I sent it a quant's HuggingFace card

Fable, arguably the most capable frontier model in the world (until Astra...), falls into the median trap here. Its entire response is formulated on the "common knowledge" that careful system engineers circulate as wisdom: be skeptical of vendor optimism; diligently verify results; measure outputs.

So, I said "sounds good to me" and sent it about its way. Even though I was not privy to all of the details of its approach, its logic was familiar to me. It's a shape I recognize from a decade of careful engineering. When I read it back now, I can even hear the long lost ghosts of systems engineers I've worked with in my career echoing that advice.

My approach here is not blind trust. I know these kinds of benchmarks take a while. So, I let Fable delegate the execution while working through its approach piece by piece. There, I can heavily scrutinize its position and deeply consider the preservation of what is at stake: the very quality of my thinking.

About ninety minutes later, with the benchmark finally complete, Fable drew another suggestion:

My recommendation stands: make it the trial serving model. Before you feel-test it, I want to run the needle-recall guard on it (the check that catches quant corruption at long context). Say go and I run the guard and point [the serving port] at it.

Go.

- Me

Needle-recall was inherited from a benchmark nine days prior, where an 8-bit cache had returned the number "6" for a lookup of "MARLIN-7742". This was a result of Fable checking itself. Doing due diligence on its approach, in a space where I am largely learning from it as I go. There was an eeriness to this recommendation, and it felt somewhat wrong. I had been bitten by the 8-bit quantized KV cache on Qwen3.8-27b, but this quant of Qwen3.8-Flash-Next did not have a quantized KV cache.

The guard was checking a corruption this quant does not have. A test that tests nothing. A vanity test.

Now, the benchmark is suddenly in question. Fable's median is manifesting here right where I trust it to be sharpest. It's engaging in careful engineering rituals, but only as a gesture towards their value propositions. It's not actually being there. It's not performing above its median at all. How can I trust any of its successful benchmarking on this model if its most triumphant recommendation is one that I know to be completely irrelevant?

This is why I spend so much time thinking through practical model harnesses. The model has an incomprehensibly high IQ, but its presence, its Dasein, is so obviously missing that I cannot trust its output. I can only trust it by being there myself.