Locals Only

Dashboard for models on Mac

Every few weeks a new open-weight model lands, and along with it arrives a fresh batch of benchmark charts about math, code, and reasoning. While I welcome this information to help set expectations, I often wonder how a new model might stack up to others I have used on tasks I can directly relate to.

I write technically educational content for a living. The question of whether a model running on hardware I own can draft acceptably useful guidance matters more to me than its score on a competition math set.

With that in mind, I set out to build Locals Only and objectively measure the effectiveness of open models at authoring technical content.

Goal

I wanted a repeatable answer to a focused question: given the same writing assignment, which local model produces the best technical document?

That meant a few things had to be true:

• Every model writes the same documents from identical prompts, so no one model gets an easier assignment than any other.

• Anything objectively checkable never goes to an LLM. Markdown validity, spelling, link health, style rules, and whether the code blocks actually run all get measured by tool with deterministic outputs.

• Nothing subjective gets decided by a single judge model.

• Every generation and judgment lands on disk with full provenance, so a re-run, audit, and comparison across hardware remains possible over time.

Process

The pipeline has four independent layers:

1. Author models/

2. Deterministic validators.

3. Judge models.

4. Human calibration.

Each layer evolves on its own schedule, which keeps a single rubric tweak from quietly invalidating a massive amount of generation runs.

The lab

I tested Locals only on in two environments detailed here.

Machine	Backend	Hardware
MacBook Air	LM Studio	M5, 32 GB unified memory
Linux server	llama.cpp in Docker	Nvidia RTX 3090, 24GB VRAM

Both environments expose an OpenAI-compatible chat API, so every stage of the pipeline runs unchanged in either one. The backend gets detected per machine, and each document records the environment that produced it with details that include hostname, OS, CPU, GPU, backend, and version.

That last detail turned out to matter more than I expected.

The weights differ per backend (MLX and QAT builds on the Mac, GGUF quants on the 3090), so a cross-machine comparison compares deployments of a model rather than identical weights. Every ranking in the reports gets broken out per environment for exactly that reason.

The assignments

The dataset holds 10 writing tasks spread across the Diátaxis document types: tutorial, how-to, reference, and conceptual.

Each prompt carries source material, a target audience, required sections, a word count target, and a companion file of checkable facts:

prompt_id: tut-python-json
facts:
  - claim: null mapping
    truth: JSON `null` maps to Python `None`.

The factuality skill grades documents against those facts rather than against its own recollection, which removes an entire category of judge hallucination from the results.

Judge without fooling yourself

LLM judges have well-documented failure modes, and I tried to design around the ones I could measure.

Every subjective criterion gets scored by at least two judge models, running at temperature 0 in JSON mode against an enforced schema. Invalid output gets exactly one repair retry with the validation error fed back; a second failure gets stored as failed rather than coerced into a number.

Rankings come from pairwise A/B comparisons rather than absolute scores, fed to a Bradley–Terry model. Every pair gets judged twice with the documents swapped. If the two orderings disagree, the pair counts as a tie and the disagreement feeds a per-judge swap-consistency metric.

A judge running the same underlying model as the author counts as a self-judge, which the pipeline allows but tracks, reporting the delta between self-scores and other-judge scores.

Best of all, eval rank refuses to declare a winner it cannot defend:

NOTE: top-2 confidence intervals overlap — no statistically
distinguishable winner.

Results

The results from runs of Locals Only get interesting and explain a lot, particularly around the behavior of reasoning models and the universal difficulty in authoring quality Markdown content that can pass linting.

Everyone writes broken Markdown

Not a single document from any model passed markdownlint in its default configuration during the runs.

author              tool          mean   pass  n
qwen3.5-9b          markdownlint  4.52   0     1
qwen3.6-35b         markdownlint  2.74   0     10
qwen3.6-27b         markdownlint  2.52   0     10
lfm2.5-8b           markdownlint  2.03   0     10
gemma-4-26b         markdownlint  1.63   0     10
gemma-4-31b         markdownlint  1.54   0     10

The same story plays out with Vale against the Google developer style guide, where mean scores land between 1.5 and 3.1 out of 10.

Meanwhile, every single model scores a perfect 10 on spelling and link checking.

These models produce prose that reads clean while quietly violating conventions which documentation tool-chains enforce.

If you plan to drop model output straight into a documentation repository with lint gates in CI, you should also plan for an extra cleanup pass.

Code blocks become the real test

The code-runner executes every code block in a sandbox and scores whether it runs. This separates the models more sharply than any judge did:

author              tool          mean   pass  n
qwen3.6-35b         code-runner   7.71   8     10
gemma-4-26b         code-runner   7.38   7     10
gemma-4-31b         code-runner   6.39   6     10
qwen3.6-27b         code-runner   6.33   7     10
lfm2.5-8b           code-runner   4.09   5     10

An 8 billion parameter model getting roughly half its code blocks to execute tells you something no style score can. This also lines up with the judges, who scored code-quality lowest of every criterion across the board.

The LLM panel noticed the same weakness the interpreter did.

The judges compress everything

Absolute judge scores turned out nearly useless for ranking:

author        style-guide  audience-fit  completeness  factuality  code-quality
gemma-4-31b   9.49         9.50          8.25          10.0        7.42
qwen3.6-27b   9.55         9.35          8.25          9.72        7.00
qwen3.6-35b   9.44         9.25          8.10          9.62        7.20
gemma-4-26b   9.28         9.45          7.60          10.0        7.25
lfm2.5-8b     8.68         7.63          8.15          6.57        5.50

Four of the five models sit within a tenth of a point of each other on style, and three of them post a perfect 10 on factuality. The only model the panel cleanly separates is the 8B, and only because it makes unmissable errors.

This is exactly why the rankings come from pairwise comparisons instead. Asked “which of these two is better?” the same judges produce far more signal than they do handing out numbers in isolation:

== Bradley-Terry leaderboard — Linux/x86_64 · llamacpp ==
rank  author         rating            95% CI     n
1     gemma-4-31b     0.334   [0.039, 0.667]      56
2     gemma-4-26b     0.329   [0.061, 0.650]      54
3     qwen3.6-35b    -0.029   [-0.326, 0.266]     56
4     lfm2.5-8b      -0.634   [-1.028, -0.289]    54
NOTE: The top-2 confidence intervals overlap with no statistically
distinguishable winner.

Even here, the top two remain statistically tied, and the honest reading is that the Gemma pair and Qwen 35B all write comparably well while the 8B model trails them.

Speed tells a different story

Dashboard for models on Linux

The fastest author in the whole matrix is also the worst writer, and the spread is even dramatic in comparison:

Author	Backend	tok/s	Avg time per doc
lfm2.5-8b	llamacpp	163.5	17.3s
qwen3.6-35b	llamacpp	131.6	35.6s
gemma-4-26b	llamacpp	124.4	23.5s
gemma-4-31b	llamacpp	34.9	55.1s
qwen3.6-27b	lmstudio	5.9	763.1s

That last row deserves a note: Qwen 3.6 27B on the MacBook Air averages nearly 13 minutes per document, because as a reasoning model, it spends most of its token budget thinking before it writes a word.

That said, it also topped the Mac leaderboard.

Quality and patience trade off against each other in a super literal way here.

Neither machine can hold two large models at once, so the pipeline serves exactly one model at a time and batches all the work for that model before switching.

On LM Studio, a swap is an unload and a load. On the llama.cpp backend, it recreates the compose service with a new GGUF and waits for the health check. Model load time dominates a full run, which is why the complete matrix is an unattended multi-hour chore.

What I learned

A few things surprised me enough to change how I use these models day to day.

Deterministic validators earn their keep: The single most actionable finding in the whole project is universal Markdown lint failure. This finding came from a tool that costs nothing to run and never hallucinates. You can use such tools to send the LLM judges only the questions that genuinely need judgment.

Absolute scores from LLM judges compress toward the top: Everything is a 9. You can use pairwise comparison to recovers the signal that absolute scoring throws away.

Refusing to declare a winner is a feature: The most valuable line in the output is the one where the tool tells me the confidence intervals overlap.

A benchmark that always produces a ranking is a benchmark that will happily produce the wrong one.

Local models write better prose than they write artifacts: The gap between how good these documents look and how well their code blocks actually execute is the gap between a draft and a publishable document.

Next steps

The pipeline works end-to-end, but my research is still in the early stages.

Ahead of me awaits a full-matrix run across all authors and prompts, a human calibration round to check the judges against my own scoring with Spearman correlation, and then a scale-out to 25 or 30 prompts along with temperature and quantization sweeps.

I suspect the model that writes the best tutorial is not the model that writes the best reference page, so further down the line, I plan to surface that with the addition of:

• Docker-isolated code execution

• Embedding similarity against reference documents

• Elo tracking over time

• A cloud model as a calibration anchor

• Per-document-type leaderboards

Rapid advances in local models make it tempting to trust the vibes and the leaderboard screenshots. Building and running the measurements myself has been the fastest and most educational way to find out where that trust actually holds.