Locals Only

Dashboard for models on Linux

Every few weeks, a new open-weight model lands and along with it arrives a fresh batch of benchmark charts about math, code, and reasoning.

I like to glance at the benchmarks to size up the model capabilities, but I often wonder how a new model might stack up to others on tasks I can directly relate to.

With that in mind, and perhaps against my better judgment, I set out to build a formal eval called Locals Only, and use the eval to objectively measure the effectiveness of open models at authoring technical content.

Dashboard for models on Mac

Goal

I wanted a repeatable answer to a focused question: given the same writing assignment, which local model produces the best technical document?

My constraints for this eval boil down to the following points:

• Every model writes the same documents from identical prompts, so no one model gets an easier assignment than any other.

• Anything objectively testable never goes to an LLM. Markdown validity, spelling, link health, style rules, and whether the code blocks actually run all get measured by a tool with deterministic outputs.

• Nothing subjective gets decided by a single judge model.

• Every generation and judgment gets persisted to enable re-runs, audits, and comparison across hardware and future models over time.

Process

The pipeline has four independent layers:

1. Author models

2. Deterministic validators

3. Judge models

4. Human calibration

Each layer evolves on its own schedule, which keeps a single rubric tweak from quietly invalidating a massive amount of generation runs.

Next steps

The pipeline works end-to-end and provided me with some concrete numbers to go with the gut instinct one tends to build up when running inference with local models on a regular basis.

This project represents my first attempt at a full bespoke eval, and ended up a good learning experience that will certainly contribute to future efforts.