Description
Using a general LLM for classification has always been awkward. You hand it a ticket, ask for JSON, and hope the format holds. Asking five questions means five requests, or one prompt where the questions contaminate each other. And the
kev replaces that path entirely. It is a LoRA adapter plus a small readout head on Qwen2.5-0.5B that reads a document once and answers every question about it in a single forward pass, with no decoding at all. The document and all the questions are packed into one sequence; a block-causal mask lets each question see the document but never its siblings. A pointer head scores each option's hidden state against the decision token, and the softmax over those scores is the answer.
So the output is a probability that was actually trained, not a phrase that sounds confident. The head learns by cross-entropy against labelled outcomes, and lands at 0.065 ECE on held-out data, 0.031 after one-parameter temperature scaling. The isolation claim is measured, not asserted: the same questions packed together and sent separately agree to 3.7e-6, and the packed request is twice as fast.
The API follows TypeSafe's System One contract.
The part that matters most in practice: it really does train on a laptop.
Three question types:
Encode once, answer many: the document's hidden states are computed a single time and every question hangs off them as its own branch. Adding questions does not scale the cost linearly — the packed request measures 2× faster than sending them one by one.
Isolation that was tested: the mask stops a question from reading its siblings. Put a secret in a sibling question and the model answers it with p = 0.03; put the same secret in the document and it is 0.99. Question order does not matter either, because each branch restarts its position ids after the document.
"None of the above" actually works: the decision token sits after every option, so the model has read the full list before it scores anything. Fake delimiters planted in option text do not change the option count or smuggle in a forged choice.
Local server with a playground: a Next.js UI lets you edit the document and the questions and re-run instantly, with built-in packed-vs-separate and option-permutation comparisons. There is also a chess page where the legal moves are the options of one Choice question, the board is the document, and a Score rates the position in the same request.
The training recipe is public: six open datasets — Banking77, AG News, MNLI, BoolQ, SST-5, Yelp — converted into TypeSafe-shaped requests, 9,000 records and 13,500 questions over two epochs. One command runs anything from a one-minute smoke test to the full run, and
A frozen evaluation suite:
The comparison against Jev, including the losses: inside the training distribution kev-0.5b scores 79.7% micro accuracy against Jev's 81.1%, a difference inside the noise. On eight datasets it never trained on, kev drops to 63.3% against Jev's 82.3% — 19.1 points behind, with the confidence interval printed in the README. Calibration goes the other way (ECE 0.052 vs 0.075) while option-order stability clearly does not (flip rate 0.208 vs 0.000).
Stated limits: the backbone is 0.5B parameters, so general knowledge is thin. Training used 384 document tokens and 1,024 branch tokens; serving caps at 8,192. On MPS it runs fp32, one request at a time, with no cross-request KV cache. The calibration numbers come from the training datasets and say nothing about a new workflow until you recalibrate on your own labelled outcomes.
What it needs to run: Python 3.12+ and uv, plus Node 20 for the playground. Tested on Apple Silicon (MPS); CUDA is untested. Weights live on Hugging Face as
confidence: 0.9 that comes back is a number the model wrote, not a number that means anything.kev replaces that path entirely. It is a LoRA adapter plus a small readout head on Qwen2.5-0.5B that reads a document once and answers every question about it in a single forward pass, with no decoding at all. The document and all the questions are packed into one sequence; a block-causal mask lets each question see the document but never its siblings. A pointer head scores each option's hidden state against the decision token, and the softmax over those scores is the answer.
So the output is a probability that was actually trained, not a phrase that sounds confident. The head learns by cross-entropy against labelled outcomes, and lands at 0.065 ECE on held-out data, 0.031 after one-parameter temperature scaling. The isolation claim is measured, not asserted: the same questions packed together and sent separately agree to 3.7e-6, and the packed request is twice as fast.
The API follows TypeSafe's System One contract.
POST /v1/systemone takes the same request and returns the same response shape, so the official typesafe-sdk talks to a local kev server after changing base_url and nothing else — their quickstart runs unmodified.The part that matters most in practice: it really does train on a laptop.
kev-0.5b finishes in about 1h45m on an Apple M5 and serves a six-question request in roughly 160 ms.Features
Three question types:
noul for yes/no, choice for 2 to 255 options, score for ordered levels. All three share one readout head and the option count comes from the request, so no per-task head has to be trained.Encode once, answer many: the document's hidden states are computed a single time and every question hangs off them as its own branch. Adding questions does not scale the cost linearly — the packed request measures 2× faster than sending them one by one.
Isolation that was tested: the mask stops a question from reading its siblings. Put a secret in a sibling question and the model answers it with p = 0.03; put the same secret in the document and it is 0.99. Question order does not matter either, because each branch restarts its position ids after the document.
"None of the above" actually works: the decision token sits after every option, so the model has read the full list before it scores anything. Fake delimiters planted in option text do not change the option count or smuggle in a forged choice.
Local server with a playground: a Next.js UI lets you edit the document and the questions and re-run instantly, with built-in packed-vs-separate and option-permutation comparisons. There is also a chess page where the legal moves are the options of one Choice question, the board is the document, and a Score rates the position in the same request.
The training recipe is public: six open datasets — Banking77, AG News, MNLI, BoolQ, SST-5, Yelp — converted into TypeSafe-shaped requests, 9,000 records and 13,500 questions over two epochs. One command runs anything from a one-minute smoke test to the full run, and
--holdout excludes sources for out-of-domain evaluation.A frozen evaluation suite:
evals/decision-v1 keeps training, calibration, development and locked test partitions apart, pins dataset and base-model revisions, and records provenance per record. Experiment runs hash the code, the suite and the git commit, refuse configurations outside an allowlist, and only touch the locked test behind an explicit --allow-test.The comparison against Jev, including the losses: inside the training distribution kev-0.5b scores 79.7% micro accuracy against Jev's 81.1%, a difference inside the noise. On eight datasets it never trained on, kev drops to 63.3% against Jev's 82.3% — 19.1 points behind, with the confidence interval printed in the README. Calibration goes the other way (ECE 0.052 vs 0.075) while option-order stability clearly does not (flip rate 0.208 vs 0.000).
Stated limits: the backbone is 0.5B parameters, so general knowledge is thin. Training used 384 document tokens and 1,024 branch tokens; serving caps at 8,192. On MPS it runs fp32, one request at a time, with no cross-request KV cache. The calibration numbers come from the training datasets and say nothing about a new workflow until you recalibrate on your own labelled outcomes.
What it needs to run: Python 3.12+ and uv, plus Node 20 for the playground. Tested on Apple Silicon (MPS); CUDA is untested. Weights live on Hugging Face as
jaredpalmer/kev-0.5b and are attached to the GitHub release. Apache-2.0.
