Description
Most decisions inside an agent are small: which queue handles this, should this retry, does the evidence support the claim. Ask a chat model and it writes a JSON blob that your program immediately parses back into an if statement — dozens or hundreds of tokens spent on a round trip. OpenJev removes the round trip: it reads the model's logits over the options you declared, normalizes them into probabilities, and returns after one forward pass without sampling an answer token.
The author measured it against the ordinary path on one RTX 3090 with the same frozen Qwen3.5-4B: 21 binary criteria took a median 1.023 s and 0 output tokens read directly, versus 5.332 s and 111 tokens to generate a compact JSON array — 5.21× slower. Criteria and option descriptions arrive with the request, so no trained classifier head is involved, and one long state can be prefilled once then branched across many criteria in parallel, which took 777 decisions from 333 s to 38.8 s. The fixtures, raw timings and row-level predictions are all committed.
Direct option logits: the model's raw scores for the declared options are read and normalized across only those options — no answer sentence, no JSON parsing, no repair loop, one forward pass to a probability.
Runtime-defined criteria: the question and the option descriptions travel with the request, so changing what is being judged is editing text rather than training another classifier.
Shared state, parallel branches: a long state can be prefilled once and branched across many criteria at once, which raised throughput from 2.33 to 20.03 decisions per second on a 37-state by 21-criterion workload.
Auditable results: every row carries the per-option scores, timing, the exact model revision and a prompt hash, so a later review can tell which run and which weights produced it.
Published benchmarks and failures: the fixtures, runners, row-level predictions, raw timings and known failure cases are committed, so the numbers can be rerun rather than taken on trust.
Try it in a browser: the project site runs a small Qwen model locally through WebGPU and lets you run the same question down both paths — direct readout and JSON generation — to compare the timings yourself, with no waitlist.
Batch scoring from the CLI: an openjev-score command reads JSONL input and writes results, takes an explicit model and revision, and switches to a shared prefill mode when every row carries the same state.
Open and self-hostable: MIT licensed, reproducing the interface pattern with open weights, and a 3090-class card is enough to hold a 4B BF16 model.
The author measured it against the ordinary path on one RTX 3090 with the same frozen Qwen3.5-4B: 21 binary criteria took a median 1.023 s and 0 output tokens read directly, versus 5.332 s and 111 tokens to generate a compact JSON array — 5.21× slower. Criteria and option descriptions arrive with the request, so no trained classifier head is involved, and one long state can be prefilled once then branched across many criteria in parallel, which took 777 decisions from 333 s to 38.8 s. The fixtures, raw timings and row-level predictions are all committed.
Features
Direct option logits: the model's raw scores for the declared options are read and normalized across only those options — no answer sentence, no JSON parsing, no repair loop, one forward pass to a probability.
Runtime-defined criteria: the question and the option descriptions travel with the request, so changing what is being judged is editing text rather than training another classifier.
Shared state, parallel branches: a long state can be prefilled once and branched across many criteria at once, which raised throughput from 2.33 to 20.03 decisions per second on a 37-state by 21-criterion workload.
Auditable results: every row carries the per-option scores, timing, the exact model revision and a prompt hash, so a later review can tell which run and which weights produced it.
Published benchmarks and failures: the fixtures, runners, row-level predictions, raw timings and known failure cases are committed, so the numbers can be rerun rather than taken on trust.
Try it in a browser: the project site runs a small Qwen model locally through WebGPU and lets you run the same question down both paths — direct readout and JSON generation — to compare the timings yourself, with no waitlist.
Batch scoring from the CLI: an openjev-score command reads JSONL input and writes results, takes an explicit model and revision, and switches to a shared prefill mode when every row carries the same state.
Open and self-hostable: MIT licensed, reproducing the interface pattern with open weights, and a 3090-class card is enough to hold a 4B BF16 model.
