From Coaching Jev to Training My Own: Plumb-4B on JevBench
I hooked TypeSafe's Jev up to a runner game, found an open 4B model that answers the same way, and spent the next day and night training my own version of it. Teacher data on rented GPUs, LoRA rounds on a 4080 Super, a public release on Hugging Face and GitHub, and a submission to JevBench.
On this page
A few days ago TypeSafe put five dollars of credit in everyone's account, and I spent mine on a game. Jev is their decision model, which is a friendly name for a classifier. You hand it some evidence, a question and a list of options, and it hands back a probability for every option. You get a decision and how sure it is, and nothing else.
I built Coach because I wanted to understand why people wanted to use Jev in the first place. A browser game gave me something concrete to try it on. It is a three-lane runner you do not play. You write a playbook in plain English, and every step Jev reads the track ahead, weighs five moves (left, right, jump, duck, hold) and picks one in about a fifth of a second. You rewrite the playbook until it stops crashing.
A full run cost fractions of a cent. That got me wondering how small a model could be and still play this well, and that question ate the next day and most of a night. By the end I had trained my own decision model, published it, and submitted it to the benchmark that ranks these things. This is the whole trip, start to finish.
September 27 update: Plumb-4B reached #1 on the official JevBench v1.4.2.1 composite ranking, ahead of Jev. I added the leaderboard screenshot, the upstream commit, and the caveats below.
The Goal
Three things, because that format keeps working:
- Run a Jev-style decision model on my own hardware, so Coach can play without an API call.
- Train a version of it that is measurably better, on an honest scoreboard.
- Ship it properly: open weights, open data, open code, and a benchmark submission somebody else can rerun from a fresh clone.
House rules: nothing trained on the benchmark, and every number in this post comes from a run I can point to.
What a Jev-Class Model Actually Does
This took me a minute to get, so it gets its own section.
Under the hood it is a classifier. The twist is that the labels arrive with every request, so the same model can sort a support email into refund or complaint and then pick a runner's next move, without being retrained for either. The usual name for that is a zero-shot classifier.
A normal chat model answers by writing text one token at a time. A Jev-class model does one forward pass over the prompt and stops. The options are labelled with letters, and the answer is read straight off the model's scores for those letters at the very last position. Softmax those scores, divide by a calibration temperature, and you have a probability for every option. Zero generated tokens.
That is why it is fast and cheap, and also why it fits a game so well. Every step of a Coach run is exactly that shape: here is the track, here are five moves, pick one.
The open one I found is JevK5, a Qwen3.5-4B model trained to answer in exactly this format, with a small runtime that does the letter readout. Its Q4 build is a 2.7 GB file, which Ollama runs on my 4080 Super without blinking. Hooked into Coach as a local engine, it finally gave me a decision model that costs nothing per step.
The Scoreboard
JevBench is the benchmark for this class of model. It has 231 public decisions split into easy (48), standard (72) and hard (111), plus 308 sealed ones only the maintainer runs. The official score is a harmonic mean of four axes: intelligence, calibration, speed and cost. Harmonic means are unforgiving. One weak axis drags the whole score down, and there is an extra penalty below 50.
On my machine JevK5 aced easy (48 of 48), nearly aced standard (70 of 72), and got 82 of 111 on hard. Hard is where every model on the board separates, and it is mostly long policy documents, trade-offs, probabilities, and dates and numbers buried in paragraphs.
So the target was simple to say. Start from JevK5, keep easy and standard where they are, and move the hard tier.
The Rule That Shapes Everything
The easiest way to beat a benchmark is to train on it. I decided up front that no JevBench item, public or sealed, would ever be used as training data, for tuning, or to pick between checkpoints. The public items are a scoreboard I look at after a round finishes, and that is all.
Two checks enforce it on every round:
- An overlap scan drops any training item that shares more than two 8-word sequences with a public item. Across the whole project that caught 9, all of them generic policy phrases.
- An exact audit normalizes text and confirms no training state or instruction equals or contains a public one. 12,143 records checked, zero hits.
Checkpoints are chosen on my own held-out set instead: questions from domains training never sees, held-out splits of public datasets, and a 65 item hand-written hard set.
Making Training Data
There is no pile of hard, typed decisions with correct answers lying around, so I made one with a teacher.
The teacher is Qwen3.8-27B. For each item it writes a realistic document (a refund policy, a shipping log, an on-call runbook) and a hard question about it in one of the three JevBench types: true or false, pick one option, or an ordinal score. Then it answers its own question twice, separately, with thinking turned on. A question is kept only when both answers agree with the one it intended. Counting failed requests, about one attempt in seven survives, and the survivors are the ones worth having.
Early rounds ran the teacher on my own GPU and my friend's box. That got me the first few hundred questions and made it clear I needed more throughput than a desk can give. So I rented it.
Runpod rents an RTX PRO 6000 (96 GB, Blackwell) for about $1.69 an hour. Each pod ran vLLM serving the teacher in FP8, and a script on my machine fed it prompts in parallel. At peak I had several pods writing at once, each with a stop timer so nothing could bill past the run. Kept questions cost roughly six tenths of a cent each.
Every round also mixes in an equal number of human-labelled questions from public datasets (MNLI, BoolQ, ARC, MMLU-Pro and friends) in the same format. They keep the easy stuff from sliding while the model chases the hard stuff.
The final set: 5,026 teacher questions, including about 500 long documents in the 1,500 to 2,800 word range. The data is public as crh225/plumb-decisions.
Training on a 4080 Super
Each round is a LoRA fine-tune. LoRA freezes the model's 4 billion numbers and trains a small set of extra ones beside them, rank 16 on the attention projections, which is well under 1% of the model. At the end the extra numbers get merged back in, so the result is one ordinary 8 GB file that runs exactly as fast as the original. That is also the only reason this fits on a 16 GB card at all.
The loss is cross-entropy on the option letters only, the same readout the model uses at answer time. For probability questions the target is the full distribution instead of a single letter, so the model learns to spread its confidence when the evidence is genuinely mixed.
After training, one calibration temperature is fitted on held-out teacher questions. For the final model that came out to T = 2.07.
A round is one script: build the set, run the overlap scan, train, calibrate, score the public items. A supervisor kept teachers writing and started rounds on a clock, so it ran for hours while I did other things. I wanted to see it without reading logs, so it got a dashboard.
The Rounds
| Round | Teacher questions | Easy | Standard | Hard | Held-out check |
|---|---|---|---|---|---|
| JevK5 (start) | 1.000 | 0.972 | 0.739 | 0.812 | |
| v2 | 495 | 1.000 | 0.972 | 0.757 | 0.824 |
| v3 | 909 | 1.000 | 0.972 | 0.739 | 0.834 |
| v4 | 2,541 | 1.000 | 0.972 | 0.775 | 0.844 |
| v5 | 5,026 | 1.000 | 0.972 | 0.802 | 0.846 |
The interesting part is v3. It nearly doubled the data, improved the held-out check, and left hard exactly where it started. So v4 and v5 changed the recipe instead of just adding volume.
Hard mining came in v4: before training, run the starting model over every teacher question and keep the ones it gets wrong or answers with less than 80% confidence, plus a quarter of the easy ones so it keeps its footing. Training time goes where the model is weak. Then long documents in v5, because JevBench's hard policies run to thousands of words and a model that only ever trained on short ones reads long ones badly.
v5 landed at 89 of 111 on hard, against JevK5's 82. Against JevK5 it fixes 8 hard items and breaks 1, which is a McNemar p of 0.039, so it is unlikely to be luck. Long policies went from 11 to 14, probability questions 7 to 10, and trade-offs 5 to 6. Dates and numbers stayed at 7 of 15, which is my next problem. Easy and standard did not move, and neither did speed.
Naming It and Shipping It
For most of its life it was called "jevy" in my terminal, which is fine for a folder and bad for a model somebody else might download. It shipped as Plumb-4B.
Shipping it properly meant four places:
- crh225/plumb-4b on Hugging Face, the bf16 weights with a model card and the calibration temperature in
jevk5_config.json, so the JevK5 runtime loads it unchanged. - crh225/plumb-4b-GGUF, Q8_0 and Q4_K_M builds with a Modelfile, so
ollama creategives you the same model Coach uses. - crh225/plumb-decisions, the 5,014 training decisions and a 131 item held-out test, after the overlap drop.
- crh225/plumb on GitHub, the whole pipeline: teacher, rounds, audits, per-item results for every round.
The model card says plainly what it is, where it came from, and how it scored. Apache-2.0 all the way down, with credit to JevK5 and Qwen in the NOTICE file.
Submitting to JevBench
JevBench takes submissions as GitHub issues, and the maintainer's requirements are fair and specific: runnable code, the exact model and license, and a straight answer about whether public items were used during development.
The first two are about reproducibility, so I treated the issue's commands as code. I cloned JevBench fresh, ran the commands in the issue exactly as written, and checked that the result matched: 231 of 231 answered, 0 failed, hard 89 of 111. The route is the simplest one I could find. Install the JevK5 runtime, download the weights at a pinned revision, start its server (which speaks TypeSafe's /v1/systemone format), and point JevBench's standard typesafe adapter at localhost. No patches to the benchmark.
The third requirement deserved a careful answer, and the honest one has two parts. No JevBench item was ever trained on or used for selection. But I did look at the public scores after every round, and that feedback is why v4 added hard mining and v5 added long documents. The issue says both, in those words.
It is issue #84. When I first wrote this, the sealed evaluation and official ranking were still pending. The maintainer has since published the result.
Update, September 27: A Day at Number One
Well, that happened. On September 27, 2026, the frozen JevBench v1.4.2.1 leaderboard put Plumb-4B first among 90 ranked systems. The model I started by wiring into a runner game ended up ahead of Jev on the benchmark's official composite score.
| Rank | Model | JevBench composite |
|---|---|---|
| 1 | Plumb-4B | 65.84 |
| 2 | decider-4b v2 | 64.13 |
| 3 | Jev 1.13.0 | 63.29 |
The official equal-weight composite, captured September 27. The versioned link preserves this ranking even when the live board moves on.
There is a Git receipt too. The benchmark maintainer's commit fb829df, titled "Release JevBench v1.4.2.1 with Plumb-4B," adds the first-place result to the changelog and README. It reached the default branch through PR #115. These are upstream benchmark records, so the evidence survives a later change at the top.
The release commit and its dated changelog. Click through for the full diff.
The scope matters: this is a win on the combination of intelligence, calibration, speed and cost. Jev still leads slightly on Intelligence (53.1 versus 53.0) and Calibration (76.3 versus 75.5); Plumb's speed and estimated cost help it finish ahead overall. The published sealed accuracy is 38.0% for Plumb and 36.7% for Jev. There is plenty of work left in both models.
There is also a backlog of benchmark requests, including new models and updated versions still waiting for evaluation. This ranking covers the systems evaluated for that release, and the order can change as the queue clears. This is a dated result, and the next release may put someone else first.
If I only get a day at number one, I will take the day. That is why I took the screenshots.
Where the work stands now
The model on that leaderboard is the published Plumb-4B v5, trained further from JevK5. Since that release, I have also been experimenting with a separate line starting from plain Qwen. Those newer runs have not produced a replacement I can justify promoting. More training has sometimes improved one group of decisions while making another worse, and matching evaluation to the way the saved model actually serves requests turned out to matter.
For now, I have stopped the latest unfinished training run and brought the desktop GPU back to Coach. The game defaults to Plumb-4B against JevK5, with Jev also available. The standalone Qwen chat-model choices are removed. The newer research models are separate from the published model you can play against.
Back to the Game
Here is the part the leaderboard screenshot does not tell you: the published Plumb-4B is not good at the game that started this project.
I also connected Coach to a larger, general-purpose 27B Qwen endpoint. In my game tests, it played better than both Jev and Plumb. Those are observations from Coach, not a claim that Qwen wins every kind of decision task. The Qwen choices were removed from the current game menu when I simplified it to the decision models; removing them does not change what the comparison showed.
The published Plumb model learned from general decision data: policies, evidence, dates, trade-offs and uncertainty. It was not trained on Coach's game dataset. Picking through a long policy and reliably choosing a safe lane are different demands, even when both end with a letter. The missing game data is a plausible explanation for the gap, rather than proof of its only cause.
That is the lesson I want to keep next to the number-one screenshot. You can train on general data, improve a benchmark score, and still fail at the original goal. The application needs its own evaluation.
Coach now has a Plumb 4B engine next to Jev and JevK5, running through Ollama on the same desktop as everything else in the self-hosting series.
I trained a separate version, plumb-4b-coach, using game decisions labelled by exhaustive search: try every sequence of moves through the visible rows, then prefer the fewest hits and the most coins. In the saved ten-course race, the published Plumb-4B finished zero courses; the game-trained version finished all ten without a hit. That is a limited game evaluation, but it makes the distinction concrete. You can now select Plumb 4B Coach in the game and compare it with Plumb 4B, the published benchmark model. They are separate checkpoints; the Coach version runs locally through Docker Ollama.
Design Decisions
Why start from JevK5 instead of plain Qwen? It already answers in the letter format and already scores well. Starting there meant every round was spent on the hard tier instead of re-teaching the basics.
Why a teacher that answers twice? A 27B model writing its own questions will sometimes write ones it gets wrong. Two independent solves that both match the intended answer throw most of those out, and a wrong label costs far more than a missing one.
Why rent GPUs for the data and not the training? The teacher is a 27B model generating thousands of long answers, which is throughput work a desk card is slow at. Training a rank 16 LoRA on a 4B model fits on the 4080 Super, and keeping it local meant every round ran in the same container on the same card, so rounds compare fairly.
Why choose checkpoints on my own held-out set? Because choosing on the benchmark is a quiet way of training on it. The held-out set is what decided v5 over v4, even though v4 had better calibration on the public items.
Why validate the issue commands on a fresh clone? The maintainer is going to paste them. If they fail, the submission fails, no matter how good the model is.
The Gotchas, So You Skip Them
- Convert Qwen3.5 to GGUF with
--no-mtp. Otherwise the file gains a speculative-decoding block and Ollama fails to load it with a missingblk.32. - Match the training format byte for byte. Python's
json.dumpsputs a space after every comma and colon, and JavaScript'sJSON.stringifydoes not. The tokens differ, and a one-pass model reading letter scores feels it. Coach now builds its questions with Python-style spacing. - JevBench's
typesafeadapter wantsTYPESAFE_API_KEYset even for a local server. Any value works. It also refuses to write results inside the JevBench checkout. - Pin a Hugging Face revision, then leave the history alone. Squashing a repo's history changes the commit hash, and a submission that pins the old hash stops resolving.
- About one rented pod in three arrived broken. Wait until the server actually answers before pointing a teacher at it, or the teacher burns its whole queue on errors in a few seconds.
Conclusion
About a day of rounds, one 4080 Super, and about $40 of rented GPU time. The result is an open 4B decision model that answers in one forward pass and scores 89 of 111 on JevBench's hard tier against its starting point's 82. It also runs my game from a file on my own disk, where its mistakes are a useful reminder that benchmark success does not guarantee success in the application.
The part I did not expect is how much of it was plumbing (sorry). The training itself is a few hundred lines. Most of the work was the teacher, the audits, the scoreboard discipline, the dashboard, and making every published command run on somebody else's machine. That is also the part that makes the model worth downloading.
If you want to try it, the weights, data and code are all linked above, and Coach is the most fun way to watch a decision model think in fifths of a second.
Found this useful?
Comments