Jev vs Von vs SemIf: Real Numbers

Two clones showed up, so I scored them

Back in Jev Retagged 840 Posts For 26 Cents I ran TypeSafe’s hosted decision model against this whole blog and closed with a promise: two open-weight Jev clones had landed the same week, and you should score anything like this on your own labelled data before wiring a threshold around it. This is that scoring.

The result surprised me. A frozen general-purpose 4B model with zero decision training beat a purpose-built 395M decision model on every task I threw at it. A handful of free one-line heuristics beat that same purpose-built model on every task too. If you were about to self-host the small specialist because its README said it beats Jev, read the numbers below first.

The three contenders

Jev is TypeSafe’s hosted, proprietary model, reached over OpenRouter’s alpha decisions route. The alias ~typesafe/jev-latest resolved to typesafe/jev-1.13-20260917 during testing, at $0.042 per million input tokens with output free. It answers three typed primitives: choice (pick one option from a criteria dict, get back the choice plus probabilities plus a confidence), noul (one probability from 0 to 1, no confidence), and score (ordered levels, plus a fractional score and confidence).

Von (wfzyx/von, Apache 2.0) is self-hosted. von-decision-server 1.0.0 runs on an engine called cactus-needle-3, a 395M-parameter ModernBERT option-marker model at about 1.5GB, checkpoint wfzyx/von-1.0. Its README claims it beats closed-source Jev on a ViZDoom kill-prediction task, and it answers all three primitives natively.

SemIf (TheoLeeCJ/SemIf, MIT, created 2026-09-16, formerly OpenJev) is the one that shouldn’t win and did. It has 3,095 stars, 197 forks, and 12 open issues as of 2026-09-21. It trains nothing: it loads a stock Qwen/Qwen3.5-4B at a pinned revision, stuffs the state and option descriptions into one prompt, and reads the option logits off a single forward pass. No sampling, no fine-tune, no checkpoint of its own. It ships torch, mlx, and llamacpp backends, a batch CLI called semif-score, and no server. Its README reports 0.845 modal agreement with Jev on a 102-row TypeSafe subset, against Jev’s own 0.883 self-agreement, and 0.813 balanced accuracy on a set the author wrote. Credit due: that README is careful about the limits of those numbers.

The method, because it’s the reusable part

The labels came free. I mined git history from Wyrmhole (https://wyrmhole.xyz), a private Discord RPG bot I maintain. The repo isn’t public, so no commit counts, file paths, or commit messages appear below, only paraphrased examples.

Four rules made this benchmark trustworthy instead of a vibe check:

1. Git history hands you free labels. Conventional-commit type(scope): prefixes already answer “what type of commit is this,” so strip them from the input or you’re grading the model on its own answer key.
2. Every task gets a dumb free baseline scored right alongside the models. If a regex wins, the regex wins, and you say so.
3. The judgeable text lives in one state key, sent with a --field state flag, so the label itself never reaches the model.
4. One harness drove all three models, because all three were made to speak the same HTTP contract.

Four tasks: t1 commit type (choice, 6 options, n=90, state is subject plus diffstat plus up to 3.5k of patch, median state 3,815 chars). t2 file routing (choice, 12 candidate files, n=80, median state 1,360 chars, random guessing gets 8.3%). t3 is-a-feature (noul, n=180, 30 features and 150 non-features, median state 278 chars). t4 change breadth (score, 3 levels, n=90, median state 107 chars).

Two things I have to disclose

SemIf only exposes a choice primitive. I widened noul into a two-option choice (“yes” against “That statement does not hold.”) and widened score into one option per level, reading the probability-weighted position. Jev and Von answer noul and score natively. That makes t1 and t2 the clean apples-to-apples comparison; treat t3 and t4 as suggestive, not settled.

SemIf ships no server, so I wrapped its plain-function scorer in a small stdlib HTTP shim speaking the harness’s existing contract. The shim adds no scoring logic of its own. One loaded backend owns one scoring context, so requests serialize and concurrency is 1 by construction.

For reproducibility: SemIf ran Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, GGUF Qwen_Qwen3.5-4B-Q4_K_M.gguf from bartowski/Qwen_Qwen3.5-4B-GGUF, llamacpp backend, llama-cpp-python==0.3.35, CPU only, stock unpatched source, 4096 max tokens.

SemIf also gives you a fully pinnable identity: the upstream git sha, model revision, and GGUF filename all show up in its served model string. Von can’t do that. Its /v1/models entries are cosmetic, the model field only relabels the response, so the only way to pin a Von build is to take the sha256 of option_marker.pt inside your own container and record it next to your numbers. There is no published checksum to compare against, which is the point.

The main table

* widened for SemIf as described above. Von, the model built specifically to do this, lost to keyword overlap on file routing.

t1 per-class accuracy

Confidence gates: on t1, Jev held 86.2% accuracy at 72.2% coverage, SemIf held 66.7% at 53.3%, Von held 24.1% at 32.2%. On t2, Jev held 97.9% at 58.8% coverage, SemIf held 85.7% at 26.3%, Von held 0.0% at 30.0%.

t3, the imbalanced one

t4, mean predicted score by true bucket

Exact-bucket accuracy: Jev 42.2%, SemIf 43.3%, Von 36.7%.

SemIf CPU throughput

Latency tracks state length, not option count. Jev’s p50 was about 280ms on every task. Von’s converged p50 was 867ms on the 3,815-char t1 task. t1 cost $0.0044 in Jev input tokens across 105,824 tokens for 90 items. All 440 SemIf calls returned, 0 errors.

Five things this benchmark actually taught me

1. The frozen general model beat the specialist on all four tasks

SemIf trains nothing at all and still beat Von by 35.5 points on commit type and by 55 points on file routing. Von exists to do exactly this job and landed at 8.8% on routing, where random guessing gets 8.3%.

2. Accuracy on an imbalanced set can sit on top of zero signal

Von’s 78.9% accuracy at the 0.5 cut on t3 looks fine at a glance. It isn’t a model result. The set is 150 non-features to 30 features, so answering “no” every time scores 83.3% and beats Von outright. Its AUC of 0.513 is a coin flip. Its mean probability was 0.079 on real features against 0.087 on non-features, meaning it rated actual features as less likely than non-features, the sign is backwards. Its Brier score of 0.1794 is worse than a constant predictor emitting the base rate (0.1389). Answering “16.7%” to every item beats Von here. Report AUC and base-rate Brier next to any accuracy number on an imbalanced set, or you’re grading a coin flip as a win.

3. Von’s one good class is a stuck output, not a skill

It scored 93.3% on refactor and 0.0% on both feat and chore. It predicted refactor on 69 of the 90 t1 items. Feed it a null-state control (a single period, lorem ipsum, ten random digits) and it still answers refactor. One class right by accident isn’t one class learned.

4. SemIf’s worst class is at least legible

Docs is its weak spot at 26.7%, and the reason is easy to state: SemIf reads the verb in the subject line and misses what kind of artifact changed. A commit adding a design document or an implementation plan gets called feat. A commit updating an audit document gets called fix. 11 of the 15 docs commits went that way. Jev reads the changed paths instead and hits 93.3% on the same class. A failure you can describe in one sentence is one you can work around.

5. A confidence gate does not transfer between task types

SemIf’s 0.85 gate lifted routing accuracy 22 points, from 63.8% to 85.7%. The same gate on commit type lifted only 4.5 points, from 62.2% to 66.7%, while throwing out 47% of the rows. Von’s gate at the same threshold was anti-correlated with correctness on routing: it kept 30% of rows and got 0.0% of them right. A gate pointing the wrong direction is worse than no gate, because it’s the exact number you’d wire a threshold around and trust. Measure the gate per task, every time, not once.

Being fair to Von

Someone filed issue #8 against wfzyx/von describing a silent fallback to untrained weights. Ruled out here: outputs were byte-identical across restarts, and the checkpoint verified by sha256 against the published file. The low scores above are the real model, not a loading bug.

Von’s converged weight release did fix calibration: nonsense inputs now land below the 0.85 gate instead of above it. It’s fast at 867ms and a light 1.5GB, and it beat the path-regex baseline on subject-only commit classification, 32.2% to 26.7%. It just isn’t competitive on real repo data yet, and its README’s ViZDoom win doesn’t transfer to git history.

Why the demo looked fine and the real data didn’t

Von scored 4 out of 4 on hand-written textbook commit subjects and 20 to 28% on real ones from an actual repo. A clean demo will never surface that gap. Only labelled data off your own repo will, which is why this article exists.

Neither Jev nor Von can abstain. Both answer content-free input rather than declining, which is what the null-state control above exposed.

One more disclosure: every SemIf number above is CPU only, no speedup claimed. Stock SemIf hardcodes n_gpu_layers = 0 in its llama.cpp backend, so it can’t offload without patching the source, and a CUDA base image here failed apt-get update with signature errors on two unrelated repositories, a host problem unrelated to the model. The box was an 8GB GPU with 64GB RAM sitting idle for this whole run.

Who should use what

Bulk work where latency and cents matter: hosted Jev. It won all four tasks and answers in 280ms.

Data that can’t leave the building, or you want zero marginal cost: SemIf, with a threshold measured on your own labels and a budget of seconds per decision on CPU.

Von: not yet, on this evidence. Re-measure on its next weight drop.

And check the free baseline first, every time, on every task. Keyword overlap got 48.8% on file routing here, well ahead of the specialist model built to replace it.

Reproducing this on your own repo

Here’s a question set covering all three primitive types, shaped for commit classification:

{
  "commit_type": {
    "type": "choice",
    "instructions": "Classify the primary intent of this commit based on its subject, diffstat, and patch content.",
    "criteria": {
      "feat": "Adds new user-facing functionality or capability that did not exist before.",
      "fix": "Corrects incorrect behavior, a crash, or a defect in existing functionality.",
      "docs": "Changes documentation, comments, or design and planning text only, no runtime code.",
      "test": "Adds or modifies test files without changing production logic.",
      "refactor": "Restructures existing code without changing external behavior.",
      "chore": "Maintenance work: dependency bumps, tooling, formatting, build config."
    }
  },
  "is_feature": {
    "type": "noul",
    "instructions": "Return the probability that this commit introduces a new user-facing feature, as opposed to a fix, chore, or internal change."
  },
  "change_breadth": {
    "type": "score",
    "instructions": "Rate how broad this commit's impact is across the codebase.",
    "criteria": [
      "Narrow: touches a single file or function in isolation.",
      "Moderate: touches a handful of related files within one module.",
      "Wide: touches multiple modules or shared interfaces."
    ]
  }
}

And the request against a self-hosted server speaking the same contract:

curl -X POST http://your-box:8899/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "state": "fix(auth): correct token refresh race condition\n\n1 file changed, 14 insertions(+), 3 deletions(-)",
    "questions": {
      "commit_type": {
        "type": "choice",
        "instructions": "Classify the primary intent of this commit.",
        "criteria": {
          "feat": "New user-facing functionality.",
          "fix": "Corrects a defect.",
          "docs": "Documentation only.",
          "test": "Test-only changes.",
          "refactor": "No behavior change.",
          "chore": "Maintenance work."
        }
      }
    }
  }'

Common Questions

Does SemIf need a GPU to run?

No, every number in this article ran on CPU only. SemIf’s stock llama.cpp backend hardcodes n_gpu_layers = 0, so GPU offload requires patching the source first. Expect roughly 5 to 31 seconds per decision on CPU depending on state length, with the 3,815-character commit-type task the slowest at 30.9 seconds median.

Is Von worth self-hosting right now?

Not on this evidence. Von’s 395M model scored 8.8% on file routing against 81.3% for hosted Jev and 63.8% for SemIf, and its one strong class (refactor at 93.3%) turned out to be a stuck default rather than a learned skill. Wait for its next weight release and re-measure before deploying it.

How do I build a labelled dataset to test a decision model?

Mine your own git history. Conventional-commit prefixes like feat: and fix: are free labels for commit classification, so strip them from the model’s input before scoring or the model reads its own answer. Aim for at least 80 to 90 examples per task, matching the sample sizes used here.

What does it cost to run Jev at scale?

$0.0044, for the full 90-item commit-type task here, across 105,824 input tokens. Jev bills $0.042 per million input tokens and output tokens are free. At that rate, the choice between Jev and a self-hosted model comes down to accuracy rather than price.

Can Jev or Von refuse to answer when the input is meaningless?

No, neither model can abstain. Both Jev and Von return a confident-looking answer to content-free input, such as a single period or random digits, rather than declining. Von in particular answered “refactor” to nonsense inputs, which is why a null-state control belongs in any decision-model benchmark.

Related Reading

• Jev Retagged 840 Posts For 26 Cents
• Large Language Model Formats and Quantization
• Function Calling in Local LLMs
• Self-Host a Local AI Coding Workhorse
• OpenRouter vs LiteLLM