Running an open Jev on a Mac: Laya, measured

Jev from TypeSafe AI is a decision model, not a chatbot. You give it text and a list of typed questions (yes/no, pick one of N, rate 1 to 5), and it answers all of them in one pass with calibrated probabilities. It never generates text, so there is nothing to parse and nothing to hallucinate. It is also closed: hosted API only, no weights.

Laya is an open reproduction under Apache 2.0, with weights on Hugging Face. I wanted to know whether it is good enough to ship inside a Mac or iPhone app, so I ran the multilingual checkpoint (mmBERT-base, 322M parameters, about 650 MB) on an Apple M5 with 32 GB and ported it to ONNX and Core ML. The numbers below are from 2026-09-20.

Quality

I hand-labeled 12 smoke cases in English and Russian and made every variant pass the same set.

English went 9 of 9. Both misses are Russian, and one of them is the bad kind: “is this urgent?” answered 0.023 with 0.977 confidence on a message that was urgent. A calibrated model being confidently wrong is exactly the failure the calibration is supposed to prevent, so Russian output gets a human-review step until it is measured on a bigger set.

The int8 row taught me something too. Quantization moved logits by up to 2.4, and every decision survived. Compare decisions, not raw logits.

Speed and size

Hosted Jev answers in 70 to 500 ms over the network. The local Core ML build is 5 to 35 times faster and needs no Python or torch at serving time, so it can live inside a Swift app. The model card claims about 33 ms per call; I got 34 on MPS.

int8 on ONNX bought size, not speed: 325 MB instead of 1290, same 148 ms. Inputs are padded to a fixed 512 tokens for export, so a 100-token message pays for 512 everywhere except the original torch path.

The Neural Engine refused the model: ANECCompile() FAILED on every attempt. The 34 ms you get when asking for cpuAndNeuralEngine is a silent fallback to CPU and GPU, not the ANE. Getting a transformer encoder through the ANE compiler means rewriting attention as convolutions, and with the GPU at 13.7 ms I stopped there.

Five converter blockers

coremltools 9.0 with torch 2.7.0 (that exact pair, they are tested together) failed five times before it produced a model:

1. new_ones has no lowering. Registered it by hand.
2. bitwise_and on a double and a bool. Overridden with cast plus logical_and, and the override has to be registered under the alias and, or it silently does nothing.
3. int() on a one-element numpy array, which numpy 2 refuses during constant folding.
4. The fused fastpath of nn.TransformerEncoderLayer. Fixed with torch.backends.mha.set_fastpath_enabled(False) before tracing.
5. A clamp that turned integer indices into floats. I dropped it, since positions are non-negative by construction, and asserted a logit diff of exactly 0 after each rewrite.

Where a decision model pays off

Vivek Haldar, who ran the OC AI Builders Lightning Hackathon in Irvine on 21.10.2026, published an experiment that answers the “why bother” question. He put hosted Jev inside Proceda, his harness that turns business procedures into agents, next to Qwen 27B, on 360 SOP-Bench cases. The rule: act on Jev’s answer when its probability is at least 0.8, otherwise ask the LLM.

• Checking “is this step finished?” after tool calls cut 605 LLM calls (23.8%), at 356 of 360 correct against 357 for the LLM alone.
• Classifying email intent, Jev took 141 of 148 decisions by itself, all 148 were correct, and the workflow made 25% fewer LLM calls.

Those are his numbers; I have not reproduced them.

A demo at the same event put Jev and Laya side by side as the router. One request, “Translate ‘the meeting moved to Thursday afternoon’ into Spanish”, went to three deciders, each picking a lane (fast or slow model) and scoring urgency and “needs a human”:

The draft then came from a small fast model in 755 ms, and the decision model checked it: safe 97%, answers the request 94%, 235 ms. This is one request photographed off a slide, not a benchmark, but it shows the trade in one row: Laya was 5 times faster than Jev and agreed with it on the soft scores, yet on the routing call it sat at 41%, below any sensible 0.8 gate, where Jev was certain. With Haldar’s rule that request would have fallen through to the LLM. The pattern carries over to Laya unchanged: wherever an LLM only classifies or checks, a small calibrated model can take the confident cases, and the LLM keeps the rest.

What it cannot do

Laya reads text only. At that same hackathon we built BroCheck, which looks at two photos and judges whether the pair are friends, a couple, or the same person twice. Hosted Jev handled the images. For the offline version (/local) Laya was no use, so it runs CLIP ViT-B/32 int8 in the browser instead: zero-shot over a side-by-side composite, plus cosine similarity between the two faces.

Verdict

For English routing, triage and scoring, Laya is ready to embed: 309 MB, 25 ms a question, no server, and an open license. For Russian, ship it behind a review gate. For images, it is the wrong model.

Related: jev-system-one, needle-tiny-tool-model, schema-guided-reasoning, hackathons.