I Ran a DLP Shootout This Weekend and Jev Somehow Won

TypeSafe shipped this thing called Jev a few weeks ago. It doesn't generate text, won't write you a poem. It's a "System One" decision engine. You hand it text and ask a typed question, and it hands back a probability. Is this document sensitive? 0.94, yes. What kind? Credentials. No paragraphs, no hedging, no "it depends." Just a decision.^[1]

Weird pitch. But here's what got me: it's general-purpose. You don't train it or fine-tune it. You just ask different questions. So I got curious — if I pointed it at a DLP problem alongside actual specialized tools, would it hold up? Or would it get demolished by models that were literally built for this?

It won 4 out of 5 categories. But the way it won is more interesting than the fact that it did.

What I actually tested

I grabbed 515 labeled documents from HuggingFace DLP datasets and a batch I wrote myself. 166 became the test set, split five ways: PII (names, SSNs, emails), financial (credit cards, IBANs, crypto wallets), credentials (API keys, passwords, connection strings), PHI (medical records, prescriptions), and legal/confidential (NDAs, privileged docs, trade secrets).

Then 87 clean documents. Synthetic emails, code without secrets, meeting notes, news articles. The stuff that shouldn't trigger anything but often does because it looks wrong. A business email to "Sarah Thompson" isn't PII. It's a fake name in synthetic data. Try telling that to a regex engine.^[2]

Four detectors, same test set: Jev (via OpenRouter), Microsoft Presidio, GLiNER (a NER model), and a regex engine with hand-rolled patterns.

On raw binary detection, Presidio technically wins at F1=0.735. But the number you actually care about in production is precision, and Jev leads at 0.609. When Jev says something is sensitive, you can trust it more. Presidio's F1 comes from brute force — it flags almost everything, including 57.5% of clean documents. That means your SOC team drowns.

Jev's false positive rate is still 49.4%, which isn't great. But it's the best of the four. The pattern sharpens when you break it down by category.

The red zeros are the real story. Presidio literally cannot detect credentials, PHI, or legal documents. It's a PII specialist. GLiNER misses legal entirely. Regex can't spot a clean document to save its life.

Jev is the only detector with non-zero F1 in every category. It detected API keys in CI/CD configs, passwords in Kubernetes secrets, connection strings in environment files — stuff Presidio and GLiNER are architecturally blind to.

The speed problem

Jev is really slow.

57× slower than Presidio. 1M documents: Presidio finishes in a workday, Jev takes over two weeks. Each Jev classification is an HTTPS round-trip to OpenRouter. Presidio runs in-process locally. You can catch things nobody else sees, but you can't scan your entire corpus with it.

The part I'm actually excited about

Building a DLP classifier used to go like this: collect thousands of labeled examples, pick a model architecture, fine-tune, evaluate, deploy, discover your training data missed edge cases, collect more labels, retrain. Weeks to months, repeat forever.

With Jev, I wrote a prompt: "Does this text contain API keys or hardcoded credentials?" I ran it against my test set. I had a working detector. That's the whole thing.

Eventually a fine-tuned model will be faster and cheaper. But you don't know if the problem is solvable when you start. You don't have labels. You don't know if your definition of "sensitive" matches reality. Jev answers those questions on a Friday afternoon.

The most expensive way to build a DLP detector is to collect training data first. The cheapest way is to start with Jev, find out which categories actually matter, and only then build specialized models for the high-volume ones.

This pattern — general-purpose reasoning as a prototyping layer, specialized models for production — is going to eat a lot of ML pipelines. Jev works especially well here because its output is structured: probabilities and typed choices you can measure against. Free-form LLM text needs parsing. Jev doesn't.

What a real pipeline looks like

You don't pick one. You cascade:

Most documents resolve in the first two layers at microsecond speed. Only the genuinely ambiguous cases reach Jev — "is this Slack message a credential or a config example?" Speed where it's cheap, depth where it matters.

The false positive problem

Every detector flagged clean data as sensitive at rates between 46% and 57.5%. Synthetic emails have fake names. Product IDs look like credit card numbers. Git hashes look like tokens. Network docs have IP addresses. That's the task.

The difference between a good DLP detector and a useless one isn't whether it has false positives. It's whether the false positives make sense. When Jev flags a document as containing credentials, you can ask why and get an answer: "This contains a JWT signing secret in what appears to be a config file." Regex matches a pattern. Presidio returns an entity type. Jev gives you a decision you can interrogate.

That matters for tuning. You can't tune a regex black box except by writing more regex. You tune Jev by refining the question: "Does this contain production credentials, or example/test credentials?" One word changes the false positive rate. Thirty seconds to implement.

Where this goes

This is a Bitter Lesson situation. Rich Sutton's point was that general methods that leverage computation eventually beat specialized, hand-crafted approaches. Every time. Jev isn't doing anything Presidio or GLiNER can't do in theory. It's just not specialized. It asks questions. The model figures out the rest at inference time. No training loop, no entity taxonomy, no regex library that needs updating every time a new API key format ships.

The specialized models will win on speed and on narrow metrics for specific categories, probably forever. Presidio on PII is a solved problem. But the moment you need to detect something new — a new credential format, a new compliance domain, a new type of sensitive data that nobody has labeled yet — the specialized model is dead in the water until someone collects data and retrains. Jev just... asks a different question.

That's the bitter lesson playing out at the product level. Not "general beats specialized on the same benchmark" but "general ships today, specialized ships in six months." The time-to-deploy gap is the real metric, and it's hard to overstate how big it is. I had a working credential detector running against my test set in the time it took to type a sentence. The regex engine took me an afternoon of pattern-writing and still missed 37% of sensitive data. The fine-tuned model would have taken weeks.

Jev is the first of a wave. The model is a single-purpose decision engine today, but the insight — that structured, calibrated outputs beat free-form generation for classification — will spread to general models. When Claude or GPT can return typed probability distributions instead of paragraphs, the prototyping advantage compounds.

The pattern I keep coming back to: define the problem with Jev (hours, not months), measure the baseline, label the 200 examples where Jev is uncertain, fine-tune a micro model on that data, deploy the cascade. The old way assumes you know the problem upfront. Jev lets you discover it.

I ran the same benchmark against a local reproduction — EdgeJev running kev-0.5b, a 0.5 billion parameter ONNX model on Apple Silicon CoreML. Zero API cost, fully local. Here's what happened.

It was slower than the API. 3,144ms per sample vs Jev's 1,570ms. The 0.5B model doing 5 ONNX forward passes on CPU takes twice as long as a round-trip to OpenRouter. Local doesn't automatically mean fast.

But it did catch credentials at Jev-level accuracy — 0.800 F1 on credentials and 0.727 on financial, matching or beating the big model on those categories. The catch: its false positive rate was 89.7%. It flagged almost every document. The smaller model has the right instincts but none of the restraint. The Bitter Lesson plays out at the model level too: bigger models discriminate better.

I'm running more local reproductions next — the laya backend (322MB, INT8, CPU-optimized) should be much faster. EdgeJev supports it, I just need to install one more dependency. If a 322MB model can match Jev's coverage at local inference speeds, the fully local cascade becomes practical today.

---

The benchmark code, test cases, and interactive dashboard are open source. All 515 labeled samples, 4 detectors, 8 interactive charts.