How accurate is Stript's on-device PII detection? A benchmark
On our German and English benchmark, Stript's on-device pipeline reaches an F1 of 97.3 (English) and 95.8 (German), with 95 to 98% recall. Checksum-validated identifiers like IBANs, credit cards, and bank accounts are caught at 100% recall. Every detection can be reviewed before anonymization.
Updated
On our benchmark of German and English documents, Stript’s on-device pipeline reaches an F1 of 97.3 on English and 95.8 on German, with 95 to 98% recall. Checksum-validated identifiers such as IBANs, credit cards, and domestic bank accounts are caught at 100% recall. Every detection can be reviewed before anything is anonymized.
What we measured
We evaluate on our own annotated benchmark of German and English documents from legal, financial, and HR contexts. The headline metric is F1 (the balance of recall and precision) on the is-this-genuinely-personal-data layer: an item counts only if it is real personal data in context, so a public court or a standalone city name is correctly not counted. The current run (July 2026) covers the complete shipping pipeline with all 20+ detected types, including the newest ones such as domestic bank accounts, VAT IDs, and company registration numbers.
How we tested
The pipeline runs three stages, all on the device:
- Pattern and validation for structured identifiers, checksum-validated where possible (IBAN, credit card, bank account, tax and VAT IDs).
- Context-aware entity detection for names, organizations, and locations.
- Context classification that judges whether each candidate is genuinely personal data in its sentence.
Every result can be reviewed by a human, who confirms, rejects, or reclassifies before anonymization. This is a transparent, in-house benchmark, not an independent third-party test.
Results
Overall, on the personal-data layer:
| Corpus | Precision | Recall | F1 |
|---|---|---|---|
| German | 96.5 | 95.0 | 95.8 |
| English | 96.8 | 97.8 | 97.3 |
Measured precision / recall for selected types:
| Type | Precision | Recall |
|---|---|---|
| 100 | 100 | |
| Bank account | 100 | 100 |
| Credit card | 100 | 100 |
| ID card & passport | 100 | 100 |
| Case number | 100 | 100 |
| License plate | 100 | 100 |
| VAT ID | 100 | 100 |
| Company registration | 100 | 100 |
| VIN | 100 | 100 |
| URL | 100 | 100 |
| Person | 98.1 | 97.2 |
| Address | 96.7 | 93.7 |
| Organization | 93.7 | 96.7 |
| IP address | 93.3 | 93.3 |
| IBAN | 92.0 | 100 |
| Date | 90.5 | 100 |
| Phone | 82.6 | 100 |
Phone shows the tuning philosophy at a glance: 100% recall with some over-flagging (reference numbers that resemble phone numbers get surfaced for review rather than silently skipped). That is the safe direction; a flagged non-issue costs one click, a missed number would leak.
Tax IDs and social-security / health-insurance numbers are detected as well, and where a checksum exists it is verified; ambiguous candidates are surfaced for review rather than silently dropped.
Dates are the hardest class, and deliberately so: a birth date is personal data, an invoice or letter date is not. Deciding that needs context, which is exactly what the third stage is for.
The adversarial stress set
Beyond the two language corpora, we maintain a set of deliberately nasty documents: mixed-language text, unusual formats, edge-case dates, and lookalike identifiers. On this stress set the pipeline measures 90.4% precision and 86.1% recall (F1 88.2). We publish this number separately on purpose: it is not representative of normal documents, but it shows honestly where the boundaries are.
Runtime, measured
Detection speed on a MacBook Pro M1 Max with 32 GB memory (Performance profile), completely offline:
| Measurement | Value |
|---|---|
| First document (includes model warm-up) | ~26 s |
| Median document | ~15 s |
| 95th percentile (long documents) | ~50 s |
| 50 documents, end to end | ~17 min |
Runtime scales with document length and the selected compute profile; lower-end hardware runs a lighter model tier.
What the context stage adds
Detection alone (finding every candidate name, date, and number) is high-recall but noisy: it flags standalone city names and administrative dates that are not personal data. The context classification stage is what turns that noisy candidate set into a precise result. In our tests, it lifted overall F1 by up to ~10 points over detection alone, chiefly by recovering genuinely personal dates and removing standalone-location false positives. All of it runs on the device.
Why recall matters more than precision here
For anonymization, the expensive mistake is a miss: a real name or IBAN that slips through. That is why the pipeline is tuned for high recall and why every detection can be reviewed. A false positive costs one click to reject; a false negative can leak data.
Limitations
- The benchmark targets direct identifiers. Indirect, quasi-identifying context (a rare combination of role, place, and date) can still allow inference and is left to human review.
- Results are measured on our own corpora; other document types may differ.
- Detection quality scales with the compute tier; higher tiers run a larger context model.
- Runtime numbers are from one machine and vary with hardware and document mix.
- This is a first pass with a human safety net, not a claim of perfection.
Conclusion
Strong PII detection is achievable entirely on-device, without sending documents to a server. High recall plus a human review step is what makes it usable for confidential documents.
Try it on your own documents, entirely on your machine. Download Stript for free →