Advanced Search Indexing for Superior Performance
Watchman maintains searchable entities in memory (with database options for ingested files) to deliver ultra-fast search responses. The index automatically rebuilds with each data refresh, ensuring your compliance checks are always based on the latest information.
What is built on refresh
When lists finish downloading and preparing, Watchman constructs an in-memory search corpus from every entity:
- Prepared fields — normalized names, tokenized name fields (stopwords removed), alt names, former names, addresses, and contact data (see pipeline and entity
Normalize()). - Source × type partitions — index slices keyed by list source and entity type so queries with
source/typeonly score the relevant subset. Empty source or type values are not double-counted into the “all” partition. - Name-token inverted index — maps each significant prepared name token to entity positions (primary name, alternate names, and historical “Former Name” values). Each entity is posted once per distinct token even if the token repeats across name fields.
- Exact prepared-name map — maps the full prepared name string to entity positions for exact-name shortcuts.
- Crypto address index — exact lookup by
CURRENCY:addressfor fast crypto screening. - Optional TF-IDF weights — when enabled, term weights for each entity’s name fields are stored on the entity so search does not recompute them per comparison.
These structures are immutable for readers until the next successful refresh replaces the corpus atomically.
Candidate selection at search time
Before Jaro-Winkler scoring, Watchman selects a candidate set:
| Query shape | Candidate strategy |
|---|---|
type and/or source set |
Start from that partition only |
| Known source, empty type (no entities of that type) | Empty result — does not scan other types or sources |
| Unknown / unregistered source | Empty result — does not fall back to the full corpus |
| Name tokens present | Union of inverted-index hits for those tokens, restricted to the partition |
| No token hits (e.g. heavy typos) | Fall back to the full partition (preserves recall within that source/type) |
| Crypto address only | Exact crypto hits only (does not expand to the full partition) |
| Crypto + name tokens | Union of crypto hits and name-token candidates |
| Exact prepared name (no tokens after stopwords) | Binary-search exact-name postings against the partition |
| Name-less / identifier-oriented (no crypto) | Full partition for the filtered source/type |
If name-token candidates would cover most of the partition (default threshold: half the partition size), Watchman scores the full partition instead—token pruning would not save work.
Candidate index membership checks use binary search over sorted partition slices (no per-query partition maps). Duplicate postings are removed with sort + compact.
Always pass type (and source when appropriate) on /v2/search for the best latency. See Performance for concurrency, admission control, and tuning.
TF-IDF
Starting with v0.57.0 Watchman builds an in-memory TF-IDF
(Term Frequency–Inverse Document Frequency) index over the corpus of searchable entities, treating each entity’s
normalized name (including primary names and alternate known-as entries from lists like OFAC SDN) as a “document”
composed of tokenized terms. This precomputes IDF values to weight terms inversely by their commonality across
the entire watchlist—rare terms (e.g., unique surnames or identifiers) receive higher weights, while common
ones (e.g., “bank” or frequent first names) are downweighted.
During name searches, TF-IDF weighting is optionally applied to boost the contribution of discriminative rare terms in similarity scoring, improving match precision for fuzzy name matching against global sanctions and watchlists without overemphasizing ubiquitous words.
When TF-IDF is enabled, weights for indexed entities are materialized at corpus build time and attached to each entity’s prepared fields. Query-side weights are computed once per search request.
The feature is opt-in via configuration, with tunable parameters like smoothing and IDF bounds to adapt to corpus size and growth.