Configuration

Practical recipe: Using Watchman.

1. File
2. Environment Variables
3. Metrics
4. Download
5. Search
6. Geocoding
7. libpostal
8. Deepparse (optional alternative to libpostal)
9. MCP
10. Ingest
11. Included Lists

The settings that matter most in production are which lists you load, which fields you send on each search, and minMatch. Name-algorithm environment flags change results very little. If you screen names in Arabic, Cyrillic, or Chinese, enable embeddings with CROSS_SCRIPT_ONLY. See Using Watchman.

Search Configuration

1. TF-IDF Configuration
2. Cross Script Embeddings Configuration
3. Similarity Configuration (SEARCH_MAX_IN_FLIGHT, Jaro-Winkler flags, etc.)

Source List Configuration

1. European Union
2. Open Sanctions
3. United Kingdom
4. United States

Data Persistence

1. Download reliability
2. Persistence Disclaimer

File

Set the APP_CONFIG environment variable with a filepath to a yaml file containing the following:

Watchman:
  Servers:
    BindAddress: ":8084"
    AdminAddress: ":9094"

  Telemetry:
    ServiceName: "watchman"

  # Database:
  #   DatabaseName: "watchman"
  #   MySQL:
  #     Address: "tcp(mysql:3306)"
  #     User: "watchman"
  #     Password: "watchman"
  #   Postgres:
  #     Address: "postgres:5432"
  #     User: "watchman"
  #     Password: "watchman"
  #     Connections:
  #       MaxOpen: 50
  #       MaxIdle: 50
  #       MaxLifetime: "60s"
  #       MaxIdleTime: "60s"

BindAddress is the business API. AdminAddress is a separate port for Prometheus metrics and /version so deployments can firewall, bind internally, or block admin without touching search. Watchman is not designed to be served directly on the internet. See Network access.

Metrics

The admin server (AdminAddress, :9094 by default) serves Prometheus metrics at /metrics, including Go runtime and process collectors plus a duration histogram for every request the API server handles. Metrics on this port are unauthenticated by design; do not expose :9094 on the public internet.

watchman_http_request_duration_seconds_bucket{method="GET",route="/v2/search",code="200",le="0.25"}
watchman_http_request_duration_seconds_sum{...}
watchman_http_request_duration_seconds_count{...}

The route label is the registered path template, not the requested path — so /v2/ingest/{fileType} stays one time series no matter how many file types are seen, and query strings never reach the labels. Requests matching no route are recorded as route="unmatched".

# p95 search latency
histogram_quantile(0.95, sum(rate(
  watchman_http_request_duration_seconds_bucket{route="/v2/search"}[5m])) by (le))

# request rate by route
sum(rate(watchman_http_request_duration_seconds_count[5m])) by (route)

Download

  Download:
    RefreshInterval: "12h"
    InitialDataDirectory: ""

    # Specify which lists to download and include in Watchman results
    # Examples: us_csl, us_ofac, us_non_sdn, us_fincen_311, uk_csl, eu_csl, un_csl
    IncludedLists:
      - "us_csl"
      - "us_ofac"

    # Source-list names for which download or parse errors are suppressed.
    # A failed list named here will log a warning but will not cause a refresh error.
    IgnoredDownloadErrors: []

    # When true, an empty list after download/parse will cause a hard error (useful for
    # detecting data problems early). Default: false (empty lists are tolerated).
    ErrorOnEmptyList: false

    # Include any senzing formatted OpenSanctions lists
    # OpenSanctions:
    #   ApiKey: "secret"
    #   Lists:
    #     - SourceList: opensanctions_peps
    #       Location: https://data.opensanctions.org/datasets/latest/us_congress/senzing.json

    # Include any senzing formatted lists in Watchman's corpus
    # Senzing:
    #   - SourceList: "senzing-persons"
    #     Location: "file://pkg/sources/senzing/testdata/persons.jsonl"
    # Use SENZING_CONCURRENT_DOWNLOADS env var (default 5) to limit parallel downloads of Senzing lists.

Location and the *_DOWNLOAD_TEMPLATE / *_DOWNLOAD_URL environment variables are operator configuration, not API input. file:// is supported for local Senzing files. Pointing templates at an internal HTTP cache (including http:// on a private network) is expected. Watchman does not allowlist download hosts or jail file:// paths. See Network access.

Search

  Search:
    # Tune these settings based on your available resources (CPUs, etc).
    # Watchman will dynamically find an optimal goroutine count for faster responses.
    # Usually a multiple (i.e. 2x, 4x) of GOMAXPROCS is optimal.
    Goroutines:
      Default: 10
      Min: 1
      Max: 25

    # Max concurrent *large* searches admitted at once (admission control).
    # Candidate selection runs first; sets with <= 100 entities bypass the queue
    # so exact/crypto hits are not blocked behind full-partition scans.
    # 0 or omit = GOMAXPROCS. Override with SEARCH_MAX_IN_FLIGHT.
    # MaxInFlight: 8

    Embeddings:
      Enabled: false # See below for Cross-Script Embeddings

See Performance for how admission control, per-search workers, and candidate indexes interact under load.

Geocoding

Optional. Off by default. Fills latitude / longitude on list addresses at refresh. Similarity scoring does not use those coordinates; see Geocoding for tradeoffs.

  Geocoding:
    Enabled: false
    Provider:
      # Specify the provider name, which can be one of "opencage", "google", or "nominatim"
      Name: ""
      ApiKey: ""  # Can also set GEOCODING_API_KEY
      BaseURL: "" # Useful for self-hosted Nominatim instances.
      Timeout: "10s"
    RateLimit:
      RequestsPerSecond: 1.5 # req/s
      Burst: 5
    Cache:
      L1MaxSize: 10000
      L1TTL: "24h"     # time-to-live for L1 cache entries.
      L2Enabled: false # Uses the database connection for a persistent cache.

libpostal

libpostal (with Senzing’s data) is how Docker images and Linux/macOS GitHub release binaries parse addresses. It loads about 3GB of models. Windows GitHub .exe files and go run without -tags libpostal use usaddress instead. make dist builds with libpostal by default (DIST_GOTAGS= to skip).

Settings live under Watchman.PostalPool in the config file (or matching environment variables). Enabled: false is in-process libpostal in the Docker image. Set Enabled: true to run extra libpostal worker processes for concurrency:

  PostalPool:
    Enabled: false
    # Extra libpostal worker processes. Most deployments keep this false
    # (in-process parse in the Docker image).
    Instances: 0
    StartingPort: 10000
    StartupTimeout: "60s"
    RequestTimeout: "10s"
    BinaryPath: "" # POSTAL_SERVER_BIN_PATH is set in the Dockerfile
    CGOSelfInstances: 1

Deepparse

Deepparse is an optional HTTP client for GRAAL-Research/deepparse, a neural-network address parser that runs outside Watchman. Default parsers (usaddress / libpostal) are unchanged until this is enabled.

Start the published image from this repository without pulling it into the default test stack:

make setup-deepparse

That runs ghcr.io/graal-research/deepparse:0.11.0 on port 8000 (docker compose --profile deepparse). First start downloads models and can take several minutes.

  Deepparse:
    Enabled: false
    BaseURL: "http://localhost:8000"  # or DEEPPARSE_URL
    Model: "bpemb-attention"          # bpemb, bpemb-attention; the image skips FastText
    Timeout: "10s"

If Deepparse is enabled, Watchman uses it instead of libpostal or usaddress.

MCP

Experimental Feature: The MCP server is experimental and may change in future releases.

Enable the Model Context Protocol server endpoints at /mcp for AI agent integration:

  MCP:
    Enabled: true

When enabled, Watchman will serve MCP endpoints at /mcp (streamable HTTP) in addition to the standard HTTP API. (The MCP server is HTTP-based; no stdio mode is used.)

Ingest

POST /v2/ingest/{fileType} reads the request body into a configured file schema and loads every parsed row into the search index. A checksum of each ingested source is stored so search does not scan the table on every query. The body is capped to limit memory use:

  Ingest:
    MaxBodyBytes: 33554432 # 32MiB; oversized uploads return HTTP 413

INGEST_MAX_BODY_BYTES overrides the YAML value when it is a positive integer. See File Dataset Ingestion and Network access.

Included Lists

Watchman integrates the following lists to help you maintain global compliance.

Environment Variables

TF-IDF Configuration

TF-IDF (Term Frequency–Inverse Document Frequency) is a technique used to measure the importance of a word in a corpus. Less frequent words are more important.

By default Watchman does not employ TF-IDF, but it can be enabled and configured with the following environmental variables. When enabled, term weights for indexed entities are computed once at list refresh and attached to each entity; query weights are computed once per search.

Cross-Script Embeddings Configuration

Watchman can use neural network embeddings to match names across different writing systems (Arabic, Cyrillic, Chinese, etc.). This feature requires configuring an embeddings API provider (Ollama, OpenAI, OpenRouter, etc.).

See Cross-Script Name Matching for detailed setup instructions.

YAML configuration (example with OpenAI):

  Search:
    Embeddings:
      Enabled: true
      Provider:
        Name: "openai"
        BaseURL: "https://api.openai.com/v1"
        APIKey: "${OPENAI_API_KEY}"
        Model: "text-embedding-3-small" # Required - no default
        Dimension: 1536                 # Required - must work with model
        NormalizeVectors: false         # OpenAI vectors are already normalized
        Timeout: "30s"
        RateLimit:
          RequestsPerSecond: 50
          Burst: 25
        Retry:
          MaxRetries: 3
          InitialBackoff: "1s"
          MaxBackoff: "30s"
        # Headers: { "HTTP-Referer": "https://example.com" }  # e.g. for OpenRouter
      Cache:
        Type: "memory" # or sql (persistent via DB, ignores Size)
        Size: 10000
      CrossScriptOnly: true
      SimilarityThreshold: 0.7
      BatchSize: 32
      IndexBuildTimeout: "10m"

Similarity Configuration

Note: DISABLE_PHONETIC_FILTERING, USE_SOUNDEX_MATCHING, and SOUNDEX_BOOST_WEIGHT are read once at process startup for hot-path performance. Restart Watchman after changing them. Search requests may still override the string-matching algorithm with ?algorithm= (see Search and Algorithm comparison) without a restart.

Source List Configuration

United States

European Union

United Kingdom

United Nations

Open Sanctions

Senzing

Download reliability

Sanctions list downloads/endpoints (OFAC, trade.gov, EU, UK, UN, etc.) are occasionally slow, return errors, or are unreachable for extended periods. A failed download on startup or during a refresh will prevent Watchman from serving requests (initial download failures cause a fatal exit).

For production deployments, consider running moov-io/watchman-cache as a reverse proxy in front of Watchman. It is an optional companion project that provides:

• Internal handling of 302 redirects (particularly relevant for OFAC and Non-SDN files that redirect to short-lived S3 pre-signed URLs)
• Hardened settings (large buffers, long timeouts, retries, SNI fixes, ipv6=off) tuned for large files such as consolidated.csv
• Long cache lifetimes (48h on major lists) plus aggressive stale serving (proxy_cache_use_stale), allowing Watchman to continue operating even when the origin is down for multiple days
• Persistent cache storage using a named Docker volume across container restarts
• An intentional allowlist — only the exact filenames Watchman requests are permitted (it is not an open proxy)

No changes are required inside Watchman. Simply override the download template environment variables to point at the cache (including http:// on a private network; that is expected, not a misconfiguration):

OFAC_DOWNLOAD_TEMPLATE=http://watchman-cache:8080/api/PublicationPreview/exports/%s
US_CSL_DOWNLOAD_TEMPLATE=http://watchman-cache:8080/downloadable_consolidated_screening_list/v1/%s
US_NON_SDN_DOWNLOAD_TEMPLATE=http://watchman-cache:8080/api/PublicationPreview/exports/%s
EU_CSL_DOWNLOAD_URL=http://watchman-cache:8080/...

A working docker-compose.yml example and Go integration test are provided in the watchman-cache repository.

This approach is complementary to the INITIAL_DATA_DIRECTORY feature (see Caching Data Files).

Data persistence

Watchman does not store search queries. There is no screening log inside the process. Your application must retain who was screened, when, against which list hashes (GET /v2/listinfo), at which minMatch, and what was returned. Restarting Watchman does not replay past searches.

Downloaded government lists live in memory. On restart Watchman loads them again from the origin, from INITIAL_DATA_DIRECTORY, or from watchman-cache. No in-memory encryption is performed.

Ingested files can be stored in MySQL or PostgreSQL if you uncomment the Database block above. That is how ingest survives a restart and how multiple Watchman processes share the same uploaded rows. Without a database, ingest exists only in that process and is gone when it exits. See File Dataset Ingestion.