Redaction

Where it runs

Queue adapters scrub the task fields they support inside the worker process, before building the event frame. The brain then scrubs every inbound event again before persistence as defence in depth. This is best-effort filtering, not a guarantee that the brain never receives cleartext: anything an adapter does not scrub arrives at the brain before the second pass examines it.

Data entered directly into the brain is outside this pipeline. In particular, schedule args and kwargs are accepted, stored, and returned without redaction. Do not put credentials or other secrets in schedule arguments.

The canonical built-in catalog is z4j_core.redaction.patterns.

Default key-name patterns

Key patterns use case-insensitive re.fullmatch. A matching key causes its entire value to become [REDACTED]:

• Passwords: password, password_confirmation, password_hash, passwd, pwd, old_password, new_password.
• Secrets: secret, secrets, client_secret / clientSecret, webhook_secret / webhookSecret, hmac_secret / hmacSecret.
• Tokens: token, access_token / accessToken, refresh_token / refreshToken, id_token / idToken, api_token / apiToken, bearer_token / bearerToken, session_token / sessionToken.
• API keys: api_key, apiKey, apikey, x-api-key, x_api_key.
• Authorization and cookies: authorization, auth, credentials, cookie, set-cookie, set_cookie.
• Personal identifiers: ssn, social_security, social_security_number, credit_card, credit_card_number, card_number, cvv, cvc.

The spelling api-key is not a built-in match. Neither are longer names such as password_hint, user_password, or last_password_reset; full-match means the complete key must match one of the patterns above.

Default value patterns

Value patterns use case-insensitive re.search. One match replaces the whole scalar, not only the matching substring. The defaults cover:

• Three-segment JWT-shaped strings beginning eyJ.
• Bearer followed by at least 16 letters, digits, ., _, or -. The Authorization: prefix is not required. Standard-base64 tokens containing +, /, or = in the first 16 characters are not covered by this rule.
• Stripe prefixes sk_live_, sk_test_, pk_live_, pk_test_, whsec_, rk_live_, and rk_test_ only when at least 24 consecutive alphanumeric characters follow. Shortened examples such as sk_live_abc123 do not match.
• Slack xoxb-, xoxp-, and xoxa- tokens and incoming-webhook URLs; GitHub ghp_, gho_, ghs_, and github_pat_ tokens; AWS AKIA and ASIA access-key IDs; Google AIza keys; and SendGrid API keys, subject to the exact lengths in the source catalog.
• SK followed by 32 hexadecimal characters. This is a Twilio API Key SID, an identifier, not the secret. Twilio Auth Tokens and API Key Secrets are unprefixed and have no default value-pattern coverage; rely on a matching field name.
• Credential-bearing database URLs using bare schemes: postgres://, postgresql://, mysql://, mariadb://, mongodb://, mongodb+srv://, redis://, rediss://, amqp://, and amqps://. SQLAlchemy driver-qualified URLs such as postgresql+psycopg:// and postgresql+asyncpg:// are not covered.
• RSA, EC, DSA, OPENSSH, ENCRYPTED, and unqualified PRIVATE KEY BEGIN lines. -----BEGIN PGP PRIVATE KEY BLOCK----- and SSH2 headers do not match.
• Email addresses with dotted domains and US SSN-shaped values.

An email address anywhere in an error message or traceback replaces the entire message or traceback with [REDACTED]; it is not reduced to an in-place placeholder. user@localhost does not match the email rule.

Customization and its limit

The Config model accepts redaction_extra_key_patterns, redaction_extra_value_patterns, and redaction_defaults_enabled, including their Django and Flask nested-setting spellings. The current runtime does not pass those resolved fields into automatically discovered queue adapters. They are therefore accepted configuration that has no redaction effect; do not rely on them for a security boundary.

For Celery, RQ, and Dramatiq tasks, use the adapter's z4j_meta decorator to force-redact named kwargs (and, where supported, keep only an allowlist):

@z4j_meta(redact_kwargs=["password_hint", "user_password"])
def send_message(*, password_hint: str, user_password: str) -> None:
    ...

Bare installations can inject a configured engine directly into a queue adapter before passing that adapter to install_agent:

from z4j_core.redaction import RedactionConfig, RedactionEngine
from z4j_celery import CeleryEngineAdapter

redaction = RedactionEngine(
    RedactionConfig(
        extra_key_patterns=(r"api-key", r"password_hint"),
        extra_value_patterns=(r"postgresql\+psycopg://[^\s]+",),
    )
)
engine = CeleryEngineAdapter(celery_app=celery_app, redaction=redaction)

The brain's second pass has its own extra key patterns: Z4J_REDACTION_EXTRA_KEY_PATTERNS, a JSON array of regexes, each matched against the whole key (a full match, case-insensitive) of task args, kwargs, results and exception payloads on ingest, in addition to the built-in patterns. customer_ssn scrubs customer_ssn and Customer_SSN but not customer_ssn_hash; write customer_ssn.* to cover derived keys. It runs on what arrives whatever the agent scrubbed, so it is the place to put a pattern you want enforced for every agent. An entry that does not compile refuses to start, and so does one that backtracks catastrophically: at startup every extra pattern is timed against adversarial keys built from the characters it mentions, and a pattern that takes more than 20 ms on one key or 100 ms across the probe keys is refused by name ((a+)+$ is the classic shape). Key matching is also capped: extra patterns run only on keys of up to 64 characters, no pattern runs on a key over 256, and a key past the cap that applies is treated as sensitive, so its value is scrubbed rather than matched. Note the format difference: the brain reads a JSON array (["^launch_code$", "customer_ssn"]), the agent-side variable of the same name is comma-separated.

Z4J_REDACTION_EXTRA_KEY_PATTERNS='["^launch_code$", "customer_ssn"]'

There is no per-project redaction setting in the brain; the setting is brain-wide.

Task, exception, and traceback fields

Task names, queue names, routing keys, and worker names have no exemption from value matching. Ordinary names survive; an email-, credentialed-URL-, or SSN-shaped substring replaces the whole name. This includes ordinary Celery node names such as celery@host.example.com, which look like email addresses.

Only Celery keeps the exception class in its own field, with the message and traceback scrubbed separately. Dramatiq, TaskIQ, and Huey combine ClassName: message, and RQ stores the job's exc_info text; a matching value therefore removes the class name as well. Dramatiq and RQ scrub that exception string in the worker process; TaskIQ and Huey do not, so it reaches the brain's second pass first.

A traceback is scrubbed as one scalar, not line by line. Only value patterns apply to its text. One matching value removes the whole traceback, including file, line, and function names. Key-name rules do not recognize text such as password = "..." inside a traceback, so it remains unless the value itself matches a value pattern.

Size behavior

• The default redaction engine keeps the first 8192 UTF-8 bytes of a scalar and appends [...N more bytes truncated]. Shorter strings are not tagged. Z4J_MAX_PAYLOAD_BYTES is accepted by agent configuration but is not wired into discovered adapters, so it does not change this limit. The brain's second pass truncates at Z4J_MAX_PAYLOAD_SIZE_BYTES (default 8192), the same setting that caps request bodies, so raising it raises the persisted scalar limit only for values an adapter left longer.
• bytes and bytearray are converted with str() and then scanned and truncated like any other scalar. They are not replaced with a length-only marker; a binary payload under a non-secret key may be recorded as its Python representation.
• There is no 2 MiB total-event truncation. A WebSocket frame above Z4J_WS_MAX_FRAME_BYTES (default 1 MiB) is undeliverable and purged by the agent transport. Long-poll uploads are HTTP requests and also face Z4J_MAX_PAYLOAD_SIZE_BYTES (default 8192 bytes); an oversized request gets HTTP 413, and a persistently oversized single frame is eventually dropped rather than shortened.

Disabling redaction

There is no working high-level setting that disables redaction. The low-level API can still do it: injecting RedactionEngine(RedactionConfig(default_patterns_enabled=False)) into an adapter disables every built-in key and value pattern. Audit any custom adapter wiring for that argument. The similarly named resolved Config field is not wired and changes nothing.