Notifications

z4j fans events out to seven channel types. Per-project channels are configured by admins and shared with the team. Personal channels are configured by each user for their own subscriptions. Both share the same filtering primitives and the same dispatch pipeline.

Triggers

Trigger	When it fires
task.failed	A task raised an exception or exited non-zero
task.succeeded	A task completed without error
task.retried	A task is being retried after a failure
agent.offline	An agent missed heartbeats and its offline episode was confirmed (see agent offline alerts)
agent.online	An agent reconnected after being offline
schedule.misfired	An enabled schedule missed its expected fire past the grace window (see misfire detection)
schedule.fire.succeeded	A z4j-scheduler fire was acked successfully by the agent
schedule.fire.failed	A z4j-scheduler fire failed (dispatch error or the agent reported failure)
schedule.task_failed	Alias of schedule.fire.failed; kept separate so fire-side and task-side failures can be split later without breaking existing subscriptions
schedule.circuit_breaker.tripped	A schedule hit its consecutive-failure threshold and was disabled automatically

Every subscription pins a single trigger plus an optional filter set (see Filters below). The schedule.* triggers are available on project subscriptions only; personal subscriptions cover the task and agent triggers.

Channel types

z4j ships seven channel types. All seven are project-scoped (admin) and user-scoped (personal); the schema is identical.

Webhook (generic HTTPS)

POSTs a JSON envelope to any URL you control. Use this when none of the named integrations fit.

Field	Required	Notes
url	yes	https://.... Loopback and RFC1918 ranges are rejected.
headers	no	Up to 20 custom headers, 1024 bytes each. Reserved headers (host, cookie, proxy-authorization, ...) are blocked.
hmac_secret	no	Any string. Each delivery carries X-Z4J-Timestamp (Unix seconds) and X-Z4J-Signature: sha256=<hex>, where the hex digest is HMAC-SHA256 over the string {timestamp}.{body}: the timestamp, a literal ., then the raw request body.

Verify on your side, passing the raw values of both headers:

import hmac, hashlib, time

def verify(body: bytes, timestamp: str, signature: str, secret: str, window: int = 300) -> bool:
    if abs(time.time() - int(timestamp)) > window:
        return False
    signed = timestamp.encode() + b"." + body
    expected = "sha256=" + hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)

z4j does not enforce a replay window on the sending side: the timestamp is there so the receiver can reject stale deliveries, and the window is the receiver's choice (300 seconds is a sensible default).

Email (SMTP)

Plain SMTP with optional STARTTLS / implicit TLS. No OAuth.

Field	Required	Notes
smtp_host	yes	Hostname only.
smtp_port	yes	One of 25, 465, 587, 2525. Other ports are rejected.
smtp_user	no	Username for AUTH. When empty, no AUTH is attempted.
smtp_pass	no	Password (masked on GET, preserve on empty PATCH).
smtp_tls	no	true (default) negotiates STARTTLS on 25, 587 and 2525. 465 always uses implicit TLS, whatever this field says. false sends plaintext on 25, 587 and 2525; it is not refused.
from_addr	no	Defaults to smtp_user.
to_addrs	yes	List of recipient addresses.

Provider notes:

• Gmail / Google Workspace: use an app password, host smtp.gmail.com, port 587.
• Mailgun: host smtp.mailgun.org, port 587, user is your domain SMTP credential.
• Brevo (ex-Sendinblue): host smtp-relay.brevo.com, port 587.

Slack

Incoming webhook with Block Kit formatting. To get the URL: Slack workspace -> Apps -> Incoming Webhooks -> add to a channel -> copy the URL (https://hooks.slack.com/services/T.../B.../...).

Field	Required	Notes
webhook_url	yes	The full https://hooks.slack.com/services/... URL.

The dispatcher posts a message that includes trigger, project slug, task name (when relevant), and a link back to the dashboard.

Telegram

Bot API sendMessage. Two values to obtain:

1. Bot token - DM @BotFather, run /newbot, follow the prompts. Token format: 123456789:AAH....
2. Chat ID - either a signed integer (group chats are negative, e.g. -100123456789) or @channel_handle. The bot must be added to the chat for messages to deliver.

Field	Required	Notes
bot_token	yes	Pattern \d+:[A-Za-z0-9_-]+.
chat_id	yes	Signed integer or @handle.

To find a chat ID, send any message to your bot then GET https://api.telegram.org/bot<TOKEN>/getUpdates and read result[].message.chat.id.

PagerDuty

Events API v2. Open a PagerDuty service, Integrations -> Add integration -> choose Events API v2, copy the Integration Key (32 chars).

Field	Required	Notes
integration_key	yes	The routing key PagerDuty issues (32 characters). z4j accepts 8 to 64 characters of [A-Za-z0-9_-].
severity_default	no	One of critical / error / warning / info. Default warning.
severity_map	no	Per-trigger override, e.g. {"agent.offline": "critical", "task.failed": "error"}.

z4j picks a sensible severity per trigger out of the box, so most operators only need to paste the integration key. Repeat firings for the same (project, trigger, task_id) collapse into a single incident via PagerDuty's dedup key.

Discord

Server Settings -> Integrations -> Webhooks -> New Webhook -> copy the URL (https://discord.com/api/webhooks/<id>/<token>).

Field	Required	Notes
webhook_url	yes	The full Discord webhook URL.

z4j POSTs Slack-compatible payloads to Discord's /slack endpoint, so the formatting renders cleanly. The dispatcher auto-appends /slack to the URL you paste.

Microsoft Teams

In a Teams channel, click ··· -> Workflows -> pick Post to a channel when a webhook request is received -> finish the wizard -> copy the URL.

Field	Required	Notes
webhook_url	yes	The full Microsoft webhook URL.

z4j accepts all three official Microsoft webhook families and rejects anything else at save time:

• https://outlook.office.com/webhook/... (classic O365 connector, legacy)
• https://<tenant>.webhook.office.com/webhookb2/... (current Workflow webhooks)
• https://prod-<NN>.<region>.logic.azure.com/workflows/... (Power Automate flows)

The dispatcher sends an Adaptive Card so the message renders identically across the classic connector and the Workflow / Power Automate endpoints. Priority colours: critical -> red header (Adaptive Card attention style), high -> amber (warning), everything else neutral. The card carries task name, state, priority, a short task ID, and a monospaced exception block when the trigger is task.failed. Workflow endpoints return 202 Accepted; the classic connector returns 200. z4j treats the full 2xx range as success.

If you need to point at a host outside those three families (a relay, a proxy, an internal collector), use the generic Webhook channel instead. The Teams channel deliberately host-locks so a tenant admin cannot register an arbitrary HTTPS host under the Teams type and use it as an exfil sink that looks like Teams in the audit log.

Filters

Every subscription can narrow the firehose with these filters:

Filter	What it does
priority	List of priorities to fire for, drawn from critical, high, normal, low (at most 8 entries).
task_name	Exact task name, at most 500 characters.
task_name_pattern	fnmatch glob, e.g. billing.* or *.deliver.
queue	Only this queue.
cooldown_seconds	Drop events that arrive within N seconds of the previous matching event.
muted_until	Hard mute the subscription until a timestamp.

z4j evaluates filters before dispatch, so a muted subscription costs nothing.

Test before saving

Two preflight endpoints let you verify credentials without committing them. The dashboard exposes both as Test buttons in the channel create / edit dialogs.

POST /api/v1/projects/{slug}/notifications/channels/test
{
  "type": "slack",
  "config": { "webhook_url": "https://hooks.slack.com/services/..." }
}

A canned z4j.test payload is dispatched through the real channel. Response:

{
  "success": true,
  "status_code": 200,
  "error": null,
  "response_body": "ok"
}

For a saved channel, hit /channels/{id}/test instead. Both project test endpoints write a notification_deliveries row with trigger=test.dispatch; a preflight test on an unsaved config stores it with a null channel_id, shown as (unsaved test). Personal-channel tests are not logged.

Limits and dispatch behavior

Knob	Default
Outbound concurrency	16 deliveries per event batch
HTTP timeout	10s overall, 5s connect
Response body cap	8 KiB (excess is discarded)
DNS cache TTL	30s with a 5s resolve timeout
Config size cap	16 KiB JSON per channel
Headers per webhook	20 max, 1024 bytes per value
Rate limiting	per-channel test endpoints are throttled per IP

Failures are written to notification_deliveries with the HTTP status, sanitized error message, and (capped) response body, so you can debug without pulling logs. The brain's own log never carries a delivery URL: the HTTP client's request line is silenced, and if you raise the httpx logger to DEBUG it prints the scheme and host only, never the webhook path or a bot token.

Security

• SSRF protection: all dispatchers validate the URL, resolve the hostname, and pin the IP at send time so a DNS rebind between create and dispatch cannot redirect to a private range.
• Header injection: webhook custom headers are validated for control characters and reserved names.
• Secret masking: smtp_pass, hmac_secret, bot_token, integration_key, etc. are returned as •••••••• from GET. On PATCH, leaving a sensitive field empty preserves the stored value, so you can rename a channel without re-entering credentials.
• Encryption at rest: the whole channel config (project channels and personal channels alike) is stored as one AES-256-GCM ciphertext under a key derived from Z4J_SECRET, the same protection stored TOTP secrets have, so a database copy without the master secret does not yield webhook URLs, SMTP passwords or integration keys. After rotating Z4J_SECRET, run z4j secrets rewrap before dropping the old value from Z4J_PREVIOUS_SECRETS; see incident response.
• Audit trail: every dispatch is logged to notification_deliveries with channel name, channel type, trigger, status, and response code. That is a plain delivery log, not part of the HMAC-chained audit_log; channel create, update, delete and test actions are what land in the chained log.

Common patterns

Page on agent offline, otherwise just chat

Two channels, two subscriptions:

1. PagerDuty channel scoped to agent.offline (severity critical).
2. Slack channel scoped to task.failed with cooldown_seconds: 300 so a flood of failures from one bad deploy posts at most once every five minutes.

Per-user mute during oncall handoff

Each user manages their own subscriptions under Settings, Notifications, Subscriptions. Setting muted_until on a personal subscription stops their phone from buzzing while a teammate is on duty without affecting anyone else.

Scope alerts to one workload

Add task_name_pattern: report.* to a task.failed subscription to catch failures in the long-running aggregations without alerting on every unrelated task in the project.

API reference

The full notification API (channels, subscriptions, defaults, deliveries, in-app inbox) is documented under API reference -> Tasks and adjacent endpoints. CRUD verbs follow the same shape as projects and memberships.