Production hardening

z4j ships with backwards-compatible defaults that work out of the box but lean permissive. For production deployments, opt into the fail-closed mode of each subsystem below. None of these are required for z4j to function -- they're defense-in-depth for operators who want hardened defaults instead of trust-the-CA / trust-the-operator-config.

Scheduler gRPC -- require explicit CN allow-list

When Z4J_SCHEDULER_GRPC_ENABLED=true, brain accepts mTLS-authenticated gRPC connections from any client cert that the configured CA bundle validates -- the "trust the CA" deployment model. For production, populate Z4J_SCHEDULER_GRPC_ALLOWED_CNS with the explicit list of CNs you've minted via z4j mint-scheduler-cert, AND set the require flag so a misconfigured boot fails closed instead of falling back to trust-the-CA:

Z4J_SCHEDULER_GRPC_ENABLED=true
Z4J_SCHEDULER_GRPC_ALLOWED_CNS='["scheduler-prod-1","scheduler-prod-2"]'
Z4J_SCHEDULER_GRPC_REQUIRE_ALLOWLIST=true

The brain never dials the scheduler: cadence fires arrive over this inbound connection, and an operator's "fire now" is dispatched by the brain itself straight to the agent, so there is no outbound client to configure.

CN project bindings (multi-project deployments)

If you run schedulers per-project (one scheduler instance per tenant, each with its own CN), bind each CN to its project list so a leaked cert can only act on the projects it was minted for:

Z4J_SCHEDULER_GRPC_CN_PROJECT_BINDINGS='{"scheduler-acme":["acme"],"scheduler-globex":["globex"]}'

Without bindings (the default), every allow-listed CN can drive RPCs for any project. With bindings, requests outside the bound project list return PERMISSION_DENIED.

Notification webhooks -- HTTPS-only by default

z4j defaults to HTTPS-only for generic webhook channels. Operator-configured http:// URLs are rejected at config-time and at dispatch-time to prevent payload + custom-header leakage in transit.

If you have a legitimate internal-network http endpoint (intranet receiver, dev rig), opt back in:

Z4J_NOTIFICATIONS_WEBHOOK_ALLOW_HTTP=true

PagerDuty and Telegram post to fixed provider endpoints and are unaffected. Slack, Discord and Teams take an operator-supplied webhook URL that passes through the same scheme check as a generic webhook, so this flag admits http:// URLs for them too; Teams stays host-locked to the Microsoft families either way.

Audit log retention

The audit log is HMAC-chained, and z4j's own write path only ever appends to it. It is not retained forever: a retention sweeper prunes it, and Z4J_AUDIT_RETENTION_DAYS defaults to 90. Set this to your real obligation before the first sweep runs, because rows past the window are deleted rather than archived:

Z4J_AUDIT_RETENTION_DAYS=365   # sweeps hourly; trims rows older than 365d

Pruning the oldest rows eventually removes the genesis row, so the sweeper advances an authenticated prune watermark on the chain state. The verifier accepts a first surviving row whose prev_row_hmac matches that watermark, while a deleted middle row or an altered row_hmac still fails, so retention cannot quietly become a clean-looking truncation. Details in hmac-audit-chain.

Verifying the audit chain

Schedule verification. An HMAC-chained log only proves anything if somebody looks, and an on-demand run only ever proves the chain was intact at the moment someone asked. Either wrap z4j audit verify in cron or a Kubernetes CronJob, or turn on the built-in worker, which is off by default:

Z4J_AUDIT_CHAIN_VERIFY_ENABLED=true
Z4J_AUDIT_CHAIN_VERIFY_INTERVAL_SECONDS=86400   # floor 900, ceiling 604800

For anything stronger than "nothing outside z4j's own write path touched this log", export the chain head on a schedule to a sink the database role cannot rewrite and verify against it with z4j audit verify --known-head. That is the one check that survives a hostile database role. The built-in audit webhook is not that sink: it buffers in memory and drops rows under backpressure, so a gap in its copy cannot tell an outage from a deletion.

Production environment flag

Z4J_ENVIRONMENT alone selects environment-sensitive behavior. The exact value dev enables development relaxations; every other value uses the non-development path. Z4J_ALLOWED_HOSTS is a separate host-header allow-list that is required outside development, but it does not participate in choosing the environment. See allowed-hosts for the four-layer host header allow-list.

Recommended summary

For a production deployment, set:

# Environment
Z4J_ENVIRONMENT=production
Z4J_ALLOWED_HOSTS='["z4j.example.com"]'

# Scheduler gRPC server (brain accepts inbound from z4j-scheduler)
Z4J_SCHEDULER_GRPC_ENABLED=true
Z4J_SCHEDULER_GRPC_REQUIRE_ALLOWLIST=true
Z4J_SCHEDULER_GRPC_ALLOWED_CNS='["..."]'

# Webhooks: HTTPS-only is the default; only set this if you need
# plaintext for an internal endpoint.
# Z4J_NOTIFICATIONS_WEBHOOK_ALLOW_HTTP=true

# Audit retention
Z4J_AUDIT_RETENTION_DAYS=365

Each setting is documented individually in the env vars reference.