Self-hosting

Sizing

There is no hardware-to-agent-count guarantee. Start with an isolated test deployment, replay representative event rates and retention, exercise the dashboard queries and administrative actions you use, and size the brain, Postgres, and disk from the resulting CPU, memory, latency, and storage growth. Repeat the benchmark when event shape, retention, or replica topology changes.

PostgreSQL is authoritative for application rows, but the brain is not fully stateless. Its persistent $Z4J_HOME volume (/data in the packaged containers) can carry generated secrets, installation and restore-recovery state, allowed-host configuration, and embedded PKI. Back up that state with PostgreSQL, or preserve the corresponding external secret-manager values when the files are not local. Multi-replica deployments rely on the load balancer's session-affinity setting to pin each agent's WebSocket to one brain pod. z4j provides no affinity-balance helper of its own.

Networking

• Inbound to brain - TCP 7700 (HTTP + WebSocket) from the internet or private network.
• Outbound from agent - agents initiate WebSocket connections to z4j. Agents need DNS + TCP 443 (if you TLS-terminate at a reverse proxy).
• Brain to Postgres - usual asyncpg connection.

z4j does not need to reach agents. Agents sit behind NAT / private networks freely.

systemd

For a pip install on one Linux host, z4j ships two hardened units in the deploy/systemd/ directory of the sdist and the source repository: z4j-brain.service and, for the standalone scheduler, z4j-scheduler.service. Each reads its settings from an EnvironmentFile (/etc/z4j/brain.env, /etc/z4j/scheduler.env; examples sit next to the units), runs as a dedicated z4j user with Z4J_HOME in a systemd StateDirectory, and applies the systemd sandbox: ProtectSystem=strict, NoNewPrivileges, an empty capability set, the @system-service syscall filter, private /tmp and devices.

sudo useradd --system --home-dir /var/lib/z4j --shell /usr/sbin/nologin z4j
sudo python3 -m venv /opt/z4j/venv && sudo /opt/z4j/venv/bin/pip install z4j
sudo install -d -m 0750 -o root -g z4j /etc/z4j
sudo install -m 0600 brain.env.example /etc/z4j/brain.env   # public URL, allowed hosts, secrets, database
sudo install -m 0644 z4j-brain.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now z4j-brain
sudo journalctl -u z4j-brain -f                              # first boot prints the setup URL

The brain binds loopback and expects the reverse proxy below in front of it. JSON-valued settings go single-quoted in the env file (Z4J_ALLOWED_HOSTS='["z4j.example.com"]') so systemd passes the brackets through. Both processes read their environment and certificate files once at start: after editing an env file or rotating certificates, systemctl restart the unit. systemd-analyze security z4j-brain shows the sandbox score; relax a single directive with systemctl edit if the journal shows a denial.

TLS

Terminate TLS at a reverse proxy (nginx / Caddy / Cloudflare). z4j speaks plain HTTP internally; set Z4J_PUBLIC_URL to the https origin, have the proxy forward X-Forwarded-For, and list the proxy in Z4J_TRUSTED_PROXIES so the brain attributes requests to the real client address.

Sample Caddy config:

z4j.example.com {
  reverse_proxy localhost:7700 {
    header_up X-Forwarded-Proto https
  }
}

Database

• Minimum: PostgreSQL 17. Recommended: PostgreSQL 18, which is what the shipped Compose file runs.
• Extensions: none required (no pgvector, no timescale).
• Connection pool defaults: pool_size=20, max_overflow=10 per engine. Size them with Z4J_DATABASE_POOL_SIZE and Z4J_DATABASE_MAX_OVERFLOW; recycle and statement-cache tuning are exposed via Z4J_DATABASE_MAX_INACTIVE_CONNECTION_LIFETIME_SECONDS and Z4J_DATABASE_STATEMENT_CACHE_SIZE (see brain memory tuning).

Secrets

For pip / SQLite installs z4j auto-mints secrets on first boot and persists them to ~/.z4j/secret.env. You don't need to set anything; restarts pick up the same values from disk.

For Docker / Postgres production deployments, generate explicit secrets so they survive container replacement and live alongside your other infra config:

Z4J_SECRET=$(openssl rand -hex 32)
Z4J_SESSION_SECRET=$(openssl rand -hex 32)
Z4J_AUDIT_CHAIN_SECRET=$(openssl rand -hex 32)

Z4J_SECRET is the master from which the per-project HMAC signing secret is derived (deterministic - z4j re-derives it on every frame and never stores it). Z4J_SESSION_SECRET is independent and signs the user-session cookie. Z4J_AUDIT_CHAIN_SECRET independently signs the audit chain and is required outside exact dev mode.

The three keys have different rotation ceremonies:

• Session key: deploy the new Z4J_SESSION_SECRET with the old value in Z4J_PREVIOUS_SESSION_SECRETS, then restart every brain. Without that carry-over window, rotation invalidates every active browser session.
• Master key: deploy the new Z4J_SECRET with the old value in Z4J_PREVIOUS_SECRETS, restart every brain, and re-credential every agent. The previous window keeps old bearer tokens and stored TOTP values readable, but frame-signing keys come from the current master alone. Existing API keys and outstanding invitation or password-reset links also stop working. This rotation does not touch the dedicated audit chain.
• Audit-chain key: stop every brain replica first. For explicit secrets, configure the new Z4J_AUDIT_CHAIN_SECRET and put the old value in Z4J_AUDIT_CHAIN_PREVIOUS_SECRETS, run z4j audit rotate-chain-key from an administrative environment using that same database and keyring, then start every brain and run z4j audit verify. For a managed SQLite install, use z4j audit rotate-chain-key --begin-managed while the brain is stopped. Changing the environment and restarting without the explicit transition is not a completed audit-key rotation.

See incident response for the complete ceremonies and the separate retirement clocks for previous keys.

Backups

• Postgres: standard pg_dump or managed service snapshots. See backups guide.
• Secrets: store in your secret manager (Vault, AWS Secrets Manager, SOPS).

Monitoring

z4j exposes /metrics in Prometheus format (auth-gated; set Z4J_METRICS_AUTH_TOKEN explicitly. Only a fresh packaged SQLite install mints a token on its first z4j serve and keeps it in ~/.z4j/secret.env; Docker and PostgreSQL deployments set it themselves). See operations § monitoring.