CLI reference

z4j ships a CLI named z4j. Runs inside the container or from a local pip install.

z4j --help

Serving

serve

z4j serve [--host 0.0.0.0] [--port 7700] [--workers N]

Starts the FastAPI application via Uvicorn. This is the container entrypoint. --workers defaults to min(4, CPU count); SQLite, the in-memory registry and the embedded scheduler each force a single worker whatever you ask for.

Health and diagnostics

check

z4j check

Validates that the configuration loads and the database answers, then prints whatever migration revision it finds, and says whether the environment is dev or production. Non-destructive. Exit 1 when the configuration does not load, 2 when the database does not answer, 3 when an alembic_version table exists but holds no row.

It does not confirm Alembic is at head. It never compares the stored revision against the head your build expects, and a database with no alembic_version table at all still exits 0. Use z4j migrate current --check-heads, which exits non-zero unless every head is applied. z4j doctor is not a substitute either: it runs check and returns its exit code.

doctor

z4j doctor

Runs check first and returns its exit code, then adds configuration warnings that check cannot raise because they are not failures. It takes no options.

What it actually warns about: Z4J_ENVIRONMENT=dev with a non-loopback Z4J_BIND_HOST, Z4J_DEBUG_HOST_ERRORS being on, a secret store existing at ~/.z4j/secret.env, Z4J_METRICS_PUBLIC being on, and an install with no users, no projects or no agents yet.

It inherits check's blind spot: no Alembic head comparison, no registry probe, no scheduler reachability check, no mTLS validation, and no --json output. z4j doctor --json exits 2 with "unrecognized arguments". For the schema head, use z4j migrate current --check-heads.

status

z4j status

Prints a summary of current brain state: version, the migration revision currently stamped in the database, environment, the database URL and row counts (users, projects, agents, tasks, sessions, audit rows). The URL is printed from the @ onwards, so a PostgreSQL URL loses its scheme, user and password and shows host:port/database; a SQLite URL has no @ and prints in full.

The revision is reported as found. status does not compare it with the migration head this build ships, so a database that is behind still prints its revision and exits 0. z4j migrate current --check-heads answers that question.

version

z4j version

Prints the installed z4j version.

First-boot setup

init

z4j init

Scaffolds $Z4J_HOME/config.env as a mode 0644, tunables-only starter file. It intentionally excludes credentials, tokens, secrets, and other bootstrap values that belong in $Z4J_HOME/secret.env or the process environment. It is not a complete runnable configuration. The command refuses to overwrite an existing file unless --force is supplied.

bootstrap-admin

printf '%s\n' "$ADMIN_PASSWORD" | \
  z4j bootstrap-admin --email you@example.com --display-name "You" \
    --password-stdin

Creates the initial admin user and the default project. First-boot only. A password is required; prefer --password-stdin with a value obtained from a secret manager so it is not exposed in the process list or shell history.

createsuperuser

z4j createsuperuser --email you@example.com --display-name "You" --password-stdin

Django-friendly alias for bootstrap-admin. Use --password-stdin to pipe the password in from a secret manager.

changepassword

z4j changepassword you@example.com --password-stdin

Changes a user's password. Intended for admin recovery / CLI-only ops when the user cannot reach the dashboard reset flow.

reset-mfa

z4j reset-mfa user@example.com
z4j reset-mfa user@example.com --confirm     # skip the interactive prompt

Operator escape hatch for the lost-phone-AND-lost-recovery-codes case. Clears the user's MFA secret, recovery codes, and trusted devices in one transaction; writes a user.mfa_reset_by_admin audit row attributed to the OS user running the CLI. Shell-only by design -- there is no REST surface so an attacker who has only the dashboard cannot trigger it. See Multi-factor authentication for context.

Configuration

config show

z4j config show

Prints the resolved effective settings (post-env-var, post-defaults, post-~/.z4j/config.env). Fields typed as SecretStr are masked unless --reveal-secrets is passed. database_url is masked by name as well. Other plain URL fields such as scheduler_trigger_url are printed as configured and can contain credentials, so treat the entire output as sensitive. The admin settings endpoint exposes the same effective settings but applies additional secret-name masking.

config validate

z4j config validate
z4j config validate /path/to/candidate.env

Takes an optional path to a candidate .env file, defaulting to $Z4J_HOME/config.env. Safely captures the candidate tunables file, rejects unknown Z4J_* keys, and validates the values through the runtime settings decoder without booting the app. Because config.env is a tunables-only layer, the command supplies inert bootstrap placeholders for required secrets, hosts, public URL, and, when absent, the database URL. A zero exit therefore means the candidate tunables are valid. It does not validate the process environment or secret.env, prove that the effective runtime configuration will load, or test database connectivity.

Backups

backup

z4j backup --output /var/backups/z4j-$(date +%Y-%m-%d).db

Snapshots the brain database to a single file. On SQLite, a VACUUM INTO clone; on PostgreSQL, a pg_dump custom-format archive.

--output / -o is the only option, and it is required. An existing destination is always refused, and there is no flag to override that - move or delete the old file, or pick a different path. The output is created private to its owner, so no umask is needed. There is no --no-clobber flag; passing one aborts the command with exit 2. Exit 1 when the destination exists, the source is missing or the snapshot fails; on success the command prints the backend, the path and the size.

restore

z4j restore /var/backups/z4j-pre-upgrade.db --force

Restores from a backup file produced by z4j backup. --force is required, and it records your assertion that the brain is stopped rather than verifying it: there is no liveness detection, so stopping the brain first is your responsibility.

The target's audit chain must verify clean under the local keyring and its head must match this release. Restore is forward-only within the five heads this release lists as restorable (the current head and the four published heads before it: v1_12_auditor_role, v1_11_audit_append_tally, v1_9_audit_action_pattern, v1_8_schedule_cursor_repair, v1_7_security_hardening): such an archive is migrated up rather than taking you back to it. An archive from any older head is refused, and the message tells you to install the release matching that head, restore there, and then upgrade. See backup and restore for the full ceremony.

Maintenance and upgrades

migrate

z4j migrate upgrade head
z4j migrate current
z4j migrate current --check-heads   # exit 0 only when every head is applied
z4j migrate history
z4j migrate downgrade <revision>

Runs an Alembic command against the brain database with the bundled configuration. Everything after the action is passed to Alembic, so current --check-heads is Alembic's own head check: it exits non-zero with "Database is not on all head revisions" when the stamped revision is not this build's head, which is the check check, doctor and status do not make.

Three more actions are accepted. z4j migrate sync re-stamps a database whose head this build does not know, after a brain downgrade across a migration boundary; with --allow-future-schema and --i-know-this-can-corrupt-data, given before the action, it also drops the tables the newer code added, which loses their rows. z4j migrate revision forwards to Alembic's own revision command with whatever arguments follow it, so a contributor adding a migration passes the autogenerate flag and a message themselves. z4j migrate prepare-runtime-rollback is a sealed, image-bound preparation ceremony for one specific release-pair rollback; its --help lists the attestation inputs it requires.

Schedules

misfires

z4j misfires --project my-app
z4j misfires --project my-app --limit 200
z4j misfires --project my-app --json

Lists a project's detected schedule misfires across ALL of its schedules, newest first. --slug is an alias of --project. A misfire is a system-detected "this enabled interval or cron schedule missed its expected slot past the grace window" event, recorded by the brain's misfire detector (see misfire detection). The default output is an aligned text table (schedule id, detected-at, name, engine, kind, lateness, grace); --json emits {"project": <slug>, "misfires": [...]} for scripting. --limit defaults to 50 and is capped at 1000. This is the shell-side twin of the VIEWER-facing GET /api/v1/projects/{slug}/schedules/misfires endpoint, so an operator can triage missed slots without the dashboard. Exit 0 on success (including an empty history); exit 2 on a bad slug, unknown project, or unreachable database.

upgrade

z4j upgrade
z4j upgrade --json
z4j upgrade --timeout 20
z4j upgrade --apply

Without flags, checks PyPI for newer releases of the packages in z4j's bundled release catalogue and changes nothing. --apply runs pip install -U z4j, which lets the umbrella package pull compatible adapters forward, and only after a complete, successful scan: a lookup or comparison error refuses to touch the environment. There is no --check option because check-only is the default. --json emits a machine-readable report instead of the text table. --timeout is the PyPI scan budget in seconds (default 10); no new lookup starts after max(5, 2 x budget) seconds, though an in-flight request may finish later. Exit 1 means at least one package is behind, exit 2 that a lookup or configuration error left the result incomplete; an installed version newer than PyPI is reported as newer, not as behind. Mirrors POST /api/v1/admin/system/versions/check on the API side.

Allowed hosts

allowed-hosts list / add / remove / path

z4j allowed-hosts list
z4j allowed-hosts add z4j.example.com
z4j allowed-hosts remove old-host.example.com
z4j allowed-hosts path

Manages the persistent allow-list file at $Z4J_HOME/allowed-hosts, which a non-PostgreSQL brain merges into the auto-detected set when z4j serve starts; a PostgreSQL brain never reads it. It is separate from Z4J_ALLOWED_HOSTS, which replaces auto-detect and the file entirely when set. add and remove take one or more hosts and are idempotent, a change takes effect on the next z4j serve start, and path prints the path of the file. See allowed hosts.

Audit log

audit verify

z4j audit verify

Walks the full HMAC-chained audit log. Exit 0 when the chain verifies, 1 on a mismatch or a verification that could not complete (an unreachable database included), and 2 when the configuration does not load or --limit is outside 1 to 5000. --limit is only the page size of the walk (default 1000); the whole active generation is walked whatever you set.

Pass --known-head with an envelope recorded earlier to detect a log that was rolled back to an older authentic state, which the chain alone cannot show. The result is one of CURRENT_MATCH, PRUNE_MATCH, CURRENT_PRUNE_MATCH, VERIFIED_ANCESTOR, INVALID or UNPROVABLE. A match and VERIFIED_ANCESTOR exit 0; INVALID and UNPROVABLE exit 1.

audit export-head

z4j audit export-head --output /secure/last-known-head
z4j audit export-head --verify --output /secure/last-known-head

Prints the authenticated current chain head as the JSON envelope audit verify --known-head accepts. The state row is proved against the configured keys first, and a head that cannot be authenticated is refused rather than exported.

Write the file with --output PATH, not a shell redirect. > file truncates the target before the command runs, so a refusal would destroy the anchor you already had, exactly when something is already wrong. --output writes through a temporary file and renames it over the target only after a complete envelope has been written, so an existing anchor survives every refusal. The envelope goes to stdout when --output is absent and every other message to stderr. --verify walks the active generation first and refuses to export a head from a chain that did not verify clean, which is the right choice for an unattended job: anchoring a compromised chain would record it as the trusted state.

Anchor that file somewhere the database role cannot rewrite. A head kept only in the same database proves nothing, because a writer can restore an older copy of it. Write the file exactly as emitted: the parser's key set is closed, so appending anything makes it unreadable.

audit prune

z4j audit prune
z4j audit prune --apply
z4j audit prune --apply --before 2026-01-31T00:00:00Z
z4j audit prune --hard --apply --before 2026-06-30T00:00:00Z

Removes expired audit rows under the authenticated prune boundary, the same verified oldest-first prefix prune the retention worker runs, driven by hand. Dry run by default: prints the rows it would remove by class, the row the prefix stops at and why, the boundary it would record, and the rows that remain. --apply executes and writes one audit.prune row about itself. --before replaces the configured windows (Z4J_AUDIT_RETENTION_DAYS, Z4J_AUDIT_RETENTION_BY_CLASS) with one explicit cutoff for every class; a timezone is required and the value must not be in the future. --hard is an epoch cut: after the prune, a fully pruned generation is replaced by a fresh signed genesis with no boundary; it refuses while any row younger than the cutoff remains or while frozen legacy rows exist.

Exit 0 when done, after a dry run, or when there is nothing to prune; 1 on a refusal (no audit-chain key, chain state missing or not authenticating, a count or link in the prefix disagreeing with the signed state, a held lease, a hard-mode precondition); 2 when settings fail to load or --before is not an aware past timestamp. The command holds the scheduled verifier's lease and the sweep's lock while it works. See audit retention for the prefix rule and what verify reports afterwards.

The audit group also carries the chain-key lifecycle commands (rotate-chain-key, retire-chain-key) and the chain-state operations (activate-chain-state, reseal-watermark, export-and-delete-frozen, fork-cleanup). Each takes --help. See HMAC audit chain for the rotation ceremony.

Secrets

secrets rewrap

z4j secrets rewrap
z4j secrets rewrap --dry-run     # report only, write nothing

Re-encrypts every value stored under Z4J_SECRET (notification channel configs in notification_channels.config and user_channels.config, and stored TOTP secrets in users.mfa_secret_encrypted) under the current master, in one transaction. A value already under the current master is counted and left alone; one that decrypts only under a Z4J_PREVIOUS_SECRETS entry is re-wrapped. Prints one line per column with scanned, already-current, re-wrapped, plaintext, changed-under-us and undecryptable counts, and writes a secrets.rewrap audit row attributed to the OS user.

This is the step that makes dropping a value from Z4J_PREVIOUS_SECRETS safe. Without it a channel config is re-wrapped only when that channel is next edited, and a TOTP secret only when that user next verifies a code, so rows that are never touched stay under the old master and are orphaned the moment it is dropped.

A channel row stored without encryption (written by a brain running a release from before the column was encrypted, or by hand) is still read: the brain parses it as plaintext JSON and logs one warning per process naming the column, never the value. secrets rewrap reports such rows as plaintext N and encrypts them under the current master in the same run; --dry-run reports them and writes nothing. The command first places the database's migration head on the chain the installed scripts describe and refuses (exit 2, nothing written, in both modes) when that head is below the migration that made the channel config columns encrypted text: there the columns are still JSON, and encrypting a row would write ciphertext into them. Run z4j migrate upgrade head first. Every write is guarded by the value the command read, so a channel edit a running brain commits meanwhile wins: the row is reported as changed under us and left as it is, and its id is listed with the undecryptable ones only when what is there now is not under the current master. A non-encrypted row that is not JSON either is listed as undecryptable too; the brain cannot read it.

Exit 0 when every stored secret is encrypted under the current master. Exit 1 when at least one row decrypts under no listed secret: the rest were still re-wrapped, the failing ids are printed, and the previous secret those rows were written under has to go back before the next run. Also exit 1, with nothing written, when encrypted rows exist and not one of them decrypts, which means the process is not running with the keys the database was written under. Exit 2 when the configuration does not load. Run it with the same Z4J_SECRET and Z4J_PREVIOUS_SECRETS the brain runs with; with multiple replicas, run it once. See incident response for where it sits in a rotation.

Tokens and certs

metrics-token

z4j metrics-token              # print the active token
z4j metrics-token show         # the same, as an explicit action
z4j metrics-token rotate       # update the managed store; restart is required

Manages the bearer that gates /metrics. Resolution order is the process environment, then ./.env, then $Z4J_HOME/config.env, then $Z4J_HOME/secret.env, where the packaged SQLite first boot mints one. With no token anywhere the command exits 2. rotate writes only secret.env and exits 2 when the environment, .env or config.env is the effective source. The command writes and prints the new token, but a running brain keeps the old token in memory. Update the Prometheus scrape credential, then restart every brain replica. The old token stops working only after those restarts.

mint-scheduler-cert

z4j mint-scheduler-cert \
  --name scheduler-prod-1 \
  --ca-cert /etc/z4j/tls/ca.crt \
  --ca-key /etc/z4j/tls/ca.key \
  --out-dir /etc/z4j/tls/

Mints scheduler-prod-1.crt and scheduler-prod-1.key for a z4j-scheduler instance to authenticate to the brain. --validity-days sets the certificate's validity (default 365). The command does not modify the brain configuration. Add scheduler-prod-1 to the JSON Z4J_SCHEDULER_GRPC_ALLOWED_CNS value and restart every brain replica before deploying the client certificate. See production hardening.

Destructive

reset-setup

z4j reset-setup --force

Deletes pending first-boot tokens, preserves existing signed setup audit rows, and appends a signed setup.tokens_reset row. It refuses if any user exists. Without --force, it exits nonzero without changing the database. Use reset --force for a full wipe.

reset

z4j reset            # refuses without --force
z4j reset --force

Deletes the domain rows (users, sessions, projects, agents, tasks, events, schedules) and replaces the audit history with one signed reset-genesis row. The schema, secret.env, the authenticated installation identity and the schedule-revision and external-epoch counters are retained, and the next z4j serve prints a fresh setup URL.

On a packaged SQLite install, z4j reset --force --nuke-secrets instead retires the existing database and key pair into a recoverable bundle and creates a fresh replacement. Existing credentials do not authenticate to the replacement, and the retained bundle is removed only by the recovery command below with the manifest digest that reset printed.

Two flags bind the wipe to a preview. --preview-manifest PATH writes an owner-private, non-mutating finalized reset manifest together with a stopped-executor attestation challenge, and changes nothing else. --attest-stopped-executors SHA256 attests that every executor named in that exact preview manifest is stopped; the value must equal the preview's challenge.

Ceremonies

Two command groups exist for multi-step operations bound to a manifest digest. Each step prints what the next one must carry, and every command takes --help.

projects rewrite-scheduler

z4j projects rewrite-scheduler --slug my-app --from <owner> --to <owner> \
  --source-scope <scope> --dry-run

Previews or finalises an explicit change of the scheduler that owns a project's schedules. The dry run prints the canonical cutover preview and its digest without writing. After both scheduler fleets are quiesced, the same command is rerun with --operation-id, --preview-manifest-digest and --attest-all-schedulers-quiesced to apply it.

recovery destroy-retired-installation

z4j recovery destroy-retired-installation --operation <uuid> \
  --confirm-manifest-digest <digest>

Removes one retained installation bundle that z4j reset --force --nuke-secrets created, after checking the bundle's authenticated binding to its replacement. Both values come from the output of that reset.

Global flags

Flag	Description
-h, --help	Show help for the command
-V, --version	Print the installed version, like z4j version

Per-command flags surface with z4j <command> --help.

z4j-scheduler CLI

When you run the standalone scheduler service, the z4j-scheduler CLI ships these commands. See z4j-scheduler for deployment context.

z4j-scheduler serve                        # run the scheduler process
z4j-scheduler version                      # print installed version
z4j-scheduler check --brain-grpc-url ...   # one-line pass/fail health
z4j-scheduler status                       # local introspection: version + configured brain URLs + mode (no network)
z4j-scheduler info [--json]                # live runtime snapshot from the running scheduler's GET /info
z4j-scheduler doctor                       # comprehensive diagnostics
z4j-scheduler restart                      # informational stub (delegates to systemd / k8s)
z4j-scheduler import --from <tool> ...     # migrate from celery / django-celery-beat / rq / apscheduler / cron / huey / arq / taskiq
z4j-scheduler export --to <tool> ...       # reverse-export to celery / rq / apscheduler / cron / huey / arq / taskiq
z4j-scheduler schedules add / list / trigger / disable / enable / edit / history ...

import --from accepts celery, django-celery-beat, rq, apscheduler, cron, huey, arq, taskiq and dramatiq. Each source has its own locator flag:

z4j-scheduler import --from celery --celery-app myapp.celery:app --project myproject
z4j-scheduler import --from django-celery-beat --django-settings myapp.settings --project myproject
z4j-scheduler import --from rq --redis-url redis://localhost:6379/0 --project myproject
z4j-scheduler import --from apscheduler --jobstore-url postgresql://... --project myproject
z4j-scheduler import --from cron --crontab /etc/crontab --user-column --task-prefix myapp.shell.exec_command --project myproject
z4j-scheduler import --from huey --huey-app myapp.tasks:huey --project myproject
z4j-scheduler import --from arq --arq-settings myapp.worker:WorkerSettings --project myproject
z4j-scheduler import --from taskiq --taskiq-broker myapp.tkq:broker --project myproject

--dry-run prints the parsed schedules as JSONL instead of pushing them, --verify diffs them against the brain and implies --dry-run, and --queue, --timezone and --engine fill in what the source does not carry.

export --to accepts celery, rq, apscheduler, cron, huey, arq, taskiq and dramatiq; --out chooses the file (default stdout) and --source / --scheduler filter the rows. Dramatiq has no native scheduler, so both dramatiq values print migration guidance instead of reading or writing a schedule store: import --from dramatiq exits 2 with that guidance, and export --to dramatiq first fetches the project's schedules from the brain like every other target (so the brain must be reachable) and then renders the guidance as comments.

All commands accept --help. Most read configuration from Z4J_SCHEDULER_* env vars; explicit flags win where provided.

Framework adapter CLIs

Each framework adapter ships its own doctor plus a few helpers under python -m z4j_<framework>:

python -m z4j_bare doctor              # canonical implementation
python -m z4j_flask doctor             # same checks; reads Z4J_* env
python -m z4j_fastapi doctor           # same checks; reads env
python -m z4j_django doctor            # reads Z4J_* env only; ./manage.py z4j_doctor reads Django settings

All four implement doctor with the same probes (buffer dir writable, brain DNS / TCP / TLS, WebSocket upgrade) and accept the same flags (--no-websocket, --json). See frameworks/bare for the canonical reference.