Upgrade and rollback

z4j ships patch releases regularly. Patch upgrades are in-place. An installation that upgraded and then merely ran can be brought back down at the schema level, because nothing on the running path rewrites the rows the older release identifies its schedules by. That is the ordinary case, not a guaranteed one: once a schedule has been created or re-saved, the downgrade refuses and the pre-upgrade backup is the only way back. Take the backup, then read Rollback - pip before you start, not after.

Pre-flight

Before any upgrade:

z4j doctor                         # config valid, DB reachable, no surprises
z4j migrate current --check-heads  # exit 0 only if the schema IS at head
z4j status                         # row counts, to compare after the restart
z4j backup --output /var/backups/z4j-pre-upgrade-$(date +%Y-%m-%d-%H%M).db

z4j doctor does not tell you the schema is at head, despite reading like it might: it runs check first and returns its exit code, and check prints whatever revision it finds without comparing it to anything. Use --check-heads for the question that has an answer. Note the revision it prints as well, because that is the value you need if you have to roll back.

The backup is the most important step. Without it, an upgrade that introduces an unexpected migration is hard to recover from.

Pip upgrade

# 1. Stop the brain
sudo systemctl stop z4j

# 2. Upgrade the wheel (no-cache so we always get the published version)
sudo -u z4j /srv/venv/bin/pip install --no-cache-dir --upgrade z4j

# 3. Confirm version and that migrations are in place
sudo -u z4j /srv/venv/bin/z4j version
sudo -u z4j /srv/venv/bin/z4j migrate current     # pre-restart sanity check

# 4. Restart - Alembic upgrades to head automatically on serve
sudo systemctl start z4j

# 5. Verify
journalctl -u z4j -n 30 --no-pager     # look for the boot banner
sudo -u z4j /srv/venv/bin/z4j check
sudo -u z4j /srv/venv/bin/z4j status   # row counts should match step-1 snapshot

z4j runs alembic upgrade head on every serve start unless you set Z4J_AUTO_MIGRATE=false. Additive migrations are bidirectional and a roundtrip test proves it against a real PostgreSQL server before each release, but that roundtrip runs below the boundary activations, not from the current head. Above them the boundary activations refuse unconditionally, so nothing crosses them downward. The schedule-control revision above those does come down, but only for an installation whose schedule rows still carry the identity the target wrote, and only when you declare what that target computes. Keep the pre-upgrade backup regardless: it is what you have left when the downgrade refuses. See database migrations for which revisions refuse and why.

Troubleshooting: `Unable to locate executable '/srv/venv/bin/z4j'`

If systemd reports this after an upgrade:

z4j.service: Unable to locate executable '/srv/venv/bin/z4j': No such file or directory
z4j.service: Failed at step EXEC spawning /srv/venv/bin/z4j: No such file or directory
z4j.service: Main process exited, code=exited, status=203/EXEC

The z4j console script is missing from the venv even though pip list shows z4j is installed. The cause is a venv where the z4j dist-info metadata exists but the wheel content was never actually unpacked, so pip install --upgrade z4j short-circuits with "Requirement already satisfied" and never drops the binary into bin/. Common triggers: a venv built across a package-rename cut, manual cleanup that deleted files but left dist-info, or a previous install was interrupted.

The fix is a force-reinstall that ignores the metadata check:

sudo -u z4j /srv/venv/bin/pip install --force-reinstall --no-deps z4j
ls -la /srv/venv/bin/z4j   # should now exist, mode 0755
sudo systemctl restart z4j

--no-deps keeps it fast since dependencies are not the issue; only z4j's own wheel needs to be replanted.

Docker upgrade

The runnable procedure is in upgrades, per stack. It is not repeated here, because the version that used to be here was generic: it ran docker compose with no -f and set no image variable, so a PostgreSQL operator following it addressed the default SQLite Compose file instead of their own and could recreate the z4j service against the wrong volume.

Both details that make it work are stack-specific, which is why a generic form cannot exist:

The Compose file, since PostgreSQL deployments need -f docker-compose.postgres.yml.
The image variable, which is Z4J_IMAGE for the default file and Z4J_BRAIN_IMAGE for the PostgreSQL one. Setting the wrong one is a silent no-op that leaves you on the old tag.

For PostgreSQL deployments the z4j image and the PostgreSQL image are independent; upgrading z4j does not touch PostgreSQL data.

Pinning vs `:latest`

Tag	When to use
`z4jdev/z4j:<X.Y.Z>`	Production -- reproducible deploys, controlled upgrade cadence.
`z4jdev/z4j:latest`	Homelab / small teams that track current stable. Pair with a `digest`-pin if you need reproducibility within "latest".

Pinning is what makes a rollback plan possible at all, because both routes back need you to name the exact release you are returning to. pip install <older> on its own does not move you back: the older binary refuses to start against the newer schema, so the schema has to come down first, and bringing it down past v1_9_schedule_control_columns means measuring the fingerprint that release computes. Know which version you are on, and keep the pre-upgrade archive that belongs to it. See Rollback - pip. Across major versions, expect a documented manual step.

Rollback - pip

An installation that upgraded and then merely ran can go back. Every schedule owned by the z4j scheduler carries a fingerprint of the cadence runtime that wrote it. Nothing on the running path rewrites that value: the fire path does not touch it, the cursor path does not touch it, and the only write sites are creating a schedule and cutting one over to a new owner. So on an installation that upgraded and then ran, the rows still carry the identity the older release wrote, and handing them back restores exactly the state that release last saw. Nothing is rewritten, so nothing is claimed, and no image, registry or signature is involved.

The migration will not take your word for which release you are returning to. You declare what it computes, and the declared value has to equal what the rows already carry.

Measure the target first

Only the v1_9_schedule_control_columns step reads Z4J_ROLLBACK_TARGET_FINGERPRINT; a plan that stops at or above that revision never reads the variable and needs no measurement. When the plan does step it down, run this in the environment you are about to install the older release into, under that release:

python -c 'from z4j_brain.domain.schedule_cadence import cadence_runtime_fingerprint as f; print(f())'

It prints a sha256 digest. That is the value the downgrade wants, and reading it out of your current database instead is the one mistake this whole step exists to prevent. See what the declaration proves below.

The sequence

# 1. Stop the brain, the scheduler, and anything else that writes.
sudo systemctl stop z4j

# 2. Take a backup even though you are rolling back. If the downgrade
#    succeeds, the dropped columns are gone, and this archive is what holds
#    the record of the pauses and revocations that went with them.
sudo -u z4j /srv/venv/bin/z4j backup --output /var/backups/z4j-pre-rollback-$(date +%Y-%m-%d-%H%M).db

# 3. Bring the schema down to the revision you noted in pre-flight, declaring
#    the fingerprint you measured. Only the v1_9_schedule_control_columns step
#    reads the variable; a plan that stops above it ignores it. `env` is
#    required: sudo does not accept a bare VAR=value in front of the command.
sudo -u z4j env Z4J_ROLLBACK_TARGET_FINGERPRINT=<measured-digest> \
  /srv/venv/bin/z4j migrate downgrade <pre-upgrade-revision>

# 4. Install the release you are returning to.
sudo -u z4j /srv/venv/bin/pip install --no-cache-dir --force-reinstall z4j==<previous-version>

# 5. Start and verify.
sudo systemctl start z4j
sudo -u z4j /srv/venv/bin/z4j migrate current --check-heads
sudo -u z4j /srv/venv/bin/z4j status
sudo -u z4j /srv/venv/bin/z4j audit verify

Step 3 comes before step 4, not after. z4j migrate downgrade has to run under the release that knows the revision it is stepping down from, and the older binary refuses to start against the newer schema anyway.

Confirm at step 5 that your schedules are still enabled and their cursors intact. They should be: the downgrade dropped columns and rewrote no schedule row. The rows it does rewrite are notification channel configs, which it turns back into plain JSON, and that is why it needs the master secret they were written under; see when it refuses.

When it refuses

Every guard is evaluated over the complete resolved downgrade plan before its first step runs, so a refused downgrade leaves the database exactly as it was. You are not stranded part way through a rollback, and there is nothing to clear. Each refusal names its own reason. Three of them (a channel config that does not decrypt, a membership holding the auditor role, an SQLite library that cannot drop a column) depend on your rows or your host rather than on the revision alone, and they are evaluated at the same point as the rest: before the first step.

The rows do not carry the identity you declared, or carry more than one identity between them:

refusing downgrade: reserved schedules carry cadence fingerprint <a>, but
Z4J_ROLLBACK_TARGET_FINGERPRINT declares <b>

That is what a schedule created or re-saved since the upgrade produces: those rows carry the current release's identity, the untouched ones carry the target's, and there is no single state to hand back. It is the minority case, and it is real, because one save is enough. Restore the pre-upgrade backup instead.

You declared what the running release computes, rather than the one you are returning to. That names no downgrade target, so it is refused on sight. It is the shape a copied-from-the-wrong-shell mistake takes, and it is the reason the measurement has to happen in the target environment.

A schedule is paused. paused_at is the only record of the hold, so dropping it would release every one of them at once, silently, and they would start firing again on the next cadence tick during whatever incident you paused them for. Resuming them to get past the guard does not help you: it produces that exact outcome, only with you having asked for it. Roll back from a backup taken before the hold.

An agent is revoked. revoked_at is the durable tombstone that keeps older cleanup code away from retained event history. Restore a backup taken before those revocations.

A notification channel config does not decrypt. Stepping v1_12_channel_config_encrypted down rewrites every channel config back to plain JSON, so it needs the master secret each row was written under: Z4J_SECRET, or an entry in Z4J_PREVIOUS_SECRETS. A row that decrypts under none of them is refused by table and id. Put the secret it was written under back into Z4J_PREVIOUS_SECRETS and run the downgrade again. That is a manual step; nothing recovers the secret for you.

A membership holds the auditor role. The release you are returning to cannot represent it, so stepping v1_12_auditor_role down is refused with the count. Move those members to another role by hand first (viewer holds strictly less authority), then run the downgrade again.

The SQLite library cannot drop a column. Stepping v1_12_export_jobs_sink down drops three columns with ALTER TABLE DROP COLUMN, which older SQLite libraries do not have. The guard reads sqlite_version() and refuses on such a library; PostgreSQL is never affected. Upgrade the library, or restore a backup taken before that revision.

You declared nothing, and the plan steps v1_9_schedule_control_columns down. Absent Z4J_ROLLBACK_TARGET_FINGERPRINT a different path governs that step, described under the container ceremony below, and it refuses with:

refusing downgrade: rollback compatibility image authority is not finalized

That message means no fingerprint was declared, not that a declared one failed.

What the declaration proves

It cannot make a bad downgrade succeed. The value has to equal what the rows already carry, and a wrong one is refused with both values named. Its purpose is to force you to measure the release you are about to install rather than assume it.

It cannot prove you measured. Nothing ties the value you set to a running target. An operator who reads the fingerprint out of their own database and pastes it into the variable satisfies the check and learns nothing from it. What the check establishes is that the rows are internally consistent and that you asserted a value. It does not establish that the value is what your target computes.

That distinction has a durable consequence. If the release you install computes something different from what the rows carry, it disables every schedule it cannot agree with. Nothing fires until an operator re-enables them, one by one, by hand, and a restart does not clear it. Defeating the check with a value you did not measure buys you a stopped fleet on the other side of the rollback.

When the downgrade is not available

Restore your pre-upgrade backup under the release you are returning to.

Two constraints shape the sequence. z4j restore is forward-only, so it has to run under the older binary rather than the newer one. And the older binary will not take a target that is already on the newer schema: each release pins the exact schema signature it authenticates, and on SQLite that refusal arrives after the durable phase has been written, which leaves the installation fenced until you clear it with z4j restore --force --rollback-operation <uuid>. So the upgraded database gets moved aside rather than restored over.

# 1. Stop the brain, the scheduler, and anything else that writes.
sudo systemctl stop z4j

# 2. Move the upgraded database aside. Do not delete it: it is the only copy of
#    everything written since the upgrade, and the older binary will not start
#    against it.

# 3. Install the release you are returning to.
sudo -u z4j /srv/venv/bin/pip install --no-cache-dir --force-reinstall z4j==<previous-version>

# 4. Let that release provision and migrate its own database, then stop it. An
#    empty database is refused as a restore target, so it has to reach that
#    release's own head the ordinary way before it can take the archive.

# 5. Restore the pre-upgrade archive into it.
sudo -u z4j /srv/venv/bin/z4j restore /var/backups/z4j-pre-upgrade-<stamp>.db --force

# 6. Start and verify.
sudo systemctl start z4j
sudo -u z4j /srv/venv/bin/z4j migrate current --check-heads
sudo -u z4j /srv/venv/bin/z4j status
sudo -u z4j /srv/venv/bin/z4j audit verify

Read backup and restore before you run step 5. It carries what the target has to satisfy, what each failure mode leaves behind, and why the fence rather than the row count is the thing to read afterwards. Rehearse this sequence on a throwaway install before you need it.

Everything written since the upgrade is gone on this path, and nothing recovers it. That is why the pre-upgrade backup is still the plan you take the upgrade with rather than a precaution. If you skipped it, the options are to roll forward (fix the bug, ship a patch) or to restore the most recent off-host backup and accept the same loss from an older point.

The container ceremony

Absent a declared fingerprint a different path governs, and it is the one built for the deployment shape where the rows have moved on: a two-phase preparation ceremony that restamps every reserved schedule against a published compatibility carrier image. Restamping is a claim rather than a check, which is why that path needs a signed carrier at all and the declared one does not.

It is not available. The ceremony binds to a carrier digest that has not been published, so every invocation refuses with "rollback compatibility image authority is not finalized". Do not plan a rollback around it. If your rows have moved on, the backup is the route.

For completeness, because the refusal names them: the ceremony is z4j migrate prepare-runtime-rollback --target <sealed target> --target-image <ref> --candidate-authority-root <dir> --rollback-evidence-root <dir>, and it reads two environment variables that nothing else reads. Z4J_RELEASE_SOURCE_REVISION binds the running release's 40 hex character Git revision and Z4J_RELEASE_IMAGE binds its image by OCI sha256: digest; an empty or malformed value exits 2, and both are checked against the finalized production carrier. The two directory arguments have environment equivalents that the downgrade step itself reads when no Z4J_ROLLBACK_TARGET_FINGERPRINT is declared: Z4J_PRODUCTION_FINALIZATION_ROOT names the read-only finalized production-container authority (signed receipt, bundle, attestation transcript and staging index) and Z4J_ROLLBACK_COMPAT_EVIDENCE_ROOT names the read-only evidence graph, verified with cosign. The downgrade checks the sealed manifest, then the production root, then the evidence root, then the ceremony's own audit receipt, and refuses at the first one that is missing.

Rollback - Docker

There is no copy-and-run recipe here, and that is deliberate. Three review rounds found three different sets of defects in the one that used to be here: it pinned the image through a variable the PostgreSQL stack does not read, it restored from a path its own preflight never wrote, and it ran pg_restore without the safety flags the supported implementation uses. A rollback procedure is read once, under pressure, by someone who cannot check it. Wrong is worse than absent.

What is true, and what you need to plan around:

Stop the brain first, and know that nothing enforces it. z4j restore requires --force, but --force only records your assertion that the brain is stopped. There is no liveness detection in the restore path at all, so a restore run against a live brain proceeds. An earlier version of this page said the command refuses while the brain is up. It does not, and that is the most dangerous kind of wrong, because an operator reading it under rollback pressure would conclude the tool was checking for them.

A downgrade that does run preserves your rows. It is not a data-wiping operation: what it removes is derived audit accounting, the API-key CIDR column, the audit forwarder's state table, the three export-job columns and, at v1_9_schedule_control_columns, the three schedule-control columns, not your projects, tasks, events, or audit history. It rebuilds the PostgreSQL role enum without auditor, and the one kind of row it rewrites is the notification channel config, back to plain JSON. That is worth knowing because it means the preflights below are protecting the meaning of those columns and rows rather than protecting your data from the migration itself.

An older brain will not start against a newer schema. It refuses, hard: the brain compares the stamped revisions against the single release head compiled into it and exits when they differ. No environment variable downgrades that to a warning. So the failure mode to plan for is a brain that will not come up until the schema matches the binary, not one that runs on half-understood data. (There is a tolerant path, but only for package-version skew inside an identical schema head, which is to say a release that shipped no migration.)

The schema rolls back conditionally, and each step has its own guard. Stepping the head, v1_12_auditor_role, down rebuilds the PostgreSQL project_role enum without auditor (SQLite stores the role as plain text and needs no change) and refuses while any membership still holds that role; move those members to another role by hand first, viewer being the nearest. Stepping v1_12_export_jobs_sink down drops the three export-job columns and refuses on an SQLite library without ALTER TABLE DROP COLUMN support. Stepping v1_12_channel_config_encrypted down rewrites every channel config back to plain JSON, which needs the master secret each row was written under (Z4J_SECRET or an entry in Z4J_PREVIOUS_SECRETS), and refuses naming the first row that decrypts under none of them. Those three refusals depend on your rows or your host rather than on the revision alone, and each is evaluated over the whole plan before its first step, the same as every other guard on this page. v1_12_audit_forward_state and v1_12_api_key_allowed_cidrs drop their table and column. Stepping v1_11_audit_append_tally down drops the audit tally triggers and functions (PostgreSQL only), the ix_commands_schedule_fire_receipt index and the audit_chain_state.observed_active_row_count column; signed audit rows are left intact. Its one preflight refuses if that index has an unexpected definition. Stepping v1_9_audit_action_pattern down drops the ix_audit_log_action_pattern index under the same kind of preflight. Stepping v1_9_schedule_control_columns down drops schedules.overlap_policy, schedules.paused_at, and agents.revoked_at, and it runs three durable-state preflights before it does. Two are about your data. It refuses if any schedule is paused, because dropping paused_at would release that hold silently. It also refuses if any agent is revoked, because dropping revoked_at would erase the tombstone that keeps older cleanup code away from retained event history. The third asks whether the schedule rows are still the ones the release you are returning to wrote. Declare what that release computes in Z4J_ROLLBACK_TARGET_FINGERPRINT and the step proceeds when the rows agree; declare nothing and a container ceremony governs instead, refusing with "rollback compatibility image authority is not finalized". Only this step reads the variable. See Rollback - pip for how to measure the value, and for what declaring it does and does not prove.

The preflights run over the whole plan before the first step, so a refusal leaves the database untouched. That is the good news: you are never stranded part way.

Restoring a backup is the route whenever one of the schedule-control guards refuses, so have the archive in hand before you start rather than after. The channel-config and auditor refusals clear by hand instead (put the secret back, reassign the members) and the SQLite one by upgrading the library. See database migrations for which revisions refuse and why.

z4j restore cannot run inside the shipped image on PostgreSQL. That image carries libpq5 and no client tools, so it refuses with "pg_restore was not found on PATH". Nor can it run from the host against the shipped Compose stack: the database publishes no port and joins a private network, deliberately. It needs to run somewhere that has both the client tools and a route to the database, with the client major version matching the server, which z4j restore checks.

Do not reach for raw pg_restore as a shortcut. The supported implementation clears the managed relations first and runs the restore single-transaction with --exit-on-error. A bare pg_restore --clean that fails partway leaves a half-cleared database that looks restored.

The image variable differs between the two shipped files. It is Z4J_BRAIN_IMAGE in docker-compose.postgres.yml and Z4J_IMAGE in docker-compose.yml. Setting the wrong one for the stack you deploy is a silent no-op that leaves you on the tag already in the file, so check with grep image: <your-compose-file> before pinning a previous version.

Both are written as ${VAR:-z4jdev/z4j:latest}, and neither variable is set by default, so editing the literal fallback in the YAML does change the image that resolves. An earlier version of this page said it changed nothing, which is wrong. Prefer the variable anyway, because it survives a git pull and the edited literal does not.

For the supported end-to-end procedure, including the client requirements and what restore verifies, see backup and restore. That page is the authority; this one lists the constraints that make a Docker rollback different from a pip one.

Major-version upgrades

A major upgrade is a documented manual operation, and the release notes for it carry the steps. Expect:

A required Z4J_* env var change (something deprecated in the current major is removed).
A breaking wire-protocol bump (agents and brain must be on the matching protocol version).
A schema change that is not backward-compatible with the older major's readers.

Treat a major upgrade as a coordinated stop, migrate, start operation, with the pre-upgrade backup in hand before you begin.

Upgrading agents

Agent packages (z4j-django, z4j-celery, etc.) version independently of the brain. Inside a major line, any patch / minor agent talks to any patch / minor brain. Patch a brain or an agent on its own; coordination between the two is only required across majors.

To bulk-upgrade every agent in your app's venv:

pip install --upgrade --no-cache-dir z4j-django z4j-celery z4j-celerybeat

Restart your web process and your worker after the upgrade. Agents reconnect to z4j automatically; no token rotation needed.