Upgrade and rollback

z4j ships patch releases regularly. Patch upgrades are designed to be in-place and reversible.

Pre-flight

Before any upgrade:

z4j doctor      # config valid, DB at head, no warnings you don't expect
z4j status      # snapshot row counts so you can verify after restart
z4j backup --output /var/backups/z4j-pre-upgrade-$(date +%Y-%m-%d-%H%M).db

The backup is the most important step. Without it, an upgrade that introduces an unexpected migration is hard to recover from.

Pip upgrade

# 1. Stop the brain
sudo systemctl stop z4j

# 2. Upgrade the wheel (no-cache so we always get the published version)
sudo -u z4j /srv/venv/bin/pip install --no-cache-dir --upgrade z4j

# 3. Confirm version and that migrations are in place
sudo -u z4j /srv/venv/bin/z4j version
sudo -u z4j /srv/venv/bin/z4j migrate current     # pre-restart sanity check

# 4. Restart - Alembic upgrades to head automatically on serve
sudo systemctl start z4j

# 5. Verify
journalctl -u z4j -n 30 --no-pager     # look for the boot banner
sudo -u z4j /srv/venv/bin/z4j check
sudo -u z4j /srv/venv/bin/z4j status   # row counts should match step-1 snapshot

z4j runs alembic upgrade head on every serve start unless you set Z4J_AUTO_MIGRATE=false. Every revision ships a working downgrade() and a CI roundtrip test enforces it, so the schema is bidirectional. Data is preserved via backup-restore, not via downgrade. See database migrations for the full policy.

Troubleshooting: `Unable to locate executable '/srv/venv/bin/z4j'`

If systemd reports this after an upgrade:

z4j.service: Unable to locate executable '/srv/venv/bin/z4j': No such file or directory
z4j.service: Failed at step EXEC spawning /srv/venv/bin/z4j: No such file or directory
z4j.service: Main process exited, code=exited, status=203/EXEC

The z4j console script is missing from the venv even though pip list shows z4j is installed. The cause is a venv where the z4j dist-info metadata exists but the wheel content was never actually unpacked, so pip install --upgrade z4j short-circuits with “Requirement already satisfied” and never drops the binary into bin/. Common triggers: a venv built across a package-rename cut, manual cleanup that deleted files but left dist-info, or a previous install was interrupted.

The fix is a force-reinstall that ignores the metadata check:

sudo -u z4j /srv/venv/bin/pip install --force-reinstall --no-deps z4j
ls -la /srv/venv/bin/z4j   # should now exist, mode 0755
sudo systemctl restart z4j

--no-deps keeps it fast since dependencies are not the issue; only z4j’s own wheel needs to be replanted.

Docker upgrade

# 1. Pull the new tag (pin a specific version in production)
docker compose pull z4j

# 2. Restart - migrations run on container start
docker compose up -d z4j

# 3. Verify
docker compose logs --tail=50 z4j
docker compose exec z4j z4j check
docker compose exec z4j z4j status

For Postgres deployments, the z4j image and the Postgres image are independent; upgrading z4j does not touch Postgres data.

Pinning vs `:latest`

Tag	When to use
`z4jdev/z4j:<X.Y.Z>`	Production — reproducible deploys, controlled upgrade cadence.
`z4jdev/z4j:<X.Y>`	Float on the latest patch within one minor line.
`z4jdev/z4j:latest`	Homelab / small teams that track current stable. Pair with a `digest`-pin if you need reproducibility within “latest”.

Each minor line guarantees bidirectional schema migrations: pip install <newer> and pip install <older> both reach a clean schema state inside the same minor. Row data is preserved via backup-restore, not via downgrade. Across major versions, expect a documented manual step.

Rollback - pip

If z4j check or z4j status shows something wrong after upgrade:

# 1. Stop the brain
sudo systemctl stop z4j

# 2. Reinstall the previous version
sudo -u z4j /srv/venv/bin/pip install --no-cache-dir --force-reinstall z4j==<previous-version>

# 3. If migrations changed schema, restore from the pre-upgrade backup
sudo -u z4j /srv/venv/bin/z4j restore /var/backups/z4j-pre-upgrade-<timestamp>.db --force

# 4. Restart
sudo systemctl start z4j

# 5. Verify
sudo -u z4j /srv/venv/bin/z4j check
sudo -u z4j /srv/venv/bin/z4j status

If you skipped the backup and the migration introduced unrecoverable data loss, the options are:

Roll forward (fix the bug in the new version, ship a patch).
Restore from the most recent off-host backup (snapshot from before the upgrade).

Rollback - Docker

# 1. Pin to the previous tag in compose.yml
sed -i 's|z4jdev/z4j:<current>|z4jdev/z4j:<previous>|' docker-compose.yml

# 2. Re-create the container with the older image
docker compose up -d --force-recreate z4j

# 3. If schema changed, restore
docker compose exec z4j z4j restore /backups/z4j-pre-upgrade-<timestamp>.dump --force

# 4. Verify
docker compose logs --tail=50 z4j

For Docker Postgres deployments, restore to the live Postgres instance via pg_restore running inside the z4j container; the connection string is already set.

Major-version upgrades

When the next major releases, the release notes will document the manual steps. Expect:

A required Z4J_* env var change (something deprecated in the current major is removed).
A breaking wire-protocol bump (agents and brain must be on the matching protocol version).
A schema change that is not backward-compatible with the older major’s readers.

Expect a documented upgrade procedure alongside the release notes; until then, treat major upgrades as a coordinated stop-migrate-start operation with a tested rollback path.

Upgrading agents

Agent packages (z4j-django, z4j-celery, etc.) version independently of the brain. Inside a major line, any patch / minor agent talks to any patch / minor brain. Patch a brain or an agent on its own; coordination between the two is only required across majors.

To bulk-upgrade every agent in your app’s venv:

pip install --upgrade --no-cache-dir z4j-django z4j-celery z4j-celerybeat

Restart your web process and your worker after the upgrade. Agents reconnect to z4j automatically; no token rotation needed.