Upgrades

Semver promise

• Major -- breaking wire-protocol changes. Plan a coordinated upgrade.
• Minor -- additive features, fully backward-compatible within the major.
• Patch -- bug fixes, no behavior changes.

See versioning for the full policy.

Upgrade order

1. Upgrade z4j (the brain) first. Newer brains accept older agents within the same major.
2. Upgrade agents on a schedule. Rolling restarts are fine; agents auto-reconnect.
3. Watch the dashboard for the per-agent "update available" badge -- the brain renders it when an agent's last advertised protocol_version is older than the brain's current protocol.

The brain and the scheduler move together

The standalone scheduler and the brain compute fire times from the same pinned set of libraries and timezone data, and each proves it to the other with a fingerprint. A release that moves that set says so in its changelog. Across such a release the two do not interoperate: the scheduler's watch stream is refused, its /ready answers 503, a scheduler started against the other release exits, and no schedule fires until both run the same release. Nothing is recorded against the schedules while they wait, and the occurrences missed in that window follow each schedule's catch-up policy once the pair agrees.

Upgrade them in this order:

1. Stop the scheduler.
2. Upgrade the brain and start it.
3. Upgrade the scheduler and start it.

Stopping the scheduler first also closes a narrow window on every brain restart: a fire handed to a brain in the instant it stops may never be delivered, and the delivery timeout then holds that schedule until an operator resolves it. With the scheduler already stopped there is no such fire.

With the packaged Compose files, docker compose down stops the schedulers before the brain, so docker compose pull, docker compose down, docker compose up -d follows this order. With the Helm chart, set scheduler.replicaCount to 0, upgrade the release, and set it back.

Database migrations

Alembic migrations run automatically on brain start when Z4J_AUTO_MIGRATE=true (the default).

On a multi-replica deployment there is one supported order, and it is not a rolling one. Stop every replica of the old release, run the migration as a one-shot job, then start replicas of the new release. Stop, not drain: draining removes a replica from HTTP load balancing while its background schedule writers keep running, and it is those writers, not request traffic, that produce the incompatible envelopes. Leaving old replicas running across a migration that adds a column the schedule-control path reads costs executions, not just latency; the reasons are below under Brain replicas have to cross a schema change together.

Set Z4J_AUTO_MIGRATE=false on the runtime replicas so only the one-shot job migrates:

The image's entrypoint is already the z4j binary, so whatever you pass after the image tag starts at the SUBCOMMAND. Repeating the binary name there makes the container invoke it twice and fail; begin with migrate. The job also needs the same /data volume and Z4J_SECRET the running brain uses, or management bootstrap refuses to start.

Run it through Compose, not a bare docker run. The shipped files put the brain's state on a named volume (z4j_data for SQLite, z4j_brain_state for PostgreSQL) and the database on a private network (z4j_net) that is not published, and the brain needs Z4J_SESSION_SECRET as well as Z4J_SECRET. compose run inherits the volume, the network and the whole environment; a bare docker run gets a fresh empty volume and cannot resolve the database host. Add -f docker-compose.postgres.yml if that is the file you deploy.

# Point at the NEW image and pull it FIRST. `compose run` uses whatever image
# the file resolves to, which is still the version you are upgrading FROM
# until you change it -- so without this the one-shot job runs the OLD brain,
# reports "upgrade head" success at the old head, and applies nothing. The
# migration then happens during new-replica startup, or the new brain refuses
# because you just turned auto-migration off.
#
# The compose file and the image variable are a PAIR: each file reads only its
# own variable. Setting one while running the other is a silent no-op, because
# compose keeps the tag already in the file and the migration then reports
# success at the old head. Run the block for the stack you deploy, whole.

# --- SQLite stack: docker-compose.yml, which reads Z4J_IMAGE ---
echo 'Z4J_IMAGE=z4jdev/z4j:<new-version>' >> .env
docker compose pull z4j
docker compose run --rm -e Z4J_AUTO_MIGRATE=false z4j migrate upgrade head
docker compose run --rm z4j migrate current   # confirm it moved

# --- PostgreSQL stack: docker-compose.postgres.yml, which reads Z4J_BRAIN_IMAGE ---
echo 'Z4J_BRAIN_IMAGE=z4jdev/z4j:<new-version>' >> .env
docker compose -f docker-compose.postgres.yml pull z4j
docker compose -f docker-compose.postgres.yml run --rm \
  -e Z4J_AUTO_MIGRATE=false z4j migrate upgrade head
docker compose -f docker-compose.postgres.yml run --rm z4j migrate current

grep image: <your-compose-file> confirms which variable your file reads if you have customised it.

Then start the brain replicas on the new image.

Brain replicas have to cross a schema change together

A multi-replica brain runs mixed versions for as long as the rollout takes, and for most migrations that is fine. It is not fine for a migration that adds a column the schedule-control path reads, because the two versions then disagree about what a schedule row means:

• An older replica does not know the column exists, so it writes change-log envelopes without it. A newer brain refuses to project those envelopes and ends the scheduler's watch stream rather than guess. The scheduler recovers on its own, by reconnecting and re-syncing from live rows, but it refuses to fire while it does, so the window shows up as schedules running late.
• Worse in the other direction: an older replica cannot honour a hold it has no column for. A schedule paused through a newer replica keeps firing through an older one.

This is not merely slower scheduling. Each refused envelope ends the watch, the scheduler will not dispatch while it is reconnecting, a fire more than the grace window late is classified as missed, and a schedule whose catch-up policy is skip (the default for imported and declarative definitions) records the slot and moves past it. A schedule that came due inside one of those windows does not run late. It does not run. A schedule on fire_one_missed, the default for one created from the dashboard or the API, runs its latest missed occurrence once the watch recovers, and any earlier missed occurrence is discarded the same way.

So this is a requirement, not a recommendation:

1. Stop every brain replica of the older release. Not "most", not "roll them quickly": a single old writer is enough to produce the envelopes.
2. Run the migration.
3. Start replicas of the new release only.

Schedulers can stay up throughout. They fail closed, reconnect and re-sync on their own once every brain is on one side of the change.

Scaling to a single replica first does NOT avoid the stop. It narrows the window to one process, which is worth doing, but that replica is still an old writer until it exits, so its own restart has to be a full stop and start rather than a rolling replace. In Kubernetes terms: strategy: Recreate, or maxSurge: 0 with maxUnavailable: 100%. There is no configuration that keeps a brain of the previous release serving while the migration runs.

This upgrade takes brief downtime on a multi-replica deployment. Plan for it rather than looking for a way around it.

The release notes say when a given upgrade adds such a column.

None of this applies when only ONE brain process is alive across the migration. That is a statement about overlapping processes, not about replicas: 1: Kubernetes defaults to RollingUpdate and rounds maxSurge UP, so a single-replica Deployment starts the new pod before stopping the old one and produces the window anyway. Set strategy: { type: Recreate }. The same caution applies to any supervisor that starts a replacement before the previous process exits.

Rolling back

An installation that upgraded and then merely ran can be brought back down. Every schedule owned by the z4j scheduler carries a fingerprint of the cadence runtime that wrote it, and nothing on the running path rewrites it: not the fire path, not the cursor path. The only write sites are creating a schedule and cutting one over to a new owner. So on an installation that upgraded and then ran, the rows still carry the identity the older release wrote, and handing them back restores exactly the state that release last saw. Nothing is rewritten, so no image, registry or signature is involved.

The downgrade makes you declare what the release you are returning to computes. Measure it in the target environment, under that release:

python -c 'from z4j_brain.domain.schedule_cadence import cadence_runtime_fingerprint as f; print(f())'

Then stop everything that writes, run z4j migrate downgrade with Z4J_ROLLBACK_TARGET_FINGERPRINT set to that digest, and install the older release afterwards. The schema has to come down first, because the older binary refuses to start against the newer one. Upgrade and rollback carries the sequence step by step.

It refuses in three cases, and every guard is evaluated over the whole downgrade plan before its first step runs, so nothing is dropped and the database is left as it was. It refuses when a schedule was created or re-saved since the upgrade, because those rows carry an identity the target did not write. It refuses when a schedule is paused, and resuming paused schedules does not clear that, it just starts them firing again, which is what the guard is there to prevent, so do not release incident holds trying to get past it. And it refuses when an agent is revoked. Above that step, three more refusals depend on your rows or your host: a channel config that decrypts under no listed secret, a membership still holding the auditor role, and an SQLite library that cannot drop a column. They are evaluated at the same point, before the first step, and upgrade and rollback names what clears each.

Declaring a fingerprint proves less than it looks like. The value has to equal what the rows already carry, so it cannot make a bad downgrade succeed, but nothing ties it to a running target: paste it out of your own database and the check passes without telling you anything. If the release you install computes something different from what the rows carry, it disables every schedule it cannot agree with, durably, until an operator re-enables each one by hand. Measure the target.

When it refuses, restore your pre-upgrade backup under the release you are returning to. z4j restore is forward-only, so it runs under the older binary against a target that release provisioned and brought to its own head. See upgrade and rollback for the full sequence and backup and restore for what the target must satisfy.

Everything written since the upgrade is lost on that path. Take the pre-upgrade backup every time, because whether the downgrade is available to you depends on what happens after the upgrade, not on anything you can check before it.

Below the boundary activations there is no downgrade at any point in the release history either; those revisions refuse unconditionally. See database migrations for which revisions refuse and why.

Version skew tolerance

Brain	Agent	Behavior
same major, same minor	same major, same minor	Full feature parity.
same major, newer	same major, one minor older	Works. New brain-side features may be unavailable on older agents; the dashboard shows the "update available" badge.
same major, newer	same major, more than one minor older	Works but flagged loudly. Plan the agent rollout.
any	advertises a wire protocol_version the brain does not support	Rejected at the handshake with WebSocket close code 4426. Package-version skew outside the contract above, a major mismatch included, is logged by the brain as a warning, not rejected; close code 4427 is reserved for it and is not sent.

Rolling-upgrade caveats

The primary WebSocket transport is protocol-compatible across a same-major minor gap, so the steady-state event, heartbeat, and command data plane keeps flowing in both directions while a fleet is mid-upgrade. Two paths need operator awareness during the mixed-version window:

• Long-poll fallback (corporate-proxy path only). The long-poll HMAC binds the agent and project UUIDs the brain advertises on the probe response (X-Z4J-Agent-Id / X-Z4J-Project-Id, the long-poll analogue of the WebSocket hello_ack), and the agent builds its frame signer from those values on connect. A brain that sends the headers therefore pairs with the agent across the mixed-version window. If the brain sends no identity headers, the agent falls back to Z4J_AGENT_ID and Z4J_PROJECT_ID, which then have to be the real UUIDs shown on the brain's agents page (the configured project id is normally a slug, and frames signed under a slug never verify); the agent logs that instruction. WebSocket deployments are unaffected.
• Queue purges. The purge_queue confirm-token is a keyed HMAC. A purge issued across a version boundary is refused (it fails safe -- it never purges the wrong queue). Do not issue queue purges until the whole fleet is on the new release. If you must purge against a not-yet-upgraded brain, set Z4J_ACCEPT_LEGACY_PURGE_TOKEN=1 on the agents for the grace window only.

Reading release notes

Every release ships a CHANGELOG entry at /reference/changelog. Breaking changes are flagged explicitly.

Upgrading Postgres

z4j supports Postgres 17 and 18. Use pg_upgrade or logical replication; z4j is agnostic to which migration path you choose.