Reconciliation

The stuck-task problem

If a worker crashes mid-task, the engine may emit task_started but never task_success or task_failure. Without reconciliation, the dashboard shows "running" forever.

The reconciliation worker

Runs in z4j every Z4J_RECONCILIATION_SWEEP_SECONDS (default 5 minutes), under a leader lock so only one brain process sweeps per interval. Each pass:

1. Find tasks in state started, pending or retry whose age anchor (started_at, else received_at, else created_at) is older than Z4J_RECONCILIATION_STALE_THRESHOLD_SECONDS (default 15 minutes), oldest first, at most 100 per sweep.
2. Group them by project and issue a reconcile_task command for each to an online agent of that project that advertises the task's engine; a task with no such agent waits for the next pass. It is an ordinary command row, audited under the action command.issue.reconcile_task, carrying an idempotency key so two sweeps inside one window collapse onto a single probe.
3. The agent asks its engine: Celery AsyncResult.status, RQ job.get_status(), and so on, and answers with one canonical state: pending, started, success, failure, or unknown.
4. unknown is a no-op. It means the adapter has no result backend to consult, which is not evidence that the task is gone, so the brain changes nothing and the task is probed again on a later sweep.
5. A terminal answer (success or failure) is applied, and the brain records the outcome the missing event would have carried. Applying it fires the task.orphaned automation trigger once for that correction.
6. A non-terminal answer is applied only when the row has not been written since the probe was issued. If it has, the brain has fresher information than the probe saw and the response is dropped. A task already in a terminal state is never moved back out of it.

Why not rely on the engine alone

Each engine reports differently:

• Celery - AsyncResult is Redis-TTL-bound; after expiry it returns PENDING regardless.
• RQ - cleanly reports lost, but requires periodic cleanup for stale jobs.
• arq - no introspection API for historical jobs; relies on application-level result store.

The reconciliation worker owns the "was it really lost?" question so no adapter has to.

Tunables

Setting	Default	Meaning
Z4J_RECONCILIATION_SWEEP_SECONDS	300	Seconds between reconciliation passes
Z4J_RECONCILIATION_STALE_THRESHOLD_SECONDS	900	How old a non-terminal task must be before the worker queries the agent

Audit impact

Each probe writes an audit log entry under the action command.issue.reconcile_task, naming the task it asked about. An applied correction writes a task.reconciled audit row, and is visible as the task's state change and, where an automation rule is armed, as a task.orphaned firing.

Limits

• Reconciliation does not retry anything. It corrects the brain's snapshot to whatever the engine says is true. Retrying is a separate operator decision, or an automation rule armed on task.orphaned.
• If the agent is offline, reconciliation skips that agent and retries next pass.
• No retroactive reconciliation across deploys - only tasks still present in tasks + events are examined.