Upgrading to 0.15

1. Serializer default is now SmartSerializer

What changed. The default serializer is now SmartSerializer, which uses msgpack for plain-data payloads (dicts, lists, strings, ints, floats, bools, None, and tuples) and falls back to cloudpickle for anything else (lambdas, closures, custom classes, generators, namedtuples). Payloads that do not require cloudpickle are smaller and faster to serialize and deserialize. msgpack is now a base dependency — no extra install required.

Breaking — rolling upgrade required. 0.15 workers prefix every payload with a 1-byte codec tag. Workers running pre-0.15 do not know about this tag and will fail to deserialize payloads written by a 0.15 worker. During a mixed-version deploy window you must take one of the following approaches:

• Recommended: stop all workers, upgrade everything at once, restart.
• If rolling deployment is required: pin the old serializer explicitly on all queues until every worker has been upgraded to 0.15, then remove the override.

from flexiq import Queue
from flexiq.serializers import CloudpickleSerializer

# Pin during the rolling-upgrade window.
# Remove this line once every worker is on 0.15.
queue = Queue(
    db_path="myapp.db",
    serializer=CloudpickleSerializer(),
)

Pre-0.15 payloads (untagged cloudpickle) are read transparently by 0.15 workers — backward compatibility for reading old data is fully maintained.

2. Per-task / per-queue stats are computed server-side

What changed. On the Redis backend, per-task and per-queue statistics are now computed on the server side using SINTERCARD (a Redis 7.0+ command that counts set intersection size without transferring the members), replacing the previous O(N) full-scan approach. SQLite and Postgres already computed these with a server-side SELECT COUNT(...) and are unaffected. This is a pure performance improvement.

Action required: none. No data migration, no API change, no configuration change.

3. Batch dequeue — scheduler_batch_size

What changed. A new optional scheduler_batch_size kwarg on Queue() controls how many jobs the scheduler claims per round-trip. Left unset, it defaults to 1 on SQLite/Postgres (preserving pre-0.15 behaviour exactly) and 8 on Redis.

# Higher throughput under sustained load:
queue = Queue(
    db_path="myapp.db",
    scheduler_batch_size=20,
)

Set scheduler_batch_size above 1 when your queue regularly accumulates a backlog and per-job overhead is low. Under light load there is no meaningful difference.

Breaking: no.

4. Opt-in push-based dispatch — push_dispatch

What changed. Setting push_dispatch=True on Queue() switches from polling to event-driven wakeups. This eliminates the dispatch latency floor and idle database load under quiet periods.

queue = Queue(
    db_path="myapp.db",
    push_dispatch=True,
)

Ships in the published wheel; on by default for Redis. No custom build is needed. A Redis-backed Queue gets event-driven dispatch automatically; pass push_dispatch=False to opt back out to polling. SQLite and Postgres keep polling unless you pass push_dispatch=True.

Backend support:

Polling remains the default on SQLite/Postgres, and the safety-net fallback everywhere. There is no correctness difference between the two modes: a missed wake is only ever caught by a 2s fallback timer rather than an immediate one, never lost.

Breaking: no.

5. Terminal jobs are archived immediately

What changed. Completed, failed, dead-lettered, and cancelled jobs are now moved to the archived_jobs table the moment they reach a terminal state, rather than being cleaned up by the periodic maintenance pass. This keeps the hot jobs table bounded regardless of job throughput.

queue.stats(), queue.get_job(), and queue.list_jobs() transparently include archived jobs — there is no API change.

On first startup, existing terminal rows in jobs are drained into archived_jobs automatically. This migration runs as part of run_migrations and may take a few seconds on large databases.

6. Job payloads side table (job_payloads) — superseded

Superseded: 0.15 briefly dual-wrote payload BLOBs to a separate job_payloads table; a later release removed it and moved payloads back inline on jobs/archived_jobs (no data moved, no upgrade window, no action needed — this is kept only as history). Redis was never affected.

Upgrade checklist

• Serializer: upgrade all workers together before relying on the new format. If rolling deployment is required, pin CloudpickleSerializer() until all workers are on 0.15, then remove the override.
• Idempotent tasks: drain idempotent work before upgrading, or accept at-least-once delivery during the upgrade window.
• Batch dequeue: decide on a scheduler_batch_size for your workload. Unset is backend-chosen (1 on SQLite/Postgres, 8 on Redis) and safe; raise it if you regularly see a backlog.
• Push dispatch: on by default for Redis; pass push_dispatch=True to opt in on SQLite/Postgres too, or push_dispatch=False to opt out on Redis. Not required for correctness.
• Downgrade floor: after upgrading any database to 0.15, do not run pre-0.15 binaries against it.