Troubleshooting

Diagnose stuck jobs, unresponsive workers, growing databases, latency, and missing tasks.

Common issues and how to fix them, organized by symptom.

Turn on debug logs first — they show the worker claiming, running, and settling jobs:

import { setLogLevel } from "@byteveda/flexiq";
setLogLevel("debug");
// or: FLEXIQ_LOG_LEVEL=debug node app.js

Turn on debug logs first — they show the worker claiming, running, and settling jobs:

java

FlexiQLogger.setLevel(LogLevel.DEBUG);
// or: FLEXIQ_LOG_LEVEL=debug java -jar app.jar

Turn on debug logs first — the Rust core's own logging is wired through RUST_LOG, independent of Python's logging module:

bash

RUST_LOG=flexiq_core=debug,flexiq_python=debug flexiq worker --app myapp:queue

A job stays pending and never runs

A few checks, in order:

No worker running — a worker process must actually be started and pointed at the same storage as the producer.
Queue mismatch — a worker only serves the queues it was started with (default "default"); a job enqueued to a different queue is invisible to it. Match them.
Queue paused — check for a paused queue and resume it.
Producer and worker use different storage — both must point at the same database (same SQLite file path, Postgres DSN, or Redis URL). A relative SQLite path resolves per working directory, so starting the two processes from different directories silently splits them onto two different files.

python

queue.list_paused_queues()
queue.resume_queue("default")

queue.listPausedQueues();
queue.resumeQueue("default");

java

flexiq.listPausedQueues();
flexiq.queue("default").resume();

Jobs stuck in `running`

Symptom: jobs stay in running status long after they should have finished.

Diagnosis:

python

stats = queue.stats()
print(stats)  # {'running': 47, 'pending': 0, ...}

stuck = queue.list_jobs(status="running", limit=20)
for job in stuck:
    d = job.to_dict()
    print(f"{d['id']} | {d['task_name']} | started {d['started_at']}")

const stats = await queue.stats();
console.log(stats); // { running: 47, pending: 0, ... }

const stuck = await queue.listJobs({ status: "running", limit: 20 });
for (const job of stuck) {
  console.log(`${job.id} | ${job.taskName} | started ${job.startedAt}`);
}

java

QueueStats stats = flexiq.stats();
System.out.println(stats.running + " running, " + stats.pending + " pending");

List<Job> stuck = flexiq.listJobs(
        JobFilter.builder().status(JobStatus.RUNNING).limit(20).build());
for (Job job : stuck) {
    System.out.println(job.id + " | " + job.taskName + " | started " + job.startedAt);
}

Cause: the worker process that claimed the job crashed or hung before marking it complete. Two independent recovery mechanisms exist, and neither depends on the other:

Timeout-based — if the task has a timeout configured, the stale reaper marks the job failed once the deadline passes (even if the worker process is still alive) and retries it, or dead-letters it once retries are exhausted.
Dead-worker recovery — if the worker that claimed the job crashed outright, any surviving scheduler notices the stale heartbeat and reclaims the job immediately — this doesn't need a timeout at all, it just needs the owning worker to actually be gone.

A job stuck because its worker is alive but hung — deadlocked, stuck in an infinite loop, blocked forever on I/O — has only the first mechanism to save it. Always set a timeout on production tasks:

python

@queue.task(timeout=300)  # 5 minutes max
def process_data(payload):
    ...

queue.task("processData", (payload) => processData(payload), { timeoutMs: 300_000 });

java

Task<Payload> PROCESS_DATA = Task.of("process_data", Payload.class)
        .timeout(Duration.ofMinutes(5));

timeout_ms is the internal, database-stored deadline in milliseconds; the timeout= kwarg above takes seconds and flexiq converts it under the hood.

A job without a timeout is only recovered automatically if its worker process actually dies. A worker that's alive but hung holds the job forever — nothing detects that case without a timeout.

For that last case — a job neither recovery mechanism will touch — force it back to pending manually:

python

queue.requeue_job(job_id)  # True if it was Running and is now Pending

requeue_job() releases the job's execution claim so a healthy worker can pick it up, and preserves the retry budget. Only use it when the owning worker is confirmed dead or hung: if the old attempt is actually still running, it may finish later and the job executes twice.

For that last case — a job neither recovery mechanism will touch — force it back to pending manually:

queue.requeueJob(jobId); // true if it was running and is now pending

requeueJob() releases the job's execution claim so a healthy worker can pick it up, and preserves the retry budget. Only use it when the owning worker is confirmed dead or hung: if the old attempt is actually still running, it may finish later and the job executes twice.

For that last case — a job neither recovery mechanism will touch — force it back to pending manually:

java

flexiq.requeueJob(jobId); // true if it was running and is now pending

Worker is unresponsive

Symptom: the worker process is alive but not processing jobs; its heartbeat is stale.

Diagnosis:

python

workers = queue.workers()
for w in workers:
    print(f"{w['worker_id']}: {w['status']} (last seen: {w['last_heartbeat']})")

const workers = await queue.listWorkers();
for (const w of workers) {
  console.log(`${w.workerId}: ${w.status} (last seen: ${w.lastHeartbeat})`);
}

java

List<WorkerInfo> workers = flexiq.listWorkers();
for (WorkerInfo w : workers) {
    System.out.println(w.workerId + ": " + w.status + " (last seen: " + w.lastHeartbeat + ")");
}

Possible causes, not tied to any one runtime:

Deadlock — a task is waiting on a resource held by another task in the same worker. Check for circular waits in your task code.
Infinite loop — a task loops without returning. A timeout is the only backstop; see Jobs stuck in running above.

GIL-bound CPU task: a long-running CPU task is holding the GIL (Global Interpreter Lock — only one thread runs Python bytecode at a time), blocking the scheduler thread from dispatching new jobs. The scheduler runs in Rust, but it still needs the GIL to call Python functions.

Fix: switch to the prefork pool for CPU-bound tasks — it runs tasks in separate subprocesses instead of threads, so one task's GIL usage can't block the scheduler.

bash

flexiq worker --app myapp:queue --pool prefork

Node handlers share one thread — the event loop. A long synchronous computation blocks every in-flight job, not just its own (see Execution Models). Offload CPU-bound work to a worker_threads pool (or a native addon) and await it, or set a timeoutMs so a stuck attempt is aborted — the timeout stops tracking the attempt, but a synchronous handler still won't yield the loop until it returns.

Handlers run on real JVM threads (see Execution Models), so one hung handler doesn't stall the others under the default cached pool — it only ties up its own thread. If you sized the pool with concurrency(n) though, enough hung handlers exhaust it and everything else queues up behind them.

close() drains in-flight handlers for 30 seconds, then interrupts and waits 30 more before closing the native worker — a stuck handler delays shutdown by up to a minute. Set a task timeout so a stuck attempt is reaped, and keep handlers interruptible.

Task not registered on the worker

Cause: the task name recorded on a job at enqueue time doesn't match a name the worker has registered. Task functions never travel through the queue — only their names and arguments do — so the worker's registry has to be populated independently.

Worker logs raise KeyError: "task 'myapp.tasks.process' not registered". Task names default to module.function_name; if you enqueue from one import path and run the worker with a different one, the names won't match.

python

# Check the task name stored in the job
job = queue.get_job(job_id)
print(job.to_dict()["task_name"])  # e.g. "myapp.tasks.process"

# Check what the worker has registered
print([t["name"] for t in queue.registered_tasks()])

Fix: use consistent import paths. If the task is myapp/tasks.py:process, always import it as myapp.tasks.process — not tasks.process (relative) or src.myapp.tasks.process (with an src prefix).

You can also set an explicit name to decouple the task name from the module path:

python

@queue.task(name="process-data")
def process(payload):
    ...

The worker dequeued a job whose task name was never registered on it — it throws TaskNotRegisteredError (No task registered with name "..."). Register the handler (queue.task(name, fn)) on the worker process.

The worker dequeued a job whose task name has no handler on it — the job fails with no handler registered for task '...'. Register the handler (handle(name, payloadType, fn)) on the worker process.

When only some workers are affected

If the same task succeeds on one worker and dead-letters on another, the registries differ rather than the task being unregistered everywhere. That is the shape a partial task discovery produces — a stale image, a replica that missed a deploy, or a task module left out of the built artifact. Compare what each worker actually registered before changing any task code, and see Executors disagree on their task registry below for the warning the scheduler logs when it can see the difference itself.

Executors disagree on their task registry

Symptom: the scheduler logs a warning at attach, and jobs for some task names sit unplaced or fail with was not dispatched.

text

[flexiq] executor python-executor-0198f3d2-4c71-7a3e-b8d5-1f2a9c4e7b60 (python 1.0.0) advertises task registry
eb5ed0d43c8f2aaa, but executor python-executor-0198f3d1-9e05-7c42-a1b7-6d3f08ce5142 (python 1.0.0) advertises
0d49390f67ebbff1; a job for a task only one of them knows fails wherever it lands.
only on python-executor-0198f3d2-4c71-7a3e-b8d5-1f2a9c4e7b60: (none). only on python-executor-0198f3d1-9e05-7c42-a1b7-6d3f08ce5142:
reports.build, reports.export

text

[flexiq] executor node-executor-0198f3d2-4c71-7a3e-b8d5-1f2a9c4e7b60 (node 1.0.0) advertises task registry
eb5ed0d43c8f2aaa, but executor node-executor-0198f3d1-9e05-7c42-a1b7-6d3f08ce5142 (node 1.0.0) advertises
0d49390f67ebbff1; a job for a task only one of them knows fails wherever it lands.
only on node-executor-0198f3d2-4c71-7a3e-b8d5-1f2a9c4e7b60: (none). only on node-executor-0198f3d1-9e05-7c42-a1b7-6d3f08ce5142:
reports.build, reports.export

text

[flexiq] executor java-executor-0198f3d2-4c71-7a3e-b8d5-1f2a9c4e7b60 (java 1.0.0) advertises task registry
eb5ed0d43c8f2aaa, but executor java-executor-0198f3d1-9e05-7c42-a1b7-6d3f08ce5142 (java 1.0.0) advertises
0d49390f67ebbff1; a job for a task only one of them knows fails wherever it lands.
only on java-executor-0198f3d2-4c71-7a3e-b8d5-1f2a9c4e7b60: (none). only on java-executor-0198f3d1-9e05-7c42-a1b7-6d3f08ce5142:
reports.build, reports.export

Cause: two attached executors advertised different task lists. The scheduler routes a task name only to peers that advertised it, so a job is never handed to one that cannot run it — instead a name only part of the fleet knows depends on that part staying attached with a free slot. When none is available the job waits, and past the placement timeout it fails retryably with task '...' was not dispatched: no attached executor advertises it. (none) on one side means that registry is a strict subset of the other's — the shape a half-finished discovery produces.

Fix: read the two only on lists — they name the tasks at risk. If the peers are meant to be interchangeable, compare the two deployments: a rollout in progress resolves itself, a stale replica or a task module missing from the image does not. The unplaced jobs retry, so they drain on their own once a peer advertising those names attaches; check the dead-letter queue for any that exhausted their retries first. The warning is a diagnostic, never a gate — a divergent executor still attaches and still receives work.

The warning fires once per distinct registry, not once per executor, so rolling a fleet onto a new task list logs a single line. See Task discovery for the full rule and the fingerprint it compares.

`SQLite database is locked`

SQLite allows many concurrent readers (WAL mode — write-ahead logging) but only one writer at a time. Under many concurrent workers writing to the same file, this surfaces as lock contention.

code

OperationalError: database is locked

flexiq sets busy_timeout=5000ms to wait for locks, but heavy write loads can still cause contention.

Causes:

Multiple processes writing to the same SQLite file simultaneously
Very large batch inserts blocking the writer
Long-running transactions from external tools (e.g. a DB browser) holding locks

Fixes:

Avoid opening the database file with other tools while the worker is running
Use enqueue_many() / task.map() for batch inserts — they use a single transaction
If running multiple processes, ensure only one worker process writes at a time, or move to Postgres for multi-machine deployments

Keep workers on one machine and moderate concurrency, or
Move to Postgres or Redis for multi-process / multi-machine deployments

Keep workers on one machine and moderate concurrency, or
Move to Postgres or Redis for multi-process / multi-machine deployments

Database growing too large

Symptom: the SQLite file keeps growing, or the job tables never shrink on Postgres/Redis. Completed job records and their result payloads accumulate unless something purges them.

Fix: purge old records.

python

# Purge completed jobs older than 7 days (older_than is in seconds)
queue.purge_completed(older_than=604800)

# Purge dead-lettered jobs older than 30 days
queue.purge_dead(older_than=2592000)

// Purge completed jobs older than 7 days (olderThanMs is in milliseconds)
await queue.purgeCompleted(7 * 24 * 60 * 60 * 1000);

// Purge dead-lettered jobs older than 30 days
await queue.purgeDead(30 * 24 * 60 * 60 * 1000);

java

// Purge completed jobs older than 7 days (olderThanMs is in milliseconds)
flexiq.purgeCompleted(Duration.ofDays(7).toMillis());

// Purge dead-lettered jobs older than 30 days
flexiq.purgeDead(Duration.ofDays(30).toMillis());

Set result_ttl on the Queue to auto-purge going forward instead of purging manually every time:

python

queue = Queue(
    db_path="myapp.db",
    result_ttl=86400,  # Purge completed/dead jobs older than 24 hours
)

There's no built-in TTL option — run the purge above on a schedule instead (a cron job, or a periodic task on the queue itself).

After purging, reclaim disk space (SQLite only — Postgres reclaims space via autovacuum, Redis via key expiry):

bash

sqlite3 myapp.db "VACUUM;"

VACUUM rewrites the entire database and requires exclusive access. Run it during low-traffic periods.

High job latency

Symptom: jobs sit in pending for longer than expected before starting.

Diagnosis: check the queue depth first.

python

stats = queue.stats()
print(f"Pending: {stats['pending']}, Running: {stats['running']}")

const stats = await queue.stats();
console.log(`Pending: ${stats.pending}, Running: ${stats.running}`);

java

QueueStats stats = flexiq.stats();
System.out.println("Pending: " + stats.pending + ", Running: " + stats.running);

Possible causes and fixes:

Scheduler poll interval too high: default is 50ms. Jobs can wait up to one poll interval before being picked up.
python
```
queue = Queue(scheduler_poll_interval_ms=10)  # Poll every 10ms
```
Lower values increase CPU/DB usage. Balance based on your latency requirements.
Not enough worker threads: all workers are busy. Increase the pool size.
python
```
queue = Queue(workers=16)
```

Rate limiting: the task or queue has a rate limit active — rate-limited jobs are rescheduled 1 second into the future.

python

pending = queue.list_jobs(status="pending", limit=10)
for job in pending:
    print(job.to_dict()["scheduled_at"])

Database performance: slow dequeue queries. Check SQLite WAL size or Postgres query plans.

Possible causes and fixes:

Batch size too small: raise batchSize and channelCapacity on runWorker to claim and buffer more per poll — see the execution model.
ts
```
queue.runWorker({ queues: ["default"], channelCapacity: 256, batchSize: 16 });
```
Not enough worker processes: add more worker processes against shared storage.
Capped by a limit: check you aren't capped by a per-task maxConcurrent or rate limit.
Database performance: slow dequeue queries. Check SQLite WAL size or Postgres query plans.

Possible causes and fixes:

Not enough concurrency: raise .concurrency(n), or let .autoscale(AutoscaleOptions.of(min, max)) grow the pool with queue depth — see Execution Models.
Batch size too small: raise .batchSize(n) and .channelCapacity(n) on the worker builder — see Batching.
Not enough worker processes: add more worker processes against shared storage.
A producer-side gate is deferring the enqueue: the Java SDK throttles on the producer side — a queue.gate(...) (including the built-in Recipes) can defer a job's creation into the future, which looks like latency but is actually delayed enqueue. Check for an active gate on the task; see Rate Limiting.
Database performance: slow dequeue queries. Check SQLite WAL size or Postgres query plans.

Memory usage growing

Symptom: worker process memory climbs over time.

Causes:

Large result payloads: task return values are stored in the database but also held in the scheduler's result buffer briefly. If tasks return large objects (images, dataframes), memory spikes.

Fix: return a reference (file path, object key) instead of the data itself.

python

# Bad — large result stored in memory and DB
@queue.task()
def process_image(path: str) -> bytes:
    return open(path, "rb").read()

# Good — return a path
@queue.task()
def process_image(path: str) -> str:
    out = path + ".processed"
    # ... write output to out ...
    return out

Accumulated job records: without result_ttl, the database grows unbounded. See Database growing too large above.
Resource leaks in tasks: a task opens a file or connection and never closes it. Use context managers.

Causes:

Large result payloads: return values are serialized and written to storage on every job. Returning large buffers or objects (images, parsed documents) repeatedly pressures the heap — return a reference (file path, object key) instead of the data itself.
Accumulated job records: without periodic purging, storage keeps growing. See Database growing too large above.
Resource leaks in handlers: a handler opens a file, socket, or worker_threads worker and never closes it. Each leaked handle accumulates for the life of the process — always release what you open.

Causes:

Large result payloads: return values are serialized and written to storage on every job. Returning large objects repeatedly pressures the JVM heap — return a reference (file path, object key) instead of the data itself.
Accumulated job records: without periodic purging, storage keeps growing. See Database growing too large above.
Resource leaks in handlers: a handler opens a connection or file directly instead of through a worker-scoped resource and never closes it. Prefer scoped resources — they're closed for you when the worker shuts down.

Periodic task running twice

Symptom: a periodic task fires more than once per interval, or appears to run on two workers simultaneously.

Behavior: this is safe by design. Each worker's scheduler runs its own maintenance loop and checks for due periodic tasks independently — when one is due, every scheduler that notices tries to enqueue it, but a per-tick dedup key ensures only one enqueue actually creates a job. The rest resolve to that same job's existing id instead of inserting a duplicate.

If you see two completed jobs for the same periodic task in the same interval, check for duplicates:

python

jobs = queue.list_jobs(status="complete", task_name="daily_report", limit=50)
for j in jobs:
    print(j.to_dict()["completed_at"])

const jobs = await queue.listJobs({ status: "complete", task: "dailyReport", limit: 50 });
for (const job of jobs) {
  console.log(job.completedAt);
}

java

List<Job> jobs = flexiq.listJobs(
        JobFilter.builder().status(JobStatus.COMPLETE).task("daily_report").limit(50).build());
for (Job job : jobs) {
    System.out.println(job.completedAt);
}

If you're genuinely seeing duplicate execution, ensure all workers use the same storage (same SQLite file path, same Postgres DSN, or same Redis URL).

Payload / codec errors

A named codec must be registered under the same name on both the producer and the worker — a task opts into it by name, and if the worker's registry doesn't have that name, the job fails.

python

from flexiq import Queue
from flexiq.codecs import GzipCodec

# Both the producer and the worker construct their Queue with the same
# codec registered under the same name.
queue = Queue(db_path="tasks.db", codecs={"gz": GzipCodec()})

@queue.task(codecs=["gz"])
def process_large_payload(data: bytes):
    ...

import { Queue, GzipCodec } from "@byteveda/flexiq";

// Both the producer and the worker construct their Queue with the same
// codec registered under the same name.
const queue = new Queue({ dbPath: "tasks.db", codecs: { gz: new GzipCodec() } });

queue.task("processLargePayload", (data: Buffer) => handle(data), { codecs: ["gz"] });

java

import org.byteveda.flexiq.serialization.GzipCodec;

// Both the producer and the worker build their FlexiQ client with the
// same codec registered under the same name.
FlexiQ flexiq = FlexiQ.builder()
        .sqlite("tasks.db")
        .codec("gz", new GzipCodec())
        .open();

Task<byte[]> PROCESS_LARGE_PAYLOAD = Task.of("process_large_payload", byte[].class)
        .codecs("gz");

Serializer mismatch — producer and worker must use the same serializer too, not just the same codecs.
Exact error: SerializationError: no codec registered named 'gz' (at dispatch time) or ValueError: no codec registered named 'gz' (at enqueue time).

Serializer mismatch — producers and workers must use the same serializer, not just the same codecs.
Exact error: SerializationError: no codec registered named "gz".

Serializer mismatch — producers and workers must use the same serializer and codec chain.
no codec registered named '...' — the task was enqueued with named codecs; register the same names with FlexiQ.builder().codec(name, codec) on the worker.
Generic payload types — a non-Jackson serializer only supports plain Class payloads; use JsonSerializer/MsgpackSerializer (or a non-generic payload type) for TypeReference payloads.

Native addon or library fails to load

code

ModuleNotFoundError: No module named 'flexiq._flexiq'

The compiled Rust extension isn't present for this platform or Python version. Reinstall from a wheel that matches both (pip install --force-reinstall flexiq), or rebuild it locally with uv run maturin develop if you're working from source.

code

Cannot find module '.../native/index.js'

The native addon isn't built. Run pnpm build:native. Workflows and mesh also require the addon to be built with their cargo features (included in build:native) — a "feature not enabled" error at queue.workflows means the addon was built without them.

code

UnsatisfiedLinkError: no bundled native library for platform '...'

The jar has no native binary for this OS/architecture. Build the native library for the platform and point -Dflexiq.native.lib=/path/to/library at it. On hardened hosts where /tmp is mounted noexec, extraction succeeds but loading fails — set -Dflexiq.native.workdir=/path/on/exec/volume instead.

Inspecting failures further

Inspect failures with queue.job_errors(id) and queue.dead_letters(); see job management.

Inspect failures with await queue.getJobErrors(id) and await queue.deadLetters(); see job management.

Inspect failures with flexiq.jobErrors(id) and flexiq.listDead(50, 0); see job management.