Skip to content

feat: add database-backed job coordinator#325

Merged
romeokienzler merged 2 commits into
mainfrom
feature/db-job-coordinator
Jul 7, 2026
Merged

feat: add database-backed job coordinator#325
romeokienzler merged 2 commits into
mainfrom
feature/db-job-coordinator

Conversation

@romeokienzler

Copy link
Copy Markdown
Member

Summary

Adds a database-backed job coordinator to CLAIMED — a sibling to the existing file-based grid wrapper that turns any shell script into a distributed worker. Instead of .LOCKED/.PROCESSED/.FAILED marker files on a shared filesystem, the queue and coordination state live in a database (SQLite or PostgreSQL).

Terminology: a job is a whole batch/run (the namespace); a task is one unit of work within it (typically a file path).

Workflow

# (a) create the tasks belonging to a job, from a glob or directory
claimed propagate_jobs --db postgresql://user:pass@host/claimed --job run-2026 'directory/**/*.tif'
#     bare path / *.db / *.sqlite -> SQLite; no server needed for local runs
claimed propagate_jobs --db /tmp/jobs.db --job run-2026 'data/**/*.tif'

# (b) work that job's tasks one-by-one; each is marked pending -> processing -> succeeded/failed
claimed work_jobs --db /tmp/jobs.db --job run-2026 --worker ./worker.sh

The worker receives the task name (file path) as $1 and $CLAIMED_TASK, with $CLAIMED_JOB and $CLAIMED_WORKER_ID also exported. Exit code 0 → succeeded, anything else → failed (error recorded).

Design

  • Backends: PostgreSQL + SQLite via SQLAlchemy Core, one URL-based abstraction. URL normalization mirrors the existing iterate2 coordinator plugins.
  • Atomic claiming: SELECT ... FOR UPDATE SKIP LOCKED on PostgreSQL; BEGIN IMMEDIATE + guarded UPDATE ... WHERE status='pending' with retry on SQLite (WAL + busy timeout enabled).
  • Job isolation: uniqueness is per (job, task_name), and all operations are scoped to --job, so multiple independent jobs share one database without contending for or stealing each other's tasks.
  • Schema is created lazily and idempotently on first connect (create_all(checkfirst=True)).
  • The two verbs are dispatched from claimed.claimed.main; existing claimed run <module> behavior is untouched.

Files

  • src/claimed/jobcoordinator/{db,cli}.py — engine/schema/claim logic + CLI
  • src/claimed/claimed.py — two-verb dispatch
  • examples/jobcoordinator_example/ — runnable worker.sh + README
  • docs/jobcoordinator.md (+ mkdocs nav)
  • tests/integration/test_jobcoordinator.py

Verification

  • flake8 clean; 9 passed, 1 skipped (PostgreSQL test skipped unless POSTGRES_URL is set).
  • End-to-end (SQLite): propagate → work with a failing task → correct succeeded/failed counts and recorded error; idempotent re-propagate; two jobs isolated in one DB; concurrent claim race (3 workers / 30 tasks → 30 unique, evenly split, no duplicates).

PostgreSQL path is covered by logic + the optional integration test; not exercised against a live server in CI here.

romeokienzler and others added 2 commits July 7, 2026 15:29
Add a DB-backed coordinator that turns any shell script into a distributed
worker, as a sibling to the file-based grid wrapper. A job is a batch/run
namespace; a task is one unit of work (typically a file path).

- claimed propagate_jobs --db <url> --job <name> <glob>: expand a glob or
  directory and insert each path as a pending task of the job (idempotent).
- claimed work_jobs --db <url> --job <name> --worker <script>: claim the
  job's tasks one-by-one (pending -> processing -> succeeded/failed) and run
  the worker per task. The task name is passed as $1 and $CLAIMED_TASK, with
  $CLAIMED_JOB and $CLAIMED_WORKER_ID also exported.

Backends: PostgreSQL and SQLite via SQLAlchemy Core, one URL abstraction.
Atomic claiming uses SELECT ... FOR UPDATE SKIP LOCKED on PostgreSQL and a
BEGIN IMMEDIATE guarded UPDATE with retry on SQLite. Uniqueness is per
(job, task_name), so independent jobs can share one database safely.

Includes a runnable example, docs page, and SQLite-backed integration tests
(lifecycle, idempotent propagate, job isolation, concurrent claim race).

Signed-off-by: Romeo Kienzler <romeo.kienzler1@ibm.com>
@romeokienzler romeokienzler merged commit 8c8b5ba into main Jul 7, 2026
10 checks passed
pull Bot pushed a commit to zongruxie4/component-library that referenced this pull request Jul 7, 2026
Release includes the database-backed job coordinator (claimed-framework#325) and the
relocation of the iterate sources to claimed.iterate (claimed-framework#326).

Signed-off-by: Romeo Kienzler <romeo.kienzler1@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants