feat: add database-backed job coordinator#325
Merged
Conversation
Add a DB-backed coordinator that turns any shell script into a distributed worker, as a sibling to the file-based grid wrapper. A job is a batch/run namespace; a task is one unit of work (typically a file path). - claimed propagate_jobs --db <url> --job <name> <glob>: expand a glob or directory and insert each path as a pending task of the job (idempotent). - claimed work_jobs --db <url> --job <name> --worker <script>: claim the job's tasks one-by-one (pending -> processing -> succeeded/failed) and run the worker per task. The task name is passed as $1 and $CLAIMED_TASK, with $CLAIMED_JOB and $CLAIMED_WORKER_ID also exported. Backends: PostgreSQL and SQLite via SQLAlchemy Core, one URL abstraction. Atomic claiming uses SELECT ... FOR UPDATE SKIP LOCKED on PostgreSQL and a BEGIN IMMEDIATE guarded UPDATE with retry on SQLite. Uniqueness is per (job, task_name), so independent jobs can share one database safely. Includes a runnable example, docs page, and SQLite-backed integration tests (lifecycle, idempotent propagate, job isolation, concurrent claim race). Signed-off-by: Romeo Kienzler <romeo.kienzler1@ibm.com>
rosielickorish
approved these changes
Jul 7, 2026
pull Bot
pushed a commit
to zongruxie4/component-library
that referenced
this pull request
Jul 7, 2026
Release includes the database-backed job coordinator (claimed-framework#325) and the relocation of the iterate sources to claimed.iterate (claimed-framework#326). Signed-off-by: Romeo Kienzler <romeo.kienzler1@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a database-backed job coordinator to CLAIMED — a sibling to the existing file-based grid wrapper that turns any shell script into a distributed worker. Instead of
.LOCKED/.PROCESSED/.FAILEDmarker files on a shared filesystem, the queue and coordination state live in a database (SQLite or PostgreSQL).Terminology: a job is a whole batch/run (the namespace); a task is one unit of work within it (typically a file path).
Workflow
The worker receives the task name (file path) as
$1and$CLAIMED_TASK, with$CLAIMED_JOBand$CLAIMED_WORKER_IDalso exported. Exit code 0 →succeeded, anything else →failed(error recorded).Design
SELECT ... FOR UPDATE SKIP LOCKEDon PostgreSQL;BEGIN IMMEDIATE+ guardedUPDATE ... WHERE status='pending'with retry on SQLite (WAL + busy timeout enabled).(job, task_name), and all operations are scoped to--job, so multiple independent jobs share one database without contending for or stealing each other's tasks.create_all(checkfirst=True)).claimed.claimed.main; existingclaimed run <module>behavior is untouched.Files
src/claimed/jobcoordinator/{db,cli}.py— engine/schema/claim logic + CLIsrc/claimed/claimed.py— two-verb dispatchexamples/jobcoordinator_example/— runnableworker.sh+ READMEdocs/jobcoordinator.md(+ mkdocs nav)tests/integration/test_jobcoordinator.pyVerification
flake8clean; 9 passed, 1 skipped (PostgreSQL test skipped unlessPOSTGRES_URLis set).succeeded/failedcounts and recorded error; idempotent re-propagate; two jobs isolated in one DB; concurrent claim race (3 workers / 30 tasks → 30 unique, evenly split, no duplicates).PostgreSQL path is covered by logic + the optional integration test; not exercised against a live server in CI here.