Wire async batch extraction into ingest + background poller + cross-doc entity resolution (follow-up to #383)

[CheeryProgrammer]

Follow-up to #383, which shipped the infrastructure: the separable async batch API (anthropic.Extractor.CreateBatch / non-blocking FetchBatchResults) and the extraction_batches job ledger (store CRUD). What remains is the wiring that turns it into a running async job.

1. Submit path (ingest)

When EXTRACTOR=claude and a batch flag is set, ingest submits the chunks-to-extract as one Message Batch (CreateBatch) instead of extracting synchronously per chunk. Store the job (InsertExtractionBatch) plus a custom_id ↔ chunk mapping (a extraction_batch_items(batch_id, custom_id, chunk_id, source_uri, discriminators, trust) table) so results can be applied with the right scope/trust later.

2. Background poller (apply)

A background loop (or kb poll-batches) that, for each submitted job, calls FetchBatchResults; when ended, applies each succeeded extraction to the graph — the same node/edge creation the synchronous extractor uses (extractChunk), keyed back to the chunk via the mapping, honoring forceReview for untrusted content (#273) — then marks the job applied. Handle failed/expired.

3. Scheduler + cost

• Poll cadence config; give up after the 24h batch window (failed).
• ~50% cost saving is the point — document when to prefer batch vs sync.

4. Cross-doc entity resolution

With a whole backfill's extractions available at once, resolve entities across documents before writing (dedupe/merge same-entity mentions) rather than one chunk at a time.

Invariant

Opt-in; default (sync per-chunk extraction, or chat-driven) unchanged.

P2, area:ingestion.

🤖 Generated with Claude Code