Skip to content

Embedding-based cross-document entity clustering (follow-up to #431) #437

Description

@CheeryProgrammer

Follow-up to #431, which shipped exact normalized-string cross-document entity resolution for the batch path (merge proposed nodes sharing a normalized canonical name or alias after a batch applies).

This issue is the semantic half: catch same-entity mentions that the normalizer cannot — near-synonyms and typos ("Postgres" vs "PostgreSQL", "k8s" vs "Kubernetes", "OrderSvc" vs "OrderService") — using embedding similarity, not string identity.

Why it's separate

  • Fresh extraction proposals carry no summary embedding. resolveOrProposeNode inserts a node with just a canonical name + aliases; summary_embedding is NULL until a summary is written. store.ProposeSemanticMergeCandidates (the existing review-time semantic pass) operates on summary_embedding, so it finds nothing among fresh proposals. Semantic batch clustering first needs an embedding for each new proposal (embed the canonical name/alias text, or a synthesized summary).
  • Threshold tuning + false-merge risk. Unlike exact identity, embedding distance is fuzzy; an over-eager threshold merges genuinely distinct entities. Needs a conservative threshold (tighter than SemanticDupThreshold = 0.15) and likely a secondary guard (type agreement, alias token overlap) before auto-merging even a proposed node.

Scope

  • Give batch-created proposed nodes an identity embedding (name/alias text).
  • After resolveBatchDuplicates' exact pass, cluster the remaining proposals by embedding distance under a conservative threshold (reuse ProposeSemanticMergeCandidates/SemanticDupThreshold machinery), merge only high-confidence pairs, leave the rest as separate proposals for Consolidate review.
  • DB-test with fixture embeddings (inject vectors directly, like existing vec() test helpers) so it's deterministic.

Constraints

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions