Follow-up to #431, which shipped exact normalized-string cross-document entity resolution for the batch path (merge proposed nodes sharing a normalized canonical name or alias after a batch applies).
This issue is the semantic half: catch same-entity mentions that the normalizer cannot — near-synonyms and typos ("Postgres" vs "PostgreSQL", "k8s" vs "Kubernetes", "OrderSvc" vs "OrderService") — using embedding similarity, not string identity.
Why it's separate
- Fresh extraction proposals carry no summary embedding.
resolveOrProposeNode inserts a node with just a canonical name + aliases; summary_embedding is NULL until a summary is written. store.ProposeSemanticMergeCandidates (the existing review-time semantic pass) operates on summary_embedding, so it finds nothing among fresh proposals. Semantic batch clustering first needs an embedding for each new proposal (embed the canonical name/alias text, or a synthesized summary).
- Threshold tuning + false-merge risk. Unlike exact identity, embedding distance is fuzzy; an over-eager threshold merges genuinely distinct entities. Needs a conservative threshold (tighter than
SemanticDupThreshold = 0.15) and likely a secondary guard (type agreement, alias token overlap) before auto-merging even a proposed node.
Scope
- Give batch-created proposed nodes an identity embedding (name/alias text).
- After
resolveBatchDuplicates' exact pass, cluster the remaining proposals by embedding distance under a conservative threshold (reuse ProposeSemanticMergeCandidates/SemanticDupThreshold machinery), merge only high-confidence pairs, leave the rest as separate proposals for Consolidate review.
- DB-test with fixture embeddings (inject vectors directly, like existing vec() test helpers) so it's deterministic.
Constraints
Follow-up to #431, which shipped exact normalized-string cross-document entity resolution for the batch path (merge proposed nodes sharing a normalized canonical name or alias after a batch applies).
This issue is the semantic half: catch same-entity mentions that the normalizer cannot — near-synonyms and typos ("Postgres" vs "PostgreSQL", "k8s" vs "Kubernetes", "OrderSvc" vs "OrderService") — using embedding similarity, not string identity.
Why it's separate
resolveOrProposeNodeinserts a node with just a canonical name + aliases;summary_embeddingis NULL until a summary is written.store.ProposeSemanticMergeCandidates(the existing review-time semantic pass) operates onsummary_embedding, so it finds nothing among fresh proposals. Semantic batch clustering first needs an embedding for each new proposal (embed the canonical name/alias text, or a synthesized summary).SemanticDupThreshold= 0.15) and likely a secondary guard (type agreement, alias token overlap) before auto-merging even a proposed node.Scope
resolveBatchDuplicates' exact pass, cluster the remaining proposals by embedding distance under a conservative threshold (reuseProposeSemanticMergeCandidates/SemanticDupThresholdmachinery), merge only high-confidence pairs, leave the rest as separate proposals forConsolidatereview.Constraints
consolidate*.goprimitives.