fix(agentic_rag): honor similarity_threshold in deduplicate_chunks

[andrewwhitecdw]

Bug: deduplicate_chunks accepted a similarity_threshold argument but only performed exact-match deduplication, so the threshold was silently ignored.

Fix: Compare chunks using Jaccard similarity over normalized token sets and only keep a chunk when it is below the threshold relative to every chunk already kept.

Test: Added regression tests covering exact duplicates, near-duplicates at the default threshold, threshold sensitivity, and empty input.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ santoshkumarradha
❌ andrewwhitecdw
_{You have signed the CLA already but the status is still pending? Let us recheck it.}