[venice-common] Add TEST-ONLY store-scoped config to inject A/A DCR divergence for consistency-checker validation#2878
Open
Mohith22 wants to merge 1 commit into
Open
Conversation
d97b098 to
73ce32e
Compare
xunyin8
reviewed
Jun 18, 2026
4e89b28 to
56b63d8
Compare
xunyin8
previously approved these changes
Jun 18, 2026
56b63d8 to
a4ff8ac
Compare
…ivergence for consistency-checker validation
a4ff8ac to
dd7a68a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Statement
We need confidence that our A/A consistency checker can actually catch real divergence between fabrics. Today we have no controlled way to produce a genuine A/A inconsistency on demand, so the checker's detection capability is effectively untested end-to-end. We are manually injecting this bug to verify the consistency checker is actually capable of flagging it.
Solution
Add a TEST-ONLY, store-scoped server config (server.aa.dcr.bug.injection.store.to.fabric.map) that deliberately diverges the two fabrics for a single store. On the server whose local region matches the configured fabric, the A/A DCR write timestamp is reflected (Long.MAX_VALUE - ts), inverting "newer wins" into "older wins". Since only one fabric inverts, the fabrics deterministically settle on different winners for the same key — a real, reproducible inconsistency the checker should flag. The DCR merge code is untouched (reflection happens at one ingestion chokepoint), it's off by default, logs a loud WARN when active, and must never be enabled in production.
Code changes
Concurrency-Specific Checks
Both reviewer and PR author to verify
synchronized,RWLock) are used where needed.ConcurrentHashMap,CopyOnWriteArrayList).How was this PR tested?
Does this PR introduce any user-facing or breaking changes?