[Data Validation] Add source_record_key column to MismatchedRecords

[aasthabharill]

b/519409134

Issue

Mismatched rows are reported in the format: [spanner_pk_columns: source_values] even for rows that are MISSING_IN_DESTINATION.

Impact

• It doesn’t make sense to report rows using spanner PK when they don't exist on spanner. They should be reported using the only source of truth we have which is the source in this case.
• It’s hard for the customer to trace back failed rows to their source, as they need to do the mental math to convert the reported primary keys to the original source primary key values (in case of PK transformation)
• if the transformation yields a different valid PK that happens to exist in Spanner, the DV pipeline will hash the payloads, find them completely different, and emit a MISSING_IN_SOURCE and MISSING_IN_DESTINATION pair for the wrong rows. Without exposing the Source PK, this looks like massive, random data corruption rather than a simple mapping logic error.
• An edge case scenario can exist where values in a Spanner PK are not unique in Source in case the primary key columns were changed in Spanner. In this case it will not only be hard, but impossible to trace back a failed row to a source row.
• Important note: Using the same logic, MISSING_IN_SOURCE rows should continue being reported using their spanner PKs

Fix

1. Introduce source_record_key columns that stores the Source PK column values in MismatchedRecords table.
2. Rename record_key to spanner_record_key for clarity
3. Ensure the pipeline hard fails if Avro schema didn't contain primaryKeys information as this is required to populate source_record_key.
4. [Follow Up PR] Introduce column isPKTransformed to clearly call out to user when they need to care about source_record_key. When this is false, source_record_key = spanner_record_key

Alternative Approaches

Store source PK details in record_key column instead of equivalent spanner PK for rows that are MISSING_IN_DESTINATION and don't have spanner row.

Reason for rejection:

• A "data mismatch" (where a row exists in both systems with the same primary key but different column values) is represented by the pipeline as two distinct entries: one MISSING_IN_DESTINATION (the source row) and one MISSING_IN_SOURCE (the Spanner row).
• Users can clearly identify these rows currently because their record_key column will have identical values.
• If we were to completely remove the equivalent spanner key data and replace it with the source PK data, it will be difficult to identify data mismatches, which is an important CUJ.

Therefore, it's important to maintain BOTH source PK data and the computed equivalent spanner PK.

Design Decisions

When to hard fail the pipeline and why?

The pipeline will hard fail in the following scenarios:

• primaryKeys metadata is absent from avro schema
• primary key columns are absent from the avro data
  We hard fail here as the source_record_key column is dependent on this metadata, and won't work without it. So it's better to quickly stop the pipeline.

The pipeline does NOT hard fail if a primary key column has NULL value as this is allowed by some databases and shouldn't be enforced by a validation pipeline IMO.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.36%. Comparing base (9992792) to head (d9b784d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3962      +/-   ##
============================================
+ Coverage     55.64%   63.36%   +7.72%     
+ Complexity     6731     2455    -4276     
============================================
  Files          1117      528     -589     
  Lines         68313    30592   -37721     
  Branches       7696     3396    -4300     
============================================
- Hits          38012    19385   -18627     
+ Misses        27829    10196   -17633     
+ Partials       2472     1011    -1461     

Components	Coverage Δ
spanner-templates	88.06% <ø> (+0.41%)	⬆️
spanner-import-export	∅ <ø> (∅)
spanner-live-forward-migration	89.75% <ø> (-0.08%)	⬇️
spanner-live-reverse-replication	83.27% <ø> (-0.07%)	⬇️
spanner-bulk-migration	92.54% <ø> (-0.05%)	⬇️
gcs-spanner-dv	90.34% <ø> (+1.48%)	⬆️
see 613 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.