Skip to content

Fix sanitizer test failures#103

Open
CarlosFelipeOR wants to merge 11 commits into
mainfrom
fix-sanitizer-test-failures
Open

Fix sanitizer test failures#103
CarlosFelipeOR wants to merge 11 commits into
mainfrom
fix-sanitizer-test-failures

Conversation

@CarlosFelipeOR

@CarlosFelipeOR CarlosFelipeOR commented Feb 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Addresses test failures in UBSAN, ASAN, TSAN, and MSAN regression runs for the 26.3 release. Combines real upstream-bug xfails with infrastructure tuning (runner sizes, timeouts) for sanitizer overhead.

Real upstream bugs (xfailed under sanitizers, not backported to stable-26.3)

Suite Symptom Upstream PR / Issue
iceberg/iceberg table engine/feature/row policies/* Logical error: 'prewhere_info' chassert in sanitizer/debug builds (Fail + Error) CH#100361
iceberg/iceberg engine/*/schema evolution/* Server crashes with "Metadata is not initialized" CH#86024
kerberos/* (MSAN only) bufferToString strips trailing NUL bytes from opaque AP-REP tokens; krb5 ASN.1 decoder reads 1 byte of uninitialized memory → MSAN aborts every request CH#103114 / CH#97414

Test framework / FP-rounding xfails

Suite Reason
aggregate_functions/part 3/state/rankCorrState/with group by (TSAN only) TSAN codegen produces same sub-ULP FP rounding as the static _binary build; routes to the existing _binary snapshot variant
base_58/memory usage/* Memory-delta assertions incompatible with sanitizer allocators (Fail + Error)
alter/attach partition/*, alter/replace partition/* Sanitizer-only OOM / job timeout / BAD_ARGUMENTS on cross-disk replace

Sanitizer timeout / runner adjustments (infrastructure)

File Change
helpers/cluster.py wait_clickhouse_healthy 90→300s, start_clickhouse 60→300s, restart_clickhouse increased — sanitizer startup is 3–4× slower
helpers/common.py Fix broken check_with_any_sanitizer / per-sanitizer helpers (were always returning False)
aggregate_functions/regression.py, parquet/regression.py Bump wait_for_async_insert_timeout / receive_timeout / send_timeout to 900s under sanitizer (default 120s too tight for heavy INSERTs)
ontime_benchmark/tests/benchmark.py Bump node.query bash timeout 1200→3600s and receive_timeout 600→1800s under sanitizer (CSV.gz ingest under MSAN)
s3/tests/common.py add_config auto-bumps reload timeout 60→300s under sanitizer; fix remote_host_filter_config to forward timeout
.github/workflows/run-regression-high-performance.yml Move 11 OOM-prone ASAN suites (alter_replace_partition, alter_attach_partition 1/2, clickhouse_keeper, clickhouse_keeper_ssl, functions, lightweight_delete, memory, parquet, s3_minio_3, selects) onto cpx62 (32 GB RAM); increase timeout_minutes to 360 for 6 long-running suites (alter_replace_partition, aggregate_functions_1/3, ldap_role_mapping, memory, tiered_storage_gcs)

Notes

  • ontime_benchmark is not skipped under sanitizers (passes on all sanitizers once the bash timeout is large enough).
  • wait_for_async_insert_timeout increases resolved earlier sanitizer flakes in aggregate_functions and parquet (not flagged in upstream-bug section because no client-visible bug remains).
  • A new run is required to validate the runner-type changes (rerun-failed-jobs uses the workflow file from the original trigger commit).

Test plan

  • Resolve sanitizer-only timeouts and OOMs in ASAN
  • Identify and isolate real upstream bugs from infrastructure issues for TSAN/MSAN
  • Trigger a fresh `workflow_dispatch` run on the branch and confirm pass set

CarlosFelipeOR and others added 11 commits February 13, 2026 15:06
- Increase server startup timeouts for sanitizer builds
- Increase job timeouts from 180 to 300 minutes
- Add xfails for known bugs exposed by sanitizers
- Skip benchmark tests with sanitizers (too slow)

Co-authored-by: Cursor <cursoragent@cursor.com>
- aggregate_functions: increase receive_timeout/send_timeout to 900s for sanitizers
- rbac: skip compiled expression cache tests (JIT disabled in sanitizers)

Co-authored-by: Cursor <cursoragent@cursor.com>
- base_58: change Fail to Error (test has UnboundLocalError)
- rbac: fix FFails format (tuple not list), combine ARM + sanitizer check
- iceberg: skip iceberg engine tests with sanitizers (server unstable)

Co-authored-by: Cursor <cursoragent@cursor.com>
MSAN builds are 10-20x slower and need more time for server startup.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ge conflict residue

Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
… ccx33

Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
…me/s3 timeouts; revert OOM suites to cpx62

Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant