Skip to content

Antalya 26.6 - Backport flaky-fix commits from upstream (2026-07-21)#2098

Merged
strtgbb merged 44 commits into
antalya-26.6from
flaky-fix-backport/antalya-26.6/2026-07-21
Jul 22, 2026
Merged

Antalya 26.6 - Backport flaky-fix commits from upstream (2026-07-21)#2098
strtgbb merged 44 commits into
antalya-26.6from
flaky-fix-backport/antalya-26.6/2026-07-21

Conversation

@strtgbb

@strtgbb strtgbb commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Automated backport of upstream flaky-fix commits.
(manual push)

  • Exclude all Regression
  • Disable CI Cache

Cherry-picked

  • b0f0a79749ff Fix flaky test_kafka_formats_with_broken_message (slow-build errors-MV flush) (committed 2026-06-23T03:05:07Z)
  • aa58245e52cc Fix flaky test_system_kafka_consumers (DROP-view-vs-streamer race) (committed 2026-06-23T22:37:59Z)
  • 401bc6c8948a Fix flaky test 04365_cancel_insert_when_client_dies_distributed (committed 2026-06-23T23:22:05Z)
  • 177ed2411dae Fix flaky test_s3_plain_rewritable: opt in to legacy REPLACE PARTITION from empty source (committed 2026-06-24T04:18:40Z)
  • d2d81c115a4b Revert "Fix flaky test_rabbitmq_big_message: keep polling while consumption progresses" (committed 2026-06-24T20:16:22Z)
  • f641ece7d2f4 Fix flaky test 04240_statistics_countmin_numeric_types (committed 2026-06-24T22:53:27Z)
  • 13b91d57a817 Fix flaky test_parallel_quorum_actually_quorum (committed 2026-06-25T08:03:00Z)
  • 16493775c22e Fix flaky test_shutdown_cancels_backup and test_long_disconnection_stops_backup (committed 2026-06-25T11:32:24Z)
  • 90e888a545ff Fix flaky test 02161_addressToLineWithInlines (committed 2026-06-25T15:14:25Z)
  • 80c8a4312037 Fix flaky test_keeper_disks::test_snapshots_with_disks (committed 2026-06-25T16:08:16Z)
  • 852a2c260ad1 Fix flaky test_distributed_frozen_replica: pin lightweight_deletes_sync=1 on DELETE (committed 2026-06-25T19:03:22Z)
  • 985c43696455 Fix flaky test 03583_rewrite_in_to_join (row ordering) (committed 2026-06-26T01:11:37Z)
  • 2e30a6b9ae66 Fix flaky test_async_backup_restore_with_max_execution_time_zero on slow CI lanes (committed 2026-06-26T03:18:31Z)
  • fb1000fcf6a7 Fix flaky test_e2e_catalogs[glue]: implement wait_for_table_ready (committed 2026-06-26T12:15:07Z)
  • 01e92533ed62 Fix flaky test_storage_s3_queue/test_1.py::test_max_set_age (committed 2026-06-26T17:02:24Z)
  • ea12275689c3 Fix flaky test_kafka_many_materialized_views (view2 result race) (committed 2026-06-26T23:36:58Z)
  • 1287728b9f92 Fix flaky test_startup_without_zookeeper: retry the recursive ZK delete (committed 2026-06-29T05:21:34Z)
  • b59a7ddf8fed Fix flaky test test_clickhouse_writes_trino_reads (No nodes available to run query) (committed 2026-06-29T17:34:52Z)
  • aa30a607b4a3 Fix flaky test_restore_db_replica_with_diffrent_table_metadata (committed 2026-06-29T21:12:07Z)
  • 24c098df579d Fix flaky test 02441_alter_delete_and_drop_column (committed 2026-06-30T11:28:21Z)
  • b6dfe3b89c4a Fix flaky test_implicit_index_upgrade_alter_replay (replica fetch race) (committed 2026-06-30T16:08:56Z)
  • 9083e3a89c80 Fix flaky test 04051_pk_analysis_stats: QCC entry evicted by parallel tests (committed 2026-06-30T16:33:51Z)
  • e78459d82170 Fix flaky test 02789_filesystem_cache_alignment (committed 2026-06-30T17:46:14Z)
  • 17e2ed58e64d Fix flaky test_backup_restore_on_cluster restore timeout (committed 2026-06-30T21:15:30Z)
  • 2f861264e4ae Fix flaky test 04312_client_chime_on_slow_query_92718 (committed 2026-06-30T23:22:37Z)
  • 9f90342fdca8 Fix flaky test_keeper_session_refuse_stale_server (committed 2026-07-01T09:16:10Z)
  • c5e22ee8f3ae Fix flaky test 00612_pk_in_tuple_perf (committed 2026-07-01T10:02:40Z)
  • aa8dda746e24 Fix flaky test 00974_query_profiler (committed 2026-07-01T13:35:23Z)
  • 0c3e7fd50d2c Fix flaky test 04053_merge_table_large_query_performance (committed 2026-07-01T18:53:11Z)
  • 925fb1cb175e Fix flaky test 03986_kill_query_mutation_sync_replicated (committed 2026-07-02T14:24:16Z)
  • 8a019722ef68 Fix flaky test test_keeper_ttl_nodes::test_many_nodes_with_different_ttls (committed 2026-07-02T14:49:44Z)
  • 610cd689085a Fix flaky test 04321_multiple_quotas_per_user (committed 2026-07-03T12:19:01Z)
  • 4aef3f83c4df Fix flaky test 04146_iceberg_orc_row_policy_prewhere on release branches (committed 2026-07-03T14:24:57Z)
  • ff85e04b78e8 Fix flaky test_failed_commit_after_success (s3_queue) (committed 2026-07-03T17:23:12Z)
  • 03765d162a91 Fix flaky test 03394_distributed_shuffle_join_with_aggregation (committed 2026-07-04T03:19:47Z)
  • 01abbd025ddd Fix flaky test test_dictionaries_postgresql::test_invalidate_query (committed 2026-07-04T05:08:06Z)
  • da4060455b06 Fix flaky test 03394_distributed_shuffle_join_with_in (committed 2026-07-04T10:26:13Z)
  • eb79a6daff00 Fix flaky test_retry_loading_parts in test_merge_tree_s3_failover (committed 2026-07-06T20:58:34Z)
  • 6edf3b2fe4a3 Fix flaky test_rabbitmq_mv_combo: keep polling while consumption progresses (committed 2026-07-09T12:45:30Z)
  • e1d7ad5f9fa1 Fix flaky test 02676_optimize_old_parts_replicated (committed 2026-07-12T19:23:23Z)
  • aac9fbc6206e Fix flaky test_storage_iceberg_disks: skip S3 disk startup access check (committed 2026-07-15T16:39:35Z)
  • 6944c75587ca Fix flaky test 02421_simple_queries_for_opentelemetry (committed 2026-07-17T11:54:59Z)
  • 5e662b6d2710 Fix flaky test 01017_uniqCombined_memory_usage (committed 2026-07-17T17:02:35Z)
  • 51eea395dd90 Fix flaky test_rename_distributed_parallel_insert_and_select (committed 2026-07-20T17:49:36Z)

Skipped (cherry-pick conflict — manual backport needed)

  • f6300c9717a0 Fix flaky test 04402_map_functions_lowcardinality (committed 2026-06-27T01:45:32Z)
  • 404b36c92268 Fix flaky test 04489_regexp_compile_error_bounded by measuring the longest line (committed 2026-06-30T06:47:44Z)
  • 0fc750e4c48e Fix flaky test 04344_server_ast_fuzzer_insert_limit (sync-drop timeout) (committed 2026-07-02T18:18:39Z)
  • 9b93db63a3c0 Fix flaky test 04061_spilling_hash_join_overflow_limits (committed 2026-07-03T16:46:15Z)
  • 4410afb70270 Fix flaky test 04001_join_reorder_through_expression under randomized join order (committed 2026-07-06T13:28:51Z)
  • 47a0bc338877 Fix flaky test_server_no_logging: poll for async "Restored console logger level" line (committed 2026-07-10T18:19:46Z)
  • 2f69c7a06ffb Fix flaky test_parallel_replicas_protocol backward-compat OPTIMIZE hang (committed 2026-07-12T01:05:34Z)
  • 1b38d200d72d Fix flaky test 04491_outfile_compression_level_range (committed 2026-07-14T09:18:49Z)

groeneai and others added 30 commits July 21, 2026 12:21
…V flush)

The error row in kafka_errors_*_mv can lag behind the data rows on slow
sanitizer/coverage builds. query_with_retry returns its last result even when
the check_callback never passed, so on a too-short budget it returns an empty
string and json.loads("") raised an opaque JSONDecodeError (Expecting value:
line 1 column 1). Over 60 days every json.loads-mode failure was on a slow
build (asan_ubsan/tsan/msan/llvm_coverage), zero on fast/release builds.

Double the error-MV poll budget (30s -> 60s) for slow-build headroom and assert
the result is non-empty before json.loads so any residual failure surfaces with
a clear message instead of an opaque JSON decode error.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit b0f0a79)
test_system_kafka_consumers created the materialized view with an
implicit inner table (CREATE MATERIALIZED VIEW ... ENGINE=MergeTree AS
SELECT) and then dropped it with DROP TABLE ... _view SYNC while the
Kafka table kept streaming. The SYNC drop removes the implicit inner
table (.inner_id.<uuid>) before the view itself. A background
streamToViews() cycle that has already locked the view but not yet
resolved its inner table then resolves a missing inner table and
InsertDependenciesBuilder::observePath throws
"Target table '...inner_id...' of view '...' doesn't exists"
(UNKNOWN_TABLE). StorageKafka propagates that to every consumer's
append-only exceptions_buffer, which is never cleared on success, so the
test's assert_eq_with_retry on system.kafka_consumers can never observe
"no exception" and fails. The window is widened on sanitizer/coverage
builds, matching the observed sanitizer-only failure distribution.

Use an explicit target table (CREATE TABLE ..._target; CREATE
MATERIALIZED VIEW ... TO ..._target) and drop only the view. The target
table always exists, so the streamer can never observe a missing inner
table, and dropping the view alone takes the graceful no-dependencies
path. Test intent and the system.kafka_consumers assertion are
unchanged. This matches the explicit-target idiom already used by the
sibling test_system_kafka_consumers_rebalance_mv.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit aa58245)
The test drives the cancelled INSERT only over the native protocol, but its
final "ensure that thread_cancel actually did something" assertion was inherited
from the HTTP-heavy sibling 02434 and matches only HTTP-chunked / socket-drop
messages (plus a predicate that exists only in the test files).

On the native protocol a SIGINT makes the client send a graceful 'Cancel' packet
and keep the connection open, so the server logs QUERY_WAS_CANCELLED_BY_CLIENT
("Received 'Cancel' packet from the client, canceling the query.") instead of a
broken pipe or connection reset. The socket-drop messages only fire when a SIGKILL
lands while the server happens to be writing back to the client. thread_cancel
picks INT or KILL with equal probability, so roughly half of the cancellations
produce no message the assertion recognizes, and the SIGKILL half matches only
when the kill races a server-side write. Over the 40s run the margin is thin and
the assertion occasionally observes zero matching messages, printing 0 instead of 1.

Add the native-protocol cancellation messages to the matcher. Verified locally:
a SIGINT interrupting an in-flight native distributed insert logs the cancel
packet message 11 out of 12 times while matching the old filter 0 times; a normal
uncancelled insert never logs it, so the assertion still requires a real
cancellation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 401bc6c)
…N from empty source

When a node's random row count (random.randint(1, MAX_ROWS)) is 1, only id=0
exists, so partition 1 of `test` is empty. After MOVE PARTITION 0 TO test_dst,
`test` is empty and REPLACE PARTITION 1 FROM test reads an empty source partition.

ClickHouse#23727 (PR ClickHouse#104939) made REPLACE PARTITION from an empty source partition throw
BAD_ARGUMENTS by default, gated by allow_replace_partition_from_empty_source. That
PR updated the affected stateless tests but missed this integration test, which then
failed intermittently (~0.05% of master runs across asan/tsan/msan once ClickHouse#104939 merged).

Pin allow_replace_partition_from_empty_source=1 on the REPLACE query to restore the
legacy no-op semantics the test relies on, matching the fix applied to the stateless
tests in ClickHouse#104939. The destination partition 1 is also empty here, so no data is lost.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 177ed24)
…mption progresses"

This reverts commit 757c39c.

(cherry picked from commit d2d81c1)
PREWHERE condition order is chosen by std::min_element over each
condition's estimated_row_count; it falls back to source order (a, b)
only on a tie. The test asserts the b-before-a order, so any tie makes
the asserted plan shape flip while results stay correct (rare,
build-spanning, CI auto-rerun passes).

Two independent tie sources existed:

1. Transiently unavailable per-part statistics. loadStatistics failures
   are swallowed (MergeTreeData.cpp), so a's estimate can fall back to
   the default_cond_equal_factor (0.01). The previous data shape had b
   matching 10/1010 rows (~0.0099), which collides with 0.01 and ties.
   Fixed by making b match exactly 1/1010 rows (selectivity ~0.001), an
   order of magnitude below 0.01, so b stays first even when a's estimate
   is the default.

2. Stale statistics cache. The background refreshStatistics task (the
   only writer of cached_estimator) is scheduled immediately at table
   startup. On a loaded runner it can run after only the first part is
   committed and snapshot that single part, where a and b each have one
   row, then with use_statistics_cache=1 every later EXPLAIN reads that
   stale 1-part snapshot, ties 1 vs 1, and flips. Pinning the session
   setting use_statistics_cache=0 was rejected: it is randomized by
   clickhouse-test and targets the cache read, not the stale population.
   Fixed by setting the per-table MergeTree setting
   refresh_statistics_interval = 0, which is the documented way to
   disable the refresh: the refresh task is never created, so
   cached_estimator stays null and getConditionSelectivityEstimator
   always fresh-loads all current parts, regardless of
   use_statistics_cache. This is immune to session-setting randomization.

Validated on a debug server: with refresh_statistics_interval=1 and a
forced 1-part snapshot the new shape still flips 5/5; adding
refresh_statistics_interval=0 gives the correct b-first order 5/5, and
the full test passes 20/20 against the reference.

(cherry picked from commit f641ece)
test_parallel_quorum_actually_quorum partitions node2 off node1/node3 on
port 9009 for the whole test. While partitioned, node2's GET_PART queue
entries fail repeatedly and accumulate the ReplicatedMergeTree exponential
fetch backoff (1<<num_tries ms, capped at
max_postpone_time_for_failed_replicated_fetches_ms, default 60000). After
the partition heals, the final "SYSTEM SYNC REPLICA q" must wait out that
accumulated postpone before node2 re-fetches the 3 parts. Under the parallel
sanitizer flaky-check (--dist=each, many concurrent clusters under ASAN) the
backoff plus contention pushes the catch-up well past the barrier's tight
10s client-side timeout, so the test fails with "Client timed out!".

Two changes, neither weakens an assertion:
- Set max_postpone_time_for_failed_replicated_fetches_ms = 0 on the table so
  node2 retries fetches without the exponential postpone once it can reach
  its peers again (same idiom as test_lost_part / test_postpone_failed_tasks,
  which disable the analogous backoff settings for fault-window flakiness).
- Raise the final SYSTEM SYNC REPLICA barrier timeout from 10s to 60s (the
  common value used by other integration tests) to absorb residual parallel
  sanitizer slowness. Correctness is still gated by the retrying
  assert_eq_with_retry(node2, count, "3") that follows, so a genuine
  "node2 never syncs" regression still fails there.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 13b91d5)
…ops_backup

Both tests trip NoTrashChecker.__exit__ when a transient ZooKeeper/network
error lands in the checker's error-collection window:

- test_shutdown_cancels_backup restarts a node mid-backup; the surviving node
  briefly loses its connection to the restarting peer and to Keeper, so
  KEEPER_EXCEPTION/NETWORK_ERROR can appear. The test declared no allow_errors,
  so the checker failed. Allow the same transient-error family the sibling
  disconnection tests already allow.

- test_long_disconnection_stops_backup populated allow_errors only near the end
  of the body. When the body raised earlier (e.g. a transient SOCKET_TIMEOUT
  during the BACKUP setup), __exit__ ran with empty lists and masked the real
  failure with a spurious assertion. Move the relaxations to right after
  entering NoTrashChecker.

Also drain backups from a previous test before creating the NoTrashChecker in
all four affected tests (tests run in random order), via a shared helper that
replaces the inline blocks duplicated in the two short-disconnection tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 1649377)
Disable the execution-time guards (max_estimated_execution_time,
max_execution_time) for the final symbolization query. It scans
system.trace_log, which can be slow (e.g. s3-backed trace_log storage);
once the scan crosses timeout_before_checking_execution_speed,
ExecutionSpeedLimits extrapolates the projected runtime past the CI
profile's max_estimated_execution_time=600 and kills the query with
TOO_SLOW (160), or TIMEOUT_EXCEEDED (159) on branches still using
max_execution_time=300. This is a symbolization-correctness test, not a
perf test, so both guards are false positives. The has-inlines assertion
and symbolization flow are unchanged.

Same change already in 03560_parallel_replicas_memory_bound_merging_projection.sql.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 90e888a)
The test asserted len(local_snapshot_files) == 1 immediately after the
"Created persistent snapshot {idx}" log line and intermittently observed 2
(assert 2 == 1).

When a separate latest_snapshot_storage_disk is configured, creating a
snapshot relocates the previous (non-latest) snapshot off the local "latest"
disk to the main disk. In KeeperStateMachine::create_snapshot that log line is
emitted in the under-lock publish step (publishWrittenSnapshot), while the move
runs afterwards in the deferred, outside-the-lock maintenance step
(runSnapshotMaintenance -> moveSnapshotCandidate). selectSnapshotsToMove also
defers a move while the old snapshot is transiently pinned (use_count > 1).
Checking the count once therefore races the still-pending move.

The move is eventually consistent, so poll for the local count to settle to 1
(up to 60s) instead of asserting once, mirroring the assert_single_local_log
helper already used by test_logs_with_disks. Test-only change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 80c8a43)
…nc=1 on DELETE

test_distributed_query and test_distributed_query_network_timeout stop fetches
on node2 to keep it stale, then run a lightweight DELETE on node1. The DELETE
inherits lightweight_deletes_sync=2 (the default), which maps to mutations_sync=2
in InterpreterDeleteQuery and makes StorageReplicatedMergeTree::waitMutation wait
for the mutation to finish on ALL replicas, including the deliberately-stalled
node2. When node2 cannot promptly apply the mutation, node1's DELETE blocks until
the 600s client timeout, failing the test.

Pin lightweight_deletes_sync=1 so the DELETE waits only on node1. This keeps node2
stale as the tests intend (they assert absolute_delay >= 5 and exercise node3's
distributed query against the stale/paused replica).

Reproduced deterministically: with a replica's queue stopped, the DELETE hangs at
sync=2 and returns immediately at sync=1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 852a2c2)
The data SELECTs read FROM numbers() without an ORDER BY. With
rewrite_in_to_join the IN/NOT IN subquery becomes a LEFT JOIN whose
output is produced by parallel JoiningTransform lanes (EXPLAIN PIPELINE
shows Resize 32->16 over 16 lanes), so row order depends on which lane
finishes first. The result multiset is always correct, only the order
varies, which trips the reference comparison.

Add ORDER BY number to the affected SELECTs. The EXPLAIN rewrite
assertions and the serverError checks are unchanged, and no result
value changes (only ordering becomes deterministic).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 985c436)
…low CI lanes

The instance_with_short_timeout node sets max_execution_time = 0.5 in its
default profile to trigger the bug under test (the profile-level timeout
must fire on the paused async BACKUP/RESTORE). That same 0.5s cap also
applies to the foreground setup INSERT and the final verification SELECT,
which are not part of what the test checks. On a slow sanitizer lane
(amd_msan) the setup INSERT's pipeline finalization exceeded 500ms and
failed with Code 159 TIMEOUT_EXCEEDED before the test could run, e.g.
elapsed 532.9ms > maximum 500ms in QueryStatus::checkTimeLimit.

Scope max_execution_time = 0 to those two foreground scaffolding queries so
a loaded lane can't time them out. The profile cap and the async query
settings are unchanged, so the bug-detection mechanism is fully preserved.

CIDB: 3 failures across 3 unrelated PRs, all on amd_msan, 0 on master.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 2e30a6b)
AwsGlueCatalogManager.wait_for_table_ready and wait_for_table_gone were
bare `pass` no-ops, while the sibling BigLake and OneLake managers
implement them as bounded pyiceberg polling loops. The module-scoped
fixtures (sales_table, customers_table, types_table, complex_table) and
several tests call wait_for_table_ready(name) right after create_table
expecting it to block until the table is catalog-visible.

AWS Glue is eventually-consistent for GetTables listing
(GlueCatalog::getTablesForDatabase issues a GetTables call per query with
no server-side cache), so for the glue backend the no-op returned
immediately and the first resolve_table_name / SHOW TABLES could race
ahead of Glue's listing propagation. resolve_table_name retries for
~15s, which was not always enough, surfacing intermittently as:

  AssertionError: Table 'tbl_xxxxxxxx' not found in SHOW TABLES output:

Implement both methods to poll self.catalog (pyiceberg glue) until the
table is loadable AND listed (ready) or load_table raises (gone), bounded
by a wall-clock deadline, mirroring the proven BigLakeCatalogManager /
OneLakeCatalogManager pattern. Glue places each table in its own
namespace, so _namespace_for resolves the namespace from _tables_created.
Assertions and resolve_table_name's existing retry are unchanged.

(cherry picked from commit fb1000f)
The local wait_for_condition helper polled with a ceiling of 1.5 * max_age
(30s). On slow instrumented runners (sanitizers, coverage) basic S3Queue
ingestion of the test's files does not finish within 30s, so the first
wait_for_condition times out and fires its bare assert False. Observed in CI on
amd_llvm_coverage and amd_msan at the first wait, across PRs unrelated to
S3Queue; the ordered-mode failures have no tracked-file-ttl removal race, so
this is runner slowness rather than a correctness issue. Same root cause class
as the earlier max_age 10s->20s bump in ClickHouse#67035 (CPU overload on parallel
runners).

Scope the longer ceiling to slow builds only, per reviewer request: sanitizer
or coverage builds get 3 * max_age (60s); fast release/debug builds keep the
tighter 1.5 * max_age (30s) so a genuine timing regression is not masked. Build
flavor is detected with the existing node.is_built_with_sanitizer() /
node.is_built_with_llvm_coverage() helpers (same idiom as
test_jemalloc_global_profiler and test_trace_collector_serverwide). The change
is non-weakening: every wait still asserts the exact same condition and returns
as soon as it is met, so the passing-case runtime is unchanged; only a genuinely
stuck condition waits longer before asserting. max_age (tracked_file_ttl_sec) is
unchanged, so no ttl-expiry wait is lengthened.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 01e9253)
test_kafka_many_materialized_views reads each MV's target table with
instance.query_with_retry(check_callback=kafka_check_result). query_with_retry
returns the last query snapshot once its retry budget is exhausted, even when
check_callback never returned True. The Kafka -> MV insert flushes all 50 rows
to a view atomically (0 -> 50) at the end of a streamToViews cycle, and under
heavy load (the 4 historical failures over 180 days are all on sanitizer
builds: tsan/msan/asan) that flush can take longer than the default budget
(20 * 0.5s = 10s). view2 is polled after view1, so it has less time to catch
up and is the consistent victim: kafka_check_result(result2, True) asserts on a
short snapshot and the test fails.

Extend the retry budget for both views (retry_count=40, sleep_time=0.75),
matching the single-view test test_kafka_materialized_view_with_subquery just
above, so each view receives all rows before the assertion. No assertion is
weakened: both views must still contain all 50 reference rows.

Verified with a local reproduction: throttling view2's MV with sleepEachRow and
shrinking the retry budget makes result2 return 0 rows and the assertion fail;
the extended budget returns all 50 and passes. The unmodified test passes 6/6.

(cherry picked from commit ea12275)
`test_replication_without_zookeeper/test.py::test_startup_without_zookeeper`
intermittently fails with `kazoo.exceptions.NotEmptyError` inside `drop_zk`,
which calls `zk.delete(path="/clickhouse", recursive=True)`.

The drop happens while `node1` (ClickHouse) is still running, so its
`ReplicatedMergeTree` background threads keep re-creating ZooKeeper nodes under
`/clickhouse`. kazoo's recursive delete races with that: between `get_children`
and `delete` a new child node can appear, and kazoo raises `NotEmptyError`
without retrying.

The test already used `cluster.run_kazoo_commands_with_retries`, but with the
default `repeats=1`. In that helper the retry loop is `for i in range(repeats - 1)`,
so `repeats=1` performs zero retries — the callback runs exactly once with no
exception handling. Pass `repeats=5` (matching the existing usages in
`helpers/cluster.py`) so the drop is retried on the transient `NotEmptyError`.

This is a pre-existing flaky test (same failure seen on unrelated PRs, e.g.
ClickHouse#103540 and ClickHouse#101757), not a regression.

CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?REF=master&sha=d40ea5d0da4103b54971ac01a9960ef9153242bb&name_0=MasterCI&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20db%20disk%2C%20old%20analyzer%2C%203%2F6%29

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 1287728)
… to run query)

The Trino readiness probe (PR ClickHouse#105561) waits until system.runtime.nodes
reports an active node, but on the single-node Trino stack that table is
served coordinator-locally: it reports the coordinator active the instant it
registers with discovery, several seconds before the scheduler's worker node
map is populated. Distributed scans issued in that window fail with
NO_NODES_AVAILABLE ("No nodes available to run query"), the exact recurring
CI signature. The probe cannot observe the scheduler gap, and even a perfect
probe leaves a check-then-act window before the next query.

Retry _trino_exec on this one transient startup signature only; every other
error still fails fast so the tests keep their sensitivity. Reproduced
against trinodb/trino:474 standalone: the scan returns NO_NODES_AVAILABLE for
~3.5-4s after the node-count probe passes; with the retry it succeeds.

Related: ClickHouse#105524

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit b59a7dd)
The assertion that test_table_1 contains 100 rows on both replicas after
SYSTEM RESTORE DATABASE REPLICA used a bare count(*) polled by
query_with_retry (retry_count=20, sleep_time=0.5s, ~10s budget). After the
restore, the replica that did not originally hold the data must fetch all
parts in the background; on slow CI builds (coverage, msan, arm) the last
part can land after that fixed budget expires, so the poll returns 90 (9 of
10 PARTITION BY n % 10 parts) and the test fails.

Use the existing check_contains_table helper, which issues SYSTEM SYNC
REPLICA before checking the count. That waits for the background fetch to
finish with no artificial timeout, the same barrier every other count check
in this file already uses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit aa30a60)
The test waits for the async MUTATE_PART log entry (assigned by
mergeSelectingTask) to appear in system.replication_queue using a bounded
poll loop of param_tries={1..10} (up to 20s). Under heavy parallel CI load
the BackgroundSchedulePool can delay the assignment past that window, so the
loop gives up before the entry is created: the queue select then misses the
MUTATE_PART line and the ALTER DELETE row is never removed, producing the
observed result-differs failure.

Extend the three poll loops from {1..10} (20s) to {1..30} (60s, matching the
test's own receive_timeout=30) so assignment is tolerated under contention.
The loops early-terminate the moment the condition is met, so a normal run is
unaffected. Test intent and the reference output are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 24c098d)
`test_implicit_index_upgrade_alter_replay` creates a second replica `node2`
and asserts `SELECT count() FROM test_alter_replay` equals `10001` immediately
after `wait_for_active_replica`. But `wait_for_active_replica` only waits for
`is_readonly = 0`, not for the freshly-joined replica to finish fetching parts.

So `node2` can observe only the small `VALUES` part (1 row, `key = 99999`)
before the 10000-row part is fetched, and the assertion fails with
`assert '1' == '10001'`. This raced once each on several unrelated PRs over
the last 30 days (e.g. ClickHouse#108084, ClickHouse#99280, ClickHouse#107566, ClickHouse#107442).

Add `SYSTEM SYNC REPLICA test_alter_replay` after the replica becomes active,
which blocks until all parts are fetched, before checking the row count.

CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=108770&sha=bb760616d54e3d8e5d20968022085506c31b5f37&name_0=PR&name_1=Integration%20tests%20%28arm_binary%2C%20distributed%20plan%2C%201%2F4%29

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit b6dfe3b)
… tests

The test pre-warms the query condition cache with one query, then a second
query asserts PK analysis processed fewer marks than the total because the QCC
dropped granules. The QCC is server-level shared state (a bounded 100 MiB SLRU,
keyed by table_uuid + part_name + condition_hash); its size is a server setting,
not per-session, so a stateless test cannot size or pin its own entry.

Roughly 40 other stateless tests run in the parallel phase with
use_query_condition_cache = 1 and write to the same cache. On slow builds
(msan, WasmEdge) the gap between the pre-warm and the measured query stretches,
and cumulative writes from those concurrent tests size-evict 04051's small
probationary entry. The measured query then misses the cache, PK analysis
processes all 25 marks, and throwIf(processed >= total_marks) fires.

Tag the test no-parallel so it runs in the sequential phase with no concurrent
QCC writers, matching the existing QCC tests 04065 and 04275 (whose comment
notes the same "messes with the server-level query condition cache" reason).
The tag is orthogonal to the existing no-parallel-replicas fix (a different
mechanism), and the assertion is unchanged.

Verified deterministically on a debug build: with the entry resident the
measured query processes 14 marks (pass); after evicting it between the two
queries (SYSTEM DROP, or flooding a shrunk QCC with 60 distinct conditions) it
processes 25 marks and the assertion fires, reproducing the CI failure exactly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 9083e3a)
The test re-derived an expensive view over system.remote_data_paths
(a full metadata walk over all disks) for every assertion: 7 times for
the file-segment checks and 4 more for the filesystem_cache join. Under
thread fuzzer each scan is slow, so the repeated evaluation pushed the
test over the 600s per-test timeout (seen on amd_tsan, s3 storage runs).

Materialize each derived view into a Memory table once and read from it
in the assertions. This cuts the remote_data_paths walk from 7x to 1x
and the filesystem_cache scan from 4x to 1x, keeping the test output
byte-for-byte identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit e78459d)
The test config lesser_timeouts.xml shrinks backups.create_table_timeout
from its 300s production default to 3s purely to speed up the suite. On slow
sanitizer / old-analyzer / db-disk CI lanes, replicating the restored CREATE
TABLE DDL from the node that won the restore-coordination race to the other
nodes can take just over 3s, so the waiting node throws CANNOT_RESTORE_TABLE
("Couldn't restore table ... on other node or sync it") right at the 3s
boundary. Master (300s default) never hits this.

Bump create_table_timeout to 30s in both copies of lesser_timeouts.xml. The
wait loop returns the instant the table appears, so larger timeouts cost
nothing on success; 30s is still 10x below the production default and only
widens the deadline for the (untested) failure path, so bug detection is
unchanged. Same fix vitlibar applied in 2022 (1s -> 3s, b211dff); the
3s value has again become too small for instrumented lanes.

Affected tests (0 master failures, flaky only on slow lanes):
test_replicated_database_async, test_replicated_database,
test_replicated_database_compare_parts,
test_replicated_database_with_special_macro_in_zk_path,
test_table_in_replicated_database_with_not_synced_def, and the same tests in
the _with_checksum_data_file_name suite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 17e2ed5)
Cases 4 and 11 ran 'SELECT sleep(1.5), throwIf(1 = 1, ...)' as two sibling
projection columns. Their evaluation order is unspecified, so throwIf could
fire before sleep ran: the query then errored at ~0s elapsed, below the
--chime 1 threshold, and no BEL was emitted. Case 11 (pty) intermittently
printed 'no BEL' instead of the expected 'BEL'; the error message was still
present, which is the signature seen in CI (amd_tsan, arm_asan_ubsan), where
thread-fuzzer slowdown widens the race window.

Nest sleep inside throwIf's argument: 'SELECT throwIf(sleep(1.5) = 0, ...)'.
sleep(1.5) returns 0, so the predicate is true and throwIf still throws
'expected error', but only after the 1.5s sleep has run. Elapsed time is
therefore always >= 1.5s and the chime fires deterministically. The threshold
gate is unchanged (verified: --chime 10 still emits no BEL), so the test still
catches a threshold regression.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 2f86126)
The test restarts all three Keeper containers, then waits for them to come
back. stop_zookeeper_nodes() stops them serially, but start_zookeeper_nodes()
went through process_integration_nodes(), which ran one `docker compose start
<node>` process per node concurrently (ThreadPoolExecutor) against the same
compose project.

Concurrent `docker compose` invocations on one project race on the project's
shared state: on a loaded CI runner (integration tests run with -n workers), the
start of one node can be silently dropped. On the amd_msan failure the Keeper's
graceful stop overran docker's stop timeout and was SIGKILLed (exit 137); the
three concurrent `docker compose start` processes then fired within 2 ms of that
stop returning, and dockerd received no /start for zoo3 (it did for zoo1/zoo2).
zoo3 never came back, so wait_zookeeper_to_start() timed out after 180 s and
raised the generic "Cannot wait ZooKeeper container ... iptables-nft" exception.
The same latent race affected kill_zookeeper_nodes().

Use a single `docker compose <action> <node1> <node2> ...` invocation instead of
N concurrent ones. This is the pattern the framework already uses for startup
(a single `docker compose up -d`); compose parallelizes the services internally,
so there is no loss of parallelism and no shared-project race.

Verified via the CI report for PR ClickHouse#108074 (amd_msan 5/8): dockerd received
zoo3's container /json inspects but no /start, while zoo1/zoo2 got /start;
docker.log shows "zoo3-1 exited with code 137" with no subsequent restart.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 9f90342)
The "non-const args in tuple" section used today() in both the INSERT
(today() - number) and the SELECT ((u, d) IN ((0, today()), (1, today()))).
today() is evaluated independently at INSERT and SELECT time; if the wall
clock crosses midnight (in the randomized session timezone) between the two
client invocations, today()@select differs from today()@insert, so row 0 no
longer matches and count() returns 0 instead of 1 (rows_read stays 2, since
the primary key still selects the same granules). The 15-25x slowdown of the
amd_msan/WasmEdge build widens the INSERT->SELECT window and makes the
midnight straddle occasionally happen.

Replace today() with a fixed toDate('2020-01-01'). It is still a non-const
function expression (preserving the test's intent to cover non-const tuple
elements), but it is timezone-independent and stable across the two queries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit c5e22ee)
The numbers_mt real-time profiler probe inherited the 100ms period
(query_profiler_real_time_period_ns = 1e8) from the sleep probe above it.
The count() over numbers_mt(1e9) finishes in tens of milliseconds, so on
fast hardware the whole multi-threaded scan can complete before the 100ms
per-thread timer (first fire randomized in [0, period]) ever fires. That
yields 0 Source samples and the assertion on line 4 flips from 1 to 0.

Set a 1ms period for that probe so the fast scan reliably spans several
profiler periods. The sleep probe (500ms >> 100ms) and the CPU-time probe
are unchanged. Reproduced on a debug build: without the change the numbers_mt
probe returns 0 samples in ~7% of runs; with it, 60/60 runs pass and the
probe captures hundreds of samples.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit aa8dda7)
Take the minimum wall-clock over 5 runs per side instead of a single sample.
Timing noise is one-sided (a run can only be slower than the true CPU-bound
planning latency, never faster), so one unlucky scheduler/allocator stall under
parallel CI load inflated the single sample past the 20x ratio with no real
regression. The minimum discards transient stalls; the query, data and 20x
threshold are unchanged, so the ~100x-200x ClickHouse#32465 regression signal is preserved.

Pin compile_expressions=0 and compile_aggregate_expressions=0 on the timed
queries. Timing the same query 5 times crosses min_count_to_compile_*=3, so
without this the recorded minimum would come from a JIT-compiled (warmed) run
and measure a different execution mode than the cold planning path this test
guards. Pinning JIT off keeps all 5 samples the identical cold workload.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 0c3e7fd)
The test intermittently failed with 'Timeout waiting for query to start' on
loaded runners.

The ALTER used sleepEachRow(3) over 100 rows to keep the mutation running
long enough to observe and kill. But the whole part is read as a single block,
so sleepEachRow requests 3s * 100 = 300s of sleep up front, which exceeds
function_sleep_max_microseconds_per_block (default 3s) and the mutation throws
error 341 almost immediately. Query-level SETTINGS on the ALTER do not
propagate to the background mutation, so the cap cannot be raised there. The
ALTER therefore only lived a few seconds instead of ~300s, and a single slow
poll in wait_for_query_to_start could miss that window and hit its timeout.

Stop merges before issuing the ALTER so the mutation entry is created but never
executed. The mutation stays incomplete, so with mutations_sync=1 the ALTER
blocks in waitMutationToFinishOnReplicas (the code path fixed by ClickHouse#97589) until
KILL QUERY cancels it. This makes the test deterministic and CPU-independent:
it no longer relies on a long-running or memory-heavy mutation, and leaves no
in-flight background work to stall teardown.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 925fb1c)
groeneai and others added 14 commits July 21, 2026 12:22
…ttls

The test created 10 TTL nodes spaced only 1000ms apart and, after
wait_nodes_gone() for one node, asserted the next one is still alive. The
margin between consecutive nodes' destroy_time was ~1s, which under load can
be consumed by wait_nodes_gone overshoot (leader GC period, Raft commit,
50ms poll granularity, client round-trips), so the survivor was already
garbage-collected by the time it was checked. Widen the per-node TTL step to
3000ms so every assertion keeps a margin well above worst-case overshoot.
The Keeper TTL engine is correct (a node is never removed before its
destroy_time, re-validated at commit); this is purely test-timing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 8a01972)
The test runs its client queries at the CI-default send_logs_level=warning.
When QuotaCache::chooseQuotaToConsume() recomputes quotas slowly under load
(>=1000ms), it emits a Warning-level log that streams onto the client's
stderr, and the runner flags any non-empty stderr as a failure. The stdout
assertions are unaffected, so reruns pass once the recompute is fast again.

Lower the client log level to error for this test so transient server
warnings cannot leak onto stderr. The assertions only depend on stdout and
on the QUOTA_EXCEEDED exception text, so this does not weaken the test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 610cd68)
StorageObjectStorage caches supportsPrewhere() at table-construction time
from input_format_parquet_use_native_reader_v3 (Parquet's prewhere-support
checker returns that setting's value). The test creates an Iceberg table with
format Parquet and relies on PREWHERE, but the CREATE TABLE block did not pin
the setting, so a randomized session with native_reader_v3=0 baked
supports_prewhere=false into the storage object. Subsequent SELECT ... PREWHERE
then failed at analysis time with ILLEGAL_PREWHERE, even though the SELECTs
pinned native_reader_v3=1 (too late, the storage was already constructed).

On master the setting is obsolete and removed from clickhouse-test
randomization, so the test never sees native_reader_v3=0 there; on the 26.3 and
26.4 release branches the setting is still live and still randomized, so the
test failed deterministically about 1 in 6 runs in ReleaseBranchCI and every
backport PR. Pin native_reader_v3=1 in the CREATE TABLE block, matching the
sibling test 04147_iceberg_orc_schema_evolution_row_policy. No-op on master.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 4aef3f8)
test_failed_commit_after_success arms the process-global ONCE failpoint
object_storage_queue_fail_commit_after_success and waits up to 100s for
its own background commit to hit it. But sibling tests in the same file
(test_deduplication[unordered], test_deduplication_with_multiple_chunks
[unordered]) use after_processing='keep' + s3queue_tracked_file_ttl_sec=1,
so their S3Queue tables keep re-processing and committing forever, and
those tests never drop their tables. The autouse teardown reset only the
'instance' and 'instance2' databases, not 'instance_without_keeper_fault_
injection' (the instance all three tests share), so a leftover streamer
survived across tests. When it committed during the ~1s window the target
had the ONCE failpoint armed, the leftover consumed the failpoint first;
the target then committed successfully and its wait timed out with
"Failpoint was not triggered". This is why the failure clustered on the
slow amd_tsan build and never reproduced in isolation.

Reset the default database on instance_without_keeper_fault_injection in
the autouse fixture too, so each test starts with no leftover streamers.

Reproduced deterministically (a continuously-committing leftover steals
the failpoint: fails without the reset, passes with it).

(cherry picked from commit ff85e04)
The test asserts an exact distributed query plan via EXPLAIN. When CI randomization
enables statistics (use_statistics + materialize_statistics_on_insert +
auto_statistics_types), the cost-based distributed planner reads column NDV from the
statistics estimator, which changes the estimated group count and flips the aggregation
strategy from Shuffle to partial+merge (see makeDistributed.cpp tryMakeDistributedAggregation
and estimateReadRowsCount, gated on use_statistics). The asserted plan then diverges.

Pin use_statistics = 0 so the plan is deterministic regardless of the statistics
randomization. Mirrors 03357_join_pk_sharding and 03279_join_choose_build_table, which
pin the same setting for the same reason.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 03765d1)
The test drives dict0, a config-based PostgreSQL dictionary with
LIFETIME(1) and an invalidate_query, and asserts that changing a row
other than the invalidate row (id != 0) does not reload the dictionary.

SYSTEM RELOAD DICTIONARY performs a forced full reload but does not run
the invalidate query, so the stored invalidate response is left empty.
The first periodic check after the forced reload therefore always sees a
"modified" source and reloads once more. If that follow-up reload lands
after the test has changed the other rows (value * 2 WHERE id != 0), the
dictionary picks up value 2 for id = 1 and the "no update should happen"
assertion fails with '2' == '1'. On slow sanitizer / old-analyzer runs
the periodic check (every 5s) lines up with the fixed sleep(5) window
often enough to fail intermittently. It never fails on master.

Wait for the invalidate baseline to settle (last_successful_update_time
stops advancing across a full check period) after each reload, and verify
the "no update" case directly by asserting last_successful_update_time
does not advance. This drains the guaranteed post-forced-reload reload
before the assertion and keeps the original intent: a change to the
invalidate row triggers a reload, a change to other rows does not.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 01abbd0)
Same fix as the sibling _with_aggregation in this PR, applied to the _with_in
variant. The test asserts an exact distributed query plan via EXPLAIN. When CI
randomization enables statistics (materialize_statistics_on_insert +
non-empty auto_statistics_types), the cost-based distributed planner reads
column NDV from the statistics estimator (estimateReadRowsCount, gated on
use_statistics in optimizeJoin.cpp), which changes the estimated group count
and can flip the aggregation strategy from Shuffle to partial+merge. The
asserted plan then diverges.

The sibling _with_aggregation has a CI --diagnose-random-settings verdict
confirming this exact culprit (materialize_statistics_on_insert True,
auto_statistics_types tdigest,minmax,basic). _with_in is structurally identical
(same table, same distributed settings, same EXPLAIN plan-shape assertion; only
the subquery predicate differs) and shares the same use_statistics-gated code
path, so it carries the same latent flake.

Pin use_statistics = 0 so the plan is deterministic regardless of the statistics
randomization. Verified the EXPLAIN output is byte-identical with use_statistics
0 vs 1 and matches the committed .reference, so the pin does not alter what the
test asserts. Mirrors 03357_join_pk_sharding and 03279_join_choose_build_table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit da40604)
test_retry_loading_parts uses fail_request(cluster, 5) without a method
filter. On the module-shared cluster, background non-GET S3 requests
(cleanup, or metadata writes such as the table_identity.json write added
by another PR on the table-path-init path during ATTACH) can occupy the
fail counter window, so the injected 500 lands on a non-retryable write
under the s3_no_retries policy instead of the intended retryable
part-loading GET. That produces a hard Code 499 (S3_ERROR) at ATTACH
rather than the expected retry.

Add a GET method filter, consistent with the existing PUT method filters
in test_write_failover and test_move_failover in the same file, so only
the part-loading read is targeted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit eb79a6d)
…resses

test_rabbitmq_mv_combo used a fixed 180s wall-clock deadline for the
row-count check. Consuming ~1 mln rows takes ~125s under thread sanitizer
(8-13k rows/s) and can exceed the deadline on a loaded runner while
consumption is still steadily progressing, causing a spurious
pytest.fail at test.py:785.

Reset the no-progress deadline whenever the total row count across the 5
materialized views increases, so the test only fails on a genuine stall.
This mirrors the check_expected_result_polling helper already used by the
other RabbitMQ streaming tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 6edf3b2)
The test relies on the automatic whole-partition force merge
(`min_age_to_force_merge_on_partition_only`) reducing the number of
active parts within a bounded wait. Under heavy parallel load two effects
could starve that merge past the wait window, leaving an extra part and a
result differing from the reference (`3` instead of `2` in the "disable
limit" case):

- `getBestPartitionToOptimizeEntire` skips the whole-partition merge while
  the server-wide background merge pool has fewer free slots than
  `number_of_free_entries_in_pool_to_execute_optimize_entire_partition`
  (default 25). On a busy runner this gate keeps firing.
- Every merge-selection round that selects nothing multiplies the
  merge-selecting sleep by `merge_selecting_sleep_slowdown_factor` up to
  `max_merge_selecting_sleep_ms` (default 60s), so retries become sparse.

Set `number_of_free_entries_in_pool_to_execute_optimize_entire_partition=1`
and `max_merge_selecting_sleep_ms=1000` on the partition-only tables - the
same guards the analogous test 03357_replacing_min_age_cleanup already uses.

Closes: ClickHouse#107506

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit e1d7ad5)
node2 crashed on startup while loading metadata: the disk_s3_common
startup access check (IDisk::startup -> checkAccess) got a transient
403 AccessDenied from MinIO (SigV4 time-skew under concurrent container
startup), which is fatal during metadata load, so the server aborted
and its TCP port never opened. cluster.start() then timed out waiting
for node2 and all 18 module tests ERRORed with 'Timed out while waiting
for instance node2 to start'. node1/node3 (no 403) started fine, so it
is not shard-wide OOM.

Add skip_access_check to the S3 disks (disk_s3_common, disk_s3_with_cache),
matching test_storage_delta_disks which already skips the access check on
all its disks. The access check is not what these tests exercise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit aac9fbc)
Test 7 reused the same query_id for the TCP 'set opentelemetry_start_trace_probability=1'
query and the subsequent HTTP SELECT. Under parallel CI load the two queries can overlap,
producing Code 216 QUERY_WITH_SAME_ID_IS_ALREADY_RUNNING.

Additionally, check_http_attributes queried the 'query' span for the query_id, but the
http.referer / http.user.agent / http.method attributes live on the HTTPHandler (SERVER)
span, which carries no clickhouse.query_id. The old substring check matched the JSONEachRow
column-name keys, so it always reported the attributes as present regardless of value,
i.e. it never actually verified them.

Give the HTTP query its own query_id, send a traceparent header (so the request is traced)
plus Referer / User-Agent headers, and verify the http.* attribute VALUES on the HTTPHandler
span (matched via clickhouse.uri). The reference output is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 6944c75)
The two K=16 positive-case budgets (5230000 for UInt32, 5900000 for
UInt64) left effectively zero headroom: CI failures showed the tracked
peak landing exactly at the budget ("would use 4.99 MiB, maximum: 4.99
MiB" and "5.63 MiB, maximum: 5.63 MiB") on arm builds. With
memory_profiler_step=1 every allocation is flushed to the query tracker,
so allocator/arena rounding variance intermittently tips these two cases
over.

Give the two flaky K=16 positive budgets ~20% headroom over the observed
peak (6300000 and 7100000). The negative-case budgets (2000000) are left
unchanged, so the assertions still catch any regression that inflates the
HLL state. The K=12 and K=18 budgets already have sufficient margin and
are untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 5e662b6)
The cleanup renames back to num2 run ON CLUSTER (async distributed DDL plus
async ReplicatedMergeTree metadata replication) with ignore_exception=True, so
a failed or late rename is swallowed. The strict final selects then run with no
wait for schema convergence. A distributed SELECT for num2 forwarded to a shard
replica whose local _replicated metadata has not yet converged (still foo2/foo3)
fails with UNKNOWN_IDENTIFIER at the analyzer. This is correct engine behaviour
for per-shard schema during async metadata replication; the test asserted before
its precondition held.

Replace the fixed cleanup renames with wait_for_rename_to_num2, which re-drives
the idempotent ON CLUSTER rename-back and polls system.columns until num2 is
present on every node before the strict selects, repairing both a swallowed
rename and replication lag.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 51eea39)
@strtgbb strtgbb added cicd Improvements and fixes to the CICD process antalya-26.6 labels Jul 21, 2026
@github-actions

Copy link
Copy Markdown

Workflow [PR], commit [a478260]

@strtgbb strtgbb changed the title Backport flaky-fix commits from upstream (2026-07-21) Antalya 26.6 - Backport flaky-fix commits from upstream (2026-07-21) Jul 21, 2026
@strtgbb
strtgbb merged commit 1055292 into antalya-26.6 Jul 22, 2026
363 of 403 checks passed
@strtgbb strtgbb added the verified Approved for release label Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antalya-26.6 cicd Improvements and fixes to the CICD process verified Approved for release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants