Skip to content

[Fix] Protect TimescaleDB first boot with startupProbe#304

Merged
royischoss merged 3 commits into
mlrun:developmentfrom
tal-haim:fix/CEML-723-timescaledb-startup-probe
Jul 15, 2026
Merged

[Fix] Protect TimescaleDB first boot with startupProbe#304
royischoss merged 3 commits into
mlrun:developmentfrom
tal-haim:fix/CEML-723-timescaledb-startup-probe

Conversation

@tal-haim

@tal-haim tal-haim commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

📝 Description

Stops TimescaleDB from being killed by the liveness probe during first-boot initdb on slow storage.

Without a startupProbe, Kubernetes can interrupt initialization and leave a corrupt PGDATA (missing global/pg_filenode.map), causing a permanent crash loop that breaks model-monitoring flows.

This change adds a configurable TimescaleDB startupProbe and forces probe checks over TCP (-h 127.0.0.1).


🛠️ Changes Made

  • charts/mlrun-ce/templates/timescaledb/statefulset.yaml: added startupProbe; added -h 127.0.0.1 to startup/liveness/readiness probes; removed initialDelaySeconds from liveness/readiness (gated by startup probe)
  • charts/mlrun-ce/values.yaml: added timescaledb.startupProbe defaults (periodSeconds: 10, timeoutSeconds: 5, failureThreshold: 30 → ~5 minute startup window)
  • charts/mlrun-ce/Chart.yaml: bumped version 0.12.0-rc.40.12.0-rc.5

✅ Checklist

  • I have tested the changes in this PR
  • I confirmed whether my changes require a change in documentation and if so, I created another PR in MLRun for the relevant documentation.
  • I confirmed whether my changes require changes in QA tests, for example: credentials changes, resources naming change and if so, I updated the relevant Jira ticket for QA.
  • I increased the Chart version in charts/mlrun-ce/Chart.yaml.
  • I confirmed that the installation works both on a local Docker Desktop environment and on a real cluster when using the required prerequisites.
    • If installation issues were found, I updated the relevant Jira ticket with the issue and steps to reproduce, or updated the prerequisites documentation if the issue is related to missing or outdated prerequisites.
  • If needed, update https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/README.md with the relevant installation instructions and version Matrix.
  • If needed, update the following values files for multi namespace support:

🧪 Testing

  • helm lint charts/mlrun-ce — passed
  • ./tests/helm-template-test.sh — 82/82 passed
  • Full cluster make tests / Kind install — not run
  • Manual reproduction of mid-initdb kill failure mode was documented on the Jira ticket

🔗 References

  • Ticket link:
  • External links:
  • Design docs links (Optional):

🚨 Breaking Changes?

  • Yes (explain below)
  • No

🔍️ Additional Notes

Install-mode values files were not updated because timescaledb.startupProbe is a default inherited from values.yaml; admin install keeps timescaledb.enabled: false, and non-admin overlays do not need a probe override.

Jenkins deploy health-gate for TimescaleDB (ticket item 3) still lives in ce-deployment and is out of scope for this PR.

tal-haim and others added 2 commits July 13, 2026 21:50
This update introduces a startup probe for the TimescaleDB component, enhancing the readiness checks during deployment. The configuration includes parameters for period, timeout, and failure threshold, ensuring better management of the database's initialization process.
Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions Bot added the bug Something isn't working label Jul 13, 2026
@tal-haim
tal-haim requested a review from alxtkr77 July 14, 2026 17:03

@royischoss royischoss left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey looks good to me small comment

Comment thread charts/mlrun-ce/values.yaml Outdated
Clarified the purpose of the failureThreshold parameter in the TimescaleDB startup probe configuration, indicating that it represents a 5-minute window for initdb on slow storage.

@royischoss royischoss left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@royischoss
royischoss merged commit 08489a8 into mlrun:development Jul 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants