variadics v0.2.0-alpha.2

New Features (BREAKING)

• place Tokio dependencies behind a feature flag
  Breaking Changes:
  • the hydro_lang::telemetry::emf module now requires the non-default
    telemetry_emf feature

Commit Statistics

• 1 commit contributed to the release.
• 32 days passed between releases.
• 1 commit was understood as conventional.
• 1 unique issue was worked on: #2960

Commit Details

view details

• #2960
  • Place Tokio dependencies behind a feature flag (dbdd2f1)

multiplatform_test v0.7.1-alpha.0

Documentation

• fix lots of small docs issues

Commit Statistics

• 1 commit contributed to the release.
• 73 days passed between releases.
• 1 commit was understood as conventional.
• 1 unique issue was worked on: #3012

Commit Details

view details

• #3012
  • Fix lots of small docs issues (36830af)

lattices v0.8.0-alpha.3

Documentation

• fix lots of small docs issues

New Features (BREAKING)

• place Tokio dependencies behind a feature flag
  Breaking Changes:
  • the hydro_lang::telemetry::emf module now requires the non-default
    telemetry_emf feature

Commit Statistics

• 2 commits contributed to the release.
• 21 days passed between releases.
• 2 commits were understood as conventional.
• 2 unique issues were worked on: #2960, #3012

Commit Details

view details

• #2960
  • Place Tokio dependencies behind a feature flag (dbdd2f1)
• #3012
  • Fix lots of small docs issues (36830af)

hydro_std v0.17.0-alpha.4

Chore

• bump Stageleft version

Documentation

• fix lots of small docs issues

New Features (BREAKING)

• place Tokio dependencies behind a feature flag
  Breaking Changes:
  • the hydro_lang::telemetry::emf module now requires the non-default
    telemetry_emf feature

Commit Statistics

• 3 commits contributed to the release.
• 21 days passed between releases.
• 3 commits were understood as conventional.
• 3 unique issues were worked on: #2960, #3012, #3025

Commit Details

view details

• #2960
  • Place Tokio dependencies behind a feature flag (dbdd2f1)
• #3012
  • Fix lots of small docs issues (36830af)
• #3025
  • Bump Stageleft version (a892c50)

hydro_lang v0.17.0-alpha.4

Chore

• bump Stageleft version

• update nightly snapshots
  [ci-full]

  This PR was automatically generated by the nightly CI run. Updated insta
  snapshots for nightly Rust builds. Please review the snapshot changes to
  ensure they are expected.

• bump Stageleft version

• make some rust,ignore blocks into rust so that they get test coverage

New Features

• add an embedded serialization type for network channels

• allow for_each on unordered/retried streams with annotations
  Changed Stream::for_each from requiring O: IsOrdered + R: IsExactlyOnce to
  using ValidCommutativityFor<O> / ValidIdempotenceFor<R> with
  AggFuncAlgebra<C, I>,
  matching the pattern used by fold and reduce.

  • On TotalOrder + ExactlyOnce streams, no annotations needed (backward
    compatible).
  • On unordered streams, developer provides commutative = ... proof.
  • On streams with retries, developer provides idempotent = ... proof.
  • Added maybe_observe_for_mut in the ForEach IR emit path for mut ref
    handling.
  • Added compile-fail test verifying the error on NoOrder streams without
    annotation.

• support merge_ordered on KeyedStream

• parallel compilation with per-job target dirs and shared artifact symlinks [ci-full]
  Each compilation job gets its own --target-dir (under
  {target}/jobs/{name}) to
  avoid cargo's global artifact-dir lock, while sharing compiled artifacts
  via symlinks
  to .fingerprint, build, and deps in the shared target dir.

  A prebuild step compiles the dylib crate (or --lib for non-dylib) into
  the shared
  target dir before the parallel final builds start, ensuring all
  dependencies are ready.

  Also adds feature forwarding to the generated dylib crate's Cargo.toml.

• multi-versioning support in the simulator

• add features support to add_localhost_docker
  Added features field and .features() / .feature() builder methods
  to
  DockerDeployProcessSpec and DockerDeployClusterSpec, matching the
  API
  available on TrybuildHost. Features are threaded through to the
  RustCrate
  at compile time during instantiate().

Bug Fixes

• make sure legacy member IDs are available for sim

• support singleton reference captures in scan APIs
  Fixes #2993. The scan, scan_async_blocking, and
  generator combinators (on both Stream and KeyedStream) spliced
  their
  q!() closures without activating a reference-capture scope, so using
  by_ref() (singleton/optional/stream reference captures) inside a scan
  closure panicked with "no reference capture scope is active".

  The IR/codegen already supports singleton refs on HydroNode::Scan and
  HydroNode::ScanAsyncBlocking (their init/acc are ClosureExpr and
  transform_children already visits their singleton_refs); only the
  builder API failed to set up the capture scope.

• add Stageleft path rewrite for core::io::error [ci-full]

• eliminate test mode rewrites by setting up proper __staged override

• sim abort on run_tick() false, hooks not ready, instead of panic hidden by bolero
  Replace the silent assert with std::process::abort() so that if a tick
  DFIR's run_tick() unexpectedly returns false (indicating a waker
  registration issue), the process terminates immediately rather than
  being silently caught by bolero's catch_unwind.

  Add comment at top of compiled.rs telling people to use abort!() instead
  of panic!().

Refactor

• share compilation logic between Maelstrom and sim targets
  The Maelstrom deployment target previously compiled its generated binary
  with a
  bespoke, statically-linked cargo build --example invocation that
  recompiled the
  whole dependency tree on every run and had no concurrency management.
  The sim
  target already had a parallel-compilation pipeline (per-job target dirs
  with
  symlinked shared artifacts, plus a prebuilt dylib of the dependencies).
  This
  change makes Maelstrom reuse that same pipeline.

  • Extract the concurrent-build machinery out of sim's compile_sim into
    a shared
    compile_trybuild_example(ExampleBuildConfig) in
    compile/trybuild/generate.rs,
    parameterized over the runtime feature, example name, crate type
    (cdylib for
    sim vs. an executable for Maelstrom), the TRYBUILD_LIB_NAME
    indirection, and
    whether fuzzing is allowed.
  • Make LinkingMode::Dynamic, TrybuildConfig::linking_mode, and the
    dynamic
    examples-dir path available under the maelstrom feature (previously
    deploy-only).
  • compile_sim is now a thin wrapper over the shared function.
  • Maelstrom instantiates with LinkingMode::Dynamic, stores the full
    TrybuildConfig, and build() uses the shared builder, persisting the
    built
    executable via TempPath::keep.
  • The maelstrom feature now depends on hydro_concurrent_cargo and
    cargo_metadata.

• remove stratum from cross_singleton, defer_signal, difference, zip_longest

New Features (BREAKING)

• make Cluster::sim_input generic over ordering/retries and add SimClusterSender::send_many_unordered
  Fixes #2994.

  Previously Cluster::sim_input was hardcoded to
  TotalOrder/ExactlyOnce,
  and SimClusterSender only exposed send/send_many for that
  combination.
  This meant a cluster input that is semantically unordered (e.g.
  downstream of
  a NoOrder network channel) could not be created directly — tests had
  to
  create a TotalOrder input and then weaken_ordering, forcing a fixed
  send
  order and adding boilerplate.

• place Tokio dependencies behind a feature flag
  Breaking Changes:

  • the hydro_lang::telemetry::emf module now requires the non-default
    telemetry_emf feature

Commit Statistics

• 19 commits contributed to the release over the course of 21 calendar days.
• 21 days passed between releases.
• 19 commits were understood as conventional.
• 19 unique issues were worked on: #2945, #2953, #2956, #2960, #2972, #2973, #2975, #2978, #2982, #2984, #2990, #2992, #2995, #2996, #2997, #2998, #3001, #3002, #3025

Commit Details

view details

• #2945
  • Multi-versioning support in the simulator (8d9c29b)
• #2953
  • Allow for_each on unordered/retried streams with annotations (f3f0202)
• #2956
  • Make some rust,ignore blocks into rust so that they get test coverage (7e3a2f1)
• #2960
  • Place Tokio dependencies behind a feature flag (dbdd2f1)
• #2972
  • Remove stratum from cross_singleton, defer_signal, difference, zip_longest (394ed2a)
• #2973
  • Add features support to add_localhost_docker (fd9f454)
• #2975
  • Parallel compilation with per-job target dirs and shared artifact symlinks [ci-full] (c128c22)
• #2978
  • Sim abort on run_tick() false, hooks not ready, instead of panic hidden by bolero (0dea596)
• #2982
  • Eliminate test mode rewrites by setting up proper __staged override (5c3d672)
• #2984
  • Bump Stageleft version (94af11b)
• #2990
  • Update nightly snapshots (bbdbd9e)
• #2992
  • Add Stageleft path rewrite for core::io::error [ci-full] (71de292)
• #2995
  • Support merge_ordered on KeyedStream (92ae19e)
• #2996
  • Support singleton reference captures in scan APIs (966a22f)
• #2997
  • Make Cluster::sim_input generic over ordering/retries and add SimClusterSender::send_many_unordered (0bcfaa5)
• #2998
  • Share compilation logic between Maelstrom and sim targets (6df729b)
• #3001
  • Make sure legacy member IDs are available for sim (77270bc)
• #3002
  • Add an embedded serialization type for network channels (c8584b6)
• #3025
  • Bump Stageleft version (a892c50)

hydro_deploy v0.17.0-alpha.3

New Features

• parallel compilation with per-job target dirs and shared artifact symlinks [ci-full]
  Each compilation job gets its own --target-dir (under
  {target}/jobs/{name}) to
  avoid cargo's global artifact-dir lock, while sharing compiled artifacts
  via symlinks
  to .fingerprint, build, and deps in the shared target dir.

  A prebuild step compiles the dylib crate (or --lib for non-dylib) into
  the shared
  target dir before the parallel final builds start, ensuring all
  dependencies are ready.

  Also adds feature forwarding to the generated dylib crate's Cargo.toml.

Commit Statistics

• 1 commit contributed to the release over the course of 17 calendar days.
• 21 days passed between releases.
• 1 commit was understood as conventional.
• 1 unique issue was worked on: #2975

Commit Details

view details

• #2975
  • Parallel compilation with per-job target dirs and shared artifact symlinks [ci-full] (c128c22)

hydro_concurrent_cargo v0.1.0-alpha.0

Chore

• prepare for release

New Features

• parallel compilation with per-job target dirs and shared artifact symlinks [ci-full]
  Each compilation job gets its own --target-dir (under
  {target}/jobs/{name}) to
  avoid cargo's global artifact-dir lock, while sharing compiled artifacts
  via symlinks
  to .fingerprint, build, and deps in the shared target dir.

  A prebuild step compiles the dylib crate (or --lib for non-dylib) into
  the shared
  target dir before the parallel final builds start, ensuring all
  dependencies are ready.

  Also adds feature forwarding to the generated dylib crate's Cargo.toml.

Commit Statistics

• 2 commits contributed to the release over the course of 17 calendar days.
• 2 commits were understood as conventional.
• 1 unique issue was worked on: #2975

Commit Details

view details

• #2975
  • Parallel compilation with per-job target dirs and shared artifact symlinks [ci-full] (c128c22)
• Uncategorized
  • Prepare for release (b4da85e)

dfir_rs v0.17.0-alpha.4

Chore

• update nightly snapshots
  [ci-full]

  This PR was automatically generated by the nightly CI run. Updated insta
  snapshots for nightly Rust builds. Please review the snapshot changes to
  ensure they are expected.

• update nightly snapshots

Documentation

• add rust: link resolver so rustdoc links can't go stale
  Replaces brittle hardcoded rustdoc URLs (pathname:///rustdoc/...,
  https://hydro.run/rustdoc/...) with a clean rust: pseudo-scheme that
  is
  resolved and validated at build time:

  [`Stream`](rust:hydro_lang::live_collections::Stream)
  [`Stream::fold`](rust:hydro_lang::live_collections::Stream::fold)
  [`setup!`](rust:hydro_lang::setup!)
  

  New remark plugin (docs/src/remark/rustdoc-links.js):

  • Resolves full module paths against the actual compiled rustdoc HTML in
    docs/static/rustdoc (populated by build_docs.bash before the site
    build), so the item kind (struct/enum/trait/...) and .html layout are
    discovered automatically and every link is checked against real pages.
  • Follows rustdoc http-equiv=refresh redirect pages (generated for
    re-exports) to their destination.
  • Associated items (Type::method) are validated against the
    id="method.x" anchors in the generated page.
  • Same-name collisions within a module are an error; disambiguate with
    rustdoc's own intra-doc-link syntax: kind@ prefixes (struct@, enum@,
    trait@, fn@/method@, macro@, mod@, type@, value@, prim@, tyalias@, ...)
    or trailing () / ! suffixes, mirroring
    librustdoc/passes/collect_intra_doc_links.rs.
  • Path segments are validated as Rust identifiers (no ../separator
    traversal out of the rustdoc root).
  • Dev mode (no static/rustdoc): links degrade to rustdoc search URLs
    with
    a one-time warning. Production (NODE_ENV=production): missing rustdoc
    or
    any unresolvable/ambiguous link fails the build with a per-file error.
  • Registered via beforeDefaultRemarkPlugins (docs + pages) so
    Docusaurus'
    own link transform never sees the raw rust: URLs.

• fix lots of small docs issues

New Features

• re-add loop scopes
  Re-adds loop { ... } blocks in dfir_syntax! inline codegen with
  loop-aware partitioning, hierarchical codegen, and proper operator
  validation.

  WIP for #2902

  Operators

  Loop ingress (windowing):

  • batch() — windows data into a loop, triggers the loop to fire
  • batch_lazy() — windows data into a loop without triggering it; data
    is dropped if the loop doesn't fire that tick

  Cross-iteration/tick deferral:

  • defer_tick() — buffers data for the next time boundary. Adapts to
    context: defers to the next tick at top-level or in root-level loops,
    defers to the next loop iteration in nested loops. Non-lazy (causes
    re-fire).
  • defer_tick_lazy() — same as defer_tick() but lazy (does not cause
    re-fire or schedule a new tick).

  Loop egress (unwindowing):

  • all_iterations() — collects output from all loop iterations and
    emits outside the loop

  Key implementation details

  Partitioning & ordering:

  • DelayType::{Loop, LoopLazy} delay types used internally for
    loop-boundary edges
  • mark_tick_boundary_handoffs remaps DelayType::Tick → Loop when
    the consumer is in a nested loop, so a single defer_tick() operator
    works in all contexts
  • make_loops_contiguous: a linear-time rearrangement pass that groups
    loop subgraphs contiguously in the topological order
  • Loop-exit handoff buffers are declared outside the while loop so
    consumers can read accumulated data

  Inline codegen (meta_graph.rs):

  • Stack-based hierarchical code generation builds loop structure
    directly during subgraph iteration
  • emit_loop_gate helper wraps loop bodies:
    • Root-level loops: if gate (fire at most once per tick)
  • Nested loops: while gate (iterate to fixpoint, conditioned on entry
    handoff + non-lazy defer back-buffer data)
  • Loop-exit handoffs are hoisted to the parent level for proper scoping

  Validation (flat_graph_builder.rs):

  • Windowing operators must be at loop entry edges; unwindowing at loop
    exit edges
  • Loop bodies must be DAGs (excluding defer_tick back-edges)
  • Sources must be at root level

• derive traits for Persistence, PersistenceKeyed, and SparseVec

• add push accum operators (fold, reduce, etc)

Bug Fixes

• drain source_stream to register waker, fixing sim partitioning/panic sensitivity
  The simulator was sensitive to tick DFIR subgraph partitioning because
  source_stream didn't always register its waker with the underlying
  tokio channel. When cross_singleton (or any downstream operator) only
  pulled one item from the singleton stream without hitting Pending, the
  waker was never registered. This meant subsequent send() calls from
  hooks couldn't wake the tick DFIR, causing run_tick() to return false
  and the simulator to terminate branches early via silent panic (fixed
  in subsequent PR)

  The fix adds a write_iterator_after cleanup step to source_stream
  (mirroring source_iter's pattern) that drains remaining items from the
  stream after the subgraph's main pipeline runs. This ensures poll_next
  eventually returns Pending, registering the waker for future sends.

  ---

Refactor

• remove DelayType::Stratum, MonotoneAccum
  removes the variants and updates graph displaying, hence the snapshot
  updates
• remove stratum from cross_singleton, defer_signal, difference, zip_longest
• remove stratum for anti_join, join_fused[_lhs/_rhs], join_multiset_half, _lattice_join_fused_join
• remove stratum, add push codegen for lattice_fold, lattice_reduce, _lattice_fold_batch
• remove stratum, add push codegen, test persist_mut, persist_mut_keyed
• remove stratum, add push codegen, test sort, sort_by_key
• remove stratum, add push codegen, test reduce, reduce_keyed, reduce_no_replay
• remove stratum, add push codegen, test fold, fold_keyed, fold_no_replay

New Features (BREAKING)

• place Tokio dependencies behind a feature flag
  Breaking Changes:
  • the hydro_lang::telemetry::emf module now requires the non-default
    telemetry_emf feature

Commit Statistics

• 17 commits contributed to the release.
• 21 days passed between releases.
• 17 commits were understood as conventional.
• 17 unique issues were worked on: #2960, #2962, #2964, #2965, #2966, #2968, #2969, #2970, #2971, #2972, #2974, #2976, #2977, #2981, #2989, #3012, #3020

Commit Details

view details

• #2960
  • Place Tokio dependencies behind a feature flag (dbdd2f1)
• #2962
  • Add push accum operators (fold, reduce, etc) (79f7aa6)
• #2964
  • Derive traits for Persistence, PersistenceKeyed, and SparseVec (dea6378)
• #2965
  • Remove stratum, add push codegen, test fold, fold_keyed, fold_no_replay (7321ac2)
• #2966
  • Remove stratum, add push codegen, test reduce, reduce_keyed, reduce_no_replay (4b18fd7)
• #2968
  • Remove stratum, add push codegen, test sort, sort_by_key (9303058)
• #2969
  • Remove stratum, add push codegen, test persist_mut, persist_mut_keyed (a241ec4)
• #2970
  • Remove stratum, add push codegen for lattice_fold, lattice_reduce, _lattice_fold_batch (fd0ebc1)
• #2971
  • Remove stratum for anti_join, join_fused[_lhs/_rhs], join_multiset_half, _lattice_join_fused_join (62e8689)
• #2972
  • Remove stratum from cross_singleton, defer_signal, difference, zip_longest (394ed2a)
• #2974
  • Remove DelayType::Stratum, MonotoneAccum (5901045)
• #2976
  • Update nightly snapshots (41a9307)
• #2977
  • Drain source_stream to register waker, fixing sim partitioning/panic sensitivity (02630fc)
• #2981
  • Update nightly snapshots (07be721)
• #2989
  • Re-add loop scopes (1e136df)
• #3012
  • Fix lots of small docs issues (36830af)
• #3020
  • Add rust: link resolver so rustdoc links can't go stale (ede4924)

dfir_pipes v0.1.0-alpha.3

New Features

• add push accum operators (fold, reduce, etc)

Commit Statistics

• 2 commits contributed to the release over the course of 21 calendar days.
• 25 days passed between releases.
• 1 commit was understood as conventional.
• 1 unique issue was worked on: #2962

Commit Details

view details

• #2962
  • Add push accum operators (fold, reduce, etc) (79f7aa6)
• Uncategorized
  • Release dfir_macro v0.17.0-alpha.3, lattices v0.8.0-alpha.2, sinktools v0.2.0-alpha.3, hydro_deploy_integration v0.17.0-alpha.2, dfir_rs v0.17.0-alpha.3, hydro_deploy v0.17.0-alpha.2, hydro_lang v0.17.0-alpha.3, hydro_std v0.17.0-alpha.3 (295c0ec)

dfir_lang v0.17.0-alpha.3

New Features

• re-add loop scopes
  Re-adds loop { ... } blocks in dfir_syntax! inline codegen with
  loop-aware partitioning, hierarchical codegen, and proper operator
  validation.

  WIP for #2902

  Operators

  Loop ingress (windowing):

  • batch() — windows data into a loop, triggers the loop to fire
  • batch_lazy() — windows data into a loop without triggering it; data
    is dropped if the loop doesn't fire that tick

  Cross-iteration/tick deferral:

  • defer_tick() — buffers data for the next time boundary. Adapts to
    context: defers to the next tick at top-level or in root-level loops,
    defers to the next loop iteration in nested loops. Non-lazy (causes
    re-fire).
  • defer_tick_lazy() — same as defer_tick() but lazy (does not cause
    re-fire or schedule a new tick).

  Loop egress (unwindowing):

  • all_iterations() — collects output from all loop iterations and
    emits outside the loop

  Key implementation details

  Partitioning & ordering:

  • DelayType::{Loop, LoopLazy} delay types used internally for
    loop-boundary edges
  • mark_tick_boundary_handoffs remaps DelayType::Tick → Loop when
    the consumer is in a nested loop, so a single defer_tick() operator
    works in all contexts
  • make_loops_contiguous: a linear-time rearrangement pass that groups
    loop subgraphs contiguously in the topological order
  • Loop-exit handoff buffers are declared outside the while loop so
    consumers can read accumulated data

  Inline codegen (meta_graph.rs):

  • Stack-based hierarchical code generation builds loop structure
    directly during subgraph iteration
  • emit_loop_gate helper wraps loop bodies:
    • Root-level loops: if gate (fire at most once per tick)
  • Nested loops: while gate (iterate to fixpoint, conditioned on entry
    handoff + non-lazy defer back-buffer data)
  • Loop-exit handoffs are hoisted to the parent level for proper scoping

  Validation (flat_graph_builder.rs):

  • Windowing operators must be at loop entry edges; unwindowing at loop
    exit edges
  • Loop bodies must be DAGs (excluding defer_tick back-edges)
  • Sources must be at root level

Bug Fixes

• drain source_stream to register waker, fixing sim partitioning/panic sensitivity
  The simulator was sensitive to tick DFIR subgraph partitioning because
  source_stream didn't always register its waker with the underlying
  tokio channel. When cross_singleton (or any downstream operator) only
  pulled one item from the singleton stream without hitting Pending, the
  waker was never registered. This meant subsequent send() calls from
  hooks couldn't wake the tick DFIR, causing run_tick() to return false
  and the simulator to terminate branches early via silent panic (fixed
  in subsequent PR)

  The fix adds a write_iterator_after cleanup step to source_stream
  (mirroring source_iter's pattern) that drains remaining items from the
  stream after the subgraph's main pipeline runs. This ensures poll_next
  eventually returns Pending, registering the waker for future sends.

  ---

Refactor

• remove DelayType::Stratum, MonotoneAccum
  removes the variants and updates graph displaying, hence the snapshot
  updates
• remove stratum from cross_singleton, defer_signal, difference, zip_longest
• remove stratum for anti_join, join_fused[_lhs/_rhs], join_multiset_half, _lattice_join_fused_join
• remove stratum, add push codegen for lattice_fold, lattice_reduce, _lattice_fold_batch
• remove stratum, add push codegen, test persist_mut, persist_mut_keyed
• remove stratum, add push codegen, test sort, sort_by_key
• remove stratum from chain, chain_first_n
• remove stratum, add push codegen, test reduce, reduce_keyed, reduce_no_replay
• remove stratum, add push codegen, test fold, fold_keyed, fold_no_replay

Commit Statistics

• 11 commits contributed to the release over the course of 21 calendar days.
• 25 days passed between releases.
• 11 commits were understood as conventional.
• 11 unique issues were worked on: #2965, #2966, #2967, #2968, #2969, #2970, #2971, #2972, #2974, #2977, #2989

Commit Details

view details

• #2965
  • Remove stratum, add push codegen, test fold, fold_keyed, fold_no_replay (7321ac2)
• #2966
  • Remove stratum, add push codegen, test reduce, reduce_keyed, reduce_no_replay (4b18fd7)
• #2967
  • Remove stratum from chain, chain_first_n (4cad807)
• #2968
  • Remove stratum, add push codegen, test sort, sort_by_key (9303058)
• #2969
  • Remove stratum, add push codegen, test persist_mut, persist_mut_keyed (a241ec4)
• #2970
  • Remove stratum, add push codegen for lattice_fold, lattice_reduce, _lattice_fold_batch (fd0ebc1)
• #2971
  • Remove stratum for anti_join, join_fused[_lhs/_rhs], join_multiset_half, _lattice_join_fused_join (62e8689)
• #2972
  • Remove stratum from cross_singleton, defer_signal, difference, zip_longest (394ed2a)
• #2974
  • Remove DelayType::Stratum, MonotoneAccum (5901045)
• #2977
  • Drain source_stream to register waker, fixing sim partitioning/panic sensitivity (02630fc)
• #2989
  • Re-add loop scopes (1e136df)