Fix sample_weighted excluding items after the first amount infinite weights#1812
Closed
teddytennant wants to merge 1 commit into
Closed
Fix sample_weighted excluding items after the first amount infinite weights#1812teddytennant wants to merge 1 commit into
amount infinite weights#1812teddytennant wants to merge 1 commit into
Conversation
… weights In sample_efraimidis_spirakis, an infinite weight produces the key ln(r) / inf = -0.0. Once the reservoir's minimum key is -0.0, the skip value x = ln(r) / -0.0 = inf, so x -= weight leaves x at inf for finite weights and NaN for infinite ones, and the replacement branch is never taken again: the first `amount` infinite-weight items win deterministically and every later item is sampled with probability 0. Divert infinite-weight items into a separate uniform reservoir (Algorithm R) that takes priority over the A-ExpJ heap of finite weights. Sampling among finite weights is unchanged.
Member
|
In the case that some input weights overflow to |
1 task
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CHANGELOG.mdentrySummary
seq::index::sample_weighted(and thereforesample_weightedon slices) never samples any item that appears after the firstamountitems with infinite weight. Those firstamountinfinite-weight items win deterministically; every later item — finite or infinite weight — is selected with probability 0.Motivation
In
sample_efraimidis_spirakis, an infinite weight produces the keyln(r) / inf = -0.0. Once the reservoir's minimum key is-0.0, the skip valuex = ln(r) / -0.0 = inf, sox -= weightleavesxatinffor finite weights andNaNfor infinite ones, and the replacement branch is never taken again:Infinity is otherwise accepted as a weight (only
w < 0and NaN are rejected), so this silently returns the least likely items instead of the most likely ones.Details
Infinite-weight items cannot be represented by A-ExpJ keys (any such key is
-0.0), so they are diverted into a separate uniform reservoir (Algorithm R) that takes priority over the heap of finite-weight candidates. The resulting semantics, now documented on bothsample_weightedvariants: items with infinite weight have priority over items with finite weight and are sampled uniformly among themselves. Sampling among finite weights is unchanged, and the case of at mostamountinfinite weights appearing early keeps its old behavior (those items were and are always selected).Tests:
seq::index::test_sample_weighted_infinite: all-infinite weights are sampled uniformly (count check over 1000 runs); infinite weights dominate finite ones; and the failing case above — infinite weights appearing afteramountfinite ones — now excludes the finite items and picks uniformly among the infinite ones.seq::slice::test_multiple_weighted_edge_cases: added a case with more infinite weights thanamountthrough the slice API.cargo test --features std,cargo fmt --checkandcargo clippy --features std,alloc --libare clean.