Fix get_data livelock in protocol_transaction_out_106#1070
Conversation
|
The subscription drop is the mechanism that prevents handling additional messages of the same protocol while one is already being processed. The intent is to resubscribe once the message is fully handled, and without gapping the strand after the last send. The ensures that the caller may feed another request following receipt of the last request response without it getting dropped in a resubscribe gap. Removing this behavior may fix the observed behavior, but unless the message is handled synchronously (which requests for multiples will not be, leaving the subscription open is a DoS bug. |
02d6ef1 to
471139d
Compare
|
Revised. The drop is the backpressure; the earlier On "without gapping the strand": the gap is one strand turn -- a request following the last response arrives a round-trip later, a pipelined one is ordered after the resubscribe by strand FIFO (instrumented: strict DISPATCH,RESUB, zero drops); flood test confirms memory stays bounded. An alternative that removes the gap entirely (converge on |
|
Yes, this looks right. The POST is to the strand, so the async loop starts iteration on the strand, with the subscription having been already dropped at by the time the loop starts. The loop sends from the strand (as required) and handles each send by sending another or resubscribing. Since the handler is queued immediately upon socket send, a request having been received by the peer as a response to the final send cannot be missed. And a request initiated by the peer prior to the send may be dropped (as intended). The bug was that the resubscription could occur within the same function call as the subscription handler, in the case where no transaction was actually sent. |
Summary
A block-only
get_data(no tx items) sent to a synced node makeshandle_receive_get_dataspin a worker thread indefinitely.Mechanism
The
get_datasubscription is anunsubscriber<get_data::cptr>over astd::list;notifyiteratesif (!(*it)(...)) it = queue_.erase(it);.handle_receive_get_datareturnsfalse(drop), and during that same callsend_transaction's completion branch resubscribes (SUBSCRIBE_CHANNEL->push_back) into the list being iterated.erasereturns the just-pushed node, so the sameget_datais re-delivered to the handler forever. This is the existing// BUGBUG: registration race.site. Reachable on a default node (enable_relaydefaults true, bip37): one block-onlyget_datawedges a worker thread and N peers exhaust the pool.Fix (revised per review)
The earlier
return truewas wrong: it left the subscription open, allowing unbounded concurrentsend_transactionwalks (memory-exhaustion DoS). The subscription drop is intentional backpressure and is kept. This revision changes one thing -- the initial send is wrapped inPOST, sosend_transactionruns on a later strand turn, outsidenotify.return falseand the resubscribe are unchanged, so drop-while-processing backpressure is preserved verbatim; the completion resubscribe simply no longerpush_backs into the listnotifyis iterating.On "resubscribe ... without gapping the strand after the last send": the deferral is one strand turn. A request fed following receipt of the last response arrives a network round-trip later, long after the resubscribe has run, so it is not dropped. A request already pipelined ahead of the response is ordered after the resubscribe by strand FIFO (verified below).
Verification (testnet3, current master)
handle_receive_get_data.get_datax 100 tx -> patched stays memory-flat (bounded, drops overlap); a stay-subscribedreturn truebuild serves all 200k and grows RssAnon ~8x.DISPATCH,RESUBordering, zero drops under a request -> response -> request pipelined probe -- the one-turn window is not enterable.Alternative
If you'd prefer these out-protocols converge on
protocol_block_out_106-- stay subscribed with a single-in-flight guard rather than desubscribe/resubscribe, eliminating the gap entirely -- that is a ~4-line variant I can substitute. Kept this minimal deliberately.Note: the underlying
unsubscriber::notifynotifywill re-enter any handler that resubscribes during its own notification, silently and unbounded;protocol_filter_out_70015makes the same mistake independently (#1071). Worth considering whethernotifyshould assert against a mid-notify resubscription, so future misuse fails fast instead of hanging a thread. Raised as a question, not addressed here.