RATIS-2512. Fix unreleased zero-copy messages during gRPC service shutdown.

[slfan1989]

What changes were proposed in this pull request?

This pull request fixes a zero-copy request leak during gRPC service shutdown.

ZeroCopyMessageMarshaller may still hold pending or in-flight zero-copy request streams when the gRPC service is being closed. Previously, closing these tracked streams did not trigger the release logic for the corresponding zero-copy messages, which could cause ZeroCopyMetrics to report unreleased messages during cluster shutdown.

The proposed changes:

• Release tracked zero-copy messages when ZeroCopyMessageMarshaller.close() is called.
• Add close logic for gRPC protocol services that own zero-copy request marshallers.
• Explicitly close these protocol services in GrpcServicesImpl.closeImpl() before unregistering zero-copy metrics.
• Ensure the same GrpcServerProtocolService instance bound to the gRPC server is also closed during shutdown.

What is the link to the Apache JIRA

RATIS-2512. Fix unreleased zero-copy messages during gRPC service shutdown

How was this patch tested?

• This patch was tested with the following commands:

./mvnw -pl ratis-test -am -Pgrpc-tests -Dtest=TestGrpcZeroCopy,TestRaftAsyncWithGrpc#testWithLoadAsync test

• Both tests passed successfully.

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.ratis.grpc.TestRaftAsyncWithGrpc
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.141 s - in org.apache.ratis.grpc.TestRaftAsyncWithGrpc
[INFO] Running org.apache.ratis.grpc.util.TestGrpcZeroCopy
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.272 s - in org.apache.ratis.grpc.util.TestGrpcZeroCopy
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------

[szetszwo]

+1 the change looks good.

[szetszwo]

This is good catch! We never call zeroCopyRequestMarshaller.close() except for tests.

[github-actions]

This PR has been marked as stale due to 60 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in ~30 days.

[spacemonkd]

@szetszwo I think this is good to merge as well?

@slfan1989 could you please address the test failures in CI?

[szetszwo]

@spacemonkd , we need to wait for passing all the tests.

[spacemonkd]

Yup, let's wait for @slfan1989 to address the test failures

[slfan1989]

Yup, let's wait for @slfan1989 to address the test failures

Thanks @spacemonkd and @szetszwo.

Since this PR is based on the old RATIS-1931_grpc-zero-copy branch, would it be okay to first complete the zero-copy migration onto the current master?

Some parts of the old branch have diverged significantly from master, which may also cause compatibility issues and flaky test failures. Addressing these failures directly on the old branch may therefore require changes that will become obsolete after the migration.

After the new branch is ready, I will reapply the RATIS-2512 fix there, rerun the relevant tests against the current codebase, and open a replacement PR.

The migration approach is being discussed here:

https://lists.apache.org/thread/n9h35mvqor5dl8vpdfnngvqjwl5bx133

I would appreciate your thoughts on whether this approach sounds reasonable before I proceed. Thanks!

cc: @adoroszlai