Clean up illegal grpc connections to reduce useless logs

[solotzg]

Enhancement

• Even if the target address is invalid, the grpc connection will continue to reconnect and output relevant logs.

• For example: after a tikv is scaled in and the following duplicate logs are meaningless.

[2024/12/20 01:21:54.260 +00:00] [INFO] [Server.cpp:385] ["/workspace/source/tiflash/contrib/grpc/src/core/ext/filters/client_channel/subchannel.cc, line number: 945, log msg : subchannel 0x7ee64a727400 {address=ipv4:172.16.48.14:20160, args=grpc.client_channel_factory=0x7ee9c46c03c0, grpc.default_authority=db-tikv-7.db-tikv-peer.tidb10096900833626871784.svc:20160, grpc.http2_scheme=https, grpc.initial_reconnect_backoff_ms=1000, grpc.internal.channel_credentials=0x7ee9a6963b40, grpc.internal.security_connector=0x7ee44dbc7070, grpc.internal.subchannel_pool=0x7ee9c48d4190, grpc.max_receive_message_length=-1, grpc.max_reconnect_backoff_ms=3000, grpc.min_reconnect_backoff_ms=1000, grpc.primary_user_agent=grpc-c++/1.44.0, grpc.resource_quota=0x7ee9c48d42b0, grpc.server_uri=dns:///db-tikv-7.db-tikv-peer.tidb10096900833626871784.svc:20160}: connect failed: {\"created\":\"@1734657714.260802248\",\"description\":\"Failed to connect to remote host: FD Shutdown\",\"file\":\"/workspace/source/tiflash/contrib/grpc/src/core/lib/iomgr/lockfree_event.cc\",\"file_line\":217,\"os_error\":\"Timeout occurred\",\"referenced_errors\":[{\"created\":\"@1734657714.260791663\",\"description\":\"connect() timed out\",\"file\":\"/workspace/source/tiflash/contrib/grpc/src/core/lib/iomgr/tcp_client_posix.cc\",\"file_line\":114}],\"target_address\":\"ipv4:172.16.48.14:20160\"}"] [source=grpc] [thread_id=3645]
[2024/12/20 01:21:54.261 +00:00] [INFO] [Server.cpp:385] ["/workspace/source/tiflash/contrib/grpc/src/core/ext/filters/client_channel/subchannel.cc, line number: 884, log msg : subchannel 0x7ee64a727400 {address=ipv4:172.16.48.14:20160, args=grpc.client_channel_factory=0x7ee9c46c03c0, grpc.default_authority=db-tikv-7.db-tikv-peer.tidb10096900833626871784.svc:20160, grpc.http2_scheme=https, grpc.initial_reconnect_backoff_ms=1000, grpc.internal.channel_credentials=0x7ee9a6963b40, grpc.internal.security_connector=0x7ee44dbc7070, grpc.internal.subchannel_pool=0x7ee9c48d4190, grpc.max_receive_message_length=-1, grpc.max_reconnect_backoff_ms=3000, grpc.min_reconnect_backoff_ms=1000, grpc.primary_user_agent=grpc-c++/1.44.0, grpc.resource_quota=0x7ee9c48d42b0, grpc.server_uri=dns:///db-tikv-7.db-tikv-peer.tidb10096900833626871784.svc:20160}: Retry immediately"] [source=grpc] [thread_id=3645]
[2024/12/20 01:21:54.261 +00:00] [INFO] [Server.cpp:385] ["/workspace/source/tiflash/contrib/grpc/src/core/ext/filters/client_channel/subchannel.cc, line number: 910, log msg : subchannel 0x7ee64a727400 {address=ipv4:172.16.48.14:20160, args=grpc.client_channel_factory=0x7ee9c46c03c0, grpc.default_authority=db-tikv-7.db-tikv-peer.tidb10096900833626871784.svc:20160, grpc.http2_scheme=https, grpc.initial_reconnect_backoff_ms=1000, grpc.internal.channel_credentials=0x7ee9a6963b40, grpc.internal.security_connector=0x7ee44dbc7070, grpc.internal.subchannel_pool=0x7ee9c48d4190, grpc.max_receive_message_length=-1, grpc.max_reconnect_backoff_ms=3000, grpc.min_reconnect_backoff_ms=1000, grpc.primary_user_agent=grpc-c++/1.44.0, grpc.resource_quota=0x7ee9c48d42b0, grpc.server_uri=dns:///db-tikv-7.db-tikv-peer.tidb10096900833626871784.svc:20160}: failed to connect to channel, retrying"] [source=grpc] [thread_id=3645]

• TiFlash needs a reasonable mechanism to clean up illegal grpc connections to reduce useless logs.