get_object may be blocked

[streetycat]

I found that My program may be blocked frequently in get_object, no return for long time.

Now, I have modify them as below, and it's running good in many days.

    async fn get_object(
        &self,
        dec_id: &ObjectId,
        object_id: &ObjectId,
        from: Option<&ObjectId>,
    ) -> BuckyResult<NONObjectInfo> {
        let fut1 = match futures::future::select(
            self.get_object_impl(dec_id, object_id, from).boxed(),
            async_std::future::timeout(TIMEOUT_HALF, futures::future::pending::<()>()).boxed(),
        )
        .await
        {
            futures::future::Either::Left((ret, _)) => return ret,
            futures::future::Either::Right((_, fut)) => fut,
        };

        log::warn!(
            "group get object timeout (type={:?}) from {:?}, local: {:?}",
            object_id.obj_type_code(),
            from,
            self.local_device_id
        );

        match futures::future::select(
            self.get_object_impl(dec_id, object_id, from).boxed(),
            async_std::future::timeout(TIMEOUT_HALF, fut1).boxed(),
        )
        .await
        {
            futures::future::Either::Left((ret, _)) => ret,
            futures::future::Either::Right((ret, _)) => ret.map_or(
                Err(BuckyError::new(BuckyErrorCode::Timeout, "timeout")),
                |ret| ret,
            ),
        }
    }

    async fn get_object_impl(
        &self,
        dec_id: &ObjectId,
        object_id: &ObjectId,
        from: Option<&ObjectId>,
    ) -> BuckyResult<NONObjectInfo> {
        log::info!(
            "get object {}, local: {}, from: {:?}",
            object_id,
            self.local_device_id,
            from
        );

        self.non_service
            .get_object(NONGetObjectInputRequest {
                common: NONInputRequestCommon {
                    req_path: None,
                    source: RequestSourceInfo {
                        protocol: RequestProtocol::DataBdt,
                        zone: DeviceZoneInfo {
                            device: None,
                            zone: None,
                            zone_category: DeviceZoneCategory::CurrentZone,
                        },
                        dec: dec_id.clone(),
                        verified: None,
                    },

                    level: NONAPILevel::Router, // from.map_or(NONAPILevel::NOC, |_| NONAPILevel::Router),

                    target: from.map(|remote| remote.clone()),
                    flags: 0,
                },
                object_id: object_id.clone(),
                inner_path: None,
            })
            .await
            .map(|resp| resp.object)
    }

I think there is a bug, Is there anyone encountered this problem?

[lurenpluto]

get_object these interfaces, if the cross-device or even cross-zone call, is likely to take longer, so your "blocked" problem here need to clarify a few points first:

1. Whether the target side of get_object has received the request? If the request was received, did it return in time?
2. Has get_object successfully established a bdt connection with the target?
   We may need to analyze it from the logs maybe
3. If it is blocked, if it waits long enough, for example, 30 minutes or an hour, will it eventually return?
   The impact of some timeout policies inside this cyfs-stack

At present, cyfs-stack uses a timeout+retry mechanism for cross-device requests, and the key code is as follows

CYFS/src/component/cyfs-stack/src/forward/forward.rs

Lines 42 to 48 in 485b0c1

	// FIXME An additional timeout here
	let requestor = RequestorWithRetry::new(
	Box::new(bdt_with_fail_handler),
	2,
	RequestorRetryStrategy::ExpInterval,
	Some(std::time::Duration::from_secs(60 * 10)),
	);

So in extreme cases, it may take longer to return

[streetycat]

It's blocked several minutes, there are many calls in short durations, and many Nodes are running in same process.

The logs is hard to read, I will reproduce it and track in detail when I am free.

It would be nice to have an simple environment to diagnose it.

by the way, 10 minutes is a long time, maybe I didn't wait so long.

[lurenpluto]

This timeout interval varies greatly for different requirements, so it is difficult to determine a "correct and reasonable" length of time, a better design is that get_object can set the default timeout time, such as adding a corresponding parameter timeout, but then all interface calls may need to add this parameter