Does set_piece_deadline() exclusively support bittorrent-integrated media players?

[AllSeeingEyeTolledEweSew]

How is the caller meant to calculate the deadline for set_piece_deadline()?

I can imagine the scenario where the caller is streaming a media file, and wants a time-based readahead buffer ("buffer the next 30 seconds of data") and parses enough metadata to reliably calculate this, such as decoding/presentation timestamps and stream bitrates.

This seems like a very narrow case, and I want to understand if it's just narrow, or I missed something about the usage.

I'm building an http/bittorrent bridge, which is content-agnostic, so it can't predict when an http client will need a piece. The best it can do is measure the rate at which the client is streaming. To make use of this I'd have to continuously readjust deadlines, calling set_piece_deadline() for every piece, every second-or-something.

Is there another way to use set_piece_deadline() I'm not thinking about? Should I try to make the API more generic?

EDIT: I dug deeper into how I might use set_piece_deadline() by measuring a stream consumption rate to calculate deadlines. I found this to be an extremely hard problem.

In particular, when using a media player like mplayer and vlc to stream an http source, I see that stream throughput has very irregular spikes. The players seem to fill the tail of their cache on some kind of timer, so the throughput drops to zero for several seconds. If I use a moving average to figure out stream rate, the period would have to be quite long, and would be slow to respond if the user seeks the media. I'm not a good enough statistician to find any better way to measure stream rate, and therefore deadlines.

I conclude that set_piece_deadline() is only useful when a real-time media player is tightly coupled to its torrent client. Change my mind!

[elgatito]

@AllSeeingEyeTolledEweSew consider a situation when media player requests seek operation over your stream, and you don't have that piece, it wants to read after seek.
In that case you should get the piece as fast as possible, then you use set_piece_deadline, because setting priority to 7 is downloading the piece slower (in general) than downloading using set_piece_deadline over that piece.

[elgatito]

I can imagine the scenario where the caller is streaming a media file, and wants a time-based readahead buffer ("buffer the next 30 seconds of data") and parses enough metadata to reliably calculate this, such as decoding/presentation timestamps and stream bitrates.

You will meet many walls trying to get "30 seconds".
Bitrate is a floating measurement. And your file can have multiple audio streams, while player will read only one of them.

Usually players operate the buffer size, depending on the approximate bitrate or even a fixed-size buffer to store readahead bytes. Usually, you should have own buffer to download it with a higher priority, and player's buffer will also read upcoming bytes as well. That is usually enough and you will not see any lags from the player side.

Also you need to take in mind that it is typical to have more that 1 connection from your player, one is usually reading the video stream, second is reading audio stream, so you will have multiple current playback points and should arrange piece download prioritization for that.

You can see my example here: https://github.com/ElementumOrg/lt2http/blob/master/src/bittorrent/torrent.cpp#L809

[AllSeeingEyeTolledEweSew]

You can see my example here: https://github.com/ElementumOrg/lt2http/blob/master/src/bittorrent/torrent.cpp#L809

https://github.com/ElementumOrg/lt2http/blob/master/src/bittorrent/torrent.cpp#L511-L522

    for (int i = start; i <= int(start) + 2; ++i) {
            ...
            set_piece_priority(i, (i - int(start)) * 10, lt::download_priority_t{6});
    }

It appears you're not calculating deadlines per se, just using set_piece_deadline() with fake deadlines, to manipulate libtorrent's m_time_critical_pieces.

That's fine of course, but it supports my theory that there is no way to choose true "deadlines" in a streaming-pipe context, and using set_piece_deadline() in such a context is really a hack.

Thus we should probably clarify its documentation, and add some new functions.

[elgatito]

@AllSeeingEyeTolledEweSew In my understanding, you should make upcoming N pieces downloaded as fast as possible, and others could be managed by libtorrent. When you move the reader forward, you need to force downloading your readahead pieces and so on.

I could not see any other way, rather than set_piece_deadline.

But it would be great to be able to use piece_priority(piece_index, 7) to make it critical and make sure to download next.
Even with my #6157, if takes more time to download piece with priority 7 than downloading it with set_piece_deadline. So I do not yet have solution that works properly for all situations.

[AllSeeingEyeTolledEweSew]

I think you agree with my sentiment that using set_piece_deadline() in this context is a hack. The deadlines aren't meaningful, they're just there to manipulate m_time_critical_pieces.

I think there must be a better interface to manipulate m_time_critical_pieces, or otherwise manipulate the time-critical logic, for uses like ours. But I can't find a consensus on how one should stream data from bittorrent, so I don't know what this interface should be.

I'm trying to explore and build on first principles in #6293.

In my understanding, you should make upcoming N pieces downloaded as fast as possible, and others could be managed by libtorrent. When you move the reader forward, you need to force downloading your readahead pieces and so on.

@elgatito Where exactly did you get this idea? Just intuition? Another project? Academic paper? I'm interested in any info. This strategy is definitely not indicated by the documentation of set_piece_deadline() or http://libtorrent.org/streaming.html; these assume you can pick a meaningful deadline.

How do you choose N? I don't mean to ask "is N user-configurable in your app", I mean how should it be chosen? Is there some target measurement that should be optimized, by changing N?

[elgatito]

@AllSeeingEyeTolledEweSew

Where exactly did you get this idea? Just intuition? Another project? Academic paper? I'm interested in any info. This strategy is definitely not indicated by the documentation of set_piece_deadline() or http://libtorrent.org/streaming.html; these assume you can pick a meaningful deadline.

That is an experience from development of elementum.
You expect to have an almost stable bytes per second read rate from a user's player, but it is not. Players now have internal buffering, bitrates can go up and down. In the end of the day there are players that request bytes sequentially, from multiple read points, and you should manage the priorities for pieces to arrive faster if they should be sent to requestor soon.

[AllSeeingEyeTolledEweSew]

I've had some similar anecdotal experience. I agree that for a case like an http-to-bittorrent bridge you want to support concurrent requests, and so you end up wanting to manage the time-critical pieces holistically for a torrent.

I've been considering a PR to add set_time_critical_pieces(), which would directly manage the m_time_critical_pieces queue. It would be equivalent to repeatedly calling set_piece_deadline(i, i), but more straightforward/readable, and avoids needing to sort m_time_critical_pieces with every update.