Thoughts on adding dead videos on --download-archive?

[ghost]

Sometimes, when downloading a list of youtube videos, some of those videos are removed or no longer work. When this happens, youtube-dl and yt-dlp do not write the video ID to the file written in download archive, so if the process were to restart, it will try to make that request to the dead video again, which is inefficent. I propose there should be some way to make it so that videos that are removed off youtube while downloading from a list of videos should be added to download archive so the requests are not made again. I will submit pull request, obviously, if people agree. If this is also a option already (Can't seem to find it though), then please point me to it.

[pukkandan]

Once you confirm that the playlist only has dead videos left, run again with --force-write-archive --flat-playlist. Is this not sufficient? Doesn't work

[ghost]

Thanks for that. I applied it to my (private) fork. The use case under which I needed this feature is very specialized and specific so I wouldn't expect into to be merged into mainline anyway.

Yes, i do have to upgrade ;) But i did try the version with your first fix as well.

[Zirro]

The idea, as I've understood it, sounds a lot like the "error archive" I suggested here (though it'd use a separate file): #689 (comment)

But this could cause issues if the user runs this option on a site that don't have ID in URL, or even on a playlist

If a new option (like --error-archive) was added for this, perhaps you could throw a fatal error if someone tries to use it together with a site/extractor where the ID for missing videos is known to be unobtainable?

[pukkandan]

The idea, as I've understood it, sounds a lot like the "error archive" I suggested here (though it'd use a separate file)

No. In this, we are not doing any extraction whatsoever. We don't know if the extraction will fail or not and so cannot write to a seperate error archive

perhaps you could throw a fatal error if someone tries to use it together with a site/extractor where the ID for missing videos is known to be unobtainable

For this, I would literally have to go through the 1000+ extractors and mark whether they have IDs obtainable from URL or not. If youtube-dl was designed from the start with such a usecase in mind, this would have been trivial. But that is not the case

I do have an idea how to automatically determine whether the extractor supports fetching id from URL. But I'll have to try implementing it to know for sure how feasible it is

[Zirro]

No. In this, we are not doing any extraction whatsoever.

I think you're referring specifically to the --flat-extract option with this. My comment was in reference to the initial post which requests a way to avoid repeatedly extracting a deleted video by recording its ID. This would require an initial extraction to find out that the video is actually missing.

For this, I would literally have to go through the 1000+ extractors and mark whether they have IDs obtainable from URL or not.

Having it for YouTube would probably cover the needs for a majority of users. Support for smaller extractors could be enabled in the future, perhaps by user request. That said, for most extractors, couldn't this be determined by checking whether their _VALID_URL includes an <id> match?

[pukkandan]

That said, for most extractors, couldn't this be determined by checking whether their _VALID_URL includes an match?

The issue is that the extractors use the id group without it being the final id but only a display id