Break metadata downloading into smaller steps for more resiliency with large channels
Thanks for making Yark! This PR solves an issue I was having that sounds like #71
The issue was a large playlist with ~600 videos wasn't fully downloading. It would only download around 300 videos, and then on subsequent runs of `yark refresh`, it would show several videos as either being deleted, or as being added, right around that limit of where it stopped downloading. It was not storing the metadata of every video, and how many it would download was fluctuating a little bit on each run.
It seems this happens when running `YoutubeDL().extract_info(url, download=False)` on a playlist or channel with too many videos. It can download the metadata of several hundred videos, but then will stop getting complete metadata, and does not get the `formats` metadata. Then, in `_parse_metadata_videos_comp`, since the entry has no `formats`, it is skipped, since it is believed to be a livestream or other edge case (see #62).
To fix this, we can first download the metadata "flat" via `YoutubeDL(params={'extract_flat':True}).extract_info`, meaning `YoutubeDL` does not follow the URLs of the videos to download their metadata, it simply gets their URLs (and a little bit of metadata) . We can then download the full metadata of the videos one by one, and if they fail, we can make a new `YoutubeDL()` object and try again, which seems to solve the problem for another several hundred videos until a new `YoutubeDL()` is needed. I don't know exactly why this works, maybe something to do with expiring tokens?
The detection for the metadata download failure is checking if the `formats` key of the entry is empty. There is probably a better way to check this, and I think that YoutubeDL may be generating warnings that are getting swallowed by `VideoLogger`.
The `entries` in the result of `extract_info` can either be a playlist (with its own `entries`), or a "url" entry. I was lazy here, and just copied your strategy of trying 3 times and catching exceptions whenever calling `extract_info`, but I'm not sure how much we're getting out of it when using it for downloading the metadata of a single video. It may make sense to pull the logic of downloading one video's metadata into its own function, but I wasn't sure if we should be passing the `YoutubeDL` object to that function for it to use, or if it would be fine to create a new `YoutubeDL` every time.
合并状态:已合并 合并于 2025-01-12 关闭于 2025-01-12 2 条评论