Undici fetch is slow because webstreams suck

[seomikewaltman]

In version 1.4.5 which used, Axios/http, I was able to run 40 node worker threads with proxies to download videos with this library. We benchmarked 100k video downloads in about 8 hours.

Since Undici/fetch was incorporated into the project we noticed a major change in download speeds. 40 node workers is out. It throws errors, had to bring it down to 16. At 16 it took over 48 hours to download just to get 2k videos. The body takes forever to read, it timesout and very unstable.

Undici/request is 10x faster.

See: nodejs/undici#2014

This is a server based library, it cannot run in a web client there really was not need to use fetch/readablestreams. Any one opposed to ditching fetch and using requests?

[absidue]

YouTube.js can run on node, deno and web browsers, so yes everyone not running it inside node would be opposed.
Even the example in this repo runs in the browser.

You can pass your own fetch function into Innertube.create, so you can use whatever http backend you want, as long as you can return the appropriate response from your custom function.

[seomikewaltman]

The download method is typed to return a fetch.body ReadableStream. Even if I switched out the fetch function in innertube.create to undici/request and converted the response to conform to a fetch response (res.ok, etc), the response's body stream is going to blow up internally. If I'm wrong please correct me.

At least with Axios node http or web xmlhttprequest converted into the same uniform response. By making fetch the full on default any node based project/FE proxy is going to be 10x slower whether its download() or a getInfo() call.

[absidue]

You don't have to use YouTube.js's download function, you can write your own one with whatever HTTP backend that you want to use.

Two things to keep in mind if you write your own one:

1. Remember to decipher the URL, using the undeciphered URL will get you throttled by YouTube.
2. You need to download the stream in chunks no larger than 10MiB, if you go over that for a single request/chunk, YouTube will throttle you. You can specify the range you want from YouTube by adding the &range=start-end query parameter to the streaming URL.

[absidue]

As your use case seems to be mass downloading videos, unless you actually need the extra metadata provided by getInfo, you can use getBasicInfo, which only has to make one request instead of two.

[seomikewaltman]

You don't have to use YouTube.js's download function, you can write your own one with whatever HTTP backend that you want to use.

That would be unfortunate. Pytube and every other OS YT library isn't as good. :)

I might be able to extend the InnerTube class for now and import the decipher functionality to get the download URL then write undici/request response handlers.

[LuanRT]

I mean, it shouldn't be difficult to implement your own downloader. Besides, you'll still be using YouTube.js' deciphering & data extraction capabilities anyway.

See this section of the readme: https://github.com/LuanRT/YouTube.js#getstreamingdatavideo_id-options

[seomikewaltman]

In case anyone else is interested, this works and is much faster 👍🏼

import { ProxyAgent, request, setGlobalDispatcher } from 'undici';
import { Innertube, UniversalCache, Constants } from 'youtubei.js';

async function handle(videoId, login = false) {

    // If using a proxy, HIGHLY recommended if doing bulk downloads
    setGlobalDispatcher('http://your_proxy_here');

    const youtube = await Innertube.create({
        cache: new UniversalCache(false);
    });

    youtube.session.on('auth', async ({ credentials }) => {
        console.log('Sign in successful:', credentials);
    });

    if (login) await youtube.session.signIn();

    const videoInfo = await youtube.getBasicInfo(videoId);

    if (videoInfo.playability_status.status === 'LOGIN_REQUIRED') {
        console.log("You must login to download age-restricted videos.");
        return;
    }

    const formats = videoInfo.streaming_data.formats;

    // Custom function that finds fallback resolutions if 1080p not available it tries 720p, etc
    const format = getBestQualityVideo(formats);

    const url = format.decipher(youtube.session.player);

    try {

        // 10x faster downloads on server side than fetch 👍🏼 
        const { body, headers, statusCode } = await request(url, {
            method: 'GET',
            headers: Constants.STREAM_HEADERS,
            maxRedirections: 2 // REQUIRED if using a proxy (default: 0)
        });

        console.log('status', statusCode);
        console.log('headers', headers);
        body.on('data', console.log);

       // pipe it to a writer

    } catch (err) {
        console.log(JSON.stringify(err));
    }