Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get session cookies when using --cookies-from-browser #5534

Open
9 tasks done
flashdagger opened this issue Nov 14, 2022 · 11 comments · Fixed by #9747
Open
9 tasks done

Get session cookies when using --cookies-from-browser #5534

flashdagger opened this issue Nov 14, 2022 · 11 comments · Fixed by #9747
Labels
bug Bug that is not site-specific

Comments

@flashdagger
Copy link
Contributor

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

  • I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

  • I'm reporting a bug unrelated to a specific site
  • I've verified that I'm running yt-dlp version 2022.11.11 (update instructions) or later (specify commit)
  • I've checked that all provided URLs are playable in a browser with the same IP and same login details
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
  • I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

While investigating #4514 I found a different behavior between --cookies cookies.txt and --cookies-from-browser chrome.
The first option injected all cookies into the request, while the latter one was missing the session cookies.

Session cookies have expires=None and discard=True. Taking a look into the codebase I see that session cookies imported from Chrome have expires=0 and discard=False.

yt-dlp/yt_dlp/cookies.py

Lines 335 to 339 in d7b460d

return is_encrypted, http.cookiejar.Cookie(
version=0, name=name, value=value, port=None, port_specified=False,
domain=host_key, domain_specified=bool(host_key), domain_initial_dot=host_key.startswith('.'),
path=path, path_specified=bool(path), secure=is_secure, expires=expires_utc, discard=False,
comment=None, comment_url=None, rest={})

Is this intended or possibly a bug?

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

NA
@flashdagger flashdagger added bug Bug that is not site-specific triage Untriaged issue labels Nov 14, 2022
@flashdagger flashdagger changed the title session cookies are not injected when using --cokies-from-browser chrome session cookies are not injected when using --cookies-from-browser chrome Nov 14, 2022
@pukkandan
Copy link
Member

Definitely not intended. I thought chrome doesn't write session cookies to disk 🤔. If it does, we should "fix" the cookies similar to https://github.com/yt-dlp/yt-dlp/blob/b96187a9603aaa7ead8ad94ac82433f085ea4895/yt_dlp/utils.py#L1654-L1667

@flashdagger
Copy link
Contributor Author

I thought chrome doesn't write session cookies to disk 🤔

It does because of the Continue where you left off startup option.

@Tblue
Copy link

Tblue commented May 28, 2023

Firefox is also affected by this issue: It seems like session cookies are not stored in its cookies.sqlite file.

Instead, they are in sessionstore.jsonlz4 (but that file only exists if Firefox is not running). That's a file in Mozilla's custom mozLz4 format, which essentially is just the LZ4 block format preceded by a lower-endian 32-bit unsigned int specifying the size of the uncompressed data in bytes.

Once uncompressed, the sessionstore data can be parsed as JSON, and the session cookies found in the top-level key cookies of the resulting object.

@StefanLobbenmeier
Copy link
Contributor

StefanLobbenmeier commented Apr 20, 2024

Was about to open the same issue, here is what I found:

Disclaimer at the start - I did this investigation with a recent version of brave, will need to check if this applies to all chromium browsers and if so from which version.

The brave cookies database has 2 columns related to cookie expiry: expires_utc and has_expires. yt-dlp only checks the expires_utc column:

  • For session cookies expires_utc is set to 0 and has_expires to 0.
  • For expiring cookies expires_utc is set to a utc timestamp and has_expires to 1

The current logic in yt-dlp takes expires_utc only and the phyton cookie will consider a value of 0 as expired:

class Cookie:
    # ...
    def is_expired(self, now=None):
        if now is None: now = time.time()
        if (self.expires is not None) and (self.expires <= now):
            return True
        return False

To support session_cookies we should therefore set expires to None, since they do not expire.

StefanLobbenmeier added a commit to StefanLobbenmeier/yt-dlp that referenced this issue Apr 20, 2024
@Grub4K Grub4K changed the title session cookies are not injected when using --cookies-from-browser chrome Get session cookies when using --cookies-from-browser Apr 21, 2024
@StefanLobbenmeier
Copy link
Contributor

StefanLobbenmeier commented Apr 23, 2024

@Tblue can you elaborate on what you mean by with the prefix? I could confirm that this file is created when I close firefox: ~/Library/Application Support/Firefox/Profiles/$profile.default-esr/sessionstore.jsonlz4

It starts like this:

mozLz40�≈¡���{"version":["ses��Ú
restore",1],"windows":[{"tab	�bentrie

I tried removing the mozLz40 or the mozLz40≈¡� using a text editor (probably not the right tool 😄) but passing that to lz4 I always get Error 44 : Unrecognized header : file cannot be decoded, what do I have to remove exactly?

@StefanLobbenmeier
Copy link
Contributor

StefanLobbenmeier commented Apr 23, 2024

Actually nevermind, I found this superuser thread on it: https://superuser.com/a/1363751

And it linked this python gist that I could test sucessfully: https://gist.github.com/Tblue/62ff47bef7f894e92ed5

I could also confirm that the cookie can be found in there, I censored mine so we have a reference for the format:

jsonpath /cookies/4

    {
      "host": ".vimeo.com",
      "value": "the cookie value",
      "path": "/",
      "name": "12345678_password",
      "secure": true,
      "originAttributes": {
        "firstPartyDomain": "",
        "geckoViewSessionContextId": "",
        "inIsolatedMozBrowser": false,
        "partitionKey": "",
        "privateBrowsingId": 0,
        "userContextId": 0
      },
      "schemeMap": 2
    },

Edit: I am blind, this is even the gist of @Tblue. 😄 Before we can make a PR with this - could you just confirm that you are also willing to publish it (or parts of it) under UNLICENSE?

I think we would only need to copy this method: https://gist.github.com/Tblue/62ff47bef7f894e92ed5#file-mozlz4a-py-L76

@Tblue
Copy link

Tblue commented Apr 23, 2024 via email

@Tblue
Copy link

Tblue commented Apr 23, 2024 via email

@Tblue
Copy link

Tblue commented Apr 24, 2024

@StefanLobbenmeier, you might also find this script useful as an inspiration (feel free to treat it as licensed under the UNLICENSE as well): dump-firefox-cookies.py

bashonly pushed a commit that referenced this issue May 11, 2024
…9747)

Partially addresses #5534
Authored by: StefanLobbenmeier
@bashonly
Copy link
Member

Keeping open until this is solved for Firefox as well

@bashonly bashonly reopened this May 11, 2024
@StefanLobbenmeier
Copy link
Contributor

Yeah did not have the time to implement for Firefox, but it’s still on my radar. If someone else wants to take it I don’t mind though :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug that is not site-specific
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants