v1.27.4

Extractors

Additions

• [sexcom] add likes extractor (#6149)
• [wikimedia] add wiki extractor (#6050)

Fixes

• [bunkr] fix file downloads (#6037)
• [cyberdrop] fix extraction
• [deviantart] fix "pagination": "manual" for cursor-based endpoints (#6079)
• [deviantart] fix "original": "images" (#6124)
• [exhentai] fix limits option (#6090)
• [flickr] make exif and context metadata extraction non-fatal (#6002, #6077)
• [flickr] make album metadata extraction non-fatal (#3441)
• [furaffinity] fix favorite pagination (#6151)
• [gofile] fix KeyError: 'childrenIds' (#5993)
• [newgrounds] fix warning for age-restricted posts (#6005)
• [toyhouse] fix extraction of image URLs
• [tumblr] fix 401 Unauthorized for likes when using api-key (#5994)
• [twitter] fix pinned Tweet extraction (#6102)
• [ytdl] fix processing playlists of playlists (#6127)

Improvements

• [bcbnsfw] use * as query when retrieving all posts (#6135)
• [bunkr] support bunkr: URL prefix (#6017)
• [e621] cache pool metadata API calls (#6001)
• [generic] better directory names (#6104)
• [koharu] improve format selection (#6088)
• [pixiv] implement downloading "original" ugoira frames (#6056)
• [pixiv] use mobile API for series (#5983)

Metadata

• [batoto] improve chapter info regex (#5988, #5997)
• [batoto] extract chapter_url metadata (#5562)
• [batoto] improve title extraction (#5988)
• [hitomi] extract extension_original metadata (#6049)
• [instagram] add post_date metadata field (#6081, #6091)
• [sankaku] restore old tags format (#6043)
• [twitter] extract type metadata (#6111)

Options

• [bunkr] add tlds option to match URLs with all possible TLDs (#5875, #6017)
• [instagram] add max-posts option (#6054)
• [instagram] add info as a possible include value
• [instagram] allow disabling cursor output
• [twitter] add info as a possible include value (#6114)
• [twitter] allow disabling cursor output (#5990)

Post Processors

• [hash] add hash post processor to compute file hash metadata (#6099)
• [metadata] add include and exclude options (#6058)
• [metadata] fix using .. in directories on Windows (#5942, #6094)
• [rename] add rename post processor to rename previously downloaded files (#5846, #6044)
• [ugoira] support converting "original" frames (#6056)
• [ugoira] add skip option (#6056)

Miscellaneous

• [cookies:firefox] extract only cookies without container by default (#5957)
• [formatter] implement A format specifier (#6036)
• [tests] fix bug when running tests in a certain order
• [util] extend CustomNone with arithmetic operators (#6007, #6009)
• add --rename and --rename-to command-line options (#5846, #6044)
• add input-files config option (#6059)

v1.27.3

Extractors

Additions

• [bunkr] support bunkr.ci and bunkrrr.org (#5970)
• [furaffinity] add submissions extractor (#5954)
• [hentaicosplays] support hentai-cosplay-xxx.com (#5959)

Fixes

• [behance] fix KeyError: 'fields' (#5965)
• [behance] fix video extraction (#5965)
• [cien] extract all files when authenticated (#5934)
• [deviantart] fix KeyError - 'category' (#5960, #5961, #5969, #5971, #5976, #5978)
• [fanbox] update pagination logic (#5949, #5951, #5956)
• [hotleak] fix AttributeError (#5950)
• [instagram] restore GraphQL API functionality (#5920)
• [twitter] update x-csrf-token header during login (#5945)

Improvements

• [bunkr] fail downloads for maintenance files (#5952)
• [zerochan] improve tag redirect handling, add redirects option (#5891)

Post Processors

• [metadata] add base-directory option (#5262, #5728)

v1.27.2

Extractors

Additions

• [agnph] add tag and post extractors (#5284, #5890)
• [aryion] add favorite extractor (#4511, #5870)
• [cien] add support (#2885, #4103, #5240)
• [instagram] add info extractor (#5262)
• [koharu] add gallery, search, and favorite extractors (#5893, #4707)
• [twitter] add info extractor (#3623)

Fixes

• [8chan] update TOS cookie name (#5868)
• [behance] fix image extraction (#5873, #5926)
• [booru] prevent crash when file URL is empty (#5859)
• [deviantart] try to work around journal/status API changes (#5916)
• [hentainexus] fix error with spread pages (#5827)
• [hotleak] fix faulty image URLs (#5915)
• [inkbunny:following] fix potentially infinite loop
• [nijie] fix image URLs of single image posts (#5842)
• [readcomiconline] fix extraction (#5866)
• [toyhouse] fix Content Warning bypass (#5820)
• [tumblr] revert to offset pagination, implement pagination option (#5880)
• [twitter] fix username-alt option name (#5715)
• [warosu] fix extraction
• [zerochan] handle KeyError - 'items' (#5826)
• [zerochan] fix error on tag redirections (#5891)
• [zerochan] fix Invalid control character errors (#5892)

Improvements

• [bunkr] support bunkr.fi domain (#5872)
• [deviantart:following] use OAuth API endpoint (#2511)
• [directlink] extend recognized file extensions (#5924)
• [exhentai] improve error message when temporarily banned (#5845)
• [gelbooru_v02] use total number of posts as pagination end marker (#5830)
• [imagefap] add enumeration index to default filenames (#1746, #5887)
• [paheal] implement fast --range support (#5905)
• [redgifs] support URLs with numeric IDs (#5898, #5899)
• [sankaku] match URLs with www subdomain (#5907)
• [sankakucomplex] update domain to news.sankakucomplex.com
• [twitter] implement cursor support (#5753)
• [vipergirls] improve thread URL pattern
• [wallpapercave] support album listings (#5925)

Metadata

• [dynastyscans] extract chapter tags (#5904)
• [erome] extract date metadata (#5796)
• [furaffinity] extract folders and thumbnail metadata (#1284, #5824)
• [sankaku] implement notes extraction (#5865)
• [subscribestar] fix date parsing in updated posts (#5783)
• [twitter] extract bookmark_count and view_count metadata (#5802)
• [zerochan] fix source metadata
• [zerochan] fix tag category extraction (#5874)
• [zerochan] delay fetching extended metadata (#5869)

Options

• [agnph] implement tags option (#5284)
• [booru] allow multiple url keys (#5859)
• [cien] add files option (#2885)
• [koharu] add cbz and format options (#5893)
• [vsco] add include option (#5911)
• [zerochan] implement tags option (#5874)

Removals

• [fallenangels] remove module

Post Processors

• [metadata] allow using format strings for directory (#5728)

Options

• add --print-traffic command-line option
• add -J/--resolve-json command-line option (#5864)
• add filters-environment option
• implement archive-event option (#5784)

Actions

• [actions] support multiple actions per pattern
• [actions] add exec action (#5619)
• [actions] add abort and terminate actions (#5778)
• [actions] allow setting a duration for wait
• [actions] emit logging messages before waiting/exiting/etc

Tests

• [tests] enable test results for external extractors (#5262)
• [tests] load results from ${GDL_TEST_RESULTS} (#5262)

Miscellaneous

• [cookies] add thorium support (#5781)
• [job] add resolve argument to DataJob (#5864)
• [path] fix moving temporary files across drives on Windows (#5807)
• [ytdl] fix --cookies-from-browser option parsing (#5885)
• make exceptions in filters/conditionals non-fatal
• update default User-Agent header to Firefox 128 ESR
• include zstd in Accept-Encoding header when supported

v1.27.1

Extractors

Additions

• [hentainexus] restore module (#5275, #5712)
• [shimmie2] support vidya.pics (#5632)
• [tcbscans] support other domains (#5774)

Fixes

• [deviantart] fix watching module ID extraction (#5696, #5772)
• [fanbox] handle KeyError for no longer existing plans (#5759)
• [kemonoparty:favorite] fix exception when sorting null objects (#5692. #5721)
• [skeb] fix 429 Too Many Requests errors (#5766)
• [speakerdeck] fix extraction (#5730)
• [twitter] fix duplicate ArkoseLogin check

Improvements

• [nijie] support downloading videos (#5707, #5617)
• [philomena] support downloading .svg files (#5643)
• [szurubooru] support empty tag searches (#5711)
• [twitter] ignore Unavailable media (#5736)

Metadata

• [hitomi] extract title_jpn metadata (#5706)
• [instagram] extract liked metadata (#5609)

Options

• [newgrounds] extend format option (#5709)
• [twitter] extend ratelimit option (#5532)
• [twitter] add username-alt option (#5715)

Removals

• [photobucket] remove module
• [nitter] remove instances
• [vichan] remove wikieat.club

Downloaders

• [ytdl] fix exception due to missing ext in unavailable videos (#5675)

Formatter

• implement C format specifier (#5647)
• implement X format specifier (#5770)

Options

• add --no-input command-line and input config option (#5733)
• add --config-open command-line option (#5713)
• add --config-status command-line option (#5713)

Miscellaneous

• [actions] fix exception when msg is not a string (#5683)

v1.27.0

Extractors

Additions

• [mastodon] add favorite, list, and hashtag extractors (#5529)
• [mastodon] add support for card images
• [pixeldrain] add support for single-file album downloads (#5641)
• [reddit] support comment embeds (#5366)
• [seiga] re-implement login with username & password
• [tapas] add creator extractor (#5306)
• [vsco] add avatar extractor (#5341)
• [wikimedia] support wiki.gg wikis

Fixes

• [4archive] fix extraction
• [8chan] fix file downloads by sending a TOS cookie (#5578)
• [artstation] disable TLS 1.2 ciphers by default (#5564, #5658)
• [bluesky] filter reposts only for user timelines (#5528)
• [common] disable check_hostname for custom SSLContexts (#3614, #4891, #5576)
• [exhentai] fix Multi-Page Viewer detection (#4969)
• [exhentai] fix blank page detection
• [hiperdex] update domain to hiperdex.top (#5635)
• [hotleak] download files returning a 404 status code (#5395)
• [imgur] match URLs with title slugs (#5593)
• [kemonoparty] fix KeyError - 'path' for posts without files (#5368, #5394, #5422, #5488)
• [kemonoparty] fix crash on posts with missing datetime info (#5422)
• [mastodon] send canonical true/false boolean values (#5516)
• [newgrounds] update and fix login procedure (#5109)
• [patreon] fix bootstrap data extraction (#5624)
• [poipiku] fix downloading R-18 posts (#5567)
• [poipoku] avoid language-specific extraction (#5590, #5591)
• [realbooru] fix videos and provide fallback URLs (#2530)
• [slideshare] fix extraction
• [subscribestar] fix file URLs (#5631)
• [twitter] update domain to x.com (#5597)
• [twitter] transfer twitter.com cookies to x.com (#5597)
• [twitter] prevent crash when extracting birdwatch metadata (#5403)
• [twitter] handle missing expanded_url fields (#5463, #5490)
• [wikimedia] suppress exception for entries without imageinfo (#5384)
• [wikimedia] fix exception for files with empty metadata

Improvements

• [exhentai] detect CAPTCHAs during login (#5492)
• [foolfuuka] improve board pattern & support pages (#5408)
• [furaffinity] match fxfuraffinity.net/fxraffinity.net/xfuraffinity.net URLs (#5511, #5568)
• [gelbooru] improve pagination logic for meta tags (#5478)
• [kemonoparty:favorite] return artists/posts in native order and support sort and order query parameters (#5375, #5620)
• [oauth] use Extractor.request() for HTTP requests to support proxy servers etc (#5433)
• [pixiv] change sanity_level debug message to a warning (#5180)
• [twitter] improve username & password login procedure (#5445)
• [twitter] wait for rate limit reset before encountering a 429 error (#5532)
• [twitter] match fixvx.com URLs (#5511)
• [twitter] match Tweet URLs with query parameters (#5371, #5372)
• [twitter] match /photo/ and /video/ Tweet URLs (#5443, #5601)

Options

• [common] add sleep-429 option (#5160)
• [common] implement skip-filter option (#5255)
• [common] implement keywords-eval option (#5621)
• [kemonoparty] add announcements option (#5262)
• [pixiv:novel] add covers option (#5373)
• [twitter] implement relogin option (#5445)

Downloaders

• [http] add MIME type and signature for .m4v files (#5505)

Post Processors

• [mtime] do not overwrite _mtime values with None (#5439)
• [ugoira] log errors for general exceptions

Archives

• [archive] move DownloadArchive code into its own module
• [archive] implement DownloadArchiveMemory class (#5255)
• [archive] add archive-mode option (#5255)

Cookies

• [cookies] use temporary file when saving cookies.txt files (#5461)
• [cookies] optimize _find_most_recently_used_file() for exact profiles (#5538)
• [cookies] set proper expires value for Chrome session cookies

Documentation

• [docs] update docs/configuration links (#5059, #5369, #5423)
• [docs] update link to "nightly" builds (#5618)
• [docs] replace AnchorJS with custom script
• [docs] update defaults of sleep-request, browser, tls12
• [docs] complete Authentication info in docs/supportedsites

Formatter

• [formatter] allow dots in '...' literals (#5539)

Output

• [output] enable colored output by default
• [output] extend output.colors (#2566)
• [output] support NO_COLOR environment variable
• [output] add --no-colors command-line option
• [output] add -w/--warning command-line option (#5474)

Tests

• [tests] select unused port number for local HTTP server
• [tests] allow filtering extractor result tests by URL or comment
• [tests] mark tests with missing auth as only_matching

Update

• implement update-related command-line options (#5233)
  • -U/--update updates an executable file to the latest release
  • --update-check checks if the local version is up to date
  • --update-to allows switching to a different release channel (stable or dev)
    as well as upgrading/downgrading to a specific tag.
    • --update-to dev
    • --update-to [email protected]
    • --update-to v1.25.2
  • (non-executable installations have only access to -U/--update-check for version checks)

Miscellaneous

• add workaround for requests 2.32.3 issues (#5665)
• fix exit status of --clear-cache/--list-extractors/--list-modules
• restore LD_LIBRARY_PATH for executables built with PyInstaller (#5421)
• store match and groups values in Extractor objects

v1.26.9

Extractors

Additions

• [artstation] support video clips (#2566, #3309, #3911)
• [artstation] support collections (#146)
• [deviantart] recognize deviantart.com/stash/… URLs
• [idolcomplex] support new pool URLs
• [lensdump] recognize direct image links (#5293)
• [skeb] add extractor for followed users (#5290)
• [twitter] add quotes extractor (#5262)
• [wikimedia] support azurlane.koumakan.jp (#5256)
• [xvideos] support /channels/ URLs (#5244)

Fixes

• [artstation] fix handling usernames with dashes in domain names (#5224)
• [bluesky] fix not spawning child extractors for followed users (#5246)
• [deviantart] handle CloudFront blocks (#5363)
• [deviantart:avatar] fix index for URLs without ? (#5276)
• [deviantart:stash] fix index values (#5335)
• [gofile] fix extraction
• [hiperdex] update URL patterns & fix manga metadata (#5340)
• [idolcomplex] fix metadata extraction
• [imagefap] fix folder extraction (#5333)
• [instagram] make accessing like_count non-fatal (#5218)
• [mastodon] fix handling null moved account field (#5321)
• [naver] fix EUC-KR encoding issue in old image URLs (#5126)
• [nijie] increase default delay between requests (#5221)
• [nitter] ignore invalid Tweets (#5253)
• [pixiv:novel] fix text extraction (#5285, #5309)
• [skeb] retry 429 responses containing a request_key cookie (#5210)
• [warosu] fix crash for threads with deleted posts (#5289)
• [weibo] fix retweets (#2825, #3874, #5263)
• [weibo] fix livephoto filename extensions (#5287)
• [xvideos] fix galleries with more than 500 images (#5244)

Improvements

• [bluesky] improve API error messages
• [bluesky] handle posts with different embed structure
• [deviantart:avatar] ignore default avatars (#5276)
• [fapello] download full-sized images (#5349)
• [gelbooru:favorite] automatically detect returned post order (#5220)
• [imgur] fail downloads when redirected to removed.png (#5308)
• [instagram] raise proper error for missing reels_media (#5257)
• [instagram] change posts are private exception to a warning (#5322)
• [reddit] improve preview fallback formats (#5296, #5315)
• [steamgriddb] raise exception for deleted assets
• [twitter] handle "account is temporarily locked" errors (#5300)
• [weibo] rework pagination logic (#4168)
• [zerochan] fetch more posts by using the API (#3669)

Metadata

• [bluesky] add instance metadata field (#4438)
• [gelbooru:favorite] add date_favorited metadata field
• [imagefap] extract folder metadata (#5270)
• [instagram] default likes to 0 (#5323)
• [kemonoparty] add revision_count metadata field (#5334)
• [naver] unescape post title and description
• [pornhub:gif] extract viewkey and timestamp metadata (#4463)
• [redgifs] make date available for directories (#5262)
• [subscribestar] fix date metadata
• [twitter] add birdwatch metadata field (#5317)
• [twitter] add protected metadata field (#5327)
• [warosu] fix board_name metadata

Options

• [bluesky] add reposts option (#4438, #5248)
• [deviantart] add comments-avatars option (#4995)
• [deviantart] extend metadata option (#5175)
• [flickr] add contexts option (#5324)
• [gelbooru:favorite] add order-posts option (#5220)
• [kemonoparty] add order-revisions option (#5334)
• [vipergirls] add like option (#4166)
• [vipergirls] add domain option (#4166)

Downloaders

• [http] add MIME type and signature for .mov files (#5287)

Docker

• build images from source instead of PyPI package
• build linux/arm64 images (#5227)
• build images on every push to master
  • tag images as YYYY.MM.DD
  • tag the most recent build from master as dev
  • tag the most recent release build as latest
• reduce image size (#5097)

Miscellaneous

• [formatter] fix local DST datetime offsets for :O
• build Linux executable on Ubuntu 22.04 LTS (#4184)
• automatically create directories for logging files (#5249)

v1.26.8

Extractors

Additions

• [bluesky] add support (#4438, #4708, #4722, #5047)
• [bunkr] support new domains (#5114, #5130, #5134)
• [fanbox] add home and supporting extractors (#5138)
• [imagechest] add user extractor (#5143)
• [imagetwist] add gallery extractor (#5190)
• [kemonoparty] add posts extractor (#5194, #5198)
• [twitter] support communities (#4913)
• [vsco] support spaces (#5202)
• [weibo] add gifs option (#5183)
• [wikimedia] support www.pidgi.net (#5205)
• [wikimedia] support bulbapedia.bulbagarden.net (#5206)

Fixes

• [archivedmoe] fix thebarchive WebM URLs (#5116)
• [batoto] fix crash when manga name or chapter contains a - (#5200)
• [bunkr] fix extraction (#5088, #5151, #5153)
• [gofile] update website_token extraction
• [idolcomplex] fix pagination for tags containing : (#5184)
• [kemonoparty] fix deleting file names when computing revision_hash (#5103)
• [luscious] fix IndexError for files without thumbnail (#5122, #5124, #5182)
• [naverwebtoon] fix title for comics with empty tags (#5120)
• [pinterest] fix section URLs for boards with /, ?, or # in their name (#5104)
• [twitter] update query hashes
• [zerochan] fix skipping every other post

Improvements

• [deviantart] skip locked/blurred posts (#4567, #5193)
• [deviantart] implement downloading PNG versions of non-original images with "quality": "png" (#4846)
• [flickr] handle non-JSON errors (#5131)
• [idolcomplex] support alphanumeric post IDs (#5171)
• [kemonoparty] implement filtering duplicate revisions with "revisions": "unique"(#5013)
• [naverwebtoon] support /webtoon/ paths for all comics (#5123)

Metadata

• [idolcomplex] extract id_alnum metadata (#5171)
• [pornpics] support multiple values for channel (#5195)
• [sankaku] add id-format option (#5073)
• [skeb] add num and count metadata fields (#5187)

Downloaders

Fixes

• [http] remove pyopenssl import (#5156)

Miscellaneous

• fix filename formatting silently failing under certain circumstances (#5185, #5186)

v1.26.7

Extractors

Additions

• [2ch] add support (#1009, #3540, #4444)
• [deviantart:avatar] add formats option (#4995)
• [hatenablog] add support (#5036, #5037)
• [mangadex] add list extractor (#5025)
• [steamgriddb] add support (#5033, #5041)
• [wikimedia] add support (#1443, #2906, #3660, #2340)
• [wikimedia] support fandom wikis (#2677, #3378)

Fixes

• [blogger] fix lh-*.googleusercontent.com URLs (#5091)
• [bunkr] update domain (#5088)
• [deviantart] fix AttributeError for URLs without username (#5065)
• [deviantart] fix KeyError: 'premium_folder_data' (#5063)
• [deviantart:avatar] fix exception when comments are enabled (#4995)
• [fuskator] make metadata extraction non-fatal (#5039)
• [gelbooru] only log "Incomplete API response" for favorites (#5045)
• [giantessbooru] update domain
• [issuu] fix extraction
• [nijie] fix download URLs of single image posts (#5049)
• [patreon] fix KeyError: 'name' (#5048, #5069, #5093)
• [pixiv] update API headers (#5029)
• [realbooru] fix download URLs of older posts
• [twitter] revert to using media timeline by default (#4953)
• [vk] transform image URLs to non-blurred versions (#5017)

Improvements

• [batoto] support more mirror domains (#5042)
• [batoto] improve v2 manga URL pattern
• [gelbooru] support all tag and URLs with empty tags (#5076)
• [patreon] download m3u8 manifests with ytdl
• [sankaku] support post URLs with alphanumeric IDs (#5073)

Metadata

• [batoto] improve manga_id extraction (#5042)
• [erome] fix count metadata
• [kemonoparty] add revision_hash metadata (#4706, #4727, #5013)
• [paheal] fix source metadata
• [webtoons] extract more metadata (#5061, #5094)

Removals

• [chevereto] remove pixl.li
• [hbrowse] remove module
• [nitter] remove nitter.lacontrevoie.fr

v1.26.6

Extractors

Additions

• [batoto] add chapter and manga extractors (#1434, #2111, #4979)
• [deviantart] add avatar and background extractors (#4995)
• [poringa] add support (#4675, #4962)
• [szurubooru] support snootbooru.com (#5023)
• [zzup] add gallery extractor (#4517, #4604, #4659, #4863, #5016)

Fixes

• [gelbooru] fix favorite extractor (#4903)
• [idolcomplex] fix extraction & update URL patterns (#5002)
• [imagechest] fix loading more than 10 images in a gallery (#4469)
• [jpgfish] update domain
• [komikcast] fix manga extractor (#5027)
• [komikcast] update domain (#5027)
• [lynxchan] update bbw-chan domain (#4970)
• [manganelo] fix extraction & recognize .to TLDs (#5005)
• [paheal] restore extension metadata (#4976)
• [rule34us] add fallback for video-cdn1 videos (#4985)
• [weibo] fix AttributeError in user extractor (#5022)

Improvements

• [gelbooru] show error for invalid API responses (#4903)
• [rule34] recognize URLs with www subdomain (#4984)
• [twitter] raise error for invalid strategy values (#4953)

Metadata

• [fanbox] add metadata option (#4921)
• [nijie] add count metadata (#146)
• [pinterest] add count metadata (#4981)

Miscellaneous

• fix and update zsh completion (#4972)
• fix --cookies-from-browser macOS Firefox profile path

v1.26.5

Extractors

Additions

• [deviantart] add intermediary option (#4955)
• [inkbunny] add unread extractor (#4934)
• [mastodon] support non-numeric status IDs (#4936)
• [myhentaigallery] recognize /g/ URLs (#4920)
• [postmill] add support (#4917, #4919)
• {shimmie2[ support rule34hentai.net (#861, #4789, #4945)

Fixes

• [deviantart] add workaround for integer client-id values (#4924)
• [exhentai] fix error for infinite fallback-retries (#4911)
• [inkbunny] stop pagination on empty results
• [patreon] fix bootstrap data extraction again (#4904)
• [tumblr] fix exception after waiting for rate limit (#4916)

Improvements

• [exhentai] output continuation URL when interrupted (#4782)
• [inkbunny] improve /submissionsviewall.php patterns (#4934)
• [tumblr] support infinite fallback-retries
• [twitter] default to tweets timeline when replies are enabled (#4953)

Metadata

• [danbooru] provide tags as list (#4942)
• [deviantart] set is_original for intermediary URLs to false
• [twitter] remove date_liked (#3850, #4108, #4657)

Docker

• add Docker instructions to README (#4850)
• fix auto-generation of latest tags