20200526

Changes

Downloader

• mdn-process-url: drop resources in devnews
• download: add experimental electronjs downloader
• download: fix redirect for electronjs
• download-electronjs: disable dns cache
• download-nodejs-cn: performance tuning
• mdn: add script to merge samples form different dir (#49)
• merge: only merge from non-empty samples
• options: retry on ESERVFAIL

Processing

• mdn-process-url: fix protocol redirecting
• mdn-process-url: refactor switch
• process: correctly handle redirecting (#47)
• link: create redirect placeholder for different protocol (#48)

Misc

• search: update deps
• package.json: update deps
• build-index-worker: correctly strip suffix for title
• inject: align blocks

sha256sum

c146886462a0f767f40e129baa3b2f46e473e98f2514d62635ec0b2ddab254fd  developer.mozilla.org_en-US.7z
5067d446fa551f9a218ad8ed96e8cbedc0eb5490c15c279441dad26b7217b97b  developer.mozilla.org_zh-CN.7z
c5d4cddaecfbf5f633477c5f1a4f3829523852cd8dd3e20548caf3035ce3cd5e  nodejs-cn_14.3.0_zh-CN.7z
d38ff62c3afda76583008124cec932ecc577640080f2e5d351152da30ed83c81  www.electronjs.org_en-US_20200527.7z
fd9b478f05e576a5bd2b2aca1ddbfb2ab04b6071308e8298ff6b30dd8161ef5d  www.electronjs.org_zh-CN_20200527.7z

Note

• images in mdn package are compressed by leanify
• large videos in mdn package is replaced by smaller ones, you can manaully restore them from mdn-video-pack.7z

20200514

Changes

Downloader

• process: allow async linkRedirectFunc
• link: use same protocol for cross-origin url
• link: manually retry on some timeouts
• index: making async
• link: better retry messages
• add downloader for nodejs cn
• adjust-concurrency: make min concurrency configurable
• link: use agentkeepalive to reduce connect timeouts
• options: add extra delay for http 429
• link: use cacheable-lookup instance created by got
• options: parse retry-after header

Processing

• process: move preProcessResource before replacePath for html
• mdn-process-html: escape tag for cheerio
• mdn-process-url: process some external links (#44)
• mdn-process-url: drop links to $toc
• mdn-process-html: remove useless styles
• mdn-process-url: keep processable external links
• mdn-process-url: skip link to large files
• mdn-process-url: redirect cross-origin links with hard-coded rules
• mdn-process-url: redirect large videos to small ones (#46)
  • You can download mdn-video-pack.7z to restore videos
• mdn-process-url: refactor maps
• mdn-process-url: skip more large files
• mdn-process-url: fix more invalid urls

Misc

• logger-config: configurable sub dir
• eslint: disable require-atomic-updates
• refactor: move common part to lib
• package.json: update deps (#45)
• search: render highlights of summary if content not matching
• eslint: change linebreak-style to unix
• readme: update

sha256sum

3ed860ff409133d1859c348e76e823d065876f0fbb6d7bbd13e72d73914b75ad  developer.mozilla.org_en-US.7z
6f34a7f59e15c5546b827675843d3467bf47a5e73ce10d976edf05b5a867eaa8  developer.mozilla.org_zh-CN.7z
73684c155b668765f4cb0b3855306335a8dbde1258beee3368c7a006db60b3e4  mdn-video-pack.7z
6adf4aa6ff1778f587fbcc4839f6817c47975a75094ffa71e8e83bde51efcabe  nodejs.cn-v12.16.2_zh-CN.7z
174ca171fc98d477684e23581f649278d647dd37f93d69d0c86688384f26ea1f  nodejs.cn-v14.0.0_zh-CN.7z
418751f729d69c3d57c9d9b8c919805852ec5117a5dd6d9d66332a563c0d88c0  nodejs.cn-v14.1.0_zh-CN.7z
7fbfa29025d654741e1e685637e46d33010a708d4d7163b71dd4b34d08a2bd1c  nodejs.cn-v14.2.0_zh-CN.7z

20200401

Changes

Downloader

• index: allow en-US download (#42)
• en-US release
• link: retry on socket timeout
• options: increase default max redirect to 15

Processing

• mdn-process-html: extract injected js and css to separate file (#40)
• mdn-process-html: keep href of red links (#41)
• index: refactor and extract mdn-process-url
• index: workaround index.html redirect
• process: localized redirect url
• process: more redirect url
• mdn-process-html: remove google cdn stuff
• mdn-process-url: redirect unpkg.com
• mdn-process-url: fix else if (#43)
• process-css: exec skipProcessFunc
• redirect-url: redirect external example img to alternative
• process-css: extract process fn
• process-css: correctly process refUrl
• redirect-url: move out locale-related redirect
• redirect-url: redirect icon-font related stuff to fake url

Misc

• package.json: add type for urijs
• search: simple search server
• build-index-worker: strip suffix of title
• compressed images in this release with leanify

20200318 zh-CN

Changes

Downloader

• got: bump to version 10 (#28)
• link: fix timeout processing
• link: fix buffer responseType
• process: implement preProcessResource
• adjust-concurrency: set minimum concurrency to 4
• link: add dnsCache
• link: workaround for premature close on node 12 (#33)
• link: move retry on empty response to getRetry
• index: redirect 3rd-party links with hardCodedRedirectUrl
• adjust-concurrency: decrease concurrency slower
• link: add workaround for cacheable-lookup error (#37)
• link: retry on lookup timeout

Processing

• mdn-process-html: remove newsletter and Hacks Blog stuff
• process: keep example iframes (#29)
• process: interactive-examples (#30)
• process: correctly process inline styles
• index: fix interactive-examples (#31)
• process: remove youtube iframe
• index: initial code for mdn.github.io (#32)
• index: more validExtensionName
• process: redirect developer.cdn.mozilla.net (#34)
• process: redirect unexpected favicons (#34)
• process: parse srcset attribute (#35)
• process: fix html links
• process: skip external links (#36)
• index: drop resources for google login
• index: skip javascript links (#38)
• process: resolve malformed links (#38)
• process: replace external iframe with external links (#39)
• process: skip about:blank links (#38)
• process: fix post process with empty src
• process: redirect samples without reversion

Logging

• link: lint logger
• link: log resource type in request and response logger (#3)

Misc

• options: increase timeout
• options: retry on premature close

20200108 zh-CN

Changes

Processing

• process: hide translationInProgress via css and fix empty page content
• process: escape html for react data

Misc

• link: remove an unused assignment
• package: install types for cheerio and update deps
• type: declare types for mdn assets and react data

20191209 zh-CN

Changes:

Downloader

• adjust-concurrency: log pending item count correctly
• index: set django_language cookie
• link: parse and process simple sitemap
• adjust-concurrency: log concurrency and queue size

Processing

• mdn-process-html: cleanup useless stuff in _react_data.stringCatalog
• mdn-process-html: fix old-styled compatibility table
• sitemap: process the links
• mdn-process-html: add icon for external links
• index: handle randomly redirect to wiki.developer.mozilla.org
• index: remove more search-related stuff
• index: try not to edit profiles

Logging

• link: replace console with logger

20191101 zh-CN

Changes

• Process scripts

  • index: process links in win.mdn.assets
  • index: process react data and adapt for new ui
  • postProcessReactData: mock fetch if documentData not exists
  • postProcessReactData: process documentData.summary
  • mdn-process-html: remove more useless scripts
  • mdn-process-html: refactor
  • mdn-process-html: process inline js-polyfill script src
  • mdn-process-html: remove apple-touch-icon-precomposed links
  • mdn-process-html: workaround documentData.raw
  • mdn-process-html: mock fetch api with es5 code

• Links

  • index: drop the video
  • downloader: try to save failed resource
  • downloader: keep non-html resource and log skipped html resource
  • linkRedirectFunc: normalize path and remove some search
  • downloader: log skipped html resource to separate file
  • index: add more non-html extensions
  • index: process wrong relative link

• Misc

  • adjust-concurrency: expected concurrency to be a number from 1 and up
  • index: extract configureLogger
  • index: extract modules
  • adjust-concurrency: log queue size if no pending items
  • index: add more entry points
  • downloader: filter resource for logging

20191008 zh-CN

Changes

• logger: initial support
• index: drop pagination links
• HtmlResource: put a redirect placeholder if redirected
• process: fix self link
• index: use hooks.beforeRedirect to avoid redirecting to another locale
• index: more entry points

20190918 zh-CN

A more complete package with lots of files and 404s.

20190912 zh-CN

A more completed snapshot with some experimental new style of mdn.