build: improvements for TS & Webpack & Docker #1225

pano9000 · 2025-02-18T21:48:41Z

Hi,

this PR (still in draft) aims to improve the build process and better utilize TS / Webpack combination

still a little bit work left to do here, but wanted to make this more visible to avoid any "duplicated work", while I work on this.

eliandoran · 2025-02-23T18:16:46Z

Now that's a lot of changes. 😮

pano9000 · 2025-02-23T21:20:07Z

hope that's not a problem ;-)
I am almost finished with this PR, just need to still figure out two things:

a) make sure swc will work with Gihub actions (currently it fails, but I have an idea why that might be)
b) make adjustments to the copy-dist.ts file, to update copying of assets (translations, stylesheets, etc.)

pano9000 · 2025-02-24T08:36:03Z

update: I'll probably split out the swc Terser plugin work onto its own PR, so that we can at least have the improved ts/webpack building – and the build speed optimization (which would've been introduced by swc TerserPlugin) can come in later, after some additional investigation on how to handle the most elegant way:

→ it is currently giving me some issues during Github CI, due to it requiring "platform" specific packages (e.g. like @swc/core-darwin-x64) – which do not seem to be installed correctly, when we use npm ci.
Other have solved it by "removing" package-lock.json prior to npm i, but that doesn't seem like a proper "elegant" solution to me.

Potentially will also try to just install the required deps in the CI, after running npm ci – but that would not be as elegant, as I'd like – it would mean extra code per platform.

https://forum.strapi.io/t/package-error-swc-code-darwin-x64-deploying-to-strapi-cloud/37922
swc-project/swc#2898
https://stackoverflow.com/a/77721414
https://nextjs.org/docs/messages/failed-loading-swc

=> will update the PR accordingly this evening

pano9000 · 2025-02-27T08:46:34Z

I'd say I am done with this PR now, just need to do a quick test on a Windows machine as well, to make sure the copy-dist script does not behave a bit unexpectedly and I'll then rebase onto latest develop to avoid merge conflicts.

edit: confirmed on Win11 -> copy-dist works exactly as on my Linux machine :-)

Quick first stats from these changes:

develop branch

Dist Folder size:
    du -sch dist/
    282M    dist/

Number of files:
    find ./dist/ | wc -l
    6321

vs

this PR


    
Dist Folder size:
    du -sch dist/
    172M    dist/

Number of files:    
    find ./dist/ | wc -l
    3781

some more details to follow on what exactly changed later this evening

edit:
Docker building needs to be updated as well, it seem - the CI is failing, but I know where and why it goes wrong -- will fix

this prevents tsc from unnecessarily transpiling the frontend part as well: previously it was transpiled by tsc, but the files got discarded and replaced by the files built by webpack. speeds up tsc command a bit as well: from 14 seconds to ~8 secs

output the bundled files directly in the build folder a) keeps the src folder clean from build output b) it saves us some "manual" copying work

as per https://webpack.js.org/configuration/devtool/#production serving the `source-map` file to "normal" users seems to be not recommended, so instead let's go with `nosources-source-map`: a) this drastically reduces app-dist folder size from 20MB down to 8.7MB b) it still allows for stack traces

not sure why, but seems like it doesn't like `[jt]s` – which causes it to skip certain .d.ts files, making tsc fail

since we build into the build folder -> we should also clean the folder before building as well also it makes sense to run tsc first, as it runs faster, so if there's any TS errors, we will have a faster failing build

…folders

these folder are already "excluded" implicitly, since we only include "./src" folder

webpack bundling already ran before this script, so there is no need to copy this file over

stop the build folder from being copied into the dist/src subfolder → there is no sense in doing that → the contents of the build folder are corretly copied previously already (see line 26ff)

we have the bundled "app-dist" already in the "dist", copying over the original unbundled "app" folder serves no benefit here

the "srcDirsToCopy" block is useless now, we can just use the previous dirsToCopy to achieve the exact same thing

… copying job there's no benefit in having them split up like before

there's no need to read the folder structure and then copy each single file in a loop => just copy the whole folder and be done with it :-)

there's no benefit from stripping "node_modules/" from the string, to later add it again using the `DEST_DIR_NODE_MODULES` constant => just copy directly into the `DEST_DIR` folder and preserver the `node_modules` part in the path

since this is a "standalone" script we are running and no other JS scritps are running "in the background", there's no real benefit for async here.

since we don't export this anywhere, might as well just call the steps directly

TS and Webpack build into the dist folder directly now

…pendencies

pano9000 · 2025-03-03T09:01:21Z

update on the docker build:
I have a working version locally, but need to clean it up a bit, before pushing – the benefit here is that the build process now actually happens inside Docker and not the previous weird "hybrid" way, where part of it was built "locally" and then copied inside Docker for further steps.

=> also I do realize that this PR is getting bigger and bigger, but the issue is: The build process was so messy, you start pulling one thread, and you end up needing to take care of every other little bit as well :-D

pano9000 force-pushed the build_improve-ts-webpack branch 2 times, most recently from 5e9a33d to 6a82a3a Compare February 23, 2025 18:02

pano9000 force-pushed the build_improve-ts-webpack branch 3 times, most recently from 5915250 to 2fdb9cb Compare February 27, 2025 08:44

pano9000 added 21 commits March 3, 2025 08:41

build(webpack): output webpack build in build folder

e7c84d1

output the bundled files directly in the build folder a) keeps the src folder clean from build output b) it saves us some "manual" copying work

build(tsconfig): fix glob for build config

5b1a4bc

not sure why, but seems like it doesn't like `[jt]s` – which causes it to skip certain .d.ts files, making tsc fail

build(scripts): add build:ts and update build:prepare-dist

90b2775

since we build into the build folder -> we should also clean the folder before building as well also it makes sense to run tsc first, as it runs faster, so if there's any TS errors, we will have a faster failing build

build(scripts): add webpack progress flag

84f2b17

build(copy-trilium): use npm run build:ts

6c781f8

build(scripts): add "build:clean" script for removing dist and build …

b8f1b50

…folders

build(tsconfig): remove unnecessary exclude lines

8677c5c

these folder are already "excluded" implicitly, since we only include "./src" folder

chore(copy-dist): removee unnecesary copy of tsconfig.webpack.json

2eea6cc

webpack bundling already ran before this script, so there is no need to copy this file over

build(copy-dist): do not copy build folder into src folder

7d2d6a9

stop the build folder from being copied into the dist/src subfolder → there is no sense in doing that → the contents of the build folder are corretly copied previously already (see line 26ff)

build(copy-dist): copy over required folders/files from "public" folder

64e981b

build(copy-dist): avoid copying over the app dir into dist

f8fda7f

we have the bundled "app-dist" already in the "dist", copying over the original unbundled "app" folder serves no benefit here

build(copy-dist): consolidate folder copying

41c7c0e

the "srcDirsToCopy" block is useless now, we can just use the previous dirsToCopy to achieve the exact same thing

build(copy-dist): consolidate files and folder copying into one asset…

81d35b7

… copying job there's no benefit in having them split up like before

build(copy-dist): simplify "build" copying

46652d8

there's no need to read the folder structure and then copy each single file in a loop => just copy the whole folder and be done with it :-)

build(copy-dist): remove unused paths

acf54b4

build(copy-dist): use sync copying

277d3c5

since this is a "standalone" script we are running and no other JS scritps are running "in the background", there's no real benefit for async here.

build(copy-dist): consolidate nodeModules copying into a single loop

68db5b5

build(copy-dist): execute code in try/catch -> get rid of function

d3f10e0

since we don't export this anywhere, might as well just call the steps directly

pano9000 added 4 commits March 3, 2025 08:41

build: get rid of intermediary "build" folder -> use "dist" directly

c965124

build(copy-dist): copying build folder is not required anymore

080412d

TS and Webpack build into the dist folder directly now

build(Docker): comment out seemingly useless installation of build de…

af30a07

…pendencies

build(Docker): move server-package.json preparation into Dockerfile

e810715

pano9000 force-pushed the build_improve-ts-webpack branch from 2fdb9cb to e810715 Compare March 3, 2025 08:37

pano9000 changed the title ~~build: improvements for TS & Webpack~~ build: improvements for TS & Webpack & Docker Mar 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: improvements for TS & Webpack & Docker #1225

build: improvements for TS & Webpack & Docker #1225

pano9000 commented Feb 18, 2025

eliandoran commented Feb 23, 2025

pano9000 commented Feb 23, 2025

pano9000 commented Feb 24, 2025 •

edited

Loading

pano9000 commented Feb 27, 2025 •

edited

Loading

pano9000 commented Mar 3, 2025

build: improvements for TS & Webpack & Docker #1225

Are you sure you want to change the base?

build: improvements for TS & Webpack & Docker #1225

Conversation

pano9000 commented Feb 18, 2025

eliandoran commented Feb 23, 2025

pano9000 commented Feb 23, 2025

pano9000 commented Feb 24, 2025 • edited Loading

pano9000 commented Feb 27, 2025 • edited Loading

develop branch

this PR

pano9000 commented Mar 3, 2025

pano9000 commented Feb 24, 2025 •

edited

Loading

pano9000 commented Feb 27, 2025 •

edited

Loading