Latest Bit version `1.6.73` causes build failures in Ripple CI / Bit Cloud

[coreyaus]

Describe the bug

Since the Ripple CI / Bit Cloud build process has started using Bit version 1.6.73 I've been seeing build failures on my React 18 environment (see Ripple CI logs here), which is based on @teambit/react.react-env@1.0.38:

• the error occurs even if I don't update my components - just running bit tag --unmodified and exporting the results throws a build error for me on the latest Bit version
• the bit build process still succeeds when I running it locally with any Bit version

Screenshots, exceptions and logs - error details from Ripple CI

The full build logs and all errors are visible here (the env is in a public scope): https://bit.cloud/sitestudio/envs/~ripple-ci/job/sitestudio-sitestudio-envs-rebuilding-scope/component/sitestudio.envs/tina-react-env@0.0.3

The main part of the error message says:

PnpmError: better-sqlite3@8.7.0 install: prebuild-install || node-gyp rebuild --release
Exit status 1

Steps to Reproduce

1. You can see an example when comparing the last 2 versions of the tina-react-env here: https://bit.cloud/sitestudio/envs/tina-react-env/~compare
   • There are no code differences between the 2 versions
   • Version 0.0.2 built perfectly 1 month ago, while version 0.0.3 is failing to build
2. You can import that tina-react-env and it will build fine locally
   • run bit tag --unmodified and bit export and the Ripple CI throws the error detailed above

Expected Behavior

• New Bit versions should not lead to build failures on Ripple CI / Bit Cloud for components that were previously building successfully.
  • Or if there are errors on Ripple CI it should ideally be possible to recreate those same errors when running bit build locally
• Alternatively a way to specify the Bit version that Ripple CI uses would provide an immediate resolution: Can we specify a Bit version to be used by Ripple CI / Bit Cloud? #8627

Specifications

• Bit version: 1.6.73
• Workspace type: harmony
• Node version: 21.2.0
• pnpm version: 7.4.0
• Platform: Bit Cloud / Ripple CI
• Relevant env: The tina-react-env is based on the React 18 env: @teambit/react.react-env@1.0.38. It only has very minor customizations

[davidfirst]

For some reason I'm unable to reproduce the issue locally with the same version.

@guysaar223 , @GiladShoham , can you take a look?

[zkochan]

Does it consistently fail? Even if you retry? Looks related to our virtual fs but we have not changed anything in our ripple CI package manager. I have tried running it locally in a container and it worked.

Sometimes the build is skipped, when the build artifacts are found in the cache, so maybe that is why it worked the last time.

This is definitely a bug but finding the reason will be hard if I won't be able to reproduce it locally. In that case the only way to find the reason will be to add debug messages to the virtual fs, so that more info is printed in ripple.

[zkochan]

I created a new simple component with better-sqlite3@8.7.0, this one builds without issues: https://bit.cloud/zoltan/dogfooding/~ripple-ci/job/zoltan-main-Gb0BJcI1A

[coreyaus]

Thanks for the quick responses on this! I really appreciate your review, particular as you say because it's always hard to debug issues when it's not possible to reproduce them locally so thanks for your time so far 🙏

In answer to your first question, it is failing consistently at the moment, even when retrying. Happily it's a pretty simple env to recreate or replace so I'll run a test now to see if it builds successfully when recreated from scratch. The only customisations in that env are these 3 extra peer dependencies added to the host-dependencies.ts and env.jsonc files:

• tinacms
• @tinacms/mdx
• @tinacms/schema-tools

The env doesn't depend on better-sqlite3 directly but that package is included indirectly due to the dependency on tinacms, via the following path:

└─ tinacms@1.5.30
   ├─ @tinacms/search@1.0.18
      ├─ sqlite-level@1.0.1
         └─ better-sqlite3@8.7.0

Let me know if those notes trigger any insights and I'll report back shortly once I've finished creating a fresh version of the env from scratch (I'm halfway through that process now) 👍

[coreyaus]

The update is that I just finished recreating that env from scratch and ran into the same error: https://bit.cloud/sitestudio/envs/~ripple-ci/job/sitestudio-main-update-test-react-env-to-include-extra-peer-dependencies

Steps to reproduce:

Here are the steps I followed to re-create that env from scratch. Please let me know if there's something I'm missing or doing wrong (and apologies in advance if that's the case!):

1. Created a new workspace and test-react-env:

   mkdir temp-new-bit-workspace && cd temp-new-bit-workspace
   bit init --default-scope=sitestudio.envs
   bit create react-env test-react-env --aspect teambit.react/react-env --scope=sitestudio.envs
   bit install --add-missing-deps
   bit tag --message "Create base test-react-env"
   bit export

2. Manually added the latest versions of the 3 extra dependencies mentioned above (tinacms, @tinacms/mdx & @tinacms/schema-tools) to the host-dependencies.ts and env.jsonc files (see this code diff), then ran the following commands to tag and export those updates:

   bit install --add-missing-deps
   bit tag --message "Update test-react-env to include extra peer dependencies"
   bit export

Please let me know if I missed something in that process, or if there are any other changes you'd like me to test. Would it be helpful to manually configure better-sqlite3 as a dependency somewhere within the workspace or env (or manually override the version as a test)?

Thanks again!

[zkochan]

Thanks for the new details. I was able to reproduce the issue locally in a ripple container using a fork of sqlite that I used in my component. Using this command.

bit sign zoltan.dogfooding/module-with-sql@0.0.2 --multiple --rebuild

[coreyaus]

Thanks for your work on this @zkochan and great to hear you were able to reproduce the issue locally.

I wanted to check in with a couple of quick questions:

1. Are there are any tests I can run to help your work delivering a fix?
2. Have you found any workarounds in the meantime?

I'm currently unable to build or update an environment if it uses a package that includes better-sqlite3 in its dependencies. Thanks for any help you can offer!

---

Things I've tried

1. Setting better-sqlite3 as an explicit dependency of the env in the host-dependencies.ts and env.jsonc files

2. Trying to remove the dependency based on the docs here: https://bit.cloud/teambit/dependencies/aspect-docs/dependency-resolver/~compositions

   "teambit.dependencies/dependency-resolver": {
     "packageManager": "teambit.dependencies/pnpm",
     "policy": {
       "dependencies": {
         "better-sqlite3": "-"
       },
       "peerDependencies": {
         "better-sqlite3": "-"
       }
     },
   },

3. Adding better-sqlite3 to the neverBuiltDependencies array (again based on these docs)

Unfortunately none of those approaches worked (you can probably tell I was mostly just trying everything I could think of!)

PS: It looks like Ripple CI is now using a not-yet-released version of Bit (1.6.84) to run the build process - is that right?

[zkochan]

Right now the only solution would be to turn off fusepm (cc @guysaar223) or somehow build the package outside of the mounted directory.

I spent about 2 days debugging it but couldn't figure out the issue. The better_sqlite3.o file seems to have the wrong file type. It is data instead of ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped. Looks like the C compiler can't complete the compilation correctly on the Fuse filesystem.

---

removing better-sqlite3 or adding it to neverBuiltDependencies should work though. I should check this.

[nardote]

Same problem

✖ (9/10) env: teambit.react/react, task "teambit.preview/preview:GeneratePreview" has failed in 5s
- (10/10) [Preview: PreBundlePreview]
✔ (10/10) [Preview: PreBundlePreview] Completed successfully in 249μs
✔ running tasks. Succeeded in 18s
  running post-build for all tasks 
✔ running post-build for all tasks. Succeeded in 502μs
✖ Running build pipeline using 1 environment(s), total 10 tasks. Failed in 18s
- all objects were persisted successfully!
The following errors were found while running the build pipeline
Failed task 1: "teambit.preview/preview:GeneratePreview" of env "teambit.react/react"
component: esui.lib/theme@0.0.24
ModuleNotFoundError: Module not found: Error: Can't resolve '@emotion/styled' in '/tmp/capsules-root/sign-capsules/2024-4-15/1lqkiw6/node_modules/.pnpm/@mui+styled-engine@6.0.0-alpha.5_react@17.0.2/node_modules/@mui/styled-engine'
ModuleNotFoundError: Module not found: Error: Can't resolve '@emotion/styled' in '/tmp/capsules-root/sign-capsules/2024-4-15/1lqkiw6/node_modules/.pnpm/@mui+styled-engine@6.0.0-alpha.5_react@17.0.2/node_modules/@mui/styled-engine'
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/Compilation.js:2022:28
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:817:13
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:10:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:275:22
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:448:22
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:118:11
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:689:25
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:893:8
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:1013:5
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/neo-async/async.js:6883:13
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:996:45
    at finishWithoutResolve (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:567:11)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:656:15
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:15:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:27:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/DescriptionFilePlugin.js:89:43
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/forEachBail.js:39:13
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/AliasPlugin.js:148:14
    at next (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/forEachBail.js:35:3)
    at forEachBail (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/forEachBail.js:49:9)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/AliasPlugin.js:61:5
    at _next0 (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:8:1)
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:30:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:27:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/DescriptionFilePlugin.js:89:43
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:15:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/DirectoryExistsPlugin.js:41:15
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)
Found 1 errors in 1 component

[itaymendel]

resolved on rippleci