Update orbitdb #2600

leblowl · 2024-09-05T14:59:36Z

Pull Request Checklist

I have linked this PR to a related GitHub issue.
I have added a description of the change (and Github issue number, if any) to the root CHANGELOG.md.

(Optional) Mobile checklist

Please ensure you completed the following checks if you did any changes to the mobile package:

I have run e2e tests for mobile
I have updated base screenshots for visual regression tests

…itdbService

…e-orbitdb

packages/backend/localEntryNodePeerId.json

islathehut · 2024-11-13T13:42:59Z

package.json

+    "pnpm": "^9.12.1",
+    "typescript": "^4.9.3"
+  },
+  "volta": {


Anywhere you see this volta config it can be ignored. It will only have an effect if you have volta installed and it just does the same thing as nvm (but faster).

islathehut · 2024-11-13T13:43:31Z

packages/backend/docker-compose.yml

@@ -2,11 +2,11 @@ version: "3.9"
 # Creates local network of peers

 x-with-tor: &with-tor
-    BOOTSTRAP_ADDRS: /dns4/ix2oumqrtjaupt53l6cqpk6ct6iaa5guconwgtvgdk2v3i5wjiyehryd.onion/tcp/7788/ws/p2p/QmRbkBkhTt2DbLMF8kAaf1oxpfKQuEfLKFzVCDzQhabwkw
+    BOOTSTRAP_ADDRS: /dns4/ix2oumqrtjaupt53l6cqpk6ct6iaa5guconwgtvgdk2v3i5wjiyehryd.onion/tcp/7788/ws/p2p/12D3KooWPYjyHnYYwe3kzEESMVbpAUHkQyEQpRHehH8QYtGRntVn


Any of these changes to peer IDs that are hardcoded are just for testing and because of the change in peer ID format.

islathehut · 2024-11-13T13:51:35Z

packages/backend/package.json

-    "test-replication-no-tor": "cross-env NODE_OPTIONS=--experimental-vm-modules ts-node -v && cross-env DEBUG='backend:dbSnap*,backend:localTest*' ts-node src/nodeTest/testReplicate.ts --nodesCount 1 --timeThreshold 200 --entriesCount 1000 --no-useTor",
-    "test-replication-tor": "cross-env NODE_OPTIONS=--experimental-vm-modules cross-env DEBUG='backend:dbSnap*,backend:localTest*' ts-node src/nodeTest/testReplicate.ts --nodesCount 1 --timeThreshold 500 --entriesCount 1000 --useTor",
-    "test-it": "cross-env NODE_OPTIONS=--experimental-vm-modules DEBUG=ipfs:*,backend:* node_modules/jest/bin/jest.js --runInBand --verbose --testPathIgnorePatterns=\".src/(!?nodeTest*)|(.node_modules*)\" --",
+    "test-nest": "cross-env NODE_OPTIONS=\"--experimental-vm-modules --experimental-global-customevent\" DEBUG=ipfs:*,backend:* node_modules/jest/bin/jest.js --detectOpenHandles --forceExit   ./src/nest/**/*.spec.ts",


The experimental global custom event flag is necessary since we aren't on node 20+ and the new libp2p relies on it.

islathehut · 2024-11-13T13:52:18Z

packages/backend/patch/bitswap.patch

@@ -0,0 +1,47 @@
+--- packages/backend/node_modules/@helia/block-brokers/node_modules/ipfs-bitswap/dist/src/bitswap.js	2024-10-03 13:15:45


This patch just adds extra debug logging for testing purposes.

islathehut · 2024-11-13T13:52:53Z

packages/backend/patch/libp2p-connection-monitor.patch

+                         conn.rtt = (Date.now() - start) / 2;
+                     }
+                 })
+                    // #PATCH: This behavior is pulled from libp2p v2.1.5 (the latest as of writing this) as the original didn't actually honor this flag


This was a weird one. This bug has been patched but not in a version we can use yet.

islathehut · 2024-11-13T13:53:22Z

packages/backend/patch/libp2p-kaddht-providers.patch

@@ -0,0 +1,50 @@
+--- packages/backend/node_modules/@libp2p/kad-dht/dist/src/providers.js	2024-10-03 13:05:48


More debug logging.

islathehut · 2024-11-13T13:54:19Z

packages/backend/patch/orbitdb-oplog.patch

@@ -0,0 +1,15 @@
+--- packages/backend/node_modules/@orbitdb/core/src/oplog/log.js	2024-10-03 11:17:39


This resolves an issue where entries that are generated by a peer aren't saved locally and when that peer is offline it throws a ton of errors and crashes.

islathehut · 2024-11-13T13:55:16Z

packages/backend/src/nest/app.module.ts

@@ -153,6 +153,11 @@ export class AppModule {
          useFactory: (baseDir: string) => path.join(baseDir, 'backendDB'),
          inject: [QUIET_DIR],
        },
+        {


This was added because I was experimenting with using Level for the Libp2p datastore. Right now I'm using a memory store but it might be worth doing a persisted store in the future.

islathehut · 2024-11-13T13:59:24Z

packages/backend/src/nest/ipfs/ipfs.service.ts

    }

    return this.ipfsInstance
  }

+  private async initializeStores(init?: StoreInit): Promise<void> {


This is more complicated than it should be because we need to ensure that valueEncoding is set to buffer no matter what. I hit so many issues due to level not properly encoding to buffer...

islathehut · 2024-11-13T14:02:09Z

packages/backend/src/nest/storage/orbitDb/eventsWithStorage.ts

@@ -0,0 +1,76 @@
+/**


This wrapper allows us to use the correct valueEncoding on all of our OrbitDB DBs.

islathehut · 2024-11-13T14:05:08Z

packages/backend/src/nest/storage/orbitDb/eventsWithStorage.ts

@@ -0,0 +1,60 @@
+import { AccessControllerType, IdentitiesType, LogEntry, Events, LogType } from '@orbitdb/core'


This wrapper allows us to ensure the correct level value encoding.

islathehut · 2024-11-13T14:05:18Z

packages/backend/src/nest/storage/orbitDb/keyValueWithStorage.ts

@@ -0,0 +1,60 @@
+import { AccessControllerType, KeyValue, IdentitiesType, LogType, LogEntry } from '@orbitdb/core'


This wrapper allows us to ensure the correct level value encoding.

islathehut · 2024-11-13T15:15:08Z

packages/backend/src/nest/ipfs/ipfs.service.ts

+      throw new Error('IPFS instance does not exist')
+    }
+
+    await this.blockstore?.db.close()


This is a little weird but there are issues that can arise from not closing the underlying level DB, apparently.

islathehut · 2024-11-13T15:16:01Z

packages/backend/src/nest/storage/certifacteRequests/certificatesRequestsStore.ts

@@ -102,7 +102,8 @@ export class CertificatesRequestsStore extends EventStoreBase<string> {
          const pubKey = keyFromCertificate(parsedCsr)

          if (filteredCsrsMap.has(pubKey)) {
-            filteredCsrsMap.delete(pubKey)
+            this.logger.warn(`Skipping csr due to existing pubkey`, pubKey)


The event store now returns records in descending order rather than ascending order.

islathehut · 2024-11-13T15:17:14Z

packages/backend/src/nest/storage/orbitDb/keystoreWithStorage.ts

+
+  const storage = await ComposedStorage(
+    await LRUStorage({ size: 1000 }),
+    await LevelStorage({ path, valueEncoding: 'buffer' })


Level value encoding strikes again.

islathehut · 2024-11-13T15:19:12Z

packages/desktop/src/main/invitation.ts

@@ -16,9 +16,11 @@ export const processInvitationCode = (mainWindow: BrowserWindow, code: string |



This had nothing to do with the upgrade but I got tired of the errors so I cleaned this up.

islathehut · 2024-11-13T15:19:54Z

packages/desktop/src/main/main.ts

+  backendProcess = fork(backendBundlePath, forkArgvs, {
+    env: {
+      NODE_OPTIONS: '--experimental-global-customevent',
+      DEBUG: 'backend*,quiet*,state-manager*,desktop*,utils*,identity*,common*,libp2p:connection-manager:auto-dial',


The debug flag doesn't seem to be inherited from desktop anymore for some reason.

islathehut · 2024-11-13T15:20:45Z

packages/e2e-tests/src/tests/multipleClients.test.ts

@@ -1,3 +1,5 @@
+import { jest } from '@jest/globals'


Cleaned up the test organization and introduced some improvements.

islathehut · 2024-11-13T15:21:43Z

packages/e2e-tests/src/utils.ts

 import { createLogger } from './logger'

 const logger = createLogger('utils')

-export const BACKWARD_COMPATIBILITY_BASE_VERSION = '2.0.1' // Pre-latest production version
+export const BACKWARD_COMPATIBILITY_BASE_VERSION = '2.3.1' // Pre-latest production version


Didn't actually end up needing this but we'll wanna update this when we know if this will be part of 3.0.0 or 4.0.0.

islathehut · 2024-11-13T15:22:45Z

packages/helia/.editorconfig

@@ -0,0 +1,9 @@
+root = true


I added helia as a package which is NOT ideal but it was the fastest way to get it integrated, from my perspective. We should ultimately update it to be a submodule like lfa but I don't have it in me to do it right now.

islathehut · 2024-11-13T15:27:23Z

packages/mobile/ios/Quiet/AppDelegate.m

@@ -176,7 +176,7 @@ - (void)applicationDidEnterBackground:(UIApplication *)application
 {
  [self stopTor];

-  NSString * message = [NSString stringWithFormat:@""];
+  NSString * message = [NSString stringWithFormat:@"app:close"];


This was just to get rid of an error that didn't break anything, it was just annoying.

leblowl and others added 26 commits August 30, 2024 11:05

WIP: Update OrbitDB/IPFS/libp2p

2d1c33b

Various updates

003e94a

Various updates

f6b24b4

Fixes

89d8bd9

Updates

724e60a

Clean up NestJS modules and fix storage.service tests

191514c

Fixes, mostly around restarting app

75d5281

Use custom storage to ensure we write data as Uint8array + rename Orb…

aeab85b

…itdbService

Ignore logs

a68eb6b

More storage related stuff

91f2e97

(Maybe) fix the ping issue and mild refactor for my sanity

3be6240

Fix reconnections and update logging

f6682f1

Get desktop to run without the customevent flag

262b19e

Mobile testing

bbacc19

Ensure mobile can function with JS customevents

22f7a3e

Merge branch 'develop' into update-orbitdb

3bcf6c3

Fix for issue with not having all blocks

533e1de

Merge branch 'update-orbitdb' of github.com:TryQuiet/quiet into updat…

f5977bd

…e-orbitdb

Fix common tests

1b8352f

Fix almost all tests on the backend, etc.

03069bf

Fix for customevent errors in ci tests

b93bd3b

Fix state manager test and backend test node options

2ff48eb

Update class-validator

6e2c387

Test mac e2e ci issue

95564cc

Fix tor binary dir issue in ci

96fa124

Try again

3301d5d

holmesworcester reviewed Oct 4, 2024

View reviewed changes

packages/backend/localEntryNodePeerId.json Show resolved Hide resolved

Random fixes for e2e tests

0695992

islathehut force-pushed the update-orbitdb branch from 6b0cea5 to 0695992 Compare October 6, 2024 16:34

Fix file manager

3a82643

islathehut added 6 commits November 8, 2024 10:31

Update e2e-linux.yml

3832532

Update e2e-linux.yml

77eb520

remove print statement

56da07c

Use windows latest

e3dbecb

Use windows latest large

6c44083

Switch back

ba8934a

islathehut reviewed Nov 13, 2024

View reviewed changes

Remove node_datachannel references

8ff55c5

islathehut reviewed Nov 13, 2024

View reviewed changes

Remove inaccurate comments

d58c7cc

islathehut reviewed Nov 13, 2024

View reviewed changes

Remove unnecessary changes

c91145e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update orbitdb #2600

Update orbitdb #2600

leblowl commented Sep 5, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

islathehut Nov 13, 2024

		@@ -0,0 +1,47 @@
		--- packages/backend/node_modules/@helia/block-brokers/node_modules/ipfs-bitswap/dist/src/bitswap.js 2024-10-03 13:15:45

		@@ -0,0 +1,50 @@
		--- packages/backend/node_modules/@libp2p/kad-dht/dist/src/providers.js 2024-10-03 13:05:48

		@@ -0,0 +1,15 @@
		--- packages/backend/node_modules/@orbitdb/core/src/oplog/log.js 2024-10-03 11:17:39

		@@ -0,0 +1,60 @@
		import { AccessControllerType, IdentitiesType, LogEntry, Events, LogType } from '@orbitdb/core'

		@@ -0,0 +1,60 @@
		import { AccessControllerType, KeyValue, IdentitiesType, LogType, LogEntry } from '@orbitdb/core'

		@@ -16,9 +16,11 @@ export const processInvitationCode = (mainWindow: BrowserWindow, code: string \|

Update orbitdb #2600

Are you sure you want to change the base?

Update orbitdb #2600

Conversation

leblowl commented Sep 5, 2024

Pull Request Checklist

(Optional) Mobile checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment