-
Notifications
You must be signed in to change notification settings - Fork 107
fix(zero-cache): view-syncer close race w/ changeDesiredQueries #5506
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
| Branch | 0xcadams/pl-change-queries |
| Testbed | Linux |
Click to view all benchmark results
| Benchmark | File Size | Benchmark Result kilobytes (KB) (Result Δ%) | Upper Boundary kilobytes (KB) (Limit %) |
|---|---|---|---|
| zero-package.tgz | 📈 view plot 🚷 view threshold | 1,830.75 KB(+0.00%)Baseline: 1,830.72 KB | 1,867.33 KB (98.04%) |
| zero.js | 📈 view plot 🚷 view threshold | 244.11 KB(0.00%)Baseline: 244.11 KB | 248.99 KB (98.04%) |
| zero.js.br | 📈 view plot 🚷 view threshold | 66.92 KB(0.00%)Baseline: 66.92 KB | 68.26 KB (98.04%) |
|
| Branch | 0xcadams/pl-change-queries |
| Testbed | self-hosted |
Click to view all benchmark results
| Benchmark | Throughput | Benchmark Result operations / second (ops/s) (Result Δ%) | Lower Boundary operations / second (ops/s) (Limit %) |
|---|---|---|---|
| 1 exists: track.exists(album) | 📈 view plot 🚷 view threshold | 13,719.61 ops/s(-1.58%)Baseline: 13,940.18 ops/s | 11,293.71 ops/s (82.32%) |
| 10 exists (AND) | 📈 view plot 🚷 view threshold | 175,141.93 ops/s(-14.18%)Baseline: 204,078.33 ops/s | 161,419.43 ops/s (92.16%) |
| 10 exists (OR) | 📈 view plot 🚷 view threshold | 3,385.91 ops/s(-15.47%)Baseline: 4,005.48 ops/s | 3,258.26 ops/s (96.23%) |
| 12 exists (AND) | 📈 view plot 🚷 view threshold | 154,359.95 ops/s(-13.94%)Baseline: 179,354.67 ops/s | 140,942.40 ops/s (91.31%) |
| 12 exists (OR) | 📈 view plot 🚷 view threshold | 2,803.39 ops/s(-17.44%)Baseline: 3,395.66 ops/s | 2,758.39 ops/s (98.39%) |
| 12 level nesting | 📈 view plot 🚷 view threshold | 2,627.69 ops/s(-11.01%)Baseline: 2,952.80 ops/s | 2,360.98 ops/s (89.85%) |
| 2 exists (AND): track.exists(album).exists(genre) | 📈 view plot 🚷 view threshold | 5,022.37 ops/s(-4.18%)Baseline: 5,241.35 ops/s | 4,271.64 ops/s (85.05%) |
| 3 exists (AND) | 📈 view plot 🚷 view threshold | 1,760.14 ops/s(-13.84%)Baseline: 2,042.96 ops/s | 1,663.16 ops/s (94.49%) |
| 3 exists (OR) | 📈 view plot 🚷 view threshold | 873.87 ops/s(-14.41%)Baseline: 1,020.97 ops/s | 822.32 ops/s (94.10%) |
| 5 exists (AND) | 📈 view plot 🚷 view threshold | 280.92 ops/s(-12.59%)Baseline: 321.39 ops/s | 258.91 ops/s (92.16%) |
| 5 exists (OR) | 📈 view plot 🚷 view threshold | 151.79 ops/s(-10.23%)Baseline: 169.09 ops/s | 135.20 ops/s (89.07%) |
| Nested 2 levels: track > album > artist | 📈 view plot 🚷 view threshold | 3,885.22 ops/s(-14.58%)Baseline: 4,548.30 ops/s | 3,628.24 ops/s (93.39%) |
| Nested 4 levels: playlist > tracks > album > artist | 📈 view plot 🚷 view threshold | 623.56 ops/s(-16.32%)Baseline: 745.22 ops/s | 598.78 ops/s (96.03%) |
| Nested with filters: track > album > artist (filtered) | 📈 view plot 🚷 view threshold | 3,375.50 ops/s(-11.31%)Baseline: 3,806.11 ops/s | 3,123.91 ops/s (92.55%) |
| planned: playlist.exists(tracks) | 📈 view plot 🚷 view threshold | 573.71 ops/s(-7.79%)Baseline: 622.15 ops/s | 519.29 ops/s (90.51%) |
| planned: track.exists(album) OR exists(genre) | 📈 view plot 🚷 view threshold | 156.25 ops/s(-5.26%)Baseline: 164.93 ops/s | 143.32 ops/s (91.72%) |
| planned: track.exists(album) where title="Big Ones" | 📈 view plot 🚷 view threshold | 7,139.17 ops/s(-5.84%)Baseline: 7,581.60 ops/s | 6,507.57 ops/s (91.15%) |
| planned: track.exists(album).exists(genre) | 📈 view plot 🚷 view threshold | 37.02 ops/s(-6.40%)Baseline: 39.55 ops/s | 33.34 ops/s (90.05%) |
| planned: track.exists(album).exists(genre) with filters | 📈 view plot 🚷 view threshold | 4,916.34 ops/s(-7.88%)Baseline: 5,336.82 ops/s | 4,475.05 ops/s (91.02%) |
| planned: track.exists(playlists) | 📈 view plot 🚷 view threshold | 3.71 ops/s(-7.52%)Baseline: 4.01 ops/s | 3.34 ops/s (90.12%) |
| unplanned: playlist.exists(tracks) | 📈 view plot 🚷 view threshold | 556.51 ops/s(-8.15%)Baseline: 605.90 ops/s | 507.35 ops/s (91.17%) |
| unplanned: track.exists(album) OR exists(genre) | 📈 view plot 🚷 view threshold | 41.60 ops/s(-7.44%)Baseline: 44.94 ops/s | 37.10 ops/s (89.19%) |
| unplanned: track.exists(album) where title="Big Ones" | 📈 view plot 🚷 view threshold | 51.46 ops/s(-8.75%)Baseline: 56.39 ops/s | 47.35 ops/s (92.02%) |
| unplanned: track.exists(album).exists(genre) | 📈 view plot 🚷 view threshold | 36.76 ops/s(-6.34%)Baseline: 39.25 ops/s | 32.87 ops/s (89.42%) |
| unplanned: track.exists(album).exists(genre) with filters | 📈 view plot 🚷 view threshold | 51.56 ops/s(-6.60%)Baseline: 55.21 ops/s | 46.51 ops/s (90.21%) |
| unplanned: track.exists(playlists) | 📈 view plot 🚷 view threshold | 3.65 ops/s(-8.67%)Baseline: 4.00 ops/s | 3.36 ops/s (92.01%) |
| zpg: all playlists | 📈 view plot 🚷 view threshold | 5.21 ops/s(-7.01%)Baseline: 5.61 ops/s | 4.93 ops/s (94.63%) |
| zql: all playlists | 📈 view plot 🚷 view threshold | 6.44 ops/s(-15.59%)Baseline: 7.63 ops/s | 5.96 ops/s (92.50%) |
| zql: edit for limited query, inside the bound | 📈 view plot 🚷 view threshold | 185,131.85 ops/s(-12.79%)Baseline: 212,274.10 ops/s | 174,912.27 ops/s (94.48%) |
| zql: edit for limited query, outside the bound | 📈 view plot 🚷 view threshold | 208,615.06 ops/s(-3.29%)Baseline: 215,717.07 ops/s | 168,465.08 ops/s (80.75%) |
| zql: push into limited query, inside the bound | 📈 view plot 🚷 view threshold | 101,683.54 ops/s(-5.92%)Baseline: 108,078.30 ops/s | 89,742.60 ops/s (88.26%) |
| zql: push into limited query, outside the bound | 📈 view plot 🚷 view threshold | 358,744.43 ops/s(-10.20%)Baseline: 399,502.66 ops/s | 303,496.98 ops/s (84.60%) |
| zql: push into unlimited query | 📈 view plot 🚷 view threshold | 297,439.24 ops/s(-8.65%)Baseline: 325,589.62 ops/s | 253,869.38 ops/s (85.35%) |
| zqlite: all playlists | 📈 view plot 🚷 view threshold | 1.50 ops/s(-14.60%)Baseline: 1.76 ops/s | 1.40 ops/s (92.97%) |
| zqlite: edit for limited query, inside the bound | 📈 view plot 🚷 view threshold | 64,669.11 ops/s(-14.87%)Baseline: 75,963.62 ops/s | 61,431.63 ops/s (94.99%) |
| zqlite: edit for limited query, outside the bound | 📈 view plot 🚷 view threshold | 67,065.17 ops/s(-11.18%)Baseline: 75,509.28 ops/s | 56,209.03 ops/s (83.81%) |
| zqlite: push into limited query, inside the bound | 📈 view plot 🚷 view threshold | 3,758.94 ops/s(-6.95%)Baseline: 4,039.88 ops/s | 3,626.31 ops/s (96.47%) |
| zqlite: push into limited query, outside the bound | 📈 view plot 🚷 view threshold | 85,611.95 ops/s(-3.26%)Baseline: 88,494.60 ops/s | 74,808.66 ops/s (87.38%) |
| zqlite: push into unlimited query | 📈 view plot 🚷 view threshold | 119,499.06 ops/s(-3.59%)Baseline: 123,949.91 ops/s | 97,945.38 ops/s (81.96%) |
|
| Branch | 0xcadams/pl-change-queries |
| Testbed | self-hosted |
Click to view all benchmark results
| Benchmark | Throughput | Benchmark Result operations / second (ops/s) x 1e3 (Result Δ%) | Lower Boundary operations / second (ops/s) x 1e3 (Limit %) |
|---|---|---|---|
| src/client/custom.bench.ts > big schema | 📈 view plot 🚷 view threshold | 133.84 ops/s x 1e3(-3.36%)Baseline: 138.49 ops/s x 1e3 | 117.66 ops/s x 1e3 (87.91%) |
| src/client/zero.bench.ts > basics > All 1000 rows x 10 columns (numbers) | 📈 view plot 🚷 view threshold | 2.23 ops/s x 1e3(-9.77%)Baseline: 2.47 ops/s x 1e3 | 2.09 ops/s x 1e3 (93.58%) |
| src/client/zero.bench.ts > pk compare > pk = N | 📈 view plot 🚷 view threshold | 61.17 ops/s x 1e3(-4.56%)Baseline: 64.09 ops/s x 1e3 | 52.26 ops/s x 1e3 (85.43%) |
| src/client/zero.bench.ts > with filter > Lower rows 500 x 10 columns (numbers) | 📈 view plot 🚷 view threshold | 3.61 ops/s x 1e3(-4.72%)Baseline: 3.79 ops/s x 1e3 | 3.27 ops/s x 1e3 (90.49%) |
darkgnotic
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for investigating and proposing a fix @0xcadams!
I think the right solution would be to wait for the all logic running in the CVR lock to complete before performing the cleanup in the main run() method.
Does something like this pass your new test?
async #cleanup(err?: unknown) {
this.#stopTTLClockInterval();
this.#stopExpireTimer();
for (const client of this.#clients.values()) {
if (err) {
client.fail(err);
} else {
client.close(`closed clientGroupID=${this.id}`);
}
}
// Wait for existing lock logic to complete before
// cleaning up the pipelines and closing db connections.
await this.#runInLockWithCVR(() => {});
this.#pipelines.destroy();
}6807b53 to
acbc8b2
Compare
OK, I made this change - makes sense and thank you for the feedback! |
darkgnotic
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @0xcadams !
We saw a race condition throwing
Database connection is not openfrom a customer when view syncer is shutting down:When the view-syncer shuts down and closes the snapshotter SQLite connections while a
changeDesiredQueriestask is still running on the lock,currentPermissions()will still try to read from a closed DB and throwsTypeError: The database connection is not open.The first commit to this branch is a test repro, and the second is a regression test + proposed fix.