Incremental Watched Queries

[stevensJourney]

Overview

Our current Watched query implementations emit results whenever a change to a dependant SQLite table occurs. The table changes might not affect the query result set, but we still query and emit a new result set for each table change. The result sets typically contain the same data, but these results are new Array/object references which will cause re-renders in certain frameworks like React.

This PR overhauls, improves and extends upon the existing watched query implementations by introducing incremental watched queries.

Incrementally Watched Queries can be constructed with varying behaviour. This PR introduces the concept of comparison and differential watched queries.

Comparison based queries behave similar to standard watched queries. These queries still query the SQLite DB under the hood on each dependant table change, but they compare the result set and only incrementally yield results if a change has been made. The latest query result is yielded as the result set.

Differential queries watch a SQL query and report detailed information on the changes between result sets. This gives additional information such as the added, removed, updated rows between result set changes.

Implementation

The logic required for incrementally watched queries requires additional computation and introduces additional complexity to the implementation. For these reasons a new concept of a WatchedQuery class is introduced, along with a new query method allows building a instances of WatchedQuerys via watch and differentialWatch methods.

// Create an instance of a WatchedQuery.
const listsQuery = powersync
  .query<EnhancedListRecord>({
    sql: /* sql */ `
      SELECT
        ${LISTS_TABLE}.*,
        COUNT(${TODOS_TABLE}.id) AS total_tasks,
        SUM(
          CASE
            WHEN ${TODOS_TABLE}.completed = true THEN 1
            ELSE 0
          END
        ) as completed_tasks
      FROM
        ${LISTS_TABLE}
        LEFT JOIN ${TODOS_TABLE} ON ${LISTS_TABLE}.id = ${TODOS_TABLE}.list_id
      GROUP BY
        ${LISTS_TABLE}.id;
    `
  })
  .differentialWatch();

The listsQuery is smart, it:

• Automatically reprocesses itself if the PowerSync schema has been updated with updateSchema
• Automatically closes itself when the PowerSync client has been closed.
• Allows for the query parameters to be updated after instantiation.
• Allows shared listening to state changes.

// The registerListener method can be used multiple times to listen for updates.
// The returned dispose function can be used to unsubscribe from the updates.
const disposeSubscriber = listsQuery.registerListener({
  onData: (data) => {
    // This callback will be called whenever the data changes.
    // The data is the result of the executor.
    console.log('Data updated:', data);
  },
  onStateChange: (state) => {
    // This callback will be called whenever the state changes.
    // The state contains metadata about the query, such as isFetching, isLoading, etc.
    console.log(
      'State changed:', 
      state.error, 
      state.isFetching, 
      state.isLoading, 
      state.data
    );
  },
  onError: (error) => {
    // This callback will be called if the query fails.
    console.error('Query error:', error);
  }
});

WatchedQuery instances retain the latest state in memory. Sharing WatchedQuery instances can be used to introduce caching and reduce the number of duplicate DB queries between components.

The incremental logic is customisable. Diff based queries can specify custom logic for performing diffs on the relevant data set. By default a JSON.stringify approach is used. Different data sets might have more optimal implementations.

const watch = powersync
  .query({
    sql: /* sql */ `
      SELECT
        *
      FROM
        assets
    `,
    mapper: (raw) => {
      return {
        id: raw.id as string,
        make: raw.make as string
      };
    }
  })
  .differentialWatch({
    differentiator: {
      identify: (item) => item.id,
      compareBy: (item) => JSON.stringify(item)
    }
  });

Updates to query parameters can be performed in a single place, affecting all subscribers.

watch.updateSettings({
      query: new GetAllQuery({ sql: `SELECT * FROM assets OFFSET ? LIMIT 100`, parameters: [newOffset] })
    });

Reactivity

The existing watch method and Reactivity packages have been updated to use incremental queries with differentiation defined as an opt-in feature (defaults to no changes).

powersync.watch(
  'select * from assets',
  [],
  {
    onResult: () => {
      // This callback will be called whenever the data changes.
      console.log('Data updated');
    }
  },
  {
    comparator: {
      checkEquality: (current, previous) => {
        // This comparator will only report updates if the data changes.
        return JSON.stringify(current) === JSON.stringify(previous);
      }
    }
  }
);

/// React hooks
const { data, isLoading, isFetching } = useQuery(`SELECT * FROM cats WHERE breed = 'tabby'`, [], {
      differentiator: {
            identify: (item) => item.id,
            compareBy: (item) => JSON.stringify(item)
      }
  })

New hooks have also been added to use shared WatchedQuery instances.

const assetsQuery = powersync.query({
/// ....
}).watch();

/// In the component
export const MyComponent = () => {
   // In React one could import the `assetsQuery` or create a context provider for various queries
   const { data } = useWatchedQuerySubscription(assetsQuery)
}

The Vue and React hooks packages have been updated to remove duplicate implementations of common watched query logic. Historically these packages relied on manually implementing watched queries based off the onChange API in order to cater for exposing additional state and custom query executors. The new WatchedQuery APIs now support all the hook packages' requirements - this effectively reduces the heavy lifting in reactivity packages.

The React Supabase Todolist demo has been updated with some best practices for reducing re-renders using comparison based incrementally watched queries.

Differential Queries

These watched queries report the changes between result sets. A WatchedQueryDifferential can be accessed by registering a listener on the query.

export interface WatchedQueryDifferential<RowType> {
  readonly added: ReadonlyArray<Readonly<RowType>>;
  /**
   * The entire current result set.
   * Array item object references are preserved between updates if the item is unchanged.
   *
   * e.g. In the query
   * ```sql
   *  SELECT name, make FROM assets ORDER BY make ASC;
   * ```
   *
   * If a previous result set contains an item (A) `{name: 'pc', make: 'Cool PC'}` and
   * an update has been made which adds another item (B) to the result set (the item A is unchanged) - then
   * the updated result set will be contain the same object reference, to item A, as the previous result set.
   * This is regardless of the item A's position in the updated result set.
   */
  readonly all: ReadonlyArray<Readonly<RowType>>;
  readonly removed: ReadonlyArray<Readonly<RowType>>;
  readonly updated: ReadonlyArray<WatchedQueryRowDifferential<Readonly<RowType>>>;
  readonly unchanged: ReadonlyArray<Readonly<RowType>>;
}

// Listening
watchedQuery.registerListener({
    onDiff: (diff) => console.log(diff)
})

A common use case for this is processing newly created items as they are added. The YJS React Supabase Text Collab demo has been updated to take advantage of this feature. Document updates are watched via a differential incremental query. New updates are passed to YJS for consolidation as they are synced.

Early Access

This can be tested by applying the following package versions to your project.

{
  "@powersync/common": "0.0.0-dev-20250714151300",
  "@powersync/node": "0.0.0-dev-20250714151300",
  "@powersync/op-sqlite": "0.0.0-dev-20250714151300",
  "@powersync/react": "0.0.0-dev-20250714151300",
  "@powersync/react-native": "0.0.0-dev-20250714151300",
  "@powersync/tanstack-react-query": "0.0.0-dev-20250714151300",
  "@powersync/vue": "0.0.0-dev-20250714151300",
  "@powersync/web": "0.0.0-dev-20250714151300"
}

[changeset-bot]

🦋 Changeset detected

Latest commit: 1f1b076

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages

Name	Type
@powersync/common	Minor
@powersync/vue	Minor
@powersync/react	Minor
@powersync/web	Minor
@powersync/node	Patch
@powersync/op-sqlite	Patch
@powersync/react-native	Patch
@powersync/tanstack-react-query	Patch
@powersync/diagnostics-app	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[stevensJourney]

Update:

I've updated the implementation to guard the new watch methods behind the new query and customQuery methods on AbstractPowerSyncDatabase, following Ralf's recommendation. The PR description now includes usage examples.

Comparison based queries are now accessed via db.query().watch(). This watched query method does not perform a full diff on result sets; instead, it can exit early as soon as a change is detected. Use this method when you don't need to know the specifics of what changed and don't need to maintain previous object references in the result set. For cases where a full diff and object reference preservation are required, use db.query(...).differentialWatch().

The DifferentialWatchedQuery has been improved to better maintain previous array object references, further reducing unnecessary renders during incremental updates.

The WatchedQuery implementation now explicitly defines WatchedQueryState as readonly, making all state members and data items immutable. The Vue hooks are still in version 0; their APIs have been updated to also convey state as readonly. The React hooks package is already at a stable v1 release, so changing the API to report readonly state would be a breaking change. Backwards compatibility is maintained by conditionally applying readonly qualifiers if a differentiator parameter is supplied to the hook. Existing user code should not be affected.

An automated React profiling benchmark suite has been added in packages/react/tests/profile.test.tsx to measure the impact of differential queries on render performance. This test validates how using React memoization together with differential queries can significantly reduce render times. According to the results, incremental updates to the list widget render 60–80% faster compared to standard query methods. For details on the testing methodology, refer to the test file.
TLDR: The test starts with an initial amount of list items being rendered. We then incrementally add new items while measuring the render time of the entire list widget. Differential queries coupled with React memoization allow the incremental renders to only render the newly added item widgets, while the standard query methods re-render the entire widget. The graph below illustrates the render time for a list widget with various query and memoization methods.

[rkistner]

I feel like the previous approach of using the Set was simpler, since it didn't require an additional field in the database to filter the updates. We could still use the Set to only filter out the local updates, or alternatively apply those updates again (should be a no-op to apply the same update again).

[stevensJourney]

I assumed the main reason for having the Set was to avoid re-applying those updates. Given it's a no-op, I've removed this. With the differential queries we will be doing less no-ops, but we will currently still re-apply local edits.

[rkistner]

When I tested this with only one tab open, I often get into a state where the initial "diff" is sent with every update, e.g. "all: 20 rows, added: 20 rows". Haven't debugged further to figure out what is going wrong.

[stevensJourney]

This was due to using the logic of the onStateChange listener. The diff on the state was relative to the last time the data changed, but the onStateChange hook fires for any state change e.g. isFetching alternating. This is a good argument for the comment below. I've removed the diff from the state now. This query has also been updated.

[rkistner]

I haven't confirmed with testing, but it feels like accessing the differential state on the query could easily lead to race conditions where updates may be missed. Specifically, there could potentially be a case where the state is processed twice before it is processed in the app, leading to the diff from the first update to be missing.

With the normal async listeners this may not be an issue if the query waits for all listeners to return before processing the next one, but React hooks for example may be called at any later time.

So I'm wondering if we could send the diff in the callback, rather than the state on the query?

React hooks may have bigger issues though, since we have no control over when/how many times they are called. I'm not sure if it is actually feasible to expose the diff over a react hook's state at all.

[stevensJourney]

FWIW the listeners here are fired from within the onChange callback which uses a ControlledExecutor to ensure sequential processing.

This is a fair point though as noted in the YJS demo comment. I've removed the diff from the state. The diff can now be accessed by registering an onDiff listener.