DB properties storage refactoring

[tiensonqin]

Context and Problem Statement

• There're two ways to store block properties for both internal properties and user-defined properties:
  • Store all properties in a map for any block
    • Currently, it's stored in :block/properties, for example: a book that has authors and a published date will be something like:

    • {:block/properties {:block/name "a book"
                          :block/properties {:authors-uuid #{author-1-uuid author-2-uuid}
                                             :published-date-uuid date-page-uuid}}}

  • Use database attributes for properties

    • The above example will be stored as:

      ```clojure
      {:block/name "a book"
       :user.property/authors #{{:db/id 1} {:db/id 2}} ;; id 1 for author 1 and id 2 for author 2
       :user.property/published-date {:db/id 3} ;; id 3 represents the date page
       }
      ```
      

• There're some issues with Store all properties in a map for any block
  • Slow queries
    • To get all the values for a property, we have to scan all blocks that have defined :block/properties, not just the blocks which have the specific property
  • Painful to modify
    • Needs custom code handling to ensure the db is valid

      • Property keys need to be removed from blocks when deleting a property, for example, the property pair :authors-uuid #{...} needs to be removed from the block as below when deleting the property page authors

        ```clojure
        ;; block
        {:block/properties {:authors-uuid #{...}}}
        ```
        

      • Property UUIDs in property values need to be removed too when deleting a property
      • Changing property value cardinality (one to many or many to one) requires updating all the existing blocks that have this property
    • Updating a property's value for a block requires rebuilding the whole :block/properties

  • Painful to validate

    > I’ve also found the current schema to be painful to query and validate. To validate a property value by its type I had to [[invent a property pair](https://github.com/logseq/logseq/blob/c98db7cd3d19e4ac407073f8a3680a29be033d4d/deps/db/src/logseq/db/frontend/malli_schema.cljs#L59-L68)](https://github.com/logseq/logseq/blob/c98db7cd3d19e4ac407073f8a3680a29be033d4d/deps/db/src/logseq/db/frontend/malli_schema.cljs#L59-L68).  A property pair could be a good abstraction. It’s probably worth doing an ADR or at least a spike or two as I remember [[there were reservations](https://logseq.slack.com/archives/C04ENNDPDFB/p1687945735955859?thread_ts=1687937426.655469&cid=C04ENNDPDFB)](https://logseq.slack.com/archives/C04ENNDPDFB/p1687945735955859?thread_ts=1687937426.655469&cid=C04ENNDPDFB) about doing a property pair approach. If we migrate to another property schema it’s going to be expensive so hopefully that schema will last for more than a year  
    
    By Gabriel  
    

  • Painful to merge property changes for RTC
    • For example, there are two clients, A and B, who modified the same block, A set the background color to red, and B set the heading to level 2. Syncing :block/properties will overwrite one client's change for that block.
      Another example is that A adds a person to the "Authors" property, which can have multiple values, while B deletes an existing value. Syncing will lose either A's or B's operation.

Decision

- Use database attributes for properties
	- Create `db/ident` for both internal and user-defined properties
		- `logseq.x/y` for internal properties, for example:
			- `logseq.property/created-by-block`
			- `logseq.task/status`
		- `user.property/x` for user-defined properties
	- Page name unique index should be deleted, which will be handled by #11196
		- After discussions with [[Zhiyuan]], we agreed that multiple unique attributes including `:block/uuid`, `:block/name` and `:db/ident` could make RTC error-prone and hard to get it right.
		Another reason to remove `:block/name` unique index is how we can enable the same property names for different classes.
		- A property can only be queried with either `:db/ident` or `:db/id`

Positive Consequences

- We got real db refs and all the benefits
- Property queries and transactions are much easier
- One property change doesn't affect the other properties, which makes conflicts handling simple

Notice

• :block/left has been replaced with :block/order
• :block/namespace has been removed for the db version
• :block/name is no longer unique for the db version
• :block/macros has been removed, we used it for getting all the dsl queries, which can be addressed by using classes.
• This will affect file-based import since properties are stored as db attributes now @logseq-cldwalker
• RTC needs to be updated @RCmerci

TODO Address the following issues in other PRs

• :block/schema :properties are stored as refs, I'll handle [:block/schema :classes] soon. https://linear.app/logseq/issue/LOG-3037/migrate-blockschema-properties-and-classes-to-actual-datascript
• file version properties import (previously, those're stored in :block/properties
• fix DSL && Datalog queries

[logseq-cldwalker]

@tiensonqin Awesome work! ❤️ 🚀