One to Many Asset Files and Asset Transforms Kickoff

[cart]

Scenario

We want to be able to read a "source" asset file (which might contain labeled subassets like scenes, meshes, images, etc), process each of these assets with their own configured processor logic, and write the results to disk as individual asset files.

For example, we need to be able to load a GLTF file, convert meshes to meshlets, save them to disk, and then load the processed versions. The final processed scenes should use the processed meshlets.

Why?

• In order to have sub-asset/labeled asset processing, we need a way to take the initial group of assets provided by an AssetLoader, selectively run logic on specific assets in that group, and then persist the results.
  • With the current system this is possible, but it requires keeping everything together through the whole process: load the initial group of assets, run logic on the group, persist the group to some "combined" file. This creates problems because a "combined" format needs to be devised to put all of the assets in the same file, and it is harder to write logic for specific asset types.
• Supporting many output files for processed assets allows us to selectively load specific subassets without needing to reload the entire root asset. In practice, this is a pretty big performance problem in the current system.
• Supporting many output files makes it easier to inspect and debug processed assets. It would allow developers to open up processed sub-asset files individually (maybe even with the "correct" file extension).

How?

1. Define a new scheme to identify labeled sub-assets in the file system. Currently when requesting either a processed or unprocessed /scene/house.gltf#Mesh0, we can just ask the filesystem for /scene/house.gltf#Mesh0. If Mesh0 has its own file, how will we store and find it in the filesystem?

   • Do we make /scene/house.gltf a folder, and then assign names based on labels for each asset? (ex: /scene/house.gltf/Mesh0, /scene/house.gltf/_ (root), etc)
     • Pros: Directly maps to asset paths to filesystem paths. No external index required.
     • Cons: File name context is stripped (Can we safely add "target file types" like Mesh0.bmesh?). For assets that have no sub assets, it adds a level of weirdness / obfuscation because processed root assets now look like /scene/house.gltf/_ instead of /scene/house.gltf. However maybe that is still an improvement because /scene/house.gltf probably isn't actually a gltf after it is processed.
   • If we choose to add "processed file extensions" (ex: Mesh0.bmesh as mentioned above), how will we find the asset in the filesystem? Will we do a linear scan over all files in the folder to find Mesh0.bmesh to match our Mesh0 query? Will we store an index from Mesh0 -> Mesh0.bmesh in the processed meta file for the root, and read that meta file for every sub-asset request? Neither option seems ideal from a performance perspective. We could store the mapping in Mesh0.meta, but that breaks the current assumption that the meta file is ASSET_PATH.meta. Not including the extension certainly makes things simpler and cheaper.
   • If we adopt a scheme like the one above, is there a way to make it compatible with the "unprocessed" scheme? Do we need a hard runtime distinction between the two? Currently "processed" vs "unprocessed asset" is solely a matter of perspective, which has been a powerful simplifying architectural concept. It also leaves the door open to future features like multiple processing "phases", as the inputs and outputs of processing are "the same type" (ex: we could have an "assets -> import" phase followed by an "import -> processed" phase). One way to think about this: do we need to split out a new ProcessedAssetReader from AssetReader? Or can we expand AssetReader to encompass these scenarios?

2. Adapt Asset loading to account for the new approach

   • If the new scheme in (1) is incompatible with the "unprocessed" scheme, we'll need to have some way for the Asset Server to pick between the two schemes.
   • We'll need AssetReader (or ProcessedAssetReader, depending on the solution to (1)) to be able to request labeled assets in the filesystem.
   • The AssetServer will also need different logic for "unprocessed" vs "processed and split" because "unprocessed" assets require loading the root to retrieve sub assets, whereas "processed and split" assets do not.

3. Adapt Asset processing to account for the new approach

   • Currently AssetMeta has a Process trait which is the "generic / least opinionated" processor trait: read input bytes, write output bytes. It enforces the appropriate constraints on data access, tracks dependencies, and generally facilitates inserting custom process logic into the AssetProcessor.

   • We have built in Process implementations that are more opinionated, such as LoadAndSave<L: AssetLoader, S: AssetSaver>, which uses an AssetLoader to load the asset and then runs a type-compatible AssetSaver on the results to convert it to the bytes that will be written (and informs the asset server about the final format of those bytes, which gets written to the processed meta file). We also have LoadTransformAndSave, which inserts an AssetTransformer between load and save steps. We need to expand the Process system and the Process implementations to support:

     1. In the meta file for the root asset, for a given asset label (or assets of a given type), we need to be able to define how an asset will be transformed and saved. For example, for the Mesh0 labeled sub asset, run it through the MeshToMeshletTransformer, then save it with the MeshletSaver.
        • I suspect that the current serde approach we're using will not be able to accommodate this level of dynamic typing (ex: a Map from Label to Box<dyn ProcessSubAsset>. We might need some custom serde logic. Note that we need to be able to serialize and deserialize asset metadata.
     2. When the AssetProcessor is running alongside a Bevy App, it "locks" the root asset and forces the Bevy App AssetServer to asynchronously wait until the processing is finished (aka until the final asset and the meta are both written). I suspect we could make this more granular + parallel (aka if you request a sub-asset, wait until the sub asset specifically is ready). However as a first impl, I think we should still use root-asset granularity to keep the changes scoped. You should be able to reuse the current system, but it might require a couple of tweaks here and there. See ProcessorGatedReader (an AssetReader impl that wraps an arbitrary AssetReader and awaits processing results) for more info.
     3. AssetTransformers and AssetSavers currently assume they are receiving both a root asset and labeled sub assets. We might want to rethink this in this new context. We do not support multiple levels of nesting when loading assets (aka a labeled subasset cannot have labeled subassets beneath it). That means that when transforming and saving sub-assets, they will never have subassets. Is it ok to use the current system and just never use transformers / savers that require labeled subassets on labeled subassets? Should we rework them to not support transforming + saving subassets at all? Should we have two sets of traits for the "unprocessed root with subassets" context and the "split out individual assets" context?

   • We also need to ensure that when cleaning up processed assets for a source asset (ex: when the source asset changes), that all processed sub assets are also cleaned up. If we use the folder approach specified in (1), then that would mean deleting the processed asset folder and its contents (ex: /scene/house.gltf).

   • In practice, Asset Metadata might look something like this:

     (
         meta_format_version: "1.0",
         asset: Process(
             processor: "ProcessAndSaveSubAssets<GltfLoader>",
             settings: (
                 loader_settings: GltfLoaderSettings {
                     // settings here
                 },
                 sub_assets: {
                     // Convert MyMesh to a Meshlet
                     "Label(MyMesh)": TransformAndSave<MeshToMeshletMesh, MeshletMeshSaver> {
                         // settings here
                     },
                     // Compress _all_ images
                     "Type(Image)": Save<CompressedImageSaver> {
                         // settings here
                     }
                 }
             )
         ),
     )

4. Allow AssetLoaders to query the final processed asset type of a given asset label, as configured in (3). This would allow things like the GltfLoader to detect that a Mesh subasset (ex: Mesh0) is configured to be processed into a MeshletMesh (via a MeshToMeshletTransformer and persisted with a MeshletSaver), and then adjust scenes to assume it will have a Handle<MeshletMesh> instead of Handle<Mesh>. Something like:

   if load_context.processed_type("Mesh0") == Some(TypeId::of::<MeshletMesh>()) {
       // this should fail if "Mesh0" is not configured to be processed into a Meshlet
       let handle: Handle<MeshletMesh> = load_context.add_processed_labeled_asset("Mesh0", Mesh::default()).unwrap();
   } else {
       let handle: Handle<Mesh> = load_context.add_labeled_asset("Mesh0", Mesh::default());
   }
   
   // This could also be phrased as
   if let Ok(handle) = load_context.add_processed_labeled_asset::<MeshletMesh>("Mesh0", Mesh::default()) {
   } else {
       let handle: Handle<Mesh> = load_context.add_labeled_asset("Mesh0", Mesh::default());
   }
   
   // Or maybe we could just make `add_labeled_asset` _require_ the type to match the processed configuration, if it is provided?
   if let Ok(handle) = load_context.add_labeled_asset::<MeshletMesh>("Mesh0", Mesh::default()) {
   } else {
       // that does slightly complicate the "normal unprocessed" case because it decouples the output handle type from the input asset value type.
       // this works
       let handle: Handle<Mesh> = load_context.add_labeled_asset("Mesh0", Mesh::default());
       // this would become a compiler failure because the type cannot be inferred.
       let handle = load_context.add_labeled_asset("Mesh0", Mesh::default());
   }