How to assign per-instance custom data to shader extending StandardMaterial

[snappa-app]

Hey folks!

I am wondering how to assign modifiable per-instance custom data to a shader. For example, controlling the color of a bunch of cubes all rendered with the same MeshMaterial3d. I've been struggling to find an example for this, so if we figure out a way to do this, I would be happy to add it to the bevy examples if we want.

Goals:

• Per-instance custom data for a bunch of meshes
• Ideally, the values in the custom data is controlled by a component so I can change the values at runtime
• Using an MaterialExtension of the StandardMaterial so I can take advantage of all of Bevy's awesome PBR

I am going to give you what I have so far in snippets, but I can extract it out to a reproducible example so we can iterate if it's helpful.

In material.rs, I have a struct Component that I'd like to embed in each instance. I also have the MaterialExtension:

use bevy::render::render_resource::ShaderType;
use bevy::{
  pbr::MaterialExtension,
  prelude::*,
  reflect::TypePath,
  render::render_resource::{AsBindGroup, ShaderRef},
};
use bevy_render::extract_component::ExtractComponent;

#[derive(Component, Clone, Copy, ExtractComponent, ShaderType, Reflect)]
pub struct CharacterMaterialParameters {
  pub shirt_color: LinearRgba,
}

#[derive(Asset, TypePath, AsBindGroup, Debug, Clone, Default)]
pub struct CharacterMaterialExtension {}

impl MaterialExtension for CharacterMaterialExtension {
  fn fragment_shader() -> ShaderRef {
    "shaders/character.wgsl".into()
  }
}

In character.wgsl I have the following to try and read from the CharacterMaterialParameters embedded in the GPU:

struct CharacterMaterialParameters {
    shirt_color: vec4<f32>,
}

@group(0) @binding(7) var<storage, read> character_material_parameters_buffer: array<CharacterMaterialParameters>;

@fragment
fn fragment(...) -> FragmentOutput {
...
    let tag = mesh_functions::get_tag(in.instance_index);
    let character_parameters = character_material_parameters_buffer[tag];
    out.color = character_parameters.shirt_color;   
}

When I spawn my entity, I add the following CharacterMaterialParameters and my extension material Components (also I do add a mesh). I also add a MeshTag:

pub fn spawn_characters(
  mut commands: Commands,
  mut character_materials: ResMut<
    Assets<ExtendedMaterial<StandardMaterial, CharacterMaterialExtension>>,
  >,
...
) {
  let material_handle = character_materials.add(ExtendedMaterial {
    base: StandardMaterial {
      base_color_texture: Some(assets.image.clone()),
      cull_mode: Some(Face::Back),
      alpha_mode: AlphaMode::Mask(0.5),
      unlit: false,
      perceptual_roughness: 0.5,
      reflectance: 0.15,
      emissive: LinearRgba::BLACK,

      ..Default::default()
    },
    extension: CharacterMaterialExtension {},
  });

  commands.spawn((
      MeshTag(index as u32),
      MeshMaterial3d(material_handle.clone()),
      CharacterMaterialParameters {
        shirt_color: LinearRgba::GREEN,
      }
  ));
}

In my plugin, I add the ExtractComponentPlugin and UniformComponentPlugin for my component:

pub struct CharacterPlugin;

impl Plugin for CharacterPlugin {
  fn build(&self, app: &mut App) {
    app.add_plugins(MaterialPlugin::<
      ExtendedMaterial<StandardMaterial, CharacterMaterialExtension>,
    >::default());
    app.register_type::<CharacterMaterialParameters>();
    app.add_plugins((
      ExtractComponentPlugin::<CharacterMaterialParameters>::default(),
      UniformComponentPlugin::<CharacterMaterialParameters>::default(),
    ));
  }
}

I get the following error with this setup:

thread 'Compute Task Pool (3)' panicked at ~/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/wgpu-24.0.3/src/backend/wgpu_core.rs:1303:26:
wgpu error: Validation Error

Caused by:
  In Device::create_render_pipeline, label = 'pbr_opaque_mesh_pipeline'
    Error matching ShaderStages(FRAGMENT) shader requirements against the pipeline
      Shader global ResourceBinding { group: 0, binding: 7 } is not available in the pipeline layout
        Binding is missing from the pipeline layout

My hunch is I'm missing something large, or mixing up concepts on my way to getting this working.

My main questions are:

• Is this setup even close to correct?
• What do I set @group(0) @binding(7) to? I've tried all combinations of groups 0-2, and bindings 0-2. I am wondering if there's a good way to inspect what is embedded in each of those bindings.
• Is there a way to configure what binding my CharacterMaterialParameters gets embedded in, or is that handled automatically by the ExtractComponentPlugin?

[snappa-app]

Upon further research, it appears that the challenge I am running into is due to mixing the StandardMaterial and per-instance data.

I am wondering how to resolve this. Is there a way to set the binding of the per-instance data?