Skip to content

Latest commit

 

History

History
257 lines (189 loc) · 12.4 KB

clatter_objects.md

File metadata and controls

257 lines (189 loc) · 12.4 KB
Clatter

Object audio data

When the Clatter add-on initializes, it will request output data from the build to determine which objects are in the scene. On the next communicate() call, Clatter will "clatterize" each object in the scene.

Every "clatterized" object has audio-physical values that will determine what sort of sound it makes when it collides with another clatterized object. In the frontend TDW API, this data is stored as a ClatterObject class.

Default object data

In TDW, many models have predefined ClatterObject values, which are stored in DEFAULT_OBJECTS:

from tdw.physics_audio.clatter_object import DEFAULT_OBJECTS

for model_name in DEFAULT_OBJECTS:
    print(model_name)
    clatter_object = DEFAULT_OBJECTS[model_name]

The Clatter add-on will automatically apply the default ClatterObject data to objects in the scene.

Derived object data

If there are objects in the scene that aren't stored in DEFAULT_OBJECTS, the Clatter add-on will automatically try to derive reasonable values for them based on existing pre-defined data. For example, if there is a fork in the scene, Clatter might use mean values of all of the forks in DEFAULT_OBJECTS for this new fork model.

There are a few ways to control how ClatterObject data is derived using constructor parameters:

  • environment is either a ClatterObject or an ImpactMaterial that sets the Clatter object data for the environment (i.e. the floor).
  • robot_material is the ImpactMaterial used for all robot joints.
  • default_object is a ClatterObject used in situations in which all other attempts to derive ClatterObject data fail.

User-defined object data

You can define your own object data by setting the objects constructor parameter. This is a dictionary where the key is an object ID and the value is a ClatterObject:

from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.audio_initializer import AudioInitializer
from tdw.add_ons.clatter import Clatter
from tdw.physics_audio.impact_material import ImpactMaterial
from tdw.physics_audio.clatter_object import ClatterObject

c = Controller()
commands = [TDWUtils.create_empty_room(12, 12)]
object_id = c.get_unique_id()
model_name = "vase_02"
mass = 2
clatter_object = ClatterObject(impact_material=ImpactMaterial.ceramic,
                               amp=0.3,
                               resonance=0.1,
                               size=3)
commands.extend(c.get_add_physics_object(model_name=model_name,
                                         object_id=object_id,
                                         position={"x": 0, "y": 2, "z": 0},
                                         default_physics_values=False,
                                         dynamic_friction=0.2,
                                         static_friction=0.3,
                                         bounciness=0.2,
                                         mass=mass))
camera = ThirdPersonCamera(avatar_id="a",
                           position={"x": 1, "y": 1.6, "z": -2},
                           look_at={"x": 0, "y": 0.5, "z": 0})
audio = AudioInitializer(avatar_id="a")

# Initialize Clatter with user-defined object audio data.
clatter = Clatter(simulation_amp=0.9, objects={object_id: clatter_object})

c.add_ons.extend([camera, audio, clatter])
# Create the scene.
c.communicate(commands)
for i in range(150):
    c.communicate([])
c.communicate({"$type": "terminate"})

impact_material and size

Every ClatterObject has an ImpactMaterial, which defines the physical material of the object and the data Clatter will use to generate audio samples.

Internally, every impact material has six varieties defined by "size bucket" values ranging from 0 to 5. For example, wood_soft is subdivided into wood_soft_0, wood_soft_1, etc.

The size parameter in the ClatterObject constructor controls the size bucket.

You should use smaller values for size for smaller objects. If you aren't sure what value to use, you can get the bounds of the object from either Bounds output data or the model record and then call Clatter.get_size(bounds):

from tdw.add_ons.clatter import Clatter
from tdw.librarian import ModelLibrarian

model_name = "vase_02"
librarian = ModelLibrarian("models_core.json")
record = librarian.get_record(model_name)
size = Clatter.get_size(record)

amp and resonance

The amp parameter affects the overall loudness of the sound and resonance affects the decay times of audio generated by this object. Both amp and resonance vary per-model rather than per-ImpactMaterial. For example, a fork and a metal tray can have different amp and resonance values despite having the same ImpactMaterial.

Internal, amp values will be clamped to be between 0 and 1, while resonance will be clamped to be at least 0. Usually, resonance should be between 0 and 1, but it can be greater than 1.

dynamic_friction and static_friction

The dynamic_friction and static_friction values are found in Controller.get_add_physics_object, not the constructor of ClatterObject.

As with non-Clatter scenarios, you can choose to either let TDW define the friction values or to set them manually. In the context of Clatter, (and in contrast to amp and resonance values), friction values are associated with certain ImpactMaterials. See: tdw.impact_material_constants:

from tdw.physics_audio.impact_material import ImpactMaterial
from tdw.physics_audio.impact_material_constants import DYNAMIC_FRICTION, STATIC_FRICTION

impact_material = ImpactMaterial.ceramic
dynamic_friction = DYNAMIC_FRICTION[impact_material]
static_fricction = STATIC_FRICTION[impact_material]

The dynamic_friction and static_friction values must be between 0 and 1.

bounciness

Like dynamic_friction and static_friction, the bounciness value is handled in Controller.get_add_physics_object, not the constructor of ClatterObject, and must be between 0 and 1.

Unlike dynamic_friction and static_friction, bounciness tends to vary per-model rather than per-ImpactMaterial.

The best way to set a reasonable bounciness value is to use values of similar objects in DEFAULT_OBJECTS.

mass and fake_mass

As with non-Clatter scenarios, the mass of the object is either automatically set or explicitly defined in Controller.get_add_physics_object.

In many Clatter use-cases, researchers will want to create implausible audio sounds, which often involves creating an object that sounds more or less massive than it actually is. The ClatterObject constructor includes an optional fake_mass parameter. If set, Clatter will generate audio using the fake mass rather than the true mass:

from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.audio_initializer import AudioInitializer
from tdw.add_ons.clatter import Clatter
from tdw.physics_audio.impact_material import ImpactMaterial
from tdw.physics_audio.clatter_object import ClatterObject

"""
Generate implausible audio.
"""

c = Controller()
commands = [TDWUtils.create_empty_room(12, 12)]
object_id = c.get_unique_id()
model_name = "vase_02"
clatter_object = ClatterObject(impact_material=ImpactMaterial.wood_soft,
                               amp=0.1,
                               resonance=0.1,
                               size=4,
                               fake_mass=100)
commands.extend(c.get_add_physics_object(model_name=model_name,
                                         object_id=object_id,
                                         position={"x": 0, "y": 2, "z": 0},
                                         default_physics_values=False,
                                         dynamic_friction=0.2,
                                         static_friction=0.3,
                                         bounciness=0.9,
                                         mass=1))
camera = ThirdPersonCamera(avatar_id="a",
                           position={"x": 1, "y": 1.6, "z": -2},
                           look_at={"x": 0, "y": 0.5, "z": 0})
audio = AudioInitializer(avatar_id="a")
clatter = Clatter(simulation_amp=0.9, objects={object_id: clatter_object})
c.add_ons.extend([camera, audio, clatter])
# Create the scene.
c.communicate(commands)
for i in range(150):
    c.communicate([])
c.communicate({"$type": "terminate"})

scrape_model and scrape audio

To generate scrape audio, at least one ClatterObject must have a defined scrape_model value. This is the "scrape surface" object and it should usually be large, flat, and kinematic (e.g. a table).

scrape_model requires a ScrapeModel data object. Each ScrapeModel has one or more ScrapeSubObject data objects that are the scrape surfaces and a ScrapeMaterial. A ScrapeMaterial is distinct from an ImpactMaterial.

TDW includes a small number of predefined ScrapeModels, which can be found in tdw.physics_audio.scrape_model:

from tdw.physics_audio.scrape_model import DEFAULT_SCRAPE_MODELS

for model_name in DEFAULT_SCRAPE_MODELS:
    print(model_name)
    scrape_model = DEFAULT_SCRAPE_MODELS[model_name]

This is an example of how to assign a pre-defined a ScrapeModel to a ClatterObject:

from tdw.physics_audio.scrape_model import DEFAULT_SCRAPE_MODELS
from tdw.physics_audio.clatter_object import ClatterObject
from tdw.physics_audio.impact_material import ImpactMaterial

clatter_object = ClatterObject(impact_material=ImpactMaterial.ceramic,
                               amp=0.3,
                               resonance=0.1,
                               size=3,
                               scrape_model=DEFAULT_SCRAPE_MODELS["wood_board"])

This is an example of how to manually define a ScrapeModel and assign it to a ClatterObject:

from tdw.physics_audio.scrape_model import ScrapeModel
from tdw.physics_audio.scrape_sub_object import ScrapeSubObject
from tdw.physics_audio.scrape_material import ScrapeMaterial
from tdw.physics_audio.clatter_object import ClatterObject
from tdw.physics_audio.impact_material import ImpactMaterial

scrape_model = ScrapeModel(model_name="wood_board",
                           sub_objects=[ScrapeSubObject(name="wood_board",
                                                        material_index=0)],
                           scrape_material=ScrapeMaterial.plywood,
                           visual_material="wood_beech_honey")
clatter_object = ClatterObject(impact_material=ImpactMaterial.ceramic,
                               amp=0.3,
                               resonance=0.1,
                               size=3,
                               scrape_model=scrape_model)

Automatically setting scrape_model

If Clatter uses default object data (from DEFAULT_OBJECTS) to create a ClatterObject, it will automatically look for a corresponding ScrapeModel to assign to the object (based on the model name).

Clatter will never automatically assign a ScrapeModel to an object using derived data.

Clatter will only assign a ScrapeModel to a user-defined object if the user explicitly sets the scrape_model constructor parameter.

Next: Recording Clatter audio with the PhysicsAudioRecorder add-on

Return to the README

Example controllers:

Python API: