[Feature Request] Intra-node UVA copy

[chengyupku]

Create the tensor and its corresponding IPC handle, then perform an all-gather operation to enable peer access. This should be done on the host side.

Proposal: Introduce a create_distributed_tensor feature that enables initialization of distributed tensors, including the creation and transmission of IPC handles.

Reference: DeepEP

[chengyupku]

See #15 .

[Rachmanino]

TODO

• Warp-level unrolled copy ([Feat] Refactor UVA mechanism and introduce metadata table to support remote device-to-device copy #15)
• Block-level unrolled copy ([Feat] Add put_block/get_block implementation & examples and rename primitives #23)
• Utilize TMA for bulk copy across devices
• Support non-contiguous tensor copy by layout inference
• Tune the performance to match with baselines