[QUESTION]Is V-ZB still not support overlap_grad_reduce and transformer_engine?

[LitLeo]

overlap_grad_reduce and transformer_engine can both bring significant performance benefits. Are they still not supported?

[QPHutu]

transformer_engine is not supported because it's not accessible to split backward pass into weight gradient and activation gradient.

For overlap_grad_reduce, I think it's already well supported. Please provide more information if you found any issue. Thanks.

[veritas9872]

Hello! @QPHutu,

May I ask which parts of Transformer Engine are causing issues?

This would be very helpful for applying Zero Bubble Pipeline Parallel to the current version of Megatron.

Thanks!

[QPHutu]

Hello! @QPHutu,

May I ask which parts of Transformer Engine are causing issues?

This would be very helpful for applying Zero Bubble Pipeline Parallel to the current version of Megatron.

Thanks!

To support Transformer Engine, you may need to change the code of TransformerEngine first, and then build from source code to support splitting backward pass in Megatron. It's doable if you want.

The reason why we don't support transformer engine is not about any technical issue, it's mainly because we don't want to make any code change in dependencies.