Currently, training loops exist both in instructlab/instructlab and in this repo. Bringing all training-related paths under one repo would be helpful, especially regarding testing.
This would include:
- CPU-only
- MPS
-
=1 GPU (from the minimum-viable card up to 8xMI300x/H100)
The implication of this change might be that this library becomes a "functional-programming" alternative to the current typical pattern of a Trainer class, opinionated by our optimizations.