[TorchAcc][Experimental] Integrate more model in torchacc #683
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* [TorchAcc] Fix batch split when padding_to is not None. * fix lint
baoleai
reviewed
Apr 28, 2024
examples/pytorch/llm/scripts/torchacc/baichuan2_13b_chat/acc_lora_dp_sft.sh
Outdated
Show resolved
Hide resolved
examples/pytorch/llm/scripts/torchacc/baichuan2_13b_chat/acc_lora_dp_sft.sh
Outdated
Show resolved
Hide resolved
examples/pytorch/llm/scripts/torchacc/baichuan2_13b_chat/acc_lora_fsdp_sft.sh
Outdated
Show resolved
Hide resolved
swift/llm/utils/argument.py
Outdated
| @@ -429,6 +429,7 @@ class SftArguments(ArgumentsBase): | |||
| logging_steps: int = 5 | |||
| dataloader_num_workers: int = 1 | |||
| dataloader_pin_memory: bool = True | |||
| dataloader_drop_last: bool = True | |||
There was a problem hiding this comment.
问一下这里为什么不使用training_args的默认值False呢
swift/llm/sft.py
Outdated
| @@ -156,6 +159,8 @@ def llm_sft(args: SftArguments) -> Dict[str, Union[str, Any]]: | |||
| logger.info(f'val_dataset_sample: {val_dataset_sample}') | |||
| val_idxs = random_state.permutation(val_dataset_sample) | |||
| val_dataset = val_dataset.select(val_idxs) | |||
| training_args.train_dataset_sample = train_dataset.shape[ | |||
There was a problem hiding this comment.
train_dataset_sample会在下个版本去除了，使用'{dataset_name}#{train_sample}|{val_sample}'来控制单个数据集的数量
There was a problem hiding this comment.
train_dataset_sample会在下个版本去除了，使用'{dataset_name}#{train_sample}|{val_sample}'来控制单个数据集的数量
- 这边是需要获取一个总的train_dataset_sample用于后面warmup_step的计算。
- 而且这个pr的train_dataset_sample是基于Refactor dataset #802 得到的train_dataset的结果，给
SwiftArgumentMixin的对应参数，和前面数据集的处理是独立的吧，不知道是否理解有误
swift/llm/utils/argument.py
Outdated
| @@ -485,6 +486,9 @@ class SftArguments(ArgumentsBase): | |||
| neftune_alpha: Optional[float] = None | |||
| deepspeed_config_path: Optional[str] = None | |||
| model_cache_dir: Optional[str] = None | |||
| metric_warmup_step: Optional[float] = 0 | |||
There was a problem hiding this comment.
这三个超参数可以解释一下是什么意思么，是一定需要在torch_acc的情况下有效么
可以注释一下只在torchacc下有效嘛
There was a problem hiding this comment.
这三个超参数可以解释一下是什么意思么，是一定需要在torch_acc的情况下有效么
可以注释一下只在torchacc下有效嘛
前两个应该是通用的，fsdp_num只在torchacc有效，我注释一下
swift/torchacc_utils.py
Outdated
|
|
||
| import einops |
There was a problem hiding this comment.
ms-swift没有这个依赖, 验证一下是否会导致报错
| @@ -27,6 +27,8 @@ class SwiftArgumentsMixin: | |||
| acc_strategy: str = field( | |||
| default='token', metadata={'choices': ['token', 'sentence']}) | |||
| additional_saved_files: Optional[List[str]] = None | |||
| metric_warmup_step: Optional[float] = 0 | |||
| train_dataset_sample: Optional[int] = -1 | |||
There was a problem hiding this comment.
同, train_dataset_sample将会在下个版本移除，可能产生影响
swift/trainers/callback.py
Outdated
| @@ -45,6 +52,24 @@ def on_log(self, | |||
| logs=None, | |||
| **kwargs): | |||
| logs['global_step'] = state.global_step | |||
| if state.global_step >= self.metric_warmup_step and self.warmup_start_time == 0: | |||
swift/trainers/callback.py
Outdated
| @@ -19,6 +21,11 @@ def on_train_begin(self, args, state, control, **kwargs): | |||
| self.training_bar = tqdm( | |||
| desc='Train', total=state.max_steps, dynamic_ncols=True) | |||
| self.current_step = 0 | |||
| self.warmup_start_time = 0 | |||
There was a problem hiding this comment.
可以解释一下这里嘛
- 这里是因为torchacc在初始的阶段需要进行图编译，导致训练速度比后期稳定慢很多，因此加入了这个warmup_step，表示从训练的第warm_up step开始再计算一个训练的平均速度。
- 这里的计算逻辑是如果当前step到达了warm_up step，调用transformers的speed_metric函数计算一个指标并进行更新。
- args.metric_warmup_step可以是int或者float，表示具体的步数或者比例
swift/llm/sft.py
Outdated
| @@ -262,7 +267,17 @@ def llm_sft(args: SftArguments) -> Dict[str, Union[str, Any]]: | |||
| indent=2) | |||
| logging_path = os.path.join(args.output_dir, 'logging.jsonl') | |||
| logger.info(f'The logging file will be saved in: {logging_path}') | |||
| trainer.train(training_args.resume_from_checkpoint) | |||
|
|
|||
| if args.use_profiler: | |||
There was a problem hiding this comment.
可以介绍一下这里的逻辑嘛，或者使用环境变量隔离
There was a problem hiding this comment.
可以介绍一下这里的逻辑嘛，或者使用环境变量隔离
这里是加了对训练的profiler功能，和使用torchacc是独立的
Jintao-Huang
approved these changes
May 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR type
PR information
Previous PR: #647
Experiment results
Paste your experiment result here(if needed).
We have test some models for torchacc and swift