You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
✨ First of all, the Llama project is a tremendous asset to the entire GenAI field. I deeply appreciate the contributions of both the Llama team and the open-source community.
Description
I've recently been delving into the source code of the Llama project and I happen to have a question. Specifically, why seed must be the same in all processes.
The current version: updated to the seedable version in this PR add seed #779.
However, I have a question: why must the seed be equal? What would happen if they are not equal? I tried searching for "why must random seeds be equal in model parallelism?", but I couldn't find any relevant information.
From an engineering perspective, I think it might be related to reproducibility? But I want to know if having different random seeds for each processor would result in a catastrophic outcome? (Because the current comment makes it seem like a very serious issue)
The current docstring lacks an explanation for seed and fails in the ruff linter. (I believe the lack of comments is also one of the reasons contributing to the confusion.)
I really hope to receive help from Meta or anyone else. I would also be happy to provide the correct docstring for seed after resolving this issue to eliminate any potential confusion.
The text was updated successfully, but these errors were encountered:
✨ First of all, the Llama project is a tremendous asset to the entire GenAI field. I deeply appreciate the contributions of both the Llama team and the open-source community.
Description
I've recently been delving into the source code of the Llama project and I happen to have a question. Specifically, why
seed must be the same in all processes
.This line of code is located in the
Llama.build
function at: https://github.com/meta-llama/llama/blob/main/llama/generation.py#L94-L95.Regarding this code, there are only two versions of changes:
llama/llama/generation.py
Lines 71 to 72 in 6d4c0c2
However, I have a question: why must the
seed
be equal? What would happen if they are not equal? I tried searching for "why must random seeds be equal in model parallelism?", but I couldn't find any relevant information.From an engineering perspective, I think it might be related to reproducibility? But I want to know if having different random seeds for each processor would result in a catastrophic outcome? (Because the current comment makes it seem like a very serious issue)
The current docstring lacks an explanation for
seed
and fails in theruff
linter. (I believe the lack of comments is also one of the reasons contributing to the confusion.)I really hope to receive help from Meta or anyone else. I would also be happy to provide the correct docstring for
seed
after resolving this issue to eliminate any potential confusion.The text was updated successfully, but these errors were encountered: