Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

false_positive_check=True need to add in ThWiki tutorial #481

Open
yangjingyi opened this issue Jan 15, 2025 · 1 comment
Open

false_positive_check=True need to add in ThWiki tutorial #481

yangjingyi opened this issue Jan 15, 2025 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@yangjingyi
Copy link

I find there is a bug in LSH part of tutorial https://github.com/NVIDIA/NeMo-Curator/blob/main/tutorials/single_node_tutorial/single_gpu_tutorial.ipynb
According to #430, false_positive_check=True should be added in the fuzzy dedup part.
#Run LSH()
lsh = LSH(
cache_dir=lsh_output_dir,
num_hashes=minhash_length,
num_buckets=num_bands,
buckets_per_shuffle=buckets_per_shuffle,
false_positive_check=True,
id_fields=["dataset_id", "doc_id"],
minhash_field=minhash_field,
logger=lsh_log_dir,
)

@yangjingyi yangjingyi added the bug Something isn't working label Jan 15, 2025
@ayushdg
Copy link
Collaborator

ayushdg commented Jan 15, 2025

Thanks for opening @yangjingyi . Is this something you'd be interested in putting in a fix for? (No worries if not).
You're right, after #326 merged in, the arg needs to be added here to ensure that the results can be consumed downstream.
I do plan on refactoring the tutorial a bit more after #386 merges in but that's still ongoing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants