docs: add operator documentation

[cmgzn]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the operator documentation by adding new examples, clarifying existing ones, and introducing documentation for newly added video processing operators. The changes aim to provide clearer guidance and better understanding for users of the system's various data processing capabilities.

Highlights

• New Operator Documentation: Added comprehensive documentation for two new video-related mappers: video_camera_pose_mapper and video_hand_reconstruction_hawor_mapper.
• Enhanced Effect Demonstrations: Introduced new and updated effect demonstration examples for several existing operators, including MetaTagsAggregator, CharacterRepetitionFilter, TokenNumFilter, VideoDurationFilter, NaiveReverseGrouper, AudioFFmpegWrappedMapper, DialogIntentDetectionMapper, DialogSentimentDetectionMapper, DialogTopicDetectionMapper, DownloadFileMapper, ImageDetectionYoloMapper, ImageFaceBlurMapper, ImageRemoveBackgroundMapper, ImageSegmentMapper, ImageTaggingVLMMapper, OptimizePromptMapper, OptimizeQAMapper, QueryIntentDetectionMapper, QuerySentimentDetectionMapper, QueryTopicDetectionMapper, RelationIdentityMapper, and VggtMapper.
• Documentation Refinements: Cleaned up formatting by removing extra blank lines in several formatter documentation files and corrected a Chinese translation in image_sam_3d_body_mapper.md.
• Parameter Configuration Updates: Updated parameter configurations and default values for AudioFFmpegWrappedMapper, VggtMapper, and TextTaggingByPromptMapper to reflect current usage and improve clarity.
• Unit Test Link Corrections: Corrected and updated unit test links for RayBtsMinhashCppDeduplicator, S3DownloadFileMapper, and S3UploadFileMapper.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• docs/operators/aggregator/meta_tags_aggregator.md
  • Added effect demonstration for MetaTagsAggregator.
• docs/operators/deduplicator/ray_bts_minhash_cpp_deduplicator.md
  • Updated unit test link.
• docs/operators/filter/character_repetition_filter.md
  • Added effect demonstration for CharacterRepetitionFilter.
• docs/operators/filter/image_pair_similarity_filter.md
  • Added __dj__stats__ meta-data to input data for ImagePairSimilarityFilter.
• docs/operators/filter/token_num_filter.md
  • Added new test cases (test_token_num_filter) and updated output data format.
• docs/operators/filter/video_duration_filter.md
  • Added new test cases (test_any, test_all, test_default_filter, test_filter_long_videos).
  • Updated input/output data to include __dj__stats__ and {PROJECT_ROOT}.
  • Renamed test_filter_videos_within_range to test_any.
• docs/operators/formatter/csv_formatter.md
  • Removed extra blank lines.
• docs/operators/formatter/empty_formatter.md
  • Removed extra blank lines.
• docs/operators/formatter/json_formatter.md
  • Removed extra blank lines.
• docs/operators/formatter/parquet_formatter.md
  • Removed extra blank lines.
• docs/operators/formatter/ray_empty_formatter.md
  • Removed extra blank lines.
• docs/operators/formatter/text_formatter.md
  • Removed extra blank lines.
• docs/operators/formatter/tsv_formatter.md
  • Removed extra blank lines.
• docs/operators/grouper/key_value_grouper.md
  • Modified KeyValueGrouper example to use default constructor.
• docs/operators/grouper/naive_reverse_grouper.md
  • Added new test cases (test_two_batch_sample, test_rm_unbatched_keys1, test_rm_unbatched_keys2) for NaiveReverseGrouper.
• docs/operators/mapper/audio_ffmpeg_wrapped_mapper.md
  • Removed atrim argument from AudioFFmpegWrappedMapper example.
  • Updated input/output paths with {PROJECT_ROOT}.
• docs/operators/mapper/dialog_intent_detection_mapper.md
  • Added new test cases (test_max_round, test_max_round_zero, test_query, test_intent_candidates, test_rename_keys) for DialogIntentDetectionMapper.
  • Updated input data format.
• docs/operators/mapper/dialog_sentiment_detection_mapper.md
  • Added new test cases (test_max_round, test_max_round_zero, test_query, test_sentiment_candidates, test_rename_keys) for DialogSentimentDetectionMapper.
  • Updated input data format.
• docs/operators/mapper/dialog_topic_detection_mapper.md
  • Added new test cases (test_max_round, test_max_round_zero, test_query, test_topic_candidates, test_rename_keys) for DialogTopicDetectionMapper.
  • Updated input data format.
• docs/operators/mapper/download_file_mapper.md
  • Added new test cases (test_image_with_savefield_and_resume, test_image_with_savefield_and_resume_and_savedir) for DownloadFileMapper.
  • Updated input/output paths with {PROJECT_ROOT}.
• docs/operators/mapper/image_detection_yolo_mapper.md
  • Added new test cases (test_cpu, test_multi_process) for ImageDetectionYoloMapper.
• docs/operators/mapper/image_face_blur_mapper.md
  • Added new test cases (test_box, test_box_radius) for ImageFaceBlurMapper.
• docs/operators/mapper/image_remove_background_mapper.md
  • Added new test cases (test_multiple_images, test_single_image) for ImageRemoveBackgroundMapper.
• docs/operators/mapper/image_sam_3d_body_mapper.md
  • Corrected Chinese translation for SAM 3D Body description.
• docs/operators/mapper/image_segment_mapper.md
  • Added new test cases (test_cpu, test_multi_process) for ImageSegmentMapper.
• docs/operators/mapper/image_tagging_vlm_mapper.md
  • Added new test case (test_api_model) for ImageTaggingVLMMapper.
• docs/operators/mapper/optimize_prompt_mapper.md
  • Added new test case (test_api_model) for OptimizePromptMapper.
• docs/operators/mapper/optimize_qa_mapper.md
  • Added new test case (test_api) for OptimizeQAMapper.
• docs/operators/mapper/query_intent_detection_mapper.md
  • Added new test cases (test_default, test_no_zh_to_en) for QueryIntentDetectionMapper.
• docs/operators/mapper/query_sentiment_detection_mapper.md
  • Added new test cases (test_default, test_no_zh_to_en) for QuerySentimentDetectionMapper.
• docs/operators/mapper/query_topic_detection_mapper.md
  • Added new test cases (test_default, test_no_zh_to_en) for QueryTopicDetectionMapper.
• docs/operators/mapper/relation_identity_mapper.md
  • Added new test cases (test_default, test_rename_key) for RelationIdentityMapper.
• docs/operators/mapper/s3_download_file_mapper.md
  • Updated unit test link.
• docs/operators/mapper/s3_upload_file_mapper.md
  • Updated unit test link.
• docs/operators/mapper/text_tagging_by_prompt_mapper.md
  • Added model_params to parameter configuration.
• docs/operators/mapper/vggt_mapper.md
  • Updated frame_dir default value and input paths with {PROJECT_ROOT}.
• docs/operators/mapper/video_camera_pose_mapper.md
  • Added new documentation for video_camera_pose_mapper.
• docs/operators/mapper/video_ffmpeg_wrapped_mapper.md
  • Removed scale argument from VideoFFmpegWrappedMapper example.
  • Updated input/output paths with {PROJECT_ROOT}.
• docs/operators/mapper/video_hand_reconstruction_hawor_mapper.md
  • Added new documentation for video_hand_reconstruction_hawor_mapper.
• docs/operators/mapper/video_object_segmenting_mapper.md
  • Updated input paths with {PROJECT_ROOT}.

Activity

• The pull request was created by cmgzn.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds extensive documentation and examples for numerous operators across different modules. The changes include adding new "Effect demonstration" sections, updating links, and minor formatting fixes. While the additions are valuable, there are several areas where the documentation could be clearer and more consistent. Specifically, some examples have misleading outputs (e.g., showing [] for an empty dataset, or "empty" for a sample with rich metadata), some have code snippets that don't match the described behavior, and others have unreadable data representations or are missing output descriptions entirely. Addressing these issues will significantly improve the quality and usability of the documentation.

_{Note: Security Review has been skipped due to the limited scope of the PR.}

[gemini-code-assist]

The output for the test_existing_stats example is a bit confusing. It shows one output sample with content []. Since both input samples have a char_rep_ratio of 0.5, which is greater than the max_ratio of 0.4, they should both be filtered out, resulting in an empty dataset. To make this clearer, I suggest removing this output sample card to represent an empty output.

[gemini-code-assist]

The output for the test_all example is presented in a confusing way. It shows "Sample 1: text" followed by []. Since all input samples are filtered out in this case, the output dataset is empty. To avoid confusion, it would be clearer to have an empty "output data" section instead of showing a sample with empty content.

[gemini-code-assist]

The example code KeyValueGrouper() is inconsistent with the provided input and output. Without arguments, KeyValueGrouper defaults to grouping by the text field (text_key). Since all texts are unique, this would result in three separate groups. The output shows grouping by language ('en' and 'zh'), which corresponds to KeyValueGrouper(group_by_keys=['meta.language']). Please update the code snippet to match the example's behavior.

Suggested change

	KeyValueGrouper()
	KeyValueGrouper(group_by_keys=['meta.language'])

[gemini-code-assist]

The example code is missing the filter_name parameter, but the explanation states that the 'atrim' filter is applied. To make the example clear and consistent, please add filter_name='atrim' to the AudioFFmpegWrappedMapper call.

Suggested change

	AudioFFmpegWrappedMapper(filter_kwargs={'end': 6}, capture_stderr=False)
	AudioFFmpegWrappedMapper('atrim', filter_kwargs={'end': 6}, capture_stderr=False)

[gemini-code-assist]

The input data in this example is presented as a string of Python code, which is difficult to read and understand. For better clarity, please format the input data to show the actual sample structure, similar to other examples in the documentation. The same issue applies to the output data and the other example in this file.

[gemini-code-assist]

The "output data" section for this example is empty. To make the documentation more helpful, it should either show the output images with blurred faces or provide a textual description of the expected output. An empty section can be confusing for users trying to understand what the operator does. This applies to the test_box_radius example as well.

[gemini-code-assist]

The "output data" section for this example is empty. To improve clarity, please include either the output images with the background removed or a description of what the output should look like. An empty output section can be misleading. This also applies to the test_single_image example.

[gemini-code-assist]

The output sample is labeled as "empty", which is misleading because it contains a large table of metadata. To improve clarity, please consider changing the label to something more descriptive, for example, as metadata or 3D information. This applies to all output samples in this file.

[gemini-code-assist]

The example code is missing the filter_name parameter, but the explanation states that the 'scale' filter is applied to resize videos. To ensure the example is correct and clear, please add filter_name='scale' to the VideoFFmpegWrappedMapper call.

Suggested change

	VideoFFmpegWrappedMapper(filter_kwargs={'width': 400, 'height': 480}, capture_stderr=False)
	VideoFFmpegWrappedMapper('scale', filter_kwargs={'width': 400, 'height': 480}, capture_stderr=False)