Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

log summarization #534

Merged
merged 6 commits into from
Jan 14, 2025
Merged

log summarization #534

merged 6 commits into from
Jan 14, 2025

Conversation

HYLcool
Copy link
Collaborator

@HYLcool HYLcool commented Jan 9, 2025

  • Unify the error log format for fault tolerance. OPs will raise errors/exceptions instead of writing logs for them.

An error occurred in {op/method_name} when processing sample(s) "{sample}" -- {error_type}: {error_message}

  • Extract debug/warning/error log items to separate log files with '_{level}' suffix. e.g.
outputs/demo-process/log
├── export_demo-processed.jsonl_time_20250109111158.txt
├── export_demo-processed.jsonl_time_20250109111158_DEBUG.txt
├── export_demo-processed.jsonl_time_20250109111158_ERROR.txt
└── export_demo-processed.jsonl_time_20250109111158_WARNING.txt
  • After OPs are done, error/warning log summary will be made and output for users, including the number of each type of logs. e.g.

image

@HYLcool HYLcool added the enhancement New feature or request label Jan 9, 2025
@HYLcool HYLcool requested review from BeachWang and yxdyc January 9, 2025 06:19
@HYLcool HYLcool self-assigned this Jan 9, 2025
Copy link
Collaborator

@BeachWang BeachWang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@yxdyc yxdyc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made some modifications about the error info, plz double check them @HYLcool before merging this PR.

@HYLcool
Copy link
Collaborator Author

HYLcool commented Jan 14, 2025

I made some modifications about the error info, plz double check them @HYLcool before merging this PR.

I resolve the unpacking error when unpacking the regex results due to the extra op_name group being added. Besides, I reformat the log summary of errors to a table for a better look.

@yxdyc
Copy link
Collaborator

yxdyc commented Jan 14, 2025

LGTM

@HYLcool HYLcool merged commit 0624d44 into main Jan 14, 2025
3 checks passed
@HYLcool HYLcool deleted the opt/log_summarization branch January 14, 2025 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants