Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: proxy crash due to data race #32970

Closed
1 task done
weiliu1031 opened this issue May 11, 2024 · 1 comment
Closed
1 task done

[Bug]: proxy crash due to data race #32970

weiliu1031 opened this issue May 11, 2024 · 1 comment
Assignees
@weiliu1031
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@weiliu1031
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

proxy crash due to data race

img_v3_02ap_e6ad6ea3-f9a2-40a1-af7e-be214945c87g

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@weiliu1031 weiliu1031 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 11, 2024
@weiliu1031
Copy link
Contributor Author

/assign

weiliu1031 added a commit to weiliu1031/milvus that referenced this issue May 11, 2024
@weiliu1031
fix: Proxy crash due to shard leader cache data race
2e3b6af 
issue: milvus-io#32970
cause InvalidateShardLeaderCache use wrong lock, which may cause data
race in meta cache, then proxy may crash

This PR fixed that use leaderMut when try to access shard leader cache.

Signed-off-by: Wei Liu <[email protected]>
@weiliu1031 weiliu1031 mentioned this issue May 11, 2024
Merged
weiliu1031 added a commit to weiliu1031/milvus that referenced this issue May 11, 2024
@weiliu1031
fix: Proxy crash due to shard leader cache data race
4606431 
issue: milvus-io#32970
cause InvalidateShardLeaderCache use wrong lock, which may cause data
race in meta cache, then proxy may crash

This PR fixed that use leaderMut when try to access shard leader cache.

Signed-off-by: Wei Liu <[email protected]>
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 11, 2024
@yanliang567 yanliang567 removed their assignment May 11, 2024
czs007 pushed a commit that referenced this issue May 11, 2024
@weiliu1031
fix: Proxy crash due to shard leader cache data race (#32971)
8a9a421 
issue: #32970
cause InvalidateShardLeaderCache use wrong lock, which may cause data
race in meta cache, then proxy may crash

This PR fixed that use leaderMut when try to access shard leader cache.

Signed-off-by: Wei Liu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@weiliu1031 weiliu1031

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yanliang567 @weiliu1031