Potential performance issue: Unreliable performance of .loc in pandas 2.0.3

[TendouArisu]

Issue Description:

Hello.
I have discovered a performance degradation in the .loc function of pandas version 2.0.3 when .loc handling big DataFrame with non-unique indexes. When using pandas more than 4 indexes, .loc drastically increases to X1000 times. And I notice in python/requirements-fate.txt, shows that it depends on pandas version 2.0.3. I am not sure whether this performance problem in pandas will affect this repository. I found some discussions on GitHub related to this issue, including #54550 and #54746.
I also found that python/fate/ml/feature_selection/hetero_feature_selection.py and python/fate/ml/statistics/statistics.py both used the influenced api. There may be more files used the influenced api.

Suggestion

I would recommend considering an upgrade to a different version of pandas >= 2.1 or exploring other solutions to optimize the performance of .loc .
Any other workarounds or solutions would be greatly appreciated.
Thank you!

[mgqa34]

Very good suggestion and finding! We will upgrade pandas to newer version in later version of fate, thanks

[github-actions]

This issue has been marked as stale because it has been open for 365 days with no activity. If this issue is still relevant or if there is new information, please feel free to update or reopen it.

[github-actions]

This issue was closed because it has been inactive for 1 days since being marked as stale. If this issue is still relevant or if there is new information, please feel free to update or reopen it.