Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update wmcore_transferor global and local quotas dynamically #914

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

haozturk
Copy link
Contributor

Currently, wmcore_transferor quotas are statically set and it's as follows [1]. Its usage reached to record high of 44.4 PB [2] and made many sites run out of space. In this issue [3], we discussed a strategy to introduce global and local limits for this account to keep its usage under control. This PR implements what's been discussed in that issue as well as my additions. To explain the changes:

  1. Implements a global limit for this account over the T1 and T2 RSEs. The global limit is set to X % of total disk capacity of T1 and T2 sites used in production. The default is 15%, which correspond to ~46 PB. Beyond this usage, rule creations will fail. This is to keep its global usage under control.
  2. Implements a local limit at every T1 and T2. This is to avoid the over-usage at individual sites. Its local quota is set to Y % of an RSE's (free + expired) space. The default is 70%.
  3. Creates a new helper file to store commonly used functions for their re-use.

The output of a dry-run can be found at [4]

Before Eric reviews it, I'd like to get Ops' and WM's feedback. That's why I'll mark the PR as draft for now. Once we agree that the PR reflects what we need, I'll open the PR for Eric's review. Let me tag @Panos512 @hassan11196 @drkovalskyi @amaltaro @eachristgr and @juanpablosalas

[1]

$ rucio account limit list -a wmcore_transferor
+---------------------+-------------+------------+--------------+
| RSE                 | USAGE       | LIMIT      | QUOTA LEFT   |
|---------------------+-------------+------------+--------------|
| T1_DE_KIT_Disk      | 3.904 PB    | 2.964 PB   | 0.000 B      |
| T1_ES_PIC_Disk      | 1.117 PB    | 1.001 PB   | 0.000 B      |
| T1_FR_CCIN2P3_Disk  | 2.552 PB    | 1.174 PB   | 0.000 B      |
| T1_IT_CNAF_Disk     | 4.798 PB    | 3.766 PB   | 0.000 B      |
| T1_RU_JINR_Disk     | 975.954 TB  | 1.650 PB   | 674.046 TB   |
| T1_UK_RAL_Disk      | 2.506 PB    | 2.000 PB   | 0.000 B      |
| T1_US_FNAL_Disk     | 10.305 PB   | 9.912 PB   | 0.000 B      |
| T2_BE_IIHE          | 125.286 TB  | 1.100 PB   | 974.714 TB   |
| T2_BE_UCL           | 101.501 TB  | 550.000 TB | 448.499 TB   |
| T2_BR_SPRACE        | 142.360 TB  | 126.708 TB | 0.000 B      |
| T2_BR_UERJ          | 621.667 GB  | 15.076 TB  | 14.454 TB    |
| T2_CH_CERN          | 10.055 PB   | 8.418 PB   | 0.000 B      |
| T2_CH_CSCS          | 381.196 TB  | 500.000 TB | 118.804 TB   |
| T2_CN_Beijing       | 0.000 B     | 87.500 TB  | 87.500 TB    |
| T2_DE_DESY          | 1.031 PB    | 611.696 TB | 0.000 B      |
| T2_DE_RWTH          | 545.601 TB  | 357.000 TB | 0.000 B      |
| T2_EE_Estonia       | 31.257 TB   | 384.023 TB | 352.766 TB   |
| T2_ES_CIEMAT        | 763.129 TB  | 672.000 TB | 0.000 B      |
| T2_ES_IFCA          | 40.097 TB   | 200.000 TB | 159.903 TB   |
| T2_FI_HIP           | 49.395 TB   | 71.100 TB  | 21.705 TB    |
| T2_FR_CCIN2P3       | 0.000 B     | 58.000 TB  | 58.000 TB    |
| T2_FR_GRIF          | 17.142 TB   | 0.000 B    | 0.000 B      |
| T2_FR_GRIF_IRFU     | 0.000 B     | 175.374 TB | 175.374 TB   |
| T2_FR_GRIF_LLR      | 0.000 B     | 158.574 TB | 158.574 TB   |
| T2_FR_IPHC          | -301.272 MB | 310.000 TB | 310.000 TB   |
| T2_GR_Ioannina      | 0.000 B     | 25.000 TB  | 25.000 TB    |
| T2_HU_Budapest      | 27.255 TB   | 142.583 TB | 115.328 TB   |
| T2_IN_TIFR          | 15.271 GB   | 759.000 TB | 758.985 TB   |
| T2_IT_Bari          | 906.862 TB  | 611.755 TB | 0.000 B      |
| T2_IT_Legnaro       | 896.027 TB  | 815.554 TB | 0.000 B      |
| T2_IT_Pisa          | 613.364 TB  | 568.503 TB | 0.000 B      |
| T2_IT_Rome          | 185.414 TB  | 185.900 TB | 485.903 GB   |
| T2_KR_KISTI         | 178.872 TB  | 183.000 TB | 4.128 TB     |
| T2_PK_NCP           | 0.000 B     | 38.000 TB  | 38.000 TB    |
| T2_PL_Swierk        | 92.325 TB   | 197.000 TB | 104.675 TB   |
| T2_PL_Warsaw        | 0.000 B     | 10.000 TB  | 10.000 TB    |
| T2_PT_NCG_Lisbon    | 74.361 TB   | 74.000 TB  | 0.000 B      |
| T2_RU_IHEP          | 5.467 TB    | 55.000 TB  | 49.533 TB    |
| T2_RU_INR           | 5.102 GB    | 24.000 TB  | 23.995 TB    |
| T2_RU_ITEP          | 208.556 GB  | 26.000 TB  | 25.791 TB    |
| T2_RU_JINR          | 80.901 TB   | 140.697 TB | 59.796 TB    |
| T2_TR_METU          | 1.976 TB    | 40.000 TB  | 38.024 TB    |
| T2_TW_NCHC          | 139.172 TB  | 73.000 TB  | 0.000 B      |
| T2_UA_KIPT          | 83.815 TB   | 94.500 TB  | 10.685 TB    |
| T2_UK_London_Brunel | 3.570 TB    | 256.058 TB | 252.489 TB   |
| T2_UK_London_IC     | 593.692 TB  | 931.574 TB | 337.881 TB   |
| T2_UK_SGrid_Bristol | 314.984 GB  | 55.000 TB  | 54.685 TB    |
| T2_UK_SGrid_RALPP   | 124.178 TB  | 385.000 TB | 260.822 TB   |
| T2_US_Caltech       | 563.274 TB  | 516.399 TB | 0.000 B      |
| T2_US_Florida       | 491.851 TB  | 417.458 TB | 0.000 B      |
| T2_US_MIT           | 1.696 PB    | 1.555 PB   | 0.000 B      |
| T2_US_Nebraska      | 989.637 TB  | 1.871 PB   | 881.363 TB   |
| T2_US_Purdue        | 681.150 TB  | 800.000 TB | 118.850 TB   |
| T2_US_UCSD          | 641.655 TB  | 451.116 TB | 0.000 B      |
| T2_US_Vanderbilt    | 1.182 PB    | 950.000 TB | 0.000 B      |
| T2_US_Wisconsin     | 560.205 TB  | 510.000 TB | 0.000 B      |
+---------------------+-------------+------------+--------------+
+------------------+---------+---------+--------------+
| RSE EXPRESSION   | USAGE   | LIMIT   | QUOTA LEFT   |
|------------------+---------+---------+--------------|
+------------------+---------+---------+--------------+

[2] https://monit-grafana.cern.ch/d/viS0q0ZSz/rucio-account-monitoring?orgId=11&from=now-90d&to=now&var-RSE=All&var-RSEType=DISK&var-AccountName=wmcore_transferor&var-AccountType=All
[3] https://its.cern.ch/jira/browse/CMSTRANSF-479
[4]

DRY-RUN: Setting global quota of wmcore_transferor to 46.04775 PB
DRY-RUN: Setting local quota of wmcore_transferor at T2_FR_GRIF_IRFU to 908.3801494266443 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_US_Caltech to 10.3643883172711 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_US_Nebraska to 0.0 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_US_Florida to 111.961294357423 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_TR_METU to 402.1906372776005 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_IT_Bari to 1527.9415495045575 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_CN_Beijing to 72.64393671994691 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_UK_SGrid_RALPP to 154.86734044376 TB
DRY-RUN: Setting local quota of wmcore_transferor at T1_IT_CNAF_Disk to 5536.303062609513 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_HU_Budapest to 372.3804527664557 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_UK_London_Brunel to 84.05657648040871 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_US_UCSD to 0.0 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_ES_IFCA to 159.149843679561 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_PK_NCP to 280.689038 TB
DRY-RUN: Setting local quota of wmcore_transferor at T1_DE_KIT_Disk to 2905.7480326598975 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_PL_Cyfronet to 68.7655335679704 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_BE_UCL to 537.6259105940695 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_IT_Pisa to 1394.2740323818446 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_ES_CIEMAT to 1285.9045730912414 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_CH_CSCS to 440.4651689627777 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_EE_Estonia to 327.9392017881406 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_LV_HPCNET to 48.0188582596149 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_FR_GRIF to 1163.764476837517 TB
DRY-RUN: Setting local quota of wmcore_transferor at T1_UK_RAL_Disk to 1245.5988708602194 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_CH_CERN to 6238.485095252353 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_BR_UERJ to 222.1043767175016 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_AT_Vienna to 73.1120920553884 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_US_Purdue to 0.0 TB
DRY-RUN: Setting local quota of wmcore_transferor at T1_US_FNAL_Disk to 739.7531760491067 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_RU_JINR to 663.9109015806939 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_KR_KISTI to 274.8220419770861 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_IT_Rome to 1263.7288931542003 TB
DRY-RUN: Setting local quota of wmcore_transferor at T1_FR_CCIN2P3_Disk to 3250.964193166811 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_RU_IHEP to 158.6565304070341 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_LB_HPC4L to 114.7963376794563 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_US_Wisconsin to 302.56426338015365 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_FR_GRIF_LLR to 1070.6349472882678 TB
DRY-RUN: Setting local quota of wmcore_transferor at T1_RU_JINR_Disk to 3303.1037834165827 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_UA_KIPT to 167.16127417200119 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_PL_Swierk to 284.5630066555451 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_UK_SGrid_Bristol to 230.88932624080542 TB
DRY-RUN: Setting local quota of wmcore_transferor at T1_ES_PIC_Disk to 778.343743660567 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_IT_Legnaro to 528.6263176154828 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_US_Vanderbilt to 1563.0938106579558 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_RU_INR to 0.0 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_UK_London_IC to 981.2110964284251 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_TW_NCHC to 242.6584056083299 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_IN_TIFR to 3778.039272398437 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_DE_RWTH to 64.7424875865341 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_US_MIT to 97.8423149637192 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_BR_SPRACE to 563.8278387305186 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_DE_DESY to 1213.128036184836 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_RU_ITEP to 87.0689732523758 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_PT_NCG_Lisbon to 73.8099469336677 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_FR_IPHC to 49.448122436852294 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_BE_IIHE to 807.5445744208745 TB
DRY-RUN: Setting local quota of wmcore_transferor at T2_FI_HIP to 124.2242144014438 TB

@haozturk haozturk marked this pull request as draft April 11, 2025 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant