Allowing programmatic downloads in jupyterlab

[JoranAngevaare]

Software

straxen.print_versions('strax straxen admix utilix admix'.split())

Working on midway2-0416.rcc.local with the following versions and installation paths:
python	v3.8.8	(default, Apr 13 2021, 19:58:26) [GCC 7.3.0]
strax	v0.15.0	/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/strax
straxen	v0.18.0	/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/straxen
admix	v0.3.1	/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/admix
utilix	v0.5.3	/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/utilix
admix	v0.3.1	/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/admix

Description

It would be nice if we could admix download using jupyter bash magic such that one does not have to go do this manually on the login node. E.g. the command below. I am not sure if this is due to some setup in rucio/the container/.. or due to the fact that the batch nodes don't have internet access.

What I Did

!admix-download 012233 raw_records --threads 10 --dir /dali/lgrandi/angevaare/download_rr/.

Then, it simply hangs on this, and nothing really happens for hours. Running the exact command on the login node solves the issue but is not suitable since we don't want everyone to start doing jobs on the login node.

Downloading xnt_012233:raw_records-rfzvpzj4mf from CCIN2P3_USERDISK

[ershockley]

Thanks for reporting @jorana. I can reproduce the error for that run, but I'm not sure what's causing it. Seems to be something with CCIN2P3, as the same command for a run on UC_OSG_USERDISK works fine:

! admix-download 8000 raw_records --chunks 0 1 2 --rse UC_OSG_USERDISK

Downloading xnt_008000:raw_records-rfzvpzj4mf from UC_OSG_USERDISK
Download successful to ./008000-raw_records-rfzvpzj4mf

[JoranAngevaare]

Thanks Evan for checking it out. I will try again for more runs next time, but I thought I was seeing it for a whole set of runs. It then seemed to work fine on the login node.

[lucascottolavina]

Hi, I have also reproduced this issue (with run 8000 since this run has both copies). With this command:

!admix-download 8000 raw_records --chunks 0 --dir /home/lucascottolavina --rse UC_OSG_USERDISK

the download is immediate, while with this command:

!admix-download 8000 raw_records --chunks 0 --dir /home/lucascottolavina --rse CCIN2P3_USERDISK

it took several hours (about 4), but then it does it. The output is the following:

mix-download 8000 raw_records --chunks 0 --dir /home/lucascottolavina --rse CCIN2P3_USERDISK
xnt_008000:raw_records-rfzvpzj4mf
Downloading xnt_008000:raw_records-rfzvpzj4mf from CCIN2P3_USERDISK
Protocol implementation not found
Download try #1 failed.
Download try #2 failed.
Download successful to /home/lucascottolavina/008000-raw_records-rfzvpzj4mf

My (temporary) conclusion is that all these tests have been done during a massive data transfer from CCIN2P3 to UC (started on April 21 morning) and this transfer had some impact on the speed performances. I will redo this test as soon as this massive data transfer is completed, to check that in normal condition it works.