Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

idr0051-fulton-tailbudlightsheet S-BIAD815 #653

Open
dominikl opened this issue Feb 27, 2023 · 14 comments
Open

idr0051-fulton-tailbudlightsheet S-BIAD815 #653

dominikl opened this issue Feb 27, 2023 · 14 comments

Comments

@dominikl
Copy link
Member

No description provided.

@dominikl dominikl moved this to test convert in NGFF conversion Feb 27, 2023
@dominikl dominikl moved this from test convert to re-import test image in NGFF conversion Feb 27, 2023
@dominikl dominikl moved this from re-import test image to convert all data to NGFF in NGFF conversion Feb 28, 2023
@dominikl
Copy link
Member Author

dominikl commented Mar 1, 2023 •
edited
Loading

Export time: 17 min / plate
Import time: 4 hours.

@dominikl
Copy link
Member Author

Currently converting on pilot-zarr1-dev , will be available in /data/idr0051.

@dominikl dominikl self-assigned this Jun 12, 2023
@will-moore
Copy link
Member

Looks done:

(base) [wmoore@pilot-zarr1-dev ~]$ ls -l /data/idr0051
total 0
drwxrwxr-x. 4 dlindner dlindner 76 Jun 12 12:18 180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr
drwxrwxr-x. 4 dlindner dlindner 76 Jun 12 12:53 180712_H2B_22ss_Courtney_p00_c00_reg_preview.klb.ome.zarr
drwxrwxr-x. 4 dlindner dlindner 76 Jun 12 14:36 2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c00_reg_preview.ome.zarr
drwxrwxr-x. 4 dlindner dlindner 76 Jun 12 14:46 embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.klb.ome.zarr
drwxrwxr-x. 4 dlindner dlindner 76 Jun 12 15:40 embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.ome.zarr

@will-moore will-moore moved this from convert all data to NGFF to upload data to s3 in NGFF conversion Jun 15, 2023
@dominikl
Copy link
Member Author

👍 Yes, finised.

@will-moore will-moore assigned will-moore and unassigned dominikl Jun 15, 2023
@will-moore
Copy link
Member

To zip each dir... (remove / etc)

for i in */; do zip -r "${i%/}.zip" "$i"; done

@will-moore
Copy link
Member 

$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 mb s3://idr0051
make_bucket: idr0051
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-policy --bucket idr0051 --policy file://policy.json
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-cors --bucket idr0051 --cors-configuration file://cors.json

Should have run in a Screen as this took too long and lost connection...

/home/wmoore/mc cp -r idr0051/ uk1s3/idr0051/zarr
...g_preview.klb.ome.zarr/0/1/45/0/338/0/0: 14.32 GiB / 14.43 GiB ━━━━━━━

@will-moore
Copy link
Member

will-moore commented Jun 26, 2023 •
edited
Loading

Started running again, in a screen this time...

/home/wmoore/mc cp -r idr0051/ uk1s3/idr0051/zarr

...
$ /home/wmoore/mc cp -r idr0051/ uk1s3/idr0051/zarr
...e.zarr/OME/METADATA.ome.xml: 18.03 GiB / 18.03 GiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.92 MiB/s 
$ /home/wmoore/mc ls uk1s3/idr0051/zarr
[2023-06-26 13:10:54 UTC]     0B 180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/
[2023-06-26 13:10:54 UTC]     0B 180712_H2B_22ss_Courtney_p00_c00_reg_preview.klb.ome.zarr/
[2023-06-26 13:10:54 UTC]     0B 2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c00_reg_preview.ome.zarr/
[2023-06-26 13:10:54 UTC]     0B embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.klb.ome.zarr/
[2023-06-26 13:10:54 UTC]     0B embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.ome.zarr/

@will-moore
Copy link
Member

Initial rendering in vizarr is poor because single channel is grey:

Image

since vizarr doesn't take into account the greyscale setting.

Seen in validator:
Image

@will-moore will-moore moved this from upload some data to s3 and test to Zip and upload to BioStudies in NGFF conversion Jun 28, 2023
@will-moore
Copy link
Member 

$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0051/idr0051 [email protected]:5f/136e8d-xxxxxxxxxxx
180712_H2B_22ss_Courtney1_20180712-163837_p00                                                                         100% 1818MB 84.0Mb/s    02:02    
180712_H2B_22ss_Courtney_p00_c00_reg_preview.                                                                         100% 5430MB  210Mb/s    07:48    
2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c                                                                         100%  878MB 87.3Mb/s    08:44    
embryo_dmso_2_new_17-00-44_p00_c00_reg_previe                                                                         100% 2156MB  205Mb/s    11:15    
embryo_dmso_2_new_17-00-44_p00_c00_reg_previe                                                                         100% 2153MB 95.9Mb/s    13:46    
Completed: 12736265K bytes transferred in 827 seconds
 (126115K bits/sec), in 5 files, 1 directory.

@will-moore will-moore moved this from Zip and upload to BioStudies to BioStudies Submission in NGFF conversion Jun 28, 2023
@will-moore will-moore assigned francesw and unassigned will-moore Jun 28, 2023
@francesw francesw moved this from BioStudies Submission to create new Fileset to replace original Fileset in NGFF conversion Jul 12, 2023
@francesw francesw removed their assignment Jul 12, 2023
@will-moore
Copy link
Member

Following link from https://www.ebi.ac.uk/biostudies/submissions/ I can open the submission at
https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD815

I was hoping that the files for that submission would be available at a URL: https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD815.html but this gives:

<Error>
  <Code>NoSuchKey</Code>
  <BucketName>bia-integrator-data</BucketName>
  <RequestId>tx000000000000001633b49-0064d1ba0d-18ba7307-default</RequestId>
  <HostId>18ba7307-default-default</HostId>
</Error>

(this URL is based on previous idr0054 study data being available at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD704.html

@will-moore will-moore moved this from create new Fileset to replace original Fileset to Data on Embassy s3 in NGFF conversion Aug 15, 2023
@will-moore will-moore changed the title idr0051-fulton-tailbudlightsheet to NGFF idr0051-fulton-tailbudlightsheet S-BIAD815 Aug 15, 2023
@will-moore will-moore moved this from Data on Embassy s3 to create new Fileset to replace original Fileset in NGFF conversion Aug 18, 2023
@will-moore
Copy link
Member

will-moore commented Aug 18, 2023 •
edited
Loading

Data available at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD815.html
E.g. https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97/51afff7c-eed4-44b4-95c7-1437d8807b97.zarr/0

To harvest the uuids we need...

let csv = "";
$("#viewable tbody tr").each(function() {
    let $this = $(this);
    if ($("a", $this).length == 0) return
    let uid = $( "a:first", $this).attr("href").replace(".html", "");
    let zarrzip = $( "td:nth-child(3)", $this).text();
    csv += `${zarrzip},${uid}\n`
});
console.log(csv);

Gives me:

idr0051/180712_H2B_22ss_Courtney_p00_c00_reg_preview.klb.ome.zarr.zip,S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97
idr0051/embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.klb.ome.zarr.zip,S-BIAD815/b2633930-86b0-489e-a845-d2a7afe6ff15
idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr.zip,S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120
idr0051/2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c00_reg_preview.ome.zarr.zip,S-BIAD815/e12a8e2a-4fce-4579-a78b-b0c4597c3ada

Only 4 out of 5 zips are "viewable" above (unzipped onto s3).
idr0051/embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.ome.zarr.zip failed to unzip

@will-moore
Copy link
Member

Moving back to EBI column since we don't yet have all 5 images on s3 yet.
Discussed via e-mail. EBI will look at getting all images processed.

@will-moore will-moore moved this from create new Fileset to replace original Fileset to Data on Embassy s3 in NGFF conversion Aug 22, 2023
@will-moore
Copy link
Member

This has been tested with mkngff at IDR/omero-mkngff#4 (comment).

@will-moore will-moore moved this from Data on Embassy s3 to create new Filesets in idr-next in NGFF conversion Sep 4, 2023
@will-moore
Copy link
Member 

(venv3) [wmoore@test120-omeroreadonly-1 scripts]$ python check_pixels.py Project:552 --max-planes=sizeC
Start: 2023-12-06 20:45:32.863680
Checking Project:552
max_planes: sizeC
max_images: 0
0/5 Check Image:4007817 180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.pattern
1/5 Check Image:4007818 180712_H2B_22ss_Courtney_p00_c00_reg_preview.klb.pattern
2/5 Check Image:4007819 2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c00_reg_preview.pattern
3/5 Check Image:4007820 embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.pattern
4/5 Check Image:4007821 embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.klb.pattern
End: 2023-12-06 20:45:49.510165

@will-moore will-moore moved this from check_pixels to pixels validated in NGFF conversion Dec 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
NGFF conversion
Status: NGFF studies
Milestone
No milestone
Development

No branches or pull requests
3 participants
@will-moore @dominikl @francesw