Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

idr0054-segura-tonsilhyperion S-BIAD800 #654

Open
dominikl opened this issue Feb 27, 2023 · 13 comments
Open

idr0054-segura-tonsilhyperion S-BIAD800 #654

dominikl opened this issue Feb 27, 2023 · 13 comments

Comments

@dominikl
Copy link
Member

No description provided.

@dominikl dominikl moved this to test convert in NGFF conversion Feb 27, 2023
@dominikl dominikl moved this from test convert to re-import test image in NGFF conversion Feb 27, 2023
@dominikl dominikl moved this from re-import test image to convert all data to NGFF in NGFF conversion Feb 27, 2023
@dominikl
Copy link
Member Author

Export time: < 1 min / image
Reimport time: a few min.

@dominikl
Copy link
Member Author

Converted all three images on pilot-zarr2-dev in /data/ngff/idr0054

@will-moore will-moore moved this from convert all data to NGFF to upload data to s3 in NGFF conversion May 18, 2023
@will-moore
Copy link
Member

Created bucket as at #656

$ /home/wmoore/mc cp -r idr0054/ uk1s3/idr0054/zarr

# realised later this included idr0054/idr0054-segura-tonsilhyperion directory which we didn't want - removed:
./mc rm --recursive uk1s3/idr0054/zarr/idr0054-segura-tonsilhyperion --force

Looks good:
https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr0054/zarr/Tonsil+1.ome.zarr/0

Screenshot 2023-05-18 at 10 16 46

Mount on idr0125-pilot:

sudo mkdir /idr0054 && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0054 /idr0054
$ ls /idr0054/zarr/
Tonsil 1.ome.zarr  Tonsil 2.ome.zarr  Tonsil 3.ome.zarr

@will-moore
Copy link
Member

will-moore commented May 18, 2023 •
edited
Loading

Downloaded metadata-only images as described.

Import command needed quotes since dir names contain spaces:

for dir in *; do   omero import --transfer=ln_s --depth=100 --name="${dir/.ome.zarr/}" --skip=all "$dir" --file "/tmp/$dir.log"  --errs "/tmp/$dir.err"; done

Created Filesets: 5287119, 5287120, 5287121 - Images: 14834093, 14834094, 14834095.
e.g.

12 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:06.171

Forgot to import into Dataset... Not displayed in webclient!

$ dataset=$(omero obj new Dataset name='idr0054_NGFF')
$ omero obj new DatasetImageLink parent=$dataset child=Image:14834093
$ omero obj new DatasetImageLink parent=$dataset child=Image:14834094
$ omero obj new DatasetImageLink parent=$dataset child=Image:14834095

Symlinks:

$ python idr-utils/scripts/managed_repo_symlinks.py Dataset:15351 /idr0054/zarr/ --report

Fileset: 5287119 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/
Render Image 14834093
fs_contents ['Tonsil 1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr to /idr0054/zarr/Tonsil 1.ome.zarr

Fileset: 5287120 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/
Render Image 14834094
fs_contents ['Tonsil 2.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/Tonsil 2.ome.zarr to /idr0054/zarr/Tonsil 2.ome.zarr

Fileset: 5287121 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/
Render Image 14834095
fs_contents ['Tonsil 3.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/Tonsil 3.ome.zarr to /idr0054/zarr/Tonsil 3.ome.zarr

Images are viewable in webclient. 👍

Fileset swap...

$ python idr-utils/scripts/swap_filesets.py Dataset:5601 Dataset:15351 /tmp/idr0054_filesetswap.sql --report
SQL writing to /tmp/idr0054_filesetswap.sql
Swapping images between Fileset 1591301 and Fileset 5287119
Moving Image: 5025551 to Fileset: 5287119
Unlinking Image: 14834093
UPDATE pixels SET name = 'METADATA.ome.xml', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME' where image in (select id from Image where fileset = 5287119);
Swapping images between Fileset 5286813 and Fileset 5287120
Moving Image: 5025552 to Fileset: 5287120
Unlinking Image: 14834094
UPDATE pixels SET name = 'METADATA.ome.xml', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/Tonsil 2.ome.zarr/OME' where image in (select id from Image where fileset = 5287120);
Swapping images between Fileset 1591303 and Fileset 5287121
Moving Image: 5025553 to Fileset: 5287121
Unlinking Image: 14834095
UPDATE pixels SET name = 'METADATA.ome.xml', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/Tonsil 3.ome.zarr/OME' where image in (select id from Image where fileset = 5287121);
SQL output added to /tmp/idr0054_filesetswap.sql

$ PGPASSWORD=**** psql -U omero -d idr -h 192.168.10.102 -f /tmp/idr0054_filesetswap.sql 
UPDATE 1
UPDATE 1
UPDATE 1

Validate pixels -

$ python idr-utils/scripts/check_pixels.py Dataset:5601 /tmp/check_pixels_idr0054.log 
$ tail -f  /tmp/check_pixels_idr0054.log 
Start: 2023-05-18 10:35:55.992703
Checking Dataset:5601
max_planes: 0
0/3 Check Image:5025551 Tonsil 1
1/3 Check Image:5025552 Tonsil 2
2/3 Check Image:5025553 Tonsil 3
End: 2023-05-18 10:38:32.605204

Deleted Dataset in webclient

@will-moore will-moore moved this from upload data to s3 to Done (tested) in NGFF conversion May 18, 2023
@will-moore
Copy link
Member

Uploaded .zip files, one for each Fileset to https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD704
in order for the unzipped zarrs to be made available on s3.

cc @mrmh2 @joshmoore @sbesson @dominikl @francesw

For some fields required for the submission I was adding info from the publication - not sure I got it all perfect, but hopefully nothing incorrect.

@will-moore
Copy link
Member

Testing mkngff on idr0125-pilot

$ sudo -u root -s
$ source /opt/omero/server/venv3/bin/activate
$ pip install git+https://github.com/joshmoore/omero-mkngff.git
...
Resolved https://github.com/joshmoore/omero-mkngff.git to commit 68436532a8d2aa3ecb07164e08376a897389afa6
...
Successfully installed omero-mkngff-0.1.0.dev0

ls /bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/
0  OME

$ sudo -u omero-server -s
$ source /opt/omero/server/venv3/bin/activate

$ omero mkngff setup


CREATE OR REPLACE FUNCTION mkngff_fileset(old_fileset bigint, uuid character varying, repo character varying, prefix character varying, info text[][])
  RETURNS integer AS
$BODY$
DECLARE
   fs_id integer;
   file_id integer;
BEGIN

    insert into fileset (id, permissions, templateprefix, creation_id, group_id, owner_id, update_id)
                 values (nextval('seq_fileset'), -120, prefix, 497, 0, 0, 497)
                 returning id into fs_id;

    for i in 1 .. array_upper(info, 1)
    loop

      insert into originalfile (id, permissions, mimetype, repo, path, name, creation_id, owner_id, group_id, update_id)
                    values (nextval('seq_originalfile'), -120, info[i][3], repo, info[i][1], uuid || info[i][2], 497, 0, 0, 500)
                    returning id into file_id;

      insert into filesetentry (id, permissions, clientpath, creation_id, update_id, owner_id, group_id, fileset, originalfile, fileset_index)
                    values (nextval('seq_filesetentry'), -120, 'unknown', 497, 497, 0, 0, fs_id, file_id, i-1);

    end loop;

    update image set fileset = fs_id where fileset = old_fileset;

    RETURN fs_id;
END;
$BODY$
  LANGUAGE plpgsql VOLATILE;


$ omero mkngff sql 5287119 /bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr --zarr_name "Tonsil 1.ome.zarr"

Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18 // 09-50-49.857 for fileset 5287119

begin;
    select new_fileset(
      5287119,
      'TBD',
      'cdf35825-def1-4580-8d0b-9c349b8f78d6',
      'demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/Tonsil 1.ome.zarr/',
      array[
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/', '09-50-49.857_converted', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/', 'Tonsil 1.ome.zarr', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '.zattrs', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '.zgroup', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '0', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '.zattrs', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '.zgroup', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '0', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/0/', '.zarray', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '1', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/1/', '.zarray', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '2', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/2/', '.zarray', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '3', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/3/', '.zarray', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '4', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/4/', '.zarray', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', 'OME', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', '.zattrs', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', '.zgroup', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', 'METADATA.ome.xml', 'application/octet-stream']
      ]::text[][]
    )
);
commit;

Tried pasting those into a psql terminal...
Got errors due to an extra closing ) and had to replace new_fileset( with mkngff_fileset(.

That got me to....

idr=> begin;
BEGIN
idr=>     select mkngff_fileset(
idr(>       5287119,
idr(>       'TBD',
idr(>       'cdf35825-def1-4580-8d0b-9c349b8f78d6',
idr(>       'demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/Tonsil 1.ome.zarr/',
idr(>       array[
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/', '09-50-49.857_converted', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/', 'Tonsil 1.ome.zarr', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '.zattrs', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '.zgroup', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '0', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '.zattrs', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '.zgroup', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '0', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/0/', '.zarray', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '1', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/1/', '.zarray', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '2', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/2/', '.zarray', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '3', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/3/', '.zarray', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '4', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/4/', '.zarray', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', 'OME', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', '.zattrs', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', '.zgroup', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', 'METADATA.ome.xml', 'application/octet-stream']
idr(>       ]::text[][]
idr(>     );
ERROR:  cannot set original repo property without secret key
CONTEXT:  PL/pgSQL function _protect_originalfile_repo_insert() line 28 at RAISE
SQL statement "insert into originalfile (id, permissions, mimetype, repo, path, name, creation_id, owner_id, group_id, update_id)
                    values (nextval('seq_originalfile'), -120, info[i][3], repo, info[i][1], uuid || info[i][2], 497, 0, 0, 500)
                    returning id"
PL/pgSQL function mkngff_fileset(bigint,character varying,character varying,character varying,text[]) line 14 at SQL statement
idr=>     commit;
ROLLBACK

cc @joshmoore

@joshmoore
Copy link
Member

Sorry, @will-moore. I'm still pushing commits (e.g. IDR/omero-mkngff@2175a18). Hopefully I'll have a commit ready for you today.

@joshmoore
Copy link
Member

@will-moore: IDR/omero-mkngff@fe4e8ae is up-to-date and works as expected up to rendering where I get the Invalid index: -1 error.

Finding the secret UUID is still a bit complicated, but:

select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 10;

is a current starting point. Once you have that, the steps are:

SECRET=3a4d3a3f-ac98-4b1a-84ab-5534e364f181
micromamba create -n omero -c conda-forge -c ome omero-py bioformats2raw
micromamba activate omero
pip install 'omero-mkngff @ git+https://github.com/joshmoore/omero-mkngff@main'
omero login demo@localhost
echo series=2 >> a.fake.ini
touch a.fake
omero import a.fake.ini
bioformats2raw -p a.fake.ini a.ome.zarr
omero mkngff setup > setup.sql
omero mkgnff sql --secret=$SECRET 5287125 a.ome.zarr/
sudo -u postgres psql idr < setup.sql
sudo -u postgres psql idr < my.sql
sudo -u omero-server mkdir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-06/22/12-46-39.975_converted/

mv a.ome.zarr /tmp
ln -s /tmp/a.ome.zarr /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-06/22/12-46-39.975_converted/a.ome.zarr
omero render test Image:14834721 # Failing here

@will-moore will-moore self-assigned this Jun 26, 2023
@will-moore will-moore moved this from Zip and upload to BioStudies to BioStudies Submission in NGFF conversion Jun 27, 2023
@will-moore will-moore assigned francesw and unassigned will-moore Jun 28, 2023
@francesw francesw moved this from BioStudies Submission to create new Fileset to replace original Fileset in NGFF conversion Jul 7, 2023
@francesw francesw removed their assignment Jul 7, 2023
@will-moore
Copy link
Member

will-moore commented Aug 8, 2023 •
edited
Loading

The data for my first BioStudies submission above is at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD704.html

You can get the mapping of zarr -> uuid by running this in the browser console:

let csv = "";
$("#viewable tbody tr").each(function() {
    let $this = $(this);
    if ($("a", $this).length == 0) return
    let uid = $( "td:nth-child(1)", $this).text();
    let zarrzip = $( "td:nth-child(3)", $this).text();
    csv += `${zarrzip},${uid}\n`
});
document.write(`<code><pre>${csv}</pre></code>`);

creates:

Tonsil 3.ome.zarr.zip,3b4a8721-1a28-4bc4-8443-9b6e145efbe9
Tonsil 2.ome.zarr.zip,36cb5355-5134-4bdc-bde6-4e693055a8f9
Tonsil 1.ome.zarr.zip,5583fe0a-bbe6-4408-ab96-756e8e96af55

@will-moore will-moore moved this from create new Fileset to replace original Fileset to Data on Embassy s3 in NGFF conversion Aug 15, 2023
@will-moore will-moore changed the title idr0054-segura-tonsilhyperion to NGFF idr0054-segura-tonsilhyperion S-BIAD704 Aug 15, 2023
@will-moore
Copy link
Member

IDR submission is at https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD800

@will-moore will-moore changed the title idr0054-segura-tonsilhyperion S-BIAD704 idr0054-segura-tonsilhyperion S-BIAD800 Aug 15, 2023
@will-moore will-moore moved this from Data on Embassy s3 to create new Fileset to replace original Fileset in NGFF conversion Aug 18, 2023
@will-moore
Copy link
Member

Data at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD800.html

let csv = "";
$("#viewable tbody tr").each(function() {
    let $this = $(this);
    if ($("a", $this).length == 0) return
    let uid = $( "a:first", $this).attr("href").replace(".html", "");
    let zarrname = $( "td:nth-child(3)", $this).text().replace(".zip", "");
    csv += `${zarrname},${uid}\n`
});
console.log(csv);

Gives:

idr0054/Tonsil 1.ome.zarr,S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39
idr0054/Tonsil 2.ome.zarr,S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573
idr0054/Tonsil 3.ome.zarr,S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3

@will-moore
Copy link
Member

mkngff test...
Removing white-space and manually adding Fileset IDs... idr0054.csv:

idr0054/Tonsil_1.ome.zarr,S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39,5287366
idr0054/Tonsil_2.ome.zarr,S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573,5287367
idr0054/Tonsil_3.ome.zarr,S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3,5287121

for r in $(cat idr0054.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3)
  omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$fsid.sql"
done

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18 // 09-50-49.857_converted for fileset 5287366
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr -> /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18 // 09-51-02.628_converted for fileset 5287367
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted_mkngff/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr -> /bia-integrator-data/S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18 // 09-51-13.702 for fileset 5287121
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702_mkngff/f49ada41-43bf-47ff-99b9-bdf8cc311ce3.zarr -> /bia-integrator-data/S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3/f49ada41-43bf-47ff-99b9-bdf8cc311ce3.zarr

for r in $(cat idr0054.csv); do
  fsid=$(echo $r | cut -d',' -f3)
  psql -U omero -d idr -h $DBHOST -f "$fsid.sql"
done

Screenshot 2023-08-29 at 16 40 06

@will-moore
Copy link
Member 

(venv3) [wmoore@test120-omeroreadonly-1 scripts]$ python check_pixels.py Project:701 --max-planes=sizeC
Start: 2023-12-06 20:40:07.302346
Checking Project:701
max_planes: sizeC
max_images: 0
0/3 Check Image:5025551 Tonsil 1
1/3 Check Image:5025552 Tonsil 2
2/3 Check Image:5025553 Tonsil 3
End: 2023-12-06 20:44:17.278682

@will-moore will-moore moved this from check_pixels to pixels validated in NGFF conversion Dec 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
NGFF conversion
Status: NGFF studies
Milestone
No milestone
Development

No branches or pull requests
4 participants
@joshmoore @will-moore @dominikl @francesw