Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reintroduce S3FileSystemStore #82

Merged
merged 25 commits into from
Jun 5, 2024
Merged

Reintroduce S3FileSystemStore #82

merged 25 commits into from
Jun 5, 2024

Conversation

dgault
Copy link
Member

@dgault dgault commented Mar 10, 2024

Re-adding the S3FileSystemStore from #38 in order to enable reading directly from Embassy S3. This will also add the awssdk dependency back into the project.

Some modifications from the original have made and a new option has been added to allow for an alternative s3 location to set in the bfoptions file.

@jburel
Copy link
Member

jburel commented Mar 13, 2024

The dependency was removed in the first instance since it was breaking the openmicroscopy build.
Re-introducing it will solve one problem but not the other. This will require some investigation, the problem noticed at the time might be fixed in newer version of the aws-sdk-java library.
The version added by this PR is 4 years old. We should should try to add a more recent version of the library (latest is 2.25.7)

@dgault
Copy link
Member Author

dgault commented Mar 14, 2024

Some initial performance testing results. First testing idr0090 plate 190206, comparing the local filesystem to s3, both via the standard setId and openBytes combination, and then via the new static openPlane function. Each test result is averaged from 10 runs

From a local filesystem via setID (with memo):

  SetID (with memo) openBytes Total time
Average time (ms): 1781.838209 71.2004399 1853.038649

From a local filesystem via the new static method

  new openPlane
Average time (ms): 634.9016599

From s3 via setID (with memo) using the new bfoptions method:

omezarr.alt_store=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
  SetID (with memo) openBytes Total time
Average time (ms): 1291.234305 3295.094909 4586.329215

Surprisingly in this case the setID time was consistenly lower which I would not have expected

From s3 via the new static method

ZarrReader.openPlane("https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr/B/2/0/0", newBuffer);
  new openPlane
Average time (ms): 5133.552719

Profiling the calls

I had expected the calls to openBytes and openPlane to be closer to that of the filesystem so I profiled the new static openPlane for both the local filesystem and s3 calls.

Local filesystem:
Screenshot 2024-03-14 at 13 34 12

s3:
Screenshot 2024-03-14 at 13 28 23

Interestingly the Blosc decompression appears to be considerably slower when using the s3 option

idr0054

I also tested idr0054 (same as Seb had used for his profiling), 10 runs

Using the new static openPlane function from s3

ZarrReader.openPlane("https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0", newBuffer);

Average time (ms): 4213.232465

Profiling the call shows about 50% of the time being taken by the BloscCompressor uncompressing the data:
Screenshot 2024-03-14 at 13 13 59

@dgault
Copy link
Member Author

dgault commented Mar 15, 2024

With the build on ome/openmicroscopy#6379 now green hopefully this PR should be ready for some integration testing.

As far as next steps, there are 2 options for testing, using the bfoptions to load direct from S3 or using the new static method and bypassing the reader initialisation and loading of the memo file.

To load directly from S3 the path should be added to the bfoptions file using the new option. This will also require the memo files to be re-generated. Note that in testing the memo files were not detected as being invalid so I had to delete them first. An example of how this would look in the bfoptions file:

omezarr.alt_store=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr

To test the new static method will require changes to how pixel data is retrieved. Currently this is specific to the ZarrReader and any non Zarr data will not be able to use this approach. Instead of initialising the reader via the Memoizer and calling openBytes you can simply call the new method, pointing to the location of a zarray that you wish to open. A buffer needs to be declared first also.

ZarrReader.openPlane("https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0", newBuffer);

@jburel
Copy link
Member

jburel commented Mar 16, 2024

Removed the exclude flag

@snoopycrimecop
Copy link
Member

Conflicting PR. Removed from build BIOFORMATS-push#4. See the console output for more details.
Possible conflicts:

--conflicts

@snoopycrimecop
Copy link
Member

Conflicting PR. Removed from build BIOFORMATS-push#808. See the console output for more details.
Possible conflicts:

--conflicts

@snoopycrimecop
Copy link
Member

Conflicting PR. Removed from build BIOFORMATS-push#5. See the console output for more details.
Possible conflicts:

--conflicts

@snoopycrimecop
Copy link
Member

snoopycrimecop commented Mar 18, 2024

Conflicting PR. Removed from build BIOFORMATS-push#809. See the console output for more details.
Possible conflicts:

--conflicts Conflict resolved in build BIOFORMATS-push#6. See the console output for more details.

@will-moore
Copy link
Member

will-moore commented Mar 19, 2024

Testing today's build... Merged in https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-push/6/console

$ ssh -A idr-testing.openmicroscopy.org -L 1080:omeroreadwrite:80
$ ssh omeroreadwrite
$ sudo -u omero-server -s
$ cd
$ wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/label=testintegration/7/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar
$ mv OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_p6_b7.jar
$ rm OMERO.server/lib/client/OMEZarrReader.jar 
$ rm OMERO.server/lib/server/OMEZarrReader.jar 
$ cp OMEZarrReader_p6_b7.jar OMERO.server/lib/client/
$ cp OMEZarrReader_p6_b7.jar OMERO.server/lib/server/

restart server...

Tested render_image performance on idr0010 - 1st plate using existing memo file (same behaviour as without this PR)

Then deleted memo file and added bfoption...

rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo
$ cat  /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr.bfoptions
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD885/e7bdccbf-5119-4d9c-b466-66d211871db8/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr

View image to regenerate memo... 13:45 -> approx 10 minutes...

Timed the render_image_region (whole plane) on idr-testing:omeroreadwrite (no microservices) via localhost:1080, selecting a Well each time, with the Preview panel open.

Average approx 2 seconds with the goofys access - green (if we ignore the outliers > 3.5 secs) and average about 1 second with this PR - blue:

Screenshot 2024-03-19 at 14 34 00

@will-moore
Copy link
Member

Repeat the testing above on idr0090 - first plate:

bash-4.2$ cat /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr.bfoptions
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/a5a2714b-bfbf-4251-95ac-5319fda4bf69/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr

memo started 15:06...

@will-moore
Copy link
Member

It seems that the idr0090 plate above is still not viewable. Grepping the logs for Fileset path finds a lot of repetition...

OMERO.server/var/log/Blitz-0.log
bash-4.2$ grep "a5a2714b-bfbf-4251-95ac-5319fda4bf69" OMERO.server/var/log/Blitz-0.log 

2024-03-19 14:40:40,587 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:40:41,727 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:40:42,319 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 0
2024-03-19 14:40:42,771 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:40:43,037 DEBUG [                   loci.formats.Memoizer] (l.Server-3) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:40:43,039 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 0
2024-03-19 14:40:43,327 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:40:43,585 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:40:43,587 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 0
2024-03-19 14:40:44,077 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-6) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:40:44,325 DEBUG [                   loci.formats.Memoizer] (l.Server-6) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:40:44,327 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 0
2024-03-19 14:41:09,656 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-7) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:09,952 DEBUG [                   loci.formats.Memoizer] (l.Server-7) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:09,954 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 32
2024-03-19 14:41:10,425 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:10,854 DEBUG [                   loci.formats.Memoizer] (l.Server-5) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:10,856 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 32
2024-03-19 14:41:11,213 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-2) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:11,530 DEBUG [                   loci.formats.Memoizer] (l.Server-2) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:11,532 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 32
2024-03-19 14:41:11,987 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-9) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:12,252 DEBUG [                   loci.formats.Memoizer] (l.Server-9) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:12,254 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 32
2024-03-19 14:41:21,649 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:21,672 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:21,682 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-9) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:21,683 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-6) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:22,157 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:22,158 DEBUG [                   loci.formats.Memoizer] (l.Server-5) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:22,159 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 32
2024-03-19 14:41:22,159 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 32
2024-03-19 14:41:22,180 DEBUG [                   loci.formats.Memoizer] (l.Server-6) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:22,182 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 32
2024-03-19 14:41:22,197 DEBUG [                   loci.formats.Memoizer] (l.Server-9) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:22,199 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 32
2024-03-19 14:41:35,698 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:36,038 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:36,039 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 64
2024-03-19 14:41:36,481 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:37,004 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:37,006 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 64
2024-03-19 14:41:37,443 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-9) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:37,790 DEBUG [                   loci.formats.Memoizer] (l.Server-9) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:37,791 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 64
2024-03-19 14:41:38,207 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:38,638 DEBUG [                   loci.formats.Memoizer] (l.Server-3) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:38,642 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 64
2024-03-19 14:41:44,822 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:44,822 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-6) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:44,822 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:44,833 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-7) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:41:46,579 DEBUG [                   loci.formats.Memoizer] (l.Server-6) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:46,620 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:46,785 DEBUG [                   loci.formats.Memoizer] (l.Server-7) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:46,824 DEBUG [                   loci.formats.Memoizer] (l.Server-3) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:41:48,272 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 64
2024-03-19 14:41:48,572 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 64
2024-03-19 14:41:48,573 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 64
2024-03-19 14:41:48,573 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 64
2024-03-19 14:42:51,231 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-6) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:42:52,183 DEBUG [                   loci.formats.Memoizer] (l.Server-6) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:42:52,893 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 96
2024-03-19 14:42:53,771 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:42:54,146 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:42:54,150 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 96
2024-03-19 14:42:54,552 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:42:54,858 DEBUG [                   loci.formats.Memoizer] (l.Server-3) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:42:54,861 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 96
2024-03-19 14:42:55,261 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:42:55,701 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:42:55,703 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 96
2024-03-19 14:43:08,942 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:43:08,945 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:43:08,948 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:43:08,982 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-2) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:43:09,380 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:43:09,381 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 96
2024-03-19 14:43:09,411 DEBUG [                   loci.formats.Memoizer] (l.Server-2) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:43:09,412 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 96
2024-03-19 14:43:09,443 DEBUG [                   loci.formats.Memoizer] (l.Server-5) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:43:09,444 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 96
2024-03-19 14:43:09,492 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:43:09,494 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 96
2024-03-19 14:57:02,382 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:03,445 DEBUG [                   loci.formats.Memoizer] (l.Server-3) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:04,095 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 128
2024-03-19 14:57:04,531 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-2) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:04,793 DEBUG [                   loci.formats.Memoizer] (l.Server-2) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:04,795 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 128
2024-03-19 14:57:05,232 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:05,547 DEBUG [                   loci.formats.Memoizer] (l.Server-5) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:05,548 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 128
2024-03-19 14:57:06,214 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:06,476 DEBUG [                   loci.formats.Memoizer] (l.Server-3) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:06,478 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 128
2024-03-19 14:57:11,983 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:12,261 DEBUG [                   loci.formats.Memoizer] (l.Server-3) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:12,262 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 160
2024-03-19 14:57:13,021 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:13,377 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:13,379 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 160
2024-03-19 14:57:13,810 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:14,073 DEBUG [                   loci.formats.Memoizer] (l.Server-4) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:14,075 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 160
2024-03-19 14:57:14,612 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:14,871 DEBUG [                   loci.formats.Memoizer] (l.Server-5) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:14,873 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 160
2024-03-19 14:57:16,255 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-7) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:16,529 DEBUG [                   loci.formats.Memoizer] (l.Server-7) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:16,531 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 192
2024-03-19 14:57:16,991 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:17,361 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:17,363 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 192
2024-03-19 14:57:17,704 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-7) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:18,135 DEBUG [                   loci.formats.Memoizer] (l.Server-7) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:18,137 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 192
2024-03-19 14:57:18,680 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:18,926 DEBUG [                   loci.formats.Memoizer] (l.Server-3) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:18,927 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 192
2024-03-19 14:57:22,905 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-6) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:23,173 DEBUG [                   loci.formats.Memoizer] (l.Server-6) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:23,175 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 224
2024-03-19 14:57:23,793 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-8) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:24,044 DEBUG [                   loci.formats.Memoizer] (l.Server-8) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:24,045 INFO  [                ome.io.nio.PixelsService] (l.Server-8) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 224
2024-03-19 14:57:24,468 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:24,836 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:24,838 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 224
2024-03-19 14:57:25,323 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:25,760 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:25,762 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 224
2024-03-19 14:57:29,008 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-7) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:29,294 DEBUG [                   loci.formats.Memoizer] (l.Server-7) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:29,296 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 256
2024-03-19 14:57:29,694 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:29,955 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:29,957 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 256
2024-03-19 14:57:30,396 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-2) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:30,698 DEBUG [                   loci.formats.Memoizer] (l.Server-2) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:30,700 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 256
2024-03-19 14:57:31,037 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-9) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:31,524 DEBUG [                   loci.formats.Memoizer] (l.Server-9) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:31,526 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 256
2024-03-19 14:57:34,858 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-9) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:35,145 DEBUG [                   loci.formats.Memoizer] (l.Server-9) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:35,146 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 288
2024-03-19 14:57:35,558 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-8) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:35,828 DEBUG [                   loci.formats.Memoizer] (l.Server-8) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:35,830 INFO  [                ome.io.nio.PixelsService] (l.Server-8) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 288
2024-03-19 14:57:36,285 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:36,687 DEBUG [                   loci.formats.Memoizer] (l.Server-5) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:36,689 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 288
2024-03-19 14:57:37,186 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:37,528 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:37,530 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 288
2024-03-19 14:57:41,018 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-8) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:41,411 DEBUG [                   loci.formats.Memoizer] (l.Server-8) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:41,413 INFO  [                ome.io.nio.PixelsService] (l.Server-8) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 320
2024-03-19 14:57:41,799 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:42,355 DEBUG [                   loci.formats.Memoizer] (l.Server-4) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:42,357 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 320
2024-03-19 14:57:42,850 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-6) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:43,203 DEBUG [                   loci.formats.Memoizer] (l.Server-6) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:43,205 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 320
2024-03-19 14:57:43,603 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:43,969 DEBUG [                   loci.formats.Memoizer] (l.Server-5) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:43,971 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 320
2024-03-19 14:57:52,187 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-2) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:52,553 DEBUG [                   loci.formats.Memoizer] (l.Server-2) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:52,554 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 352
2024-03-19 14:57:53,014 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-6) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:53,260 DEBUG [                   loci.formats.Memoizer] (l.Server-6) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:53,262 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 352
2024-03-19 14:57:53,720 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:54,002 DEBUG [                   loci.formats.Memoizer] (l.Server-4) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:54,004 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 352
2024-03-19 14:57:54,372 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-8) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:57:54,705 DEBUG [                   loci.formats.Memoizer] (l.Server-8) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:57:54,707 INFO  [                ome.io.nio.PixelsService] (l.Server-8) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 352
2024-03-19 14:58:02,736 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:03,241 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:07,718 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 384
2024-03-19 14:58:08,395 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-2) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:08,756 DEBUG [                   loci.formats.Memoizer] (l.Server-2) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:08,758 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 384
2024-03-19 14:58:09,279 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-8) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:09,691 DEBUG [                   loci.formats.Memoizer] (l.Server-8) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:09,692 INFO  [                ome.io.nio.PixelsService] (l.Server-8) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 384
2024-03-19 14:58:10,167 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:10,427 DEBUG [                   loci.formats.Memoizer] (l.Server-3) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:10,429 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 384
2024-03-19 14:58:17,231 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:17,580 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:17,582 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 416
2024-03-19 14:58:17,982 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-7) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:18,398 DEBUG [                   loci.formats.Memoizer] (l.Server-7) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:18,400 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 416
2024-03-19 14:58:18,922 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-6) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:19,178 DEBUG [                   loci.formats.Memoizer] (l.Server-6) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:19,180 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 416
2024-03-19 14:58:20,347 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:20,740 DEBUG [                   loci.formats.Memoizer] (l.Server-4) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:20,742 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 416
2024-03-19 14:58:24,749 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:25,178 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:25,179 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 448
2024-03-19 14:58:25,577 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:25,834 DEBUG [                   loci.formats.Memoizer] (l.Server-4) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:25,835 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 448
2024-03-19 14:58:26,329 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-2) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:26,617 DEBUG [                   loci.formats.Memoizer] (l.Server-2) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:26,619 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 448
2024-03-19 14:58:27,150 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:27,526 DEBUG [                   loci.formats.Memoizer] (l.Server-5) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:27,527 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 448
2024-03-19 14:58:31,177 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:31,445 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:31,447 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 480
2024-03-19 14:58:31,863 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-6) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:32,298 DEBUG [                   loci.formats.Memoizer] (l.Server-6) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:32,300 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 480
2024-03-19 14:58:32,811 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:33,065 DEBUG [                   loci.formats.Memoizer] (l.Server-4) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:33,066 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 480
2024-03-19 14:58:33,521 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:33,769 DEBUG [                   loci.formats.Memoizer] (l.Server-4) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:33,771 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 480
2024-03-19 14:58:38,691 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-7) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:39,080 DEBUG [                   loci.formats.Memoizer] (l.Server-7) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:39,083 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 512
2024-03-19 14:58:39,507 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:39,982 DEBUG [                   loci.formats.Memoizer] (l.Server-1) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:39,984 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 512
2024-03-19 14:58:40,547 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:40,942 DEBUG [                   loci.formats.Memoizer] (l.Server-5) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:40,943 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 512
2024-03-19 14:58:41,411 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-7) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 14:58:41,771 DEBUG [                   loci.formats.Memoizer] (l.Server-7) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 14:58:41,774 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 512
2024-03-19 15:01:01,568 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-2) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:01:02,626 DEBUG [                   loci.formats.Memoizer] (l.Server-2) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 15:01:03,253 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 544
2024-03-19 15:01:03,648 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:01:03,939 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 15:01:03,940 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 544
2024-03-19 15:01:04,503 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:01:04,788 DEBUG [                   loci.formats.Memoizer] (l.Server-4) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 15:01:04,790 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 544
2024-03-19 15:01:05,255 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:01:05,683 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 15:01:05,685 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 544
2024-03-19 15:01:09,938 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-7) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:01:10,222 DEBUG [                   loci.formats.Memoizer] (l.Server-7) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 15:01:10,224 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 576
2024-03-19 15:01:10,578 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:01:11,060 DEBUG [                   loci.formats.Memoizer] (l.Server-0) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 15:01:11,062 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 576
2024-03-19 15:01:11,618 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-2) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:01:11,879 DEBUG [                   loci.formats.Memoizer] (l.Server-2) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 15:01:11,881 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 576
2024-03-19 15:01:12,354 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:01:12,615 DEBUG [                   loci.formats.Memoizer] (l.Server-3) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/..zattrs.bfmemo (10105126 bytes)
2024-03-19 15:01:12,616 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs Series: 576
2024-03-19 15:01:31,185 INFO  [        ome.services.util.ServiceHandler] (l.Server-5)  Rslt:	([demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fd, .zattrs, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/a5a2714b-bfbf-4251-95ac-5319fda4bf69/a], [demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fd, .zgroup, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/a5a2714b-bfbf-4251-95ac-5319fda4bf69/a], [demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fd, .zgroup, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/a5a2714b-bfbf-4251-95ac-5319fda4bf69/a], ... 12646 more)
2024-03-19 15:04:41,242 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-6) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:04:41,249 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:04:41,251 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-7) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:04:41,259 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-9) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:04:41,904 INFO  [                loci.formats.ImageReader] (l.Server-7) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:04:41,910 INFO  [                loci.formats.ImageReader] (l.Server-5) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:04:41,911 INFO  [                loci.formats.ImageReader] (l.Server-6) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:04:41,911 INFO  [                loci.formats.ImageReader] (l.Server-9) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:19:43,189 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:19:43,198 INFO  [                loci.formats.ImageReader] (l.Server-4) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:19:43,221 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:19:43,221 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-2) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:19:43,227 INFO  [                loci.formats.ImageReader] (l.Server-0) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:19:43,227 INFO  [                loci.formats.ImageReader] (l.Server-2) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:34:43,277 INFO  [      ome.services.OmeroFilePathResolver] (.Server-12) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:34:43,294 INFO  [                loci.formats.ImageReader] (.Server-12) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:48:05,689 INFO  [      ome.services.OmeroFilePathResolver] (.Server-11) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:48:05,699 INFO  [                loci.formats.ImageReader] (.Server-11) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:50:03,526 INFO  [      ome.services.OmeroFilePathResolver] (.Server-10) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:50:03,537 INFO  [                loci.formats.ImageReader] (.Server-10) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:50:12,315 INFO  [      ome.services.OmeroFilePathResolver] (.Server-13) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 15:50:12,321 INFO  [                loci.formats.ImageReader] (.Server-13) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 16:04:38,402 INFO  [      ome.services.OmeroFilePathResolver] (.Server-16) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 16:04:38,413 INFO  [                loci.formats.ImageReader] (.Server-16) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 16:11:01,286 INFO  [      ome.services.OmeroFilePathResolver] (.Server-15) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 16:11:01,295 INFO  [                loci.formats.ImageReader] (.Server-15) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 22:29:36,708 INFO  [      ome.services.OmeroFilePathResolver] (.Server-17) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 22:29:36,745 INFO  [                loci.formats.ImageReader] (.Server-17) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 22:31:09,680 INFO  [      ome.services.OmeroFilePathResolver] (.Server-18) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 22:31:09,688 INFO  [                loci.formats.ImageReader] (.Server-18) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 22:34:32,083 INFO  [      ome.services.OmeroFilePathResolver] (.Server-20) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-19 22:34:32,093 INFO  [                loci.formats.ImageReader] (.Server-20) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-20 09:38:13,874 INFO  [      ome.services.OmeroFilePathResolver] (.Server-19) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-20 09:38:13,912 INFO  [                loci.formats.ImageReader] (.Server-19) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr/.zattrs
2024-03-20 09:38:42,158 INFO  [        ome.services.util.ServiceHandler] (.Server-14)  Rslt:	([demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fd, .zattrs, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/a5a2714b-bfbf-4251-95ac-5319fda4bf69/a], [demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fd, .zgroup, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/a5a2714b-bfbf-4251-95ac-5319fda4bf69/a], [demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fd, .zgroup, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/a5a2714b-bfbf-4251-95ac-5319fda4bf69/a], ... 12646 more)

@will-moore
Copy link
Member

will-moore commented Mar 20, 2024

Going to try on a different study - idr0016, testing the memo file regeneration with and without the alt_store setting in bfoptions....

Delete the memo files for first 2 plates from idr0016...

bash-4.2$ rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-08/18/18-11-19.764_mkngff/171961d7-6cca-4201-87e4-449ff3880d48.zarr/..zattrs.bfmemo 
bash-4.2$ rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-06/15/22-36-23.585_mkngff/f1a84265-b1fa-4ef5-b8fa-fb0c01e2c24e.zarr/..zattrs.bfmemo 

Update the bfoptions of the first plate only...

$ cat /data/OMERO/ManagedRepository/demo_2/2017-08/18/18-11-19.764_mkngff/171961d7-6cca-4201-87e4-449ff3880d48.zarr.bfoptions
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/171961d7-6cca-4201-87e4-449ff3880d48/171961d7-6cca-4201-87e4-449ff3880d48.zarr

Then view images to trigger memo file...

After ~40 mins I noticed lots of DatabaseBusyException in the Blitz logs. Seems the server is overwhelmed trying to create memos for 2 plates from idr0016 (and still trying for the idr0090 plate above)?

Restarted the server... ~ 10:50

Compared time to render_image when initially loading each Well in Preview panel:

Screenshot 2024-03-20 at 15 10 59

@will-moore
Copy link
Member

Comparing the log files from idr0090 with the regeneration of idr0010 memo file above, idr0010 looks like this at the time that "memo saved" (there was no "memo saved" seen yet for idr0090 logs above):

idr0010: $ grep "e7bdccbf-5119-4d9c-b466-66d211871db8" OMERO.server/var/log/Blitz-0.log...

...
2024-03-19 13:36:10,073 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml Series: 66
2024-03-19 13:36:10,481 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-9) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml
2024-03-19 13:36:10,513 DEBUG [                   loci.formats.Memoizer] (l.Server-9) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo (520047 bytes)
2024-03-19 13:36:10,515 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml Series: 66
2024-03-19 13:36:58,664 INFO  [        ome.services.util.ServiceHandler] (l.Server-4)  Rslt:	([demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/, .zattrs, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD885/e7bdccbf-5119-4d9c-b466-66d211871db8/e], [demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/, .zgroup, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD885/e7bdccbf-5119-4d9c-b466-66d211871db8/e], [demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/A/, .zgroup, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD885/e7bdccbf-5119-4d9c-b466-66d211871db8/e], ... 2702 more)
2024-03-19 13:44:48,528 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml
2024-03-19 13:44:49,615 INFO  [                loci.formats.ImageReader] (l.Server-1) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml
2024-03-19 13:52:15,347 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml
2024-03-19 13:52:15,561 INFO  [                loci.formats.ImageReader] (l.Server-5) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml
2024-03-19 13:57:24,720 DEBUG [                   loci.formats.Memoizer] (l.Server-1) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo17125964820390949463
2024-03-19 13:57:24,720 DEBUG [                   loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo (520210 bytes)

The line with ome.services.util.ServiceHandler] (l.Server-4) Rslt: .... ... 2702 more) is followed closely by memo saved but for idr0090 logs above, the Rslt.... ... 12646 more) is not followed by memo saved.

@will-moore
Copy link
Member

will-moore commented Mar 20, 2024

Re-started memo file generation for idr0090 (first plate) by clicking on Preview panel....

Worked this time:
On idr-testing:readonly server (no microservices as above). render_image_region times for initial Well loaded in Preview panel for idr0090 first plate (s3 vv goofys):

Screenshot 2024-03-20 at 17 42 04

@will-moore
Copy link
Member

will-moore commented Mar 26, 2024

As expected, once we have memo file generated with the alt_store option, we can unmount the goofys file system and continue to render images.
However, less expected is that if we unmount goofys before the memo file is re-generated then we get a ResourceError.
Using these commands on idr-testing...

sudo umount /bia-integrator-data
sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

@will-moore
Copy link
Member

Updating ZarrReader on all omeroreadonly servers on idr-testing...
Ran this on each as omero-server...

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/label=testintegration/16/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar

mv OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_b16.jar
$ rm OMERO.server/lib/client/OMEZarrReader.jar 
$ rm OMERO.server/lib/server/OMEZarrReader.jar 
$ cp OMEZarrReader_b16.jar OMERO.server/lib/client/
$ cp OMEZarrReader_b16.jar OMERO.server/lib/server/

Then restarted...

@will-moore
Copy link
Member

I have added the alt_source to the .bfoptions for all NGFF filesets on idr-testing with script and commands at IDR/mkngff_upgrade_scripts#1

@will-moore
Copy link
Member

will-moore commented Mar 26, 2024

Started memo file generation for all NGFF filesets on idr-testing, using ngff_ids.txt from IDR/idr-metadata#686 (comment)

$ screen -dmS cache parallel --eta --sshloginfile nodes -a ngff_ids.txt --results /tmp/ngff_cache_20240326/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'

EDIT: 27th March 15:12 - getting DatabaseBusyExceptions.

[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240326/1/**/* | wc
    407    1221   32202

Cancel screen, Restart all servers":

for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-server restart"; done

21:14: restarted...

screen -dmS cache parallel --eta --sshloginfile nodes -a ngff_ids.txt --results /tmp/ngff_cache_20240327/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'

9:00 on 28th - current status: 647 Images (Plates) processed, 576 of them are "ok" out of a total of 6763 in the total list (doesn't include Datasets).

[wmoore@test120-proxy ~]$ ls /tmp/ngff_cache_20240327/1/ | wc
    647     647    9012
[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240327/1/**/* | wc
    576    1728   45506
[wmoore@test120-proxy ~]$ grep Image ids.txt | wc
   6763    6763   95877

10:30 pm on 29th March:

[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240327/1/**/* | wc
    758    2274   59980

10:00 pm on 1st April:

[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240327/1/**/* | wc
   1107    3321   87656

@will-moore
Copy link
Member

will-moore commented Apr 1, 2024

It feels like memo file generation is taking longer than before, but I don't have comparable numbers...

Since we have some timings for idr0090 memo generation at IDR/idr-metadata#686 (comment), and it would be useful to have the memo files for testing idr0090, let's cancel running above and run idr0090 only...

Status before cancelling...

screen -r
...
Computer:jobs running/jobs completed/%of started jobs
ETA: 155479s Left: 431 AVG: 360.76s  1:10/152/12%/2877.0s  2:10/196/16%/2231.1s  3:10/449/36%/973.9s  4:10/251/20%/1742.2s  5:10/164/13%/2666.4s 

Restarted on idr0090 only...

screen -dmS cache parallel --eta --sshloginfile nodes -a idr0090_ids.txt --results /tmp/ngff_cache_20240401/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'

After 7 hours:

[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240401/1/**/*
/tmp/ngff_cache_20240401/1/Image:12542037/stdout:ok: 12542037 11.96731948852539 
/tmp/ngff_cache_20240401/1/Image:12546037/stdout:ok: 12546037 4475.028658628464 
/tmp/ngff_cache_20240401/1/Image:12548245/stdout:ok: 12548245 4676.609424352646 
/tmp/ngff_cache_20240401/1/Image:12552053/stdout:ok: 12552053 4447.661913156509 
/tmp/ngff_cache_20240401/1/Image:12554709/stdout:ok: 12554709 3401.6648321151733 

These times (avg 4249 if we omit the 11.96 where the memo already exists) look comparable to times at IDR/idr-metadata#686 (comment)
which were 5772.45, 2015.81, 5618.20, 3435.90, 3453.46 (avg 4059).

Still running....

Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably the biggest unclarity on my side is to understand how the reader will behave in the different backend scenarios. Based on my reading of the code I am expecting the following

1- OME-Zarr on the local filesystem: JZarrServiceImpl.s3fs will be null and all underlying calls will use ZarrArray.open(String) or ZarrArray.open(String)
2- OME-Zarr on public AWS S3 bucket: JZarrServiceImpl.s3fs will initialize a S3FileSystemStore using a client with anonymous credentials and all underlying calls will use ZarrArray.open(Store) and ZarrGroup.open(Store)
3- OME-Zarr on S3 compatible object store backend (like EBI Embassy 4): as per the endpoint protocol check, JZarrServiceImpl.s3fs should be null and we are falling back to scenario 1. How does the opening logic work in the context of the IDR testing in that case? Are we effectively making direct HTTPS calls using Jzarr?

A separate question is related to the new openPlane API. Is that something that has been exposed / tested anywhere?

src/loci/formats/S3FileSystemStore.java Outdated Show resolved Hide resolved
src/loci/formats/S3FileSystemStore.java Outdated Show resolved Hide resolved
src/loci/formats/S3FileSystemStore.java Show resolved Hide resolved
client = AmazonS3ClientBuilder.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endpoint, "auto"))
.withPathStyleAccessEnabled(true)
.withCredentials(new AWSStaticCredentialsProvider(new AnonymousAWSCredentials())).build();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably something worth documenting in the readme limitations i.e. the S3 functionality will exclusively work unauthenticated

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is certainly a very limited solution that is tailored for the IDR use case, i would not particularly recommend this workflow to standard users, so the limitations would need to be clearly documented.

zarrArray = ZarrArray.open(file);
}
else {
LOGGER.warn("S3 access currently not supported");
s3fs.updateRoot(getZarrRoot(s3fs.getRoot()) + stripZarrRoot(file));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The repetition of this call for all commands makes me wonder whether it should be part of the constructor?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The update root is really setting the location of the particular zarr array to be opened in each case, so unfortunately it is needed in multiple locations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The array opening in lines 97-103 / 125-131, and group opening in lines 113-119 / 137-143 / 149-155 do appear to be duplicate code though. It might be worth thinking about refactoring those into methods that can just open a ZarrArray and ZarrGroup.

src/loci/formats/in/ZarrReader.java Outdated Show resolved Hide resolved
@will-moore
Copy link
Member

I tried testing whether the goofys mount on idr-testing was still being used by anything, or if this PR completely replaces access via the filesystem...

Testing only on omeroreadwrite:

$ ssh -A idr-testing.openmicroscopy.org -L 1080:omeroreadwrite:80
$ sudo umount /bia-integrator-data

Then browsed all NGFF data at http://localhost:1080/webclient/
All studies were browsable without errors except most of the idr0010 plates. Investigation revealed that the .bfoptions was missing the alt_store:

[wmoore@test120-omeroreadwrite ~]$ cat /data/OMERO/ManagedRepository/demo_2/2016-10/14/05-44-15.120_mkngff/8008bd41-f7f5-4e03-9129-fcf8218bc46e.zarr.bfoptions 
omezarr.list_pixels=false
omezarr.quick_read=true

So, this confirms that the goofys mount is NOT needed for viewing images with this PR

@melissalinkert
Copy link
Member

Before I start reviewing in depth, what's the best way to test this outside of IDR? Is it sufficient to just add the reader jar to the CLASSPATH and try showinf on a few local/s3 zarrs?

@dgault
Copy link
Member Author

dgault commented May 2, 2024

Yeah that should be sufficient to test, or using the 'with dependencies' jar from the build might be easiest.

@melissalinkert
Copy link
Member

I think I'm maybe misunderstanding how to evaluate these changes then. With OMEZarrReader-0.4.1-jar-with-dependencies.jar added to the CLASSPATH and a local dataset nucleus3.zarr (generated from test&sizeX=10000&sizeY=10000&sizeC=3.fake with bioformats2raw 0.9.2):

$ showinf -no-upgrade nucleus3.zarr
Exception in thread "main" java.io.FileNotFoundException: nucleus3.zarr (Is a directory)
	at java.base/java.io.RandomAccessFile.open0(Native Method)
	at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
	at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:130)
	at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:151)
	at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:165)
	at loci.common.Location.getHandle(Location.java:522)
	at loci.common.Location.getHandle(Location.java:462)
	at loci.common.Location.getHandle(Location.java:443)
	at loci.common.Location.getHandle(Location.java:426)
	at loci.common.Location.checkValidId(Location.java:551)
	at loci.formats.ImageReader.getReader(ImageReader.java:184)
	at loci.formats.ImageReader.getFormat(ImageReader.java:164)
	at loci.formats.tools.ImageInfo.configureReaderPreInit(ImageInfo.java:440)
	at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1038)
	at loci.formats.tools.ImageInfo.main(ImageInfo.java:1129)

which suggests that showinf is not a suitable way to test (I tried picking specific files as well, and just got UnknownFormatException instead).

Using some simple test code:

import loci.formats.MetadataTools;
import loci.formats.meta.IMetadata;
import loci.formats.in.ZarrReader;

public class ZarrReaderTest {
  public static void main(String[] args) throws Exception {

    ZarrReader reader = new ZarrReader();
    try {
      IMetadata metadata = MetadataTools.createOMEXMLMetadata();
      reader.setMetadataStore(metadata);
      reader.setFlattenedResolutions(false);
      reader.setId(args[0]);
      System.out.println("# series = " + reader.getSeriesCount());
      for (int s=0; s<reader.getSeriesCount(); s++) {
        reader.setSeries(s);
        System.out.println("series #" + s + ":");
        System.out.println("  # resolutions = " + reader.getResolutionCount());
        for (int r=0; r<reader.getResolutionCount(); r++) {
          reader.setResolution(r);
          System.out.println("  resolution #" + r + ":");
          System.out.println("    X = " + reader.getSizeX());
          System.out.println("    Y = " + reader.getSizeY());
          System.out.println("    image count = " + reader.getImageCount());
          for (int i=0; i<reader.getImageCount(); i++) {
            long t0 = System.currentTimeMillis();
            reader.openBytes(i, 0, 0, 1, 1);
            long t1 = System.currentTimeMillis();
            System.out.println("    reading one pixel from plane #" + i + " took " + (t1-t0) + " ms");
          }
        }
      }
    }
    finally {
      reader.close();
    }
  }
}

then running java -mx2g ZarrReaderTest nucleus3.zarr on the same dataset prints the dimensions I'd expect.

Running the same test code on a s3 dataset referenced above:

$ java -mx2g ZarrReaderTest s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:25:35.447 [main] DEBUG loci.common.services.ServiceFactory -- Loaded properties from: services.properties
13:25:35.463 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface ome.codecs.services.JAIIIOService and implementation class ome.codecs.services.JAIIIOServiceImpl
13:25:35.467 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JPEGXRService and implementation class loci.formats.services.JPEGXRServiceImpl
13:25:35.468 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.MDBService and implementation class loci.formats.services.MDBServiceImpl
13:25:35.469 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.MetakitService and implementation class loci.formats.services.MetakitServiceImpl
13:25:35.469 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.EXIFService and implementation class loci.formats.services.EXIFServiceImpl
13:25:35.470 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JHDFService and implementation class loci.formats.services.JHDFServiceImpl
13:25:35.471 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.NetCDFService and implementation class loci.formats.services.NetCDFServiceImpl
13:25:35.472 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JAIIIOService and implementation class loci.formats.services.JAIIIOServiceImpl
13:25:35.481 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.OMEXMLService and implementation class loci.formats.services.OMEXMLServiceImpl
13:25:35.483 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.POIService and implementation class loci.formats.services.POIServiceImpl
13:25:35.483 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.common.services.S3ClientService and implementation class loci.common.services.S3ClientServiceImpl
13:25:35.495 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JPEGTurboService and implementation class loci.formats.services.JPEGTurboServiceImpl
13:25:35.772 [main] DEBUG loci.formats.FormatHandler -- ZarrReader initializing s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:25:35.772 [main] DEBUG loci.formats.FormatHandler -- loci.formats.in.ZarrReader.initFile(s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr)
13:25:52.754 [main] DEBUG loci.common.S3Handle -- Object not found: [server:https://uk1s3.embassy.ebi.ac.uk port:0 bucket:bia-integrator-data path:S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions found:false] {}
loci.common.services.S3ClientServiceException: error occurred
ErrorResponse(code=NoSuchKey, message=Object does not exist, bucketName=bia-integrator-data, objectName=S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, resource=/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, requestId=tx000000000000000f46227-006633daae-23b77beb-default, hostId=null)
request={method=HEAD, url=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, headers=Host: uk1s3.embassy.ebi.ac.uk
User-Agent: Minio (amd64; amd64) minio-java/dev Bio-Formats/dev
x-amz-date: 20240502T182536Z
}
response={code=404, headers=Content-Length: 234
x-amz-request-id: tx000000000000000f46227-006633daae-23b77beb-default
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 02 May 2024 18:25:50 GMT
Connection: Keep-Alive
}

	at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:100)
	at loci.common.S3Handle.initialize(S3Handle.java:275)
	at loci.common.S3Handle.<init>(S3Handle.java:228)
	at loci.common.S3Handle.<init>(S3Handle.java:138)
	at loci.common.Location.getHandle(Location.java:501)
	at loci.common.Location.getHandle(Location.java:462)
	at loci.common.Location.getHandle(Location.java:443)
	at loci.common.Location.getHandle(Location.java:426)
	at loci.common.Location$URLLocationProperties.<init>(Location.java:133)
	at loci.common.Location.exists(Location.java:768)
	at loci.formats.in.DynamicMetadataOptions.loadOptions(DynamicMetadataOptions.java:493)
	at loci.formats.FormatReader.initFile(FormatReader.java:256)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:184)
	at loci.formats.FormatReader.setId(FormatReader.java:1480)
	at ZarrReaderTest.main(ZarrReaderTest.java:13)
Caused by: io.minio.errors.ErrorResponseException: Object does not exist
	at io.minio.MinioClient.execute(MinioClient.java:1134)
	at io.minio.MinioClient.executeHead(MinioClient.java:1240)
	at io.minio.MinioClient.statObject(MinioClient.java:1384)
	at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:92)
	... 14 common frames omitted
13:25:52.763 [main] DEBUG loci.formats.FormatHandler -- ZarrReader attempting to initialize file: s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:25:53.501 [main] DEBUG loci.common.S3Handle -- Object not found: [server:https://uk1s3.embassy.ebi.ac.uk port:0 bucket:bia-integrator-data path:S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions found:false] {}
loci.common.services.S3ClientServiceException: error occurred
ErrorResponse(code=NoSuchKey, message=Object does not exist, bucketName=bia-integrator-data, objectName=S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, resource=/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, requestId=tx0000000000000010599b6-006633daaf-23b74c3c-default, hostId=null)
request={method=HEAD, url=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, headers=Host: uk1s3.embassy.ebi.ac.uk
User-Agent: Minio (amd64; amd64) minio-java/dev Bio-Formats/dev
x-amz-date: 20240502T182552Z
}
response={code=404, headers=Content-Length: 234
x-amz-request-id: tx0000000000000010599b6-006633daaf-23b74c3c-default
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 02 May 2024 18:25:51 GMT
Connection: Keep-Alive
}

	at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:100)
	at loci.common.S3Handle.initialize(S3Handle.java:275)
	at loci.common.S3Handle.<init>(S3Handle.java:228)
	at loci.common.S3Handle.<init>(S3Handle.java:138)
	at loci.common.Location.getHandle(Location.java:501)
	at loci.common.Location.getHandle(Location.java:462)
	at loci.common.Location.getHandle(Location.java:443)
	at loci.common.Location.getHandle(Location.java:426)
	at loci.common.Location$URLLocationProperties.<init>(Location.java:133)
	at loci.common.Location.exists(Location.java:768)
	at loci.formats.in.DynamicMetadataOptions.loadOptions(DynamicMetadataOptions.java:493)
	at loci.formats.in.ZarrReader.reloadOptionsFile(ZarrReader.java:1238)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:195)
	at loci.formats.FormatReader.setId(FormatReader.java:1480)
	at ZarrReaderTest.main(ZarrReaderTest.java:13)
Caused by: io.minio.errors.ErrorResponseException: Object does not exist
	at io.minio.MinioClient.execute(MinioClient.java:1134)
	at io.minio.MinioClient.executeHead(MinioClient.java:1240)
	at io.minio.MinioClient.statObject(MinioClient.java:1384)
	at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:92)
	... 14 common frames omitted
Exception in thread "main" java.io.IOException: Path 's3:/uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr' is not a valid path or not a directory.
	at com.bc.zarr.ZarrUtils.ensureDirectory(ZarrUtils.java:158)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:95)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:88)
	at loci.formats.services.JZarrServiceImpl.getGroupAttr(JZarrServiceImpl.java:114)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:203)
	at loci.formats.FormatReader.setId(FormatReader.java:1480)
	at ZarrReaderTest.main(ZarrReaderTest.java:13)

If I then update the code to set the omezarr.alt_store reader option as in #82 (comment):

import loci.formats.MetadataTools;
import loci.formats.meta.IMetadata;
import loci.formats.in.DynamicMetadataOptions;
import loci.formats.in.ZarrReader;

public class ZarrReaderTest {
  public static void main(String[] args) throws Exception {

    ZarrReader reader = new ZarrReader();
    DynamicMetadataOptions options = (DynamicMetadataOptions) reader.getMetadataOptions();
    options.set("omezarr.alt_store", "https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr");
    reader.setMetadataOptions(options);
    try {
      IMetadata metadata = MetadataTools.createOMEXMLMetadata();
      reader.setMetadataStore(metadata);
      reader.setFlattenedResolutions(false);
      reader.setId(args[0]);
      System.out.println("# series = " + reader.getSeriesCount());
      for (int s=0; s<reader.getSeriesCount(); s++) {
        reader.setSeries(s);
        System.out.println("series #" + s + ":");
        System.out.println("  # resolutions = " + reader.getResolutionCount());
        for (int r=0; r<reader.getResolutionCount(); r++) {
          reader.setResolution(r);
          System.out.println("  resolution #" + r + ":");
          System.out.println("    X = " + reader.getSizeX());
          System.out.println("    Y = " + reader.getSizeY());
          System.out.println("    image count = " + reader.getImageCount());
          for (int i=0; i<reader.getImageCount(); i++) {
            long t0 = System.currentTimeMillis();
            reader.openBytes(i, 0, 0, 1, 1);
            long t1 = System.currentTimeMillis();
            System.out.println("    reading one pixel from plane #" + i + " took " + (t1-t0) + " ms");
          }
        }
      }
    }
    finally {
      reader.close();
    }
  }
}

and retry:

$ java -mx2g ZarrReaderTest s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:28:17.511 [main] DEBUG loci.common.services.ServiceFactory -- Loaded properties from: services.properties
13:28:17.539 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface ome.codecs.services.JAIIIOService and implementation class ome.codecs.services.JAIIIOServiceImpl
13:28:17.539 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JPEGXRService and implementation class loci.formats.services.JPEGXRServiceImpl
13:28:17.540 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.MDBService and implementation class loci.formats.services.MDBServiceImpl
13:28:17.541 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.MetakitService and implementation class loci.formats.services.MetakitServiceImpl
13:28:17.541 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.EXIFService and implementation class loci.formats.services.EXIFServiceImpl
13:28:17.542 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JHDFService and implementation class loci.formats.services.JHDFServiceImpl
13:28:17.543 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.NetCDFService and implementation class loci.formats.services.NetCDFServiceImpl
13:28:17.544 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JAIIIOService and implementation class loci.formats.services.JAIIIOServiceImpl
13:28:17.546 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.OMEXMLService and implementation class loci.formats.services.OMEXMLServiceImpl
13:28:17.548 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.POIService and implementation class loci.formats.services.POIServiceImpl
13:28:17.548 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.common.services.S3ClientService and implementation class loci.common.services.S3ClientServiceImpl
13:28:17.556 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JPEGTurboService and implementation class loci.formats.services.JPEGTurboServiceImpl
13:28:17.811 [main] DEBUG loci.formats.FormatHandler -- ZarrReader initializing s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:28:17.811 [main] DEBUG loci.formats.FormatHandler -- loci.formats.in.ZarrReader.initFile(s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr)
13:28:35.136 [main] DEBUG loci.common.S3Handle -- Object not found: [server:https://uk1s3.embassy.ebi.ac.uk port:0 bucket:bia-integrator-data path:S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions found:false] {}
loci.common.services.S3ClientServiceException: error occurred
ErrorResponse(code=NoSuchKey, message=Object does not exist, bucketName=bia-integrator-data, objectName=S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, resource=/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, requestId=tx000000000000000f46246-006633db50-23b77beb-default, hostId=null)
request={method=HEAD, url=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, headers=Host: uk1s3.embassy.ebi.ac.uk
User-Agent: Minio (amd64; amd64) minio-java/dev Bio-Formats/dev
x-amz-date: 20240502T182818Z
}
response={code=404, headers=Content-Length: 234
x-amz-request-id: tx000000000000000f46246-006633db50-23b77beb-default
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 02 May 2024 18:28:32 GMT
Connection: Keep-Alive
}

	at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:100)
	at loci.common.S3Handle.initialize(S3Handle.java:275)
	at loci.common.S3Handle.<init>(S3Handle.java:228)
	at loci.common.S3Handle.<init>(S3Handle.java:138)
	at loci.common.Location.getHandle(Location.java:501)
	at loci.common.Location.getHandle(Location.java:462)
	at loci.common.Location.getHandle(Location.java:443)
	at loci.common.Location.getHandle(Location.java:426)
	at loci.common.Location$URLLocationProperties.<init>(Location.java:133)
	at loci.common.Location.exists(Location.java:768)
	at loci.formats.in.DynamicMetadataOptions.loadOptions(DynamicMetadataOptions.java:493)
	at loci.formats.FormatReader.initFile(FormatReader.java:256)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:184)
	at loci.formats.FormatReader.setId(FormatReader.java:1480)
	at ZarrReaderTest.main(ZarrReaderTest.java:17)
Caused by: io.minio.errors.ErrorResponseException: Object does not exist
	at io.minio.MinioClient.execute(MinioClient.java:1134)
	at io.minio.MinioClient.executeHead(MinioClient.java:1240)
	at io.minio.MinioClient.statObject(MinioClient.java:1384)
	at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:92)
	... 14 common frames omitted
13:28:35.144 [main] DEBUG loci.formats.FormatHandler -- ZarrReader attempting to initialize file: s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:28:35.209 [main] WARN loci.formats.services.JZarrServiceImpl -- Zarr Reader is not using S3FileSystemStore as this is currently for use with S3 configured with a https endpoint
13:28:35.805 [main] DEBUG loci.common.S3Handle -- Object not found: [server:https://uk1s3.embassy.ebi.ac.uk port:0 bucket:bia-integrator-data path:S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions found:false] {}
loci.common.services.S3ClientServiceException: error occurred
ErrorResponse(code=NoSuchKey, message=Object does not exist, bucketName=bia-integrator-data, objectName=S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, resource=/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, requestId=tx0000000000000010599cf-006633db51-23b74c3c-default, hostId=null)
request={method=HEAD, url=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, headers=Host: uk1s3.embassy.ebi.ac.uk
User-Agent: Minio (amd64; amd64) minio-java/dev Bio-Formats/dev
x-amz-date: 20240502T182835Z
}
response={code=404, headers=Content-Length: 234
x-amz-request-id: tx0000000000000010599cf-006633db51-23b74c3c-default
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 02 May 2024 18:28:33 GMT
Connection: Keep-Alive
}

	at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:100)
	at loci.common.S3Handle.initialize(S3Handle.java:275)
	at loci.common.S3Handle.<init>(S3Handle.java:228)
	at loci.common.S3Handle.<init>(S3Handle.java:138)
	at loci.common.Location.getHandle(Location.java:501)
	at loci.common.Location.getHandle(Location.java:462)
	at loci.common.Location.getHandle(Location.java:443)
	at loci.common.Location.getHandle(Location.java:426)
	at loci.common.Location$URLLocationProperties.<init>(Location.java:133)
	at loci.common.Location.exists(Location.java:768)
	at loci.formats.in.DynamicMetadataOptions.loadOptions(DynamicMetadataOptions.java:493)
	at loci.formats.in.ZarrReader.reloadOptionsFile(ZarrReader.java:1238)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:195)
	at loci.formats.FormatReader.setId(FormatReader.java:1480)
	at ZarrReaderTest.main(ZarrReaderTest.java:17)
Caused by: io.minio.errors.ErrorResponseException: Object does not exist
	at io.minio.MinioClient.execute(MinioClient.java:1134)
	at io.minio.MinioClient.executeHead(MinioClient.java:1240)
	at io.minio.MinioClient.statObject(MinioClient.java:1384)
	at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:92)
	... 14 common frames omitted
Exception in thread "main" java.io.IOException: Path 's3:/uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr' is not a valid path or not a directory.
	at com.bc.zarr.ZarrUtils.ensureDirectory(ZarrUtils.java:158)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:95)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:88)
	at loci.formats.services.JZarrServiceImpl.getGroupAttr(JZarrServiceImpl.java:114)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:203)
	at loci.formats.FormatReader.setId(FormatReader.java:1480)
	at ZarrReaderTest.main(ZarrReaderTest.java:17)

That looks like a bfoptions file must exist in s3 no matter what? Or am I misunderstanding how to initialize the reader?

@dgault
Copy link
Member Author

dgault commented May 3, 2024

The showinf UnknownFormatException seems like the classpath has likely not been picked up correctly. I normally test bftools by modifying bf.sh to detect the jar but I would have expected the classpath to work.

For this PR (and general usage), I would expect the ID used for setID to be a location on a local filesyetm. If setting through ImageReader rather than ZarrReader then it must be one of the files in the .zarr folder rather than the directory itself.

The real goal of the PR was to find a way to provide access to open s3 data from embassy without having to reimport. It is definitely a specific solution for this unique use case rather than a general purpose solution. So for now the s3 access is very specific for use with the new alt store option with a public https endpoint such as:
https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr

If we want to add support for more general s3 handling then one option would be using then using an existing s3 filesystem with jzarr such as https://jzarr.readthedocs.io/en/latest/amazonS3.html. Though I would suggest that would be a follow up.

@sbesson
Copy link
Member

sbesson commented May 10, 2024

Tried to construct a minimal test scenario by downloading a public OME-Zarr IDR dataset 6001240.zarr locally i.e.

aws s3 --endpoint-url=https://uk1s3.embassy.ebi.ac.uk cp --recursive s3://idr/zarr/v0.4/idr0062A/6001240.zarr/ 6001240.zarr/ --no-sign-request

With this PR included, running the following command

% BF_CP=./target/OMEZarrReader-0.4.1-jar-with-dependencies.jar ~/Downloads/bftools/showinf ~/Downloads/6001240.zarr/.zattrs -autoscale

works as expected, reading the metadata and all planes from the local OME-Zarr dataset

Screenshot 2024-05-10 at 10 32 41

Constructed a secondary OME-Zarr where all the chunks (named 0 and 1) were deleted manually so that the dataset only includes .zarray and .zgroup files. The following

% BF_CP=./target/OMEZarrReader-0.4.1-jar-with-dependencies.jar ~/Downloads/bftools/showinf ~/Downloads/6001240_nochunks.zarr/.zattrs -autoscale

also works as I expect, reading the metadata but filling the missing chunks with zeros when navigating through the planes.

Trying to combine this metadata-only OME-Zarr with the alternative store option as introduced in this PR via the command-line -option

% BF_CP=./target/OMEZarrReader-0.4.1-jar-with-dependencies.jar ~/Downloads/bftools/showinf ~/Downloads/6001240_nochunks.zarr/.zattrs -autoscale -option omezarr.alt_store https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr

still gives me black planes and the initialization includes the following warning

Zarr Reader is not using S3FileSystemStore as this is currently for use with S3 configured with a https endpoint

Is that the expectation?

@dgault
Copy link
Member Author

dgault commented May 10, 2024

Sorry @sbesson, the test you did is spot on but the result was not as expected. That was a mistake on my part in the last commit, it should be corrected now. Rerunning your last test should display the data as before except being read from embassy.

@sbesson
Copy link
Member

sbesson commented May 10, 2024

With the last commit

% BF_CP=./target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar ~/Downloads/bftools/showinf ~/Downloads/6001240_nochunks.zarr/.zattrs -autoscale

still returns black planes as per the missing chunks but

% BF_CP=./target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar ~/Downloads/bftools/showinf ~/Downloads/6001240_nochunks.zarr/.zattrs -autoscale -option omezarr.alt_store https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr

now loads the planes from the Embassy object store as expected

Screenshot 2024-05-10 at 17 27 13

Copy link
Member

@melissalinkert melissalinkert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test outlined in #82 (comment) does appear to work for me as well. If it's possible to capture that in an automated test, that might be worth doing here or in a follow-up PR.

zarrArray = ZarrArray.open(file);
}
else {
LOGGER.warn("S3 access currently not supported");
s3fs.updateRoot(getZarrRoot(s3fs.getRoot()) + stripZarrRoot(file));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The array opening in lines 97-103 / 125-131, and group opening in lines 113-119 / 137-143 / 149-155 do appear to be duplicate code though. It might be worth thinking about refactoring those into methods that can just open a ZarrArray and ZarrGroup.

src/loci/formats/in/ZarrReader.java Outdated Show resolved Hide resolved
src/loci/formats/S3FileSystemStore.java Outdated Show resolved Hide resolved
src/loci/formats/S3FileSystemStore.java Outdated Show resolved Hide resolved
src/loci/formats/S3FileSystemStore.java Show resolved Hide resolved
return keys;
}

public ArrayList<String> getFiles() throws IOException {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method appears to be nearly identical to getKeysFor(String) above, and might be worth refactoring.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems the getFiles method is no longer being used anywhere so I have now removed it

Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functionally retested as described in #82 (comment) i.e. local OME-Zarr with chunks, local OME-Zarr without chunks and local OME-Zarr without chunks and with options file pointing to the S3 endpoint. Everything worked as expected.

The latest cleanup commits are useful to identify the changes introduced by this PR i.e. new S3 file system store, new reader option and its implementation in ZarrReader and underlying changes to the JZarrServiceImpl to read the alternate storage location. No objections to getting this merged in its current form. Eventually the ongoing IDR upgrade work will be the validation of this effort in a large scale deployments. Usability/performance problems arising as part of the testing can be captured and tackled as follow-up issues.

@dgault dgault merged commit 802b89c into ome:main Jun 5, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants