Reworking LMSMetadata package and adding a new lif reader

[XLEFReaderForBioformats]

Main changes:

• reworking metadata translation (and class architecture) in the LMSMetadata package, improving metadata translation especially for SP8 and STELLARIS images, also reworking metadata translation for other images (SP5, widefield)
• Adding a new LMSLIFReader for reading .lif files that also uses the LMSMetadata package

In general, metadata translation will be an ongoing topic here, and there might also be images from systems that are not yet fully regarded and where metadata translation needs to be improved. I'll try to fix all major bugs that might be noticed in the context of this PR, but concerning minor bugs and "missing metadata" , I guess I have to discuss with my colleagues how much of that I will be able to fix within this PR.

@dgault as we have discussed before, since you suggested that we could use a DelegateReader here to provide both the existing LIFReader and the new LMSLIFReader to users, I think that's a reasonable approach, so this PR at least won't create a deterioration of existing .lif files reading.
I removed my temporal changes to readers.txt and LIFReader.isThisType(...) for testing so now the LIFReader is the standard reader for .lif files again. Could you let me know how using a delegate reader would look like so both lif readers can be used?

Please let me know if you need any additional information or explanations from my side.

[dgault]

This PR was included in the CI last night, unfortunately a number of files threw exceptions at the setID stage so the full test suite didn't actually get to run. I have only had a quick look at the failures but a lot of them look to be NullPointerExceptions. Unfortunately the files that are failing are not ones that are public.

The full console output from the test run is at:
https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-test-folder/lastFailedBuild/console

[XLEFReaderForBioformats]

hi @dgault, okay thanks for sharing, I'll try to fix all the issues where exceptions are thrown.

[XLEFReaderForBioformats]

hi @dgault , I tried to address all the exceptions that happened in the tests, trying to increase the robustness against unexpected input.
However, I do not yet understand the root causes of these errors, which are images with data that the reader does not expect.
I understand that you cannot share these image data publicly, however is it maybe possible to make this image data available to me directly?

[NicoKiaru]

@XLEFReaderForBioformats I'm curious to know whether your request will be approved because I've had exactly the same issue in here:
#4092

This point has also been raised and answered in the image.sc forum:
https://forum.image.sc/t/pb-when-i-open-czi-stitching-image/87585/24

[XLEFReaderForBioformats]

@NicoKiaru thanks for sharing, good to know...

@dgault I have a suggestion... I assume that the problems of the new reader with your test data arise from unexpected XML layout and not so much from the image binary data itself. Would it be an option for you to share only the image XML with me privately?

If you have only LIF files available, you could copy the XML using a hex editor, or alternatively you could e.g. use a LASX Offline to open LIF files and save them as e.g. XLEF/TIF files, and then it would suffice for me if you sent me the XLIF files, which can be found in metadata folders.
You can also obfuscate any image names, file paths and user-comments in the XML.

Having the image XML, I can create some "fake" image datasets with it for local testing.

Would this be an option?

[XLEFReaderForBioformats]

hi @dgault @melissalinkert,
could you possibly support me with the latest build errors? my changes built locally, and for some reason I cannot see the build logs of the failed builds (it only displays "this job failed")...

[dgault]

@XLEFReaderForBioformats, Im not sure why those jobs failed in that way but I re-triggered them again and they should now all be green

[XLEFReaderForBioformats]

@dgault I'd like to ask if there are any updates regarding the test data? It would be helpful for me to know at least if you are planning to share any data with me or not, right now it is a bit unclear to me how to proceed with this PR.

[dgault]

Hi @XLEFReaderForBioformats, the next step will be to reinclude the PR in the nightly CI, testing the new reader rather than the legacy reader. If we can get a full run of the test suite then we will at least be able to identify the number of files which are causing failures and judge the scale of the impact. From there we can decide how best to proceed.

I will re-include the PR now so we can have the new reader in the nightly CI for the purposes of the testing.

[dgault]

Hi @XLEFReaderForBioformats, as a follow up, the PR has been included in our test suite for a number of days and we are seeing failures for a number of files, though most are for the same exception. You can see the LIF failures here:
https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-test-folder/7512/consoleFull

   [testng] java.lang.IndexOutOfBoundsException: Index 751 out of bounds for length 13
   [testng] 	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
   [testng] 	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
   [testng] 	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
   [testng] 	at java.base/java.util.Objects.checkIndex(Objects.java:385)
   [testng] 	at java.base/java.util.ArrayList.get(ArrayList.java:427)
   [testng] 	at loci.formats.in.LMSLIFReader.initFile(LMSLIFReader.java:461)
   [testng] 	at loci.formats.FormatReader.setId(FormatReader.java:1480)
   [testng] 	at loci.formats.in.LIFDelegateReader.setId(LIFDelegateReader.java:37)
   [testng] 	at loci.formats.ImageReader.setId(ImageReader.java:864)

The failures seem to be caused by the following line when retrieving the offsets:
https://github.com/ome/bioformats/pull/4000/files#diff-79e103bcccaa0d842ec61c1bf11e438d3ace6607e364135a7e08b3947f2dec7cR461

The following sample files which failed are available for testing from here. These should be good samples to test for the IndexOutOfBoundsException

• michael/PR2729_frameOrderCombinedScanTypes.lif
• imagesc-30856/20191025 Test FRET 585. 423, 426.lif
• seanwarren/150519_FRAP_test_ROIs_chromagreen/150519_FRAP_test_ROIs_chromagreen.lif

[XLEFReaderForBioformats]

Hi @dgault,

sorry for the long delay on my side! After a longer phase of release support that I had to prioritize, I can now hopefully return to finishing this PR, and I'm really looking forward to it!

I fixed two bugs that led to the exceptions in the test log that you kindly provided.

Since it's been a few months since the last update from my side, I am unsure as to how we best proceed now. Would you simply include the PR in your test suite again?

[XLEFReaderForBioformats]

Hi @melissalinkert ,

I'd like to kindly ask again how we can proceed with this PR, as I fixed all issues that were found in your automated tests.
Is there anything else missing before this PR can be merged?

[melissalinkert]

@XLEFReaderForBioformats : we're currently evaluating what is needed to proceed with reviewing this pull request. Given the size and scope of changes, we expect review and testing to require a substantial amount of time and effort. Due to holidays and other priorities, it will likely be after the new year before we have an update here.

Please note too that we have had some staffing changes in the last few months, details of which are mentioned here.