Prepare for integration in Bio-Formats

[melissalinkert]

Get this to a point where it could be a nearly drop-in replacement for https://github.com/ome/bioformats/tree/develop/components/forks/turbojpeg.

• Update supported architectures
  • Currently Linux/Windows/Mac OS X x86_64 are built
  • Would be nice to include M1/M2 and Linux aarch64
  • Drop 32-bit and PPC support
• Agree on artifact IDs and version number strategy; version numbers in build.gradle do not match https://github.com/glencoesoftware/libjpeg-turbo-java/releases
• Explore upgrading underlying libjpeg-turbo (https://github.com/libjpeg-turbo/libjpeg-turbo/releases/tag/3.0.0 just released)
• Migrate to OME organization?

We had decided against changing package names for the moment given that it requires more than just changes to the Java classes.

[melissalinkert]

I also can't assign or add labels to this issue, so we may wish to review the permissions on this repository as well.

[CGDogan]

@melissalinkert With my Pull request "ARM64 libturbojpeg linux" ome/bioformats#4044 I was actually planning to contribute the Dockerfile with which I generated the library (in the pull request description). Would it help if I provide a Dockerfile that runs on x64 and crosscompiles to arm64 which I can then test?

[sbesson]

A few thoughts on the set of questions to help us move forward

Update supported architectures

This is timely as M1 runners are now starting to be available in beta - see https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/
Note general availability is currently planned for Q1 2024 so to simplify the adoption, it might make sense to define an initial minimal viable product that has the same support matrix as the current fork (possibly only removing the legacy architectures) and add arm64/aarch64 support in follow-up releases.

Agree on artifact IDs and version number strategy; version numbers in build.gradle do not match https://github.com/glencoesoftware/libjpeg-turbo-java/releases

Immediate thought re versioning is that trying to maintain the relationship with the upstream C library version is going to be tricky and will create more confusion that anything else. Thinking of a potential 1.2.1->3.0.0 upgrade as an example, we wouldn't want this to increment the major version of this library by 2. Instead we can use the readme as well as release notes to communicate this similarly to what other libraries are doing e.g. jhdf5.

I think there are two possibilities:

• semantic versioning e.g. trying to aim towards a 1.0.0 release for integration in Bio-Formats
• some calendar versioning similar to the tag above. If we want to preserve compatibility with the x.y.z scheme, this could also be expressed as YYYY.MM.DD
  My slight preference goes towards the first option, that's actually what we used for jxrlib

Explore upgrading underlying libjpeg-turbo (https://github.com/libjpeg-turbo/libjpeg-turbo/releases/tag/3.0.0 just released)

Similar to the conversation above about arm64/aarch64, unless this significantly helps the process, I would stick to using the current library for the initial integration and look into upgrading in a follow-up release

Migrate to OME organization?

I would link this decision to the choice of group ID. Obviouslyorg.libjpegturbo is not what we want to use as the publishing group. Instead, I think the available options are:

• com.glencoesoftware (or com.glencoesoftware.turbojpeg) similarly to bioformats2raw & raw2ometiff
  _ ome similarly to jxrlib
  _ org.openmicroscopy similarly to other Bio-Formats low-level components e.g. ome-codecs
  Note the third option would also open the possibility to deploy the artifact on Maven Central if this was valuable

[CGDogan]

Some notes about doing a major version bump to TurboJpeg 2 or 3:

I compiled TurboJpeg v3 latest and the latest v1.2.X on my ARM system. I used the following code on CMU-3.ndpi.tiff from OpenSlide test repo:

import loci.formats.ImageReader;

public class Test {
     public static void main(String args[]) throws Exception {
        ImageReader r = new ImageReader();
	r.setId(new String("CMU-3.ndpi.tiff"));
	int x = r.getSizeX(); int y = r.getSizeY();
	System.out.println("X,Y"); System.out.println(x); System.out.println(y);

	int fx = /*r.getOptimalTileWidth()*/ 10;
	int fy = /*r.getOptimalTileHeight()*/ 10;
	int s1 = 0; int s2 = 0;
	int max1 = (x/fx)*fx; int max2 = (y/fy)*fy;
	int sum = 0;
	while(s1 < max1) {
		while(s2 < max2) {
			byte[] bytes = r.openBytes(0, s1, s2, fx, fy);
			for (int i = 0; i < bytes.length; i++) sum += bytes[i];
			s2 += fy;
		}
		s1 += fx;
	}
	System.out.println(sum);
    }
}

(1) This took 40 seconds with 1.2.X and 18 second with 3.X.X. On x64, the difference might be less.

(2) Upgrading to v3.X.X is easy, it's simply recompiling the shared libraries and copy-pasting the java folder from libjpeg turbo repository then ant compile && ant jars. No need to update BioFormats JpegTurboImpl code.

(3) The 2010 MSVC patch can be removed from this repository if upgrading to v1.3.1 or higher

(4) v3.X.X or maybe some earlier version switched to CMake. NASM or YASM needs to be installed when building for x86/x64 but not required on ARM.

[sbesson]

With the integration of #5, the first bullet point should be addressed with the ability to built a JAR shipping libjpeg-turbo libraries built against multiple platforms. Next step will be to look into the process for publishing into Maven Central which will force to resolve the question of the component versioning. Once we have a deployed artifact, we can look into using it as a replacement for ome:turbojpeg.

I would propose to split the upgrade to libjpeg-turbo 3.x and the migration to OME as separate issues.