Release for Scala Native 0.5.x

[hejfelix]

Now cats has been updated, I'm wondering if there are still blockers left?

[keynmol]

The discussion about 0.5 is covered in governance discussions

The latest update was published in dev channel on discord:

---

@djspiewak:
Arman and I discussed at length. A few takeaways:

We're going to bury the hatchet on IOLocal propagation and PollingSystem. The latter will land mostly as-is, the former will get a check: Poller => Boolean function to fast-path cancelation and otherwise land as-is.
There's a lot of "nice to have" stuff in the 3.6 milestone which we keep kicking down the road due to lack of time and attention… and we're going to keep kicking that can down the road. This includes stuff like retry, metrics, fiber dumps in test frameworks, and other stuff. To be clear, this is all really nice to have stuff and we would love to have it, but it needs more attention than we can give right now.+ Before we do that, we're going to try to take a quick pass over things to see if anything is ready-ish to land
unsafe.Queue will land in 3.6
We're going to release 3.6 against Scala Native 0.4. This is a big one but ultimately the decision came down to the fact that there's a ton of Native-shifting stuff in this release and we're just piling on more and more changes if we decide to take on 0.5. Cats can move ahead without us, but we really need to get 3.6 out the door first.
The first Scala Native 0.5 CE release will be 3.7, and it's expected that SN 0.5 support will be the main focus of that release (thus hopefully reducing the timeline considerably). We'll start out trying to do multithreading and fall back from that position if it proves insane.
Idk if there were other things we decided? I think that was it

---

TLDR - CE will have to wait for a bit longer, and when it comes, it will likely be multi-threaded

[kubukoz]

What's the latest on this? Is the plan still to release 3.6 with polling system changes and 3.7 with SN0.5 + (hopefully) multithreading?

Personally, I'd rather get SN0.5 without multithreading than wait months more :)

edit: I see this was confirmed on discord 4 days ago

[armanbilge]

Is the plan still to release 3.6 with polling system changes and 3.7 with SN0.5 + (hopefully) multithreading

Yes. The polling system work already requires non-trivial changes downstream. I don't want to pile the SN 0.5 changes onto that. I'm sorry that we didn't get the polling system work out (years) earlier, it's been mostly ready.

Personally, I'd rather get SN0.5 without multithreading than wait months more :)

We plan to make some breaking changes to support multi-threading. We really have only one chance to do that, when we bump to SN 0.5.

[kubukoz]

Fair enough. Anything I can do to help speed things up?

[armanbilge]

Nevermind Cats Effect, Cats itself is still limping on SN 0.5, and it doesn't even do any multi-threading. Figuring out whatever is wrong there and fixing it will almost certainly help with migrating the rest of the ecosystem. We're bound to hit that bug again somewhere else in the stack.

• Sporadic test errors since upgrading to Native 0.5  cats#4647

[djspiewak]

Fair enough. Anything I can do to help speed things up?

Another thing that you could do is poke at #4201 and try to advance the ball a bit. We're sort of ad hoc working on branches right now. The main blocker to this is 3.6 though, which is pending some work on polling system metrics (#3686).

[djspiewak]

Oh, another way to help would be to get "replace Specs2 with MUnit" work over the line (started in #3787). I like Specs2 a lot, but it's clearly not the mainstream answer anymore, and particularly with stuff like this I would rather bound our set of variables. Note that this can (and should) be done entirely independently of any SN work.

[danicheg]

FWIW, I've recently run into not-so-pleasant java.lang.NoSuchMethodError: 'java.lang.Object cats.syntax.ApplyOps$.$times$greater$extension(java.lang.Object, java.lang.Object, cats.Apply)' because dependency on Cats-Core wasn't explicitly defined in a project containing both Cats-Effect (depends on Cats-Core 2.11.0) and Redis4Cats (depends on Cats-Core 2.12.0). Perhaps we should coordinate updates to Cats-Core across the ecosystem more smoothly. 🤷🏻

UPD. That issue was investigated, and redis4cats was patched profunktor/redis4cats#955. All credit to @kubukoz. Ultimately, this doesn't appear to be a Cats-Effect issue itself.

[djspiewak]

FWIW, I've recently run into not-so-pleasant java.lang.NoSuchMethodError: 'java.lang.Object cats.syntax.ApplyOps$.$times$greater$extension(java.lang.Object, java.lang.Object, cats.Apply)'

Wait, did cats-core break bincompat? Or did it get resolved downward (to 2.11.0)? The solution to the problem you're describing should be to always resolve forward to the latest version (i.e. your project should depend on cats-core 2.12.0). If that doesn't work then it suggests that we broke compatibility and that's very much a Cats problem that needs to be fixed.

[danicheg]

@djspiewak I'm getting the cats-core 2.9.0, as per https://github.com/typelevel/cats-effect/blob/v3.5.7/build.sbt#L300

[info]   +-org.typelevel:cats-effect_2.13:3.5.7 [S]
[info]     +-org.typelevel:cats-effect-kernel_2.13:3.5.7 [S]
[info]     | +-org.typelevel:cats-core_2.13:2.9.0 [S]
[info]     |   +-org.typelevel:cats-kernel_2.13:2.9.0 [S]
[info]     |   
[info]     +-org.typelevel:cats-effect-std_2.13:3.5.7 [S]
[info]       +-org.typelevel:cats-effect-kernel_2.13:3.5.7 [S]
[info]         +-org.typelevel:cats-core_2.13:2.9.0 [S]
[info]           +-org.typelevel:cats-kernel_2.13:2.9.0 [S]

[djspiewak]

I genuinely don't understand why cats-core is being evicted backwards like that. Question for @eed3si9n maybe. The solution is pretty simple: explicitly specify the cats-core dependency in your own build.sbt (I recommend just depending on the absolute latest).

In general, the Typelevel ecosystem guarantees:

• Backward binary compatibility across all minor versions within a major lineage (e.g. Cats 2.x.y)
• Forward binary compatibility across all patch versions within a minor lineage (e.g. Cats 2.9.x)
• Forward and backward source compatibility across all patch versions within a minor lineage

Backward compatibility means that code compiled with a version which is older than the one on the classpath will link and run with proper semantics (except for a few weird and unfortunate exceptions involving bugs) on a more recent version. Forward compatibility means that code compiled with a version which is newer than the one on the classpath will link and run with proper semantics (again, except for bugs mostly) on an older version. So backward means code can be compiled against Cats 2.9.0 and run with Cats 2.12.0, while forward means that code compiled against Cats 2.9.3 can be run with Cats 2.9.1.

In your case, Coursier is making the assumption that Cats Core supports forward compatibility across minor versions (it doesn't) and thus it is safe to evict to the older version, despite the fact that Redis4Cats was compiled against the newer version. This just doesn't work, but the inverse would have worked fine.

So I think it's less a matter of ecosystem coordination and more a matter of somehow making sure Coursier doesn't do weird things. And, in the event that it does do weird things (as it did here), you can always override it to fix the problem. The guarantees sum up such that it is effectively always safe to depend on the absolute latest version of everything unless there's something in your own code which causes incompatibility.

[danicheg]

Hmmm, it's likely not Coursier, as disabling it changed nothing (useCoursier := false)

[info] 	[SUCCESSFUL ] org.typelevel#cats-effect-std_2.13;3.5.7!cats-effect-std_2.13.jar (5342ms)
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.15/scala-compiler-2.13.15.jar ...
[info] 	[SUCCESSFUL ] org.typelevel#cats-kernel_2.13;2.9.0!cats-kernel_2.13.jar (34119ms)
[info] 	[SUCCESSFUL ] org.jline#jline;3.26.3!jline.jar (11289ms)
[info] 	[SUCCESSFUL ] org.typelevel#cats-core_2.13;2.9.0!cats-core_2.13.jar (54476ms)

I was also mistaken in assuming that Redis4Cats explicitly defines a dependency on Cats-Core — it does not. This has left me a bit confused. However, adding an explicit dependency on Cats-Core 2.11.0 or 2.12.0 resolves the issue.
But! That's a bikeshedding to the original problem. Thank you for taking the time, and I’m sorry about that!

[kubukoz]

I can reproduce that downward resolution with:

cs resolve org.typelevel::cats-effect:3.5.7 dev.profunktor::redis4cats-effects:1.7.2

dev.profunktor:redis4cats-core_3:1.7.2:default
dev.profunktor:redis4cats-effects_3:1.7.2:default
io.lettuce:lettuce-core:6.5.1.RELEASE:default
io.netty:netty-buffer:4.1.115.Final:default
io.netty:netty-codec:4.1.115.Final:default
io.netty:netty-common:4.1.115.Final:default
io.netty:netty-handler:4.1.115.Final:default
io.netty:netty-resolver:4.1.115.Final:default
io.netty:netty-transport:4.1.115.Final:default
io.netty:netty-transport-native-unix-common:4.1.115.Final:default
io.projectreactor:reactor-core:3.6.6:default
org.reactivestreams:reactive-streams:1.0.4:default
org.scala-lang:scala-library:2.13.14:default
org.scala-lang:scala3-library_3:3.3.4:default
org.typelevel:cats-core_3:2.9.0:default
org.typelevel:cats-effect-kernel_3:3.5.7:default
org.typelevel:cats-effect-std_3:3.5.7:default
org.typelevel:cats-effect_3:3.5.7:default
org.typelevel:cats-kernel_3:2.9.0:default
org.typelevel:literally_3:1.2.0:default

Spooky.

Edit: not spooky at all, actually - dev.profunktor::redis4cats-effects:1.7.2 depends on 2.9.0. @danicheg do you have something else with Cats in the build?

[danicheg]

Wow, what luck to catch the Redis4Cats contributor (or maintainer?)! But let’s discuss this in the appropriate place.

[kubukoz]

Wow, what luck to catch the Redis4Cats contributor (or maintainer?)!

good/bad news, I haven't contributed in years 😂

[kubukoz]

the issue reported by @danicheg turned out to have been a problem with redis4cats itself (using an Optional dependency, see discussion) - just in case. @danicheg would you mind editing this comment to say this wasn't a CE/Cats caused problem?

[danicheg]

@kubukoz yeah but, at the end of the story, having such binary compatibility issues on the end-user side is still weird since the major Cats-Core version was consistent across all the dependencies. The original issue can be reproduced even if the Cats-Core v2.9.0 is explicitly set in the dependencies. Also, the Cats-Core v2.10.0 release notes request that libraries be recompiled and republished to use 'new allocation-free syntax' but not to fix binary compatibility issues. So, haven't we overlooked something crucial? On the other hand, my concern about having the Cats-Core up-to-date among the central ecosystem building blocks is still valid.

[armanbilge]

So, haven't we overlooked something crucial?

I don't think so.

Also, the Cats-Core v2.10.0 release notes request that libraries be recompiled and republished to use 'new allocation-free syntax' but not to fix binary compatibility issues.

Yes, I wrote that. It's a recommendation, not a request. The problem is that the allocation was happening inside of library/user code, not inside of Cats. This is why we can't remove the allocation on their behalf, they must recompile.

However, Cats v2.10.0 is still binary-compatible with all previous versions through 1.0, and it still supports the old, allocation-based syntax too, for libraries that didn't recompile.

The original issue can be reproduced even if the Cats-Core v2.9.0 is explicitly set in the dependencies.

I want to help, but let's discuss this anywhere besides right here :) can you open an issue somewhere else and tag me? Thanks!