Skip to content

Conversation

@JaroslavTulach
Copy link
Member

@JaroslavTulach JaroslavTulach commented Dec 11, 2025

Pull Request Description

  • Removing snowflake from enso native image executable.
  • Fixing code to deal with dual JVM objects

Important Notes

Execute in mock dual JVM mode via:

sbt:enso> runEngineDistribution 
  --vm.D=polyglot.enso.classLoading=Standard.Snowflake:guest,hosted 
  --env ENSO_SNOWFLAKE_USER=xxx
  --env ENSO_SNOWFLAKE_PASSWORD=yyy
  --env ENSO_SNOWFLAKE_ACCOUNT=zzz
  --env ENSO_SNOWFLAKE_DATABASE=bbb
  --run test/Snowflake_Tests/

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

obrazek

@JaroslavTulach JaroslavTulach self-assigned this Dec 11, 2025
@JaroslavTulach JaroslavTulach added the CI: No changelog needed Do not require a changelog entry for this PR. label Dec 11, 2025
@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Dec 11, 2025

There is currently 40 failures:

[FAILED] [Snowflake] Info: [4/5, 15346ms]                             
  | => enso / runEngineDistribution 191s
    - [FAILED] correctly handles Decimal and Float types [5812ms]
        Reason: [1.234568] did not equal [(Decimal.Value 1.234568)]; first difference at index 0  (at test/Snowflake_Tests/src/Snowflake_Spec.enso:260:13-74).

[FAILED] [Snowflake] Edge Cases: [3/4, 24526ms]                       
  | => enso / runEngineDistribution 220s
    - [FAILED] should be able to round-trip a BigInteger column [2185ms]
        Reason: (1.1805916207174113E21): Expected a value of type Standard.Base.Data.Numbers.Integer but got a value [1.1805916207174113E21] of type Standard.Base.Data.Numbers.Float instead (at test/Snowflake_Tests/src/Snowflake_Spec.enso:371:64-84).

[FAILED] [Snowflake] Table.aggregate should correctly select result types: [2/4, 11500ms]
  | => enso / runEngineDistribution 370s
    - [FAILED] widening to decimals on Average [3637ms]
        Reason: 0.666667 did not equal 0.6666666666666666 (at test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1221:13-70).


    - [FAILED] widening to decimals on Percentile [2629ms]
        Reason: [2.50000] did not equal [2.5]; first difference at index 0  (at test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1244:13-59).


    - [FAILED] should work when stacked one after another [2421ms]
        Reason: [7.000000] did not equal [7]; first difference at index 0  (at test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1480:13-64).


    - [FAILED] round should work correctly on Decimals [70ms]
        Reason: An unexpected panic was thrown: (Unsupported_Argument_Types.Error [] 'Cannot convert \'1.3\'(language: Java, type: com.oracle.truffle.polyglot.PolyglotMap) to Java type \'java.math.BigDecimal\': Unsupported target type.')
        at <enso> case_branch(distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Statement_Setter.enso:38:30-73)
        at <enso> Statement_Setter.type.create_fill_hole.fill_hole_jdbc(/distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Statement_Setter.enso:34-58)

[FAILED] [Snowflake] (Derived_Columns_Spec) Table.set with Simple_Expression: [9/10, 18911ms]
  | => enso / runEngineDistribution 1082s
    - [FAILED] arithmetics [3589ms]
        Reason: [10.000000, 20.000000] did not equal [10, 20]; first difference at index 0  (at enso/test/Table_Tests/src/Common_Table_Operations/Derived_Columns_Spec.enso:53:13-170).

[FAILED] [Snowflake] (Upload_Spec) Uploading an in-memory Table: [10/11, 80795ms]
  | => enso / runEngineDistribution 
    - [FAILED] should include the created table in the tables directory [3206ms]
        Reason: An unexpected panic was thrown: java.lang.IllegalArgumentException: Unsupported StorageType for `is_in`: org.enso.table.data.column.storage.type.TextType@896
        at <Unknown Location> related to com.oracle.truffle.host.HostObject.doInvoke
        at <enso> case_branch.java_result(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Table/0.0.0-dev/src/Internal/In_Memory_Column_Implementation.enso:586:25-102)
        at <enso> Java_Problems.with_map_operation_problem_aggregator.Java_Problems.with_map_operation_problem_aggregator(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Java_Problems.enso:128:9-25)
        at <enso> Java_Problems.with_problem_aggregator(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Java_Problems.enso:118:14-25)
        at <enso> Java_Problems.with_map_operation_problem_aggregator(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Java_Problems.enso:126-128)

need to be addressed.

@github-actions github-actions bot added the -libs-API-change-Base Marks a PR that changes the public API of Standard.Base label Dec 11, 2025
@JaroslavTulach
Copy link
Member Author

@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Dec 11, 2025

14 failures remaining:

- [FAILED] a numeric column (Decimals) supports abs operation [3201ms]
         Reason: [10.500000000000, 0E-12, Nothing, 1.200000000000, 12.340000000000, 123454567890.123456789000, 12345678901890.123456789000] did not equal [(Decimal.Value 10.5), (Decimal.Value 0.0), Nothing, (Decimal.Value 1.2), (Decimal.Value 12.34), (Decimal.Value 123454567890.1234567890), (Decimal.Value 12345678901890.1234567890)]; first difference at index 0  (at /home/runner/work/enso/enso/test/Table_Tests/src/Common_Table_Operations/Numeric_Column_Spec.enso:64:9-98).
Error: An unexpected panic was thrown: org.enso.jvm.interop.impl.OtherJvmException: org.enso.base.enso_cloud.HideableValue is a sealed interface
          at <Unknown Location> related to case_branch
          at <enso> Enso_Secret.as_hideable_value(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Base/2025.3.1-dev/src/Enso_Cloud/Enso_Secret.enso:334-344)
          at <enso> if_then_else(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Internal/JDBC_Connection.enso:332:17-76)
          at <enso> JDBC_Connection.create.properties_as_java_props.case properties_as_java_props(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Internal/JDBC_Connection.enso:331-332)
          at <enso> Array_Like_Helpers.map.Array_Like_Helpers.map(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Base/2025.3.1-dev/src/Internal/Array_Like_Helpers.enso:277:56-77)
          at <enso> Array_Like_Helpers.vector_from_function(Internal)
          at <enso> JDBC_Connection.create(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Internal/JDBC_Connection.enso:336:18-86)
          at <enso> Snowflake_Connection.type.create(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Snowflake/2025.3.1-dev/src/Snowflake_Connection.enso:79:27-96)
          at <enso> Snowflake_Details.connect<arg-1>(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Snowflake/2025.3.1-dev/src/Connection/Snowflake_Details.enso:59:13-106)
          at <enso> Snowflake_Details._enhance_connection_errors(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Snowflake/2025.3.1-dev/src/Connection/Snowflake_Details.enso:102:14-19)
          at <enso> Snowflake_Details.connect(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Snowflake/2025.3.1-dev/src/Connection/Snowflake_Details.enso:58-59)
          at <enso> case_branch(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Connection/Database.enso:35:14-36)
          at <enso> Database.connect(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Connection/Database.enso:33-35)
          at <enso> Auth_Spec.add_specs.Auth_Spec.add_specs.table(/home/runner/work/enso/enso/test/Snowflake_Tests/src/Auth_Spec.enso:77:34-108)
          at <enso> Auth_Spec.with_temp_user<arg-1>(/home/runner/work/enso/enso/test/Snowflake_Tests/src/Auth_Spec.enso:25:10-27)
- [FAILED] division should be aligned with the Enso arithmetic [2965ms]
         Reason: [0.500000, 2.500000, 2.500000, 20.000000] did not equal [0.5, 2.5, 2.5, 20.0]; first difference at index 0  (at /home/runner/work/enso/enso/test/Table_Tests/src/Common_Table_Operations/Column_Operations_Spec.enso:996:13-43).
- [FAILED] should work when stacked one after another
Error: [7.000000] did not equal [7];
 Reason: An unexpected panic was thrown: java.lang.UnsupportedOperationException: executeLargeUpdate not implemented
        at <enso> case_branch<arg-2>(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Connection/Connection.enso:425:71-93)

@JaroslavTulach JaroslavTulach changed the title Removing snowflake from native image Running Standard.Snowflake in dual JVM mode Dec 11, 2025
@JaroslavTulach JaroslavTulach changed the title Running Standard.Snowflake in dual JVM mode Run Standard.Snowflake in dual JVM mode Dec 11, 2025
@JaroslavTulach JaroslavTulach added -libs Libraries: New libraries to be implemented -compiler labels Dec 11, 2025
@JaroslavTulach JaroslavTulach moved this to 🔧 Implementation in Issues Board Dec 11, 2025
@jdunkerley jdunkerley added this to the 2026.1 Release milestone Dec 11, 2025
@JaroslavTulach
Copy link
Member Author

With e59ef9c we are down to 10 failures:

sbt:enso> runEngineDistribution --vm.D=polyglot.enso.classLoading=Standard.Snowflake:guest,hosted --env ENSO_SNOWFLAKE_USER=aaa --env ENSO_SNOWFLAKE_PASSWORD=222 --env ENSO_SNOWFLAKE_ACCOUNT=555 --env ENSO_SNOWFLAKE_DATABASE=dddd --run test/Snowflake_Tests/

1179 tests succeeded.
10 tests failed.
37 tests skipped.
11 groups skipped.

Failed tests: 'correctly.handles.Decimal.and.Float.types|should.be.able.to.round-trip.a.BigInteger.column|widening.to.decimals.on.Average|widening.to.decimals.on.Percentile|should.work.when.stacked.one.after.another|should.work.with.mixed.types|division.should.be.aligned.with.the.Enso.arithmetic|should.allow.round.on.a.\(Decimal.Nothing.Nothing\).column.\(to.>0.decimal.places\)|arithmetics|a.numeric.column.\(Decimals\).supports.abs.operation'


    correctly handles Decimal and Float types
        Reason: [1.234568] did not equal [(Decimal.Value 1.234568)]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Snowflake_Tests/src/Snowflake_Spec.enso:260:13-74).


    should be able to round-trip a BigInteger column [2533ms]
            Reason: (1.1805916207174113E21): Expected a value of type Standard.Base.Data.Numbers.Integer but got a value [1.1805916207174113E21] of type Standard.Base.Data.Numbers.Float instead (at /home/devel/NetBeansProjects/enso/enso/test/Snowflake_Tests/src/Snowflake_Spec.enso:371:64-84).

  - [FAILED] widening to decimals on Average [2572ms]
        Reason: 0.666667 did not equal 0.6666666666666666 (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1221:13-70).


    - [FAILED] widening to decimals on Percentile [2416ms]
        Reason: [2.50000] did not equal [2.5]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1244:13-59).
       
    - [FAILED] should work when stacked one after another [2778ms]
        Reason: [7.000000] did not equal [7]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1480:13-64).
       
   - [FAILED] should work with mixed types [3392ms]
        Reason: {Table: X->Decimal (precision=38, scale=0), Y->Decimal (precision=38, scale=0)} [50.000000, 5.000000] did not equal [50, 5]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Column_Operations_Spec.enso:872:17-56).


    - [FAILED] division should be aligned with the Enso arithmetic [1702ms]
        Reason: [0.500000, 2.500000, 2.500000, 20.000000] did not equal [0.5, 2.5, 2.5, 20.0]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Column_Operations_Spec.enso:996:13-43).


   - [FAILED] should allow round on a (Decimal Nothing Nothing) column (to >0 decimal places) [396ms]
        Reason: [0.5, 0.6, 3.5, 3.6, -0.5, -0.6, -3.5, -3.6] did not equal [0.5, 0.6, 3.5, 3.6, -0.5, -0.6, -3.5, -3.6]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Column_Operations_Spec.enso:1099:17-90).

 - [FAILED] arithmetics [3517ms]
        Reason: [10.000000, 20.000000] did not equal [10, 20]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Derived_Columns_Spec.enso:53:13-170).
        
    - [FAILED] a numeric column (Decimals) supports abs operation [2419ms]
        Reason: [10.500000000000, 0E-12, Nothing, 1.200000000000, 12.340000000000, 123454567890.123456789000, 12345678901890.123456789000] did not equal [(Decimal.Value 10.5), (Decimal.Value 0.0), Nothing, (Decimal.Value 1.2), (Decimal.Value 12.34), (Decimal.Value 123454567890.1234567890), (Decimal.Value 12345678901890.1234567890)]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Numeric_Column_Spec.enso:64:9-98).        

@JaroslavTulach JaroslavTulach moved this from 🔧 Implementation to 👁️ Code review in Issues Board Jan 5, 2026
f = case Environment.get "ENSO_CLOUD_CREDENTIALS_FILE" of
Nothing -> File.home / ".enso" / "credentials"
path -> File.new path
Authentication_Service.log_message level=..Warning "credentials_file is "+f.to_text
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why Warning ? That feels more like Trace or Debug at most.

Copy link
Member Author

@JaroslavTulach JaroslavTulach Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • having access to logs with detailed and related information about a CI test failure is the motivation
  • I needed to see these messages in the output of failed CI run.
  • I don't think there is a way to see Trace or Debug messages for failed tests, right?
  • Thus Trace or Debug wasn't enough and I had to use Warning

_ ->
type_name = (Meta.type_of x).to_text
case type_name of
"java.math.BigDecimal" ->
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this a bug in pattern matcher if something that is "java.math.BigDecimal" doesn't match the previous _: BigDecimal case?
If it is some kind of a special case it deserves some comment at least.

Copy link
Member Author

@JaroslavTulach JaroslavTulach Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • here is the "motivation" 3907ebf
  • I wish we could come up with some better way to deal with this obstacle!
  • preferably to not leak java.math.BigDecimal to Enso code from a different library
    • for example by using Pair Integer Integer
    • such a library may be using "other JVM"
    • when something like that happens, the Java type is different to the value's one
    • and the case of branch check fails
  • is something like that possible, @jdunkerley?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hubertp wrote elsewhere:

Pretty sure the "java.math.BigDecimal" pattern will be the source of bugs

Exactly! Just like any other attempt to exchange Java objects between libraries when running in Dual JVM mode. However that's not something I can fix myself. It needs grand refactoring by the libraries team and that's gonna take time.

Ironically, the more bugs we have, the more incentive there's going to be to redesign the current system built in days when "Java know no borders".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@jdunkerley jdunkerley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly some nits to change to keep stuff consistent with libs code then we can get this is in and see what unravels!

var localStorage =
switch (localType) {
case BooleanType type -> BoolBuilder.fromAddress(size, data, validity).seal(storage);
case BooleanType _ -> BoolBuilder.fromAddress(size, data, validity).seal(storage);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _ variable has caused issues with debugging in the past which is why we avoid due to the targetJavaVersion being locked to 17.

Suggested change
case BooleanType _ -> BoolBuilder.fromAddress(size, data, validity).seal(storage);
case BooleanType type -> BoolBuilder.fromAddress(size, data, validity).seal(storage);

Copy link
Member Author

@JaroslavTulach JaroslavTulach Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • what kind of issue?
  • wasn't that caused by previous version of Frgaal?
  • how can I verify we still need to avoid using _?
  • related usage of _ to address together with this inquiry

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only blocks JUnit work in IntelliJ. And there are other issues that were merged over the last few weeks.

This can be worked around by changing the targetJavaVersion so this isn't a blocker for anyone who isnt me

@JaroslavTulach JaroslavTulach force-pushed the wip/jtulach/DualSnowflake branch from 7f6ba8c to 1bbb22a Compare January 5, 2026 17:12
private val macX64Release = NativeImageSize(200, 457)
private val macARM64Release = NativeImageSize(200, 473)
private val testNISize = NativeImageSize(100, 592)
private val windowsX64Release = NativeImageSize(200, 390)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@enso-bot
Copy link

enso-bot bot commented Jan 6, 2026

Jaroslav Tulach reports a new STANDUP for yesterday (2026-01-05):

Progress: .

@JaroslavTulach JaroslavTulach requested a review from hubertp January 6, 2026 06:09
get_big_decimal (that : Decimal) = that.big_decimal
Returns two numbers: the actual value and the scale associated to it

value_with_scale (that : Decimal) -> Pair Integer Integer =
Copy link
Member Author

@JaroslavTulach JaroslavTulach Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • This is the safe way to transfer BigDecimal between two JVMs
  • Pair Integer Integer is a regular Enso atom
    • with either long or BigInteger fields
    • both those types are transferable between the JVMs
  • atom instance is "transferable" as well

return new CloudAPI(apiRootUri, cloudProjectId, cloudSessionId);
}

public static void flushCloudCaches() {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • this method should be removed by Flush system caches via Runtime.gc #14557
  • those who call this method should call Runtime.gc flush_caches=True
  • the cached field in this class should use Managed_Resource
  • such a resource will be cleared by the system when caches are flushed

@@ -1,4 +1,5 @@
from Standard.Base import all
import Standard.Base.Runtime.Ref.Ref
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I don't like this Ref.Ref duplication
  • it is needlessly verbose
  • I'd like to change the code to allow just import Standard.Base.Runtime.Ref
  • opinions?

@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Jan 6, 2026

Waiting on code owner review from @4e6, @Akirathan, and/or @hubertp.

  • I believe all review comments have been addressed
    • or at least replied to
  • please review again

Copy link
Collaborator

@hubertp hubertp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure the "java.math.BigDecimal" pattern will be the source of bugs as it is easy to miss in the grand scheme of things that some values might be coming from other JVM and don't match on the regular type.
But I'm not going to block the PR because of this concern.

@github-project-automation github-project-automation bot moved this from 👁️ Code review to 🌟 Q/A review in Issues Board Jan 6, 2026
@JaroslavTulach JaroslavTulach merged commit 8390c99 into develop Jan 6, 2026
73 checks passed
@github-project-automation github-project-automation bot moved this from 🌟 Q/A review to 🟢 Accepted in Issues Board Jan 6, 2026
@JaroslavTulach JaroslavTulach deleted the wip/jtulach/DualSnowflake branch January 6, 2026 11:22
@enso-bot
Copy link

enso-bot bot commented Jan 7, 2026

Jaroslav Tulach reports a new STANDUP for yesterday (2026-01-06):

Progress: .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

-compiler -libs Libraries: New libraries to be implemented -libs-API-change-Base Marks a PR that changes the public API of Standard.Base CI: No changelog needed Do not require a changelog entry for this PR.

Projects

Status: 🟢 Accepted

Development

Successfully merging this pull request may close these issues.

4 participants